SGAR: Structural Generative Augmentation for 3D Human Motion Retrieval
Authors: Jiahang Zhang
Lilang Lin
Shuai Yang
Jiaying Liu
Venue: NeurIPS 2025 poster ()
Keywords: Motion Representation
Motion Retrieval
Primary area: applications
TL;DR: A novel structural generation-augmented retrieval method for motion-text retrieval.

Abstract: 3D human motion-text retrieval is essential for accurate motion understanding, targeted at cross-modal alignment learning. Existing methods typically align the global motion-text concepts directly, suffering from sub-optimal generalization due to the uncertainty of correspondence learning between multiple motion concepts coupled in a single motion/text sequence. Therefore, we study the explicit fine-grained concept decomposition for alignment learning and present a novel framework, Structural Generative Augmentation for 3D Human Motion Retrieval (SGAR), to enable generation-augmented retrieval. Specifically, relying on the strong priors of existing large language model (LLM) assets, we effectively decompose human motions structurally into subtler semantic units, \ie, body parts, for fine-grained motion modeling. Based on this, we develop part-mixture learning to better decouple the local motion concept learning, boosting part-level alignment. Moreover, a directional relation alignment strategy exploiting the correspondence between full-body and part motions is incorporated to regularize feature manifold for better consistency. Extensive experiments on three benchmarks, including motion-text retrieval as well as recognition and generation applications, demonstrate the superior performance and promising transferability of our method.

SGAR: Structural Generative Augmentation for 3D Human Motion Retrieval
Jiahang Zhang
Lilang Lin Shuai Yang Jiaying Liu"
Wangxuan Institute of Computer Technology. Peking University (zjh2020, linlilang, williamyang, liujiaying @pku. edu.
Abstract
3D human motion-text retrieval is essential for accurate motion understanding, targeted at cross-modal alignment learning. Existing methods typically align the global motion-text concepts directly, suffering from sub-optimal generaliza- tion due to the uncertainty of correspondence learning between multiple motion concepts coupled in a single motion/text sequence. Therefore, study the ex- plicit fine-grained concept decomposition for alignment learning and present a novel framework, Structural Generative Augmentation for 3D Human Motion Retrieval (SGAR), to enable generation-augee auggeeted retrieval. Specifically, re- lying the strong priors of existing large language model (LLM) assets, effectively decompose human motions structurally into subtler semantic units, ie., body parts, for fine- fine-grained motion modeling. Based this, develop part-mixture learning to better decouple the local motion concept learning, boost- ing part-level alignment. Moreover, directional relation alignment strategy exploiting the correspondence between full-body and part motions is incorpo- rated to regularize feature manifold for better consistency. Extensive experi- ments on three benchmarks, including motion-text retrieval as well as recog- nition and generation applications, demonstrate the superior performance and promising transferability of our method. Our project page can be found at .it.ub..itttbb..... iotPoojectPos.s.. SGAR/ SGAR. html
Introduction
3D human motion understanding is a crucial topic in computer vision. To deal with the complex nature of human motions, natural language widely adopted medium to achieve fine- fine-grained motion modeling. Among these, motion-text retrieval is fundamental task, which aims to search relevant motion/text samples based on queries of another modality, with wide applications including film production [22, health-care
43 and character animation
Following the practice of the image-text [28] 3 and video-text
retrieval, many efforts have
been made to establish semantic-aligned latent space of 3D motions and texts. TMR [25] first proposes the motion-text retrieval benchmark and presents framework by jointly contrasting and reconstructing. However, the scarcity of high-quality motion-text data heavily limits the model performance. To this end, many works propose leverage the pre-trained vision/language models as priors to boost the motion-text retrieval. For instance, some methods [35] 44 render the 3D motions into images/videos to leverage large-scale pre-trained vision-language models, e.g., CL.IP as priors to bridge the motion and text modalities.
However, we argue that the above approaches can still suffer from two points. First, the significant disparity between rendered and natural images results in sub-optimal representations for the vision- bridged methods. More importantly, they neglect the characteristies of human motion data. For
"Corresponding Author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Part Motion Library
Retrieval Auggeneie Generation
Arm
bend
swing back
swing side
liAt
stand
step
bend
Tarso
Query
LLM
DataBase
Relevant Info.
Augmented Generation
Generation- Auggented Retrieval (Ours)
Query
He walks
He prepares jump.
mum throws something.
DataBase
waves hand.
LLM
Relevant Info:
Augmented Retrieval
Figure I: Left: A motion sequence typically comprises multiple motion concepts, e.g., body part movements. Meanwhile, these fine-grained concepts often exhibit higher cross-motion generality, facilitating more precise and generalized motion-tes alignment. Right: Inspired by the Retrieval Augmented Generation [7] (RAG), we explore the Generation-Augmenne Retrieval paradigm, which relies on LLMs available assets, and propose series of effective alignment strategies.
instance, the subtler semantic units, i.e., body parts that depict motions in finer granularity are ignored, leading to sub-optimal alignment. Taking "walks and waves hand" shown Figure aas an example, with single global alignment objective, existing methods may incorrectly correspond the motion pattern of arm swinging to the verb "walk" rather than "wave hand", especially training with limited data scale. This means the difficulty of motion-text retrieval arises not only from the cross-modal alignment but also from the reorganization and decomposition within the same modality. In this context, learning the correspondence of local concepts primitives can lead to more precise, generalized local alignment, and ultimately enhance global understanding. Therefore, key motivation is to explicitly decompose the holistic human motions into fine-graine une-grained local concepts by leveraging available priors, e.g., LLMs, to achieve better motion-text alignment.
sematic concepts
To this end, study new Generation-Augmenn Augmented Retrieval (GAR) paradigm as shown in Figure [ which is less explored by previous work, i.e., leveraging the generated information (part motion texts) relevant to the queries (full motion text) to augment retrieval performance. Specifically, to exploit the generated structural knowledge, we propose a novel Structural Generative Augmentation Retrieval framework, SGAR. SGAR aligns linguistic motion knowledge of body parts generated by LLMs for fine-grained concept modeling. Based on this, a part-mixture learning strategy is developed to better de-correlate the motion pattern learning of different parts and the full-body. By virtue of such way, the consistency learning at both full-body and part levels is boosted. In addition to this independent intra-hierarchy alignment, we propose directional consistency regularization, introducing relational knowledge between full-body and part motions to enhance correspondence. Finally, we conduct extensive experiments on three large-scale benchmarks demonstrate our strong performance on motion-tex. retrieval and promising transfer capacity to a series of downstream tasks.
Our contributions can be summarized as follows:
We propose SGAR, an effective motion-text retrieval framework with structural generative knowledge augmentation. By modeling part motion concept correspondence as more general knowledge, effectively boosts the cross-moda alignment performance of motion-language models.
To enable part-based motion alignment, we integrate structural linguistic knowledge from LLMs and propose a part-mixture learning strategy. By decoupling the motion representations of the global body and local parts, our method can facilitate alignment within different structural levels.
In addition to independently minimizing the Euclidean distance of embeddings at global and local motion levels, a directional alignment objective introduced to model the relational knowledge between them. This further alleviates over-fitting and leads to better representation consistency.
2 Related Works
2.1 Motion-Language Retrieval and Alignment Learning
Cross-modal representation alignment enables entities from different modalities, which share concep- tually similar meanings, to have comparable feature embeddings. However, different from images,

---

#Foort Modion
Full Body Port Representations
The right om exlands from shouddr, trying regain balance.
right
forword.
Text Encoder
The tonso...
Global Allgiament
Port Aligument
Purt Mixture
Decompositin
#Full Motion person walks forward unbolanced and olmost Tulls
Part Mix
Encoder
Figure 2: Overview of SGAR pipeline following a GAR paradigm. We utilize LLMs to generate the part motion corpus and construct mixed motion data, to enable the part motion modeling. Three objectives are integrated: 1) global alignment for full body motion-text pairs, 2) part alignment with soft contrastive learning: 3) part-mixture decomposition for further de-correlation of different part motions. For clarity, the directional relation regularization is omitted here.
human motion-text data collection presents more challenges, e.g., the diverse semantics and the limited motion-text data scale. Most existing approaches focus incorporating pre-trained vi- sion/language models to guide motion representation alignment. For instance, TMR [25] directly utilizes frozen pre-trained text encoder
[32 accelerate learning of the motion encoder. Mo- tionCLIP 35) and TriModal 44] convert motion data into images or videos, enabling the use of pre-trained vision-language models for guidance. Yu et al. 45] introduced novel motion represen- tation that structures 3D motion data into patches, making it compatible with pre-trained ViT [4] models. Other studies [33 42] take model-centric approach, aiming to achieve more informative motion representations. However, these methods overlook the inherent structural properties of human motion, which can lead suboptimal representations and hinder generalization performance.
Meanwhile, some recent methods [1] 39] also study the part kinematics to boost motion generation performance. However, this cross moosl alignment is unidirectional, i.e., the text encoder is often fixed, while the model is dominated by generation objectives, making it difficult to achieve explicit semantic alignment. Furthermore, the part knowledge is injected by guiding motion generation through conditioning, which still differs from our case, that is, achieving bidirectional fine-grained motion-language alignment.
2.2 Collaboration between Generation and Retrieval
Retrieval and generation are closely connected not only because of their similar function, i.e., new content presentation, but also the potential collaboration. Retrieval-Augmented Generation (RAG) 10] is widely studied to enhance accuracy and credibility of LLM generation by knowledge from external databases. In contrast, with more and more impressive L.LMs available knowledge expert assets, we explore a new RenerrationAugmenne leverage the strong priors in LLMs augment data-scarce 3D human motion retrieval, which has been studied by few works. Mao [21] discussed generation-augmentee retrieval to deal with language question answering (QA), where, however, "retrieval" is still evaluated by and serves for the generative QA task. Therefore, it is still confined to RAG in a sense. In contrast, we focus on the retrieval task exactly and aims to boost by the generation knowledge following the GAR paradigm. Meanwhile, our modal setting also brings new challenges in how to effectively utilize the generated information.
Retrieval (GAR) paradigm, especially to
The Proposed Method
Our goal is to achieve semantic-aligned representations with paired lation-language dataset {m,, t to support cross- moss-modal retrieval, where (m, t.) are the igh paired motion and text data.

---

3.1 Contrastive Motion-Language Learning
Data Representation. The human motions can be generally formulated as skeleton joint positions defined in SMPL [18] model, i.e., m; E R Rxx33 where T is the sequence length and J x 3 is the 3D positions of J joints in Cartesian coordinates. Following previous work, we further structure the motion data into series of "patches" to leverage the pre-trained ViT's as priors. Typically, a patch aggregates the joint information of body part in some adjacent frames. The final input motion can be formulated as E RTxxPX33 for patchification, where P = 5 is the body part number, i... (left/right arm, left/right leg, torso), and N is the patch size. More details can be found in 45).
Model Architecture. Following the canonical design, our model consists of two modality-specific encoders, (-) and ft(-). Specifically, for the motion encoder, we adopt ViT-B/16 4 with 12 layers pre-trained on ImageNet-2 (31] to better utilize the existing prior vision knowledge. For the text encoder, we employ the pre-trained Distill-BERT
To obtain the motion/text embeddings, we append a [cls] token to inputs, which is finally projected into the shared motion- -lagunnlanguage latent space, i.e., 2m,i, 24,5 for contrastive learning. The contrastive learning objective [23] can be formulated as the sum of the following two terms, denoted as L
fall E log exp(Em. (T,/TT
exp(zm.i 21,/TT) ]
(I) where T is the temperature coefficient. For clarity, only text-to-motion (+2m) term is presented in the following, which shares similar definition of motion-to-text (m2t) term.
E
log Σ, exp(2m.j 2
E3 exp(zm. : /T,1/T)
3.2 Structural Knowledge Generation
With the rapid development of LLMs, it is promising to leverage LLMs as available knowledge experts to boost the data-scarce motion-text retrieval scenario as discussed in Sec. [ Specifically, we aim to generate the motion descriptions for each body part based on 4,-
Remark: Leveraging LLMs to generate part texts is also studied in other motion generation and Differently, we focus on the retrieval task following GAR paradigm, especially jointly considering the testing GAR application. Specifically, since there are often no paired motion-text data for the query side or database during testing, only texts can be conveniently utilized. It means some complex prompting strategies, e.g., feeding LLMs with both motion pose descriptors and texts, are inapplicable. While there are inevitably some less accurate descriptions generated by LLMs, especially for complex motions, the output content is still relevant partially matched to serve as supplementary and boost the retrieval learning. More discussions and analysis on the part texts generated by LLMs and model robustness can be found in Appendix A. and A.2
recognition works 48 40 5
To this end, we develop clear and effective generation strategy. We select LLaMA3-70B with carefully crafted few-shot prompts for our implementation. Meanwhile, to improve part text quality for training, we provide LLMs with all global texts annotated by different subjects as detailed and comprehensive descriptions to jointly reason (for training data generation only). Besides, to deal with the uncertainty in generation for the parts with confusing or hard-to-infer motions, we introduce a special indicator, [udf). and require the LLMs to output this indicator for the parts with high uncertainty. Subsequently, the texts with this indicator are excluded during training. Overall, our GAR paradigm is compatible with different LLMs and generation strategies, where the retrieval performance can be expected to be boosted as the generation capability improves.
(37
Then, the generated part motion descriptions are fed into the text encoder, yielding the embeddings z!, for the jeh body part text of the ieh sample. On the motion encoder side, we register an additional token for each body part. i.e., [cls - pyl. to feed the encoder to obtain part motion embeddings = 1, ..., P. denoting the jeh part. Along with the [cls] tokens, the motion encoder aligns both the global and local motion embeddings. However, the local part motions are much less diverse, and different body motions can share similar part motions. Therefore, adopt soft contrastive learning objective by replacing the hard label with the intra-modality similarity scores:
exp(2 - 2T.
1 P
p" (d.) logp" (2mor .pk (z, (3,)
C12m = E
(2)
Σ expl2 =1,/7)
4

---

3.3 Part-Mixture Decorrelation Learning
Due to the inherent correlation between global and part motions, can be undesirably learned by model to derive the part embeddings based on the common concurrence of motion patterns, instead of the exact part kinematics. For instance, the model can associate arm swinging with leg stepping, which can impair knowledge generalization. Therefore, we propose a part-mixture learning policy to construct unusual motion samples to force the representation decorrelation and precise modeling.
With the part "patches" representation, we can simply perform Cut-Mix [46] by exchanging the part patches of different motions. Specifically, given two motion data m,, my, we randomly sample K body parts (K < P) and replace the corresponding patches of m, with my to generate the mixing motion data. Then we obtain the full body and part motion embeddings as Sec. 3.2 by feeding the mixed motions into the motion encoder.
Similarly, optimize the global and part alignment objectives. For global alignment, utilize the linear blending of text embeddings, ž,,i and 2r,k. as the target text positives for contrastive learning. Specifically, it is calculated 2t,miz Zt. (1 > where Am is the mixing ratio determined by the proportion of replaced part number. For part alignment, the target part text embeddings of mixed data can be reassembled from =i,; and = following the replacement in mixing process. Building on this, we calculate the LJu and part for optimization similar Eq. (1 and Eq. 22 to boost the part-mixture learning. More detailed formulations can be found in Appendix D.1
Remark: Although this direct copy-paste approach can lead to globally unreal motions, recent work [16] has emphasized the local motion patterns as a more essential aspect for human motion modeling. Our experiments in Table 2 also verify the effectiveness of this copy-paste implementation.
3.4 Directional Relation Alignment Strategy
As introduced before, we encourage the model to jointly learn both full-body and part alignment. However, the two objectives optimize the Euclidean distance of embeddings independently, which ignores the relational knowledge between them, leading to over-fitting and limited performance. Inspired by the inter-word directional knowledge discovered in the neighbor topology structure of language model latent space [38] we explore the motion-tex. analogies between the relevant global-local motion concepts as a beneficial representation regularization.
Specifically, we model the relational knowledge between the full body and parts as the direction of their difference in the normalized space. Formally, the directional relation alignment objective can be denoted as
C, =
E
1
m
P
m.
where Azh 2m,i mm. 22t, 21,5 2 This can be regarded as a part-conditioned motion knowledge consistency regularization, leading to more meaningful representations. Overall, the full optimization objective of our framework can be formulated as (), is the loss weight)
= Lfull + Cport + Amin Ciica + >miis full
Cmáz part Cp
(4)
3.5
Generation-Augmentee Retrieval: An Optional Testing-Augment Strategy
The above designs naturally enable a destin-aaummettttioon option by providing additional part information. Specifically, giving one or more additional part texts embeddings, we calculate the similarity matrixes between both global and local motion-text pairs, which are then summed with weight a, i.e.,
8g + a Eje,, 8%, where s/ and are the global, part, and final similarity matrix and Jrp is the set of parts with available descriptions. Finally, the retrieval results can be derived -aggaeene retrieval policy during testing. In implementation, is set as 0.1 to control the intensity of the pert-level effect.
from s. This is an optional
Next, some possible application scenarios are discussed. For text-to-motion retrieval, part texts can be provided by users as additional queries generated by LLMs. For motion-to-ted retrieval, the additional part texts can automatically generated by feeding global texts the database to L.MMs and maintained in the database.
5

---

Table 1: Motion-text retrieval results on HML3D and KIT benchmark. "SGAR" follows the same test setting as other methods (i.e., no part information available for testing), while "SGAR++" employs part information during testing as discussed in Sec. 3.5 to show optional generation augmented retrieval capacity.
Text-to-motion retrieval
Motion-to-text retrieval
Protocol
Methods
R@1
R@3 1 R@5
R@ 10
MedR
R@1
R@3 1 R@5 f
10
MedR_..
TMR [25
8.92
16,33
22.06
33.37
25,00
9,44
16,90
22.92
32.21
26.00
MoPatch 45
10.80
20.00
26.72
38.02
19.00
11.25
19.98
26.86
37.40
20,50
AII-HML3D
ChronRet 6
8.30
16.70
23.65
35.31
22.00
9.08
11.22
23.68
34.31
23.00
SGAR
12.86
23.52
30.75
43.00
15.00
13.82
23.38
30.09
41.83
16.00
SGAR++
14.07
25.32
33.03
44.78
13,00
14.64
25.52
32.55
44.50
14,00
TMR 25
67.45
86,22
91.56
95.46
1.03
68.59
86.75
91.10
95.39
1.02
MoPutch 45
71.61
90.02
94.35
97.69
1,00
72.11
90,21
94.44
97.76
1.00
Small-HML33
TriModal
44
68.58
85,02
88.77
92.58
1.01
68.64
85,52
88.76
92.82
1.01
ChronRet
70.10
87.07
91.13
94.32
1.01
70.19
86.66
90.69
94.23
1.01
SGAR
75.66
92.11
95.55
98.06
1.00
76.35
92.38
95.69
98.02
1.00
SGAR++
76.51
92.77
95.87
98.27
1.00
77.24
92.79
96.01
98.20
1.00
TEMOS 24
7.11
17.59
24.10
35.66
24.00
11.69
20.12
26.63
36.39 66.39
26,50
TMR 25
10.05
20.74
30.03
44.66
14.00
11.83
22.14
29.39
38.55
16.00
AII-IIT
MoPatch 45
14.02
28.91
34.10
50.00
10.50
13.61
27.54
33.33
44.77
13.00
SGAR
16.75
30,48
40,00
53.61
9,00
14.46
25.54
35,54
48.31
11.00
SGAR++
16.51
30,96
40.48
56.99
8.00
16.14
29.28
38.00 67.67
52.29
9.00
TMR 25
50.00
78.02
87.97
94.87
1.50
51,21
78.64
89.00
95.31
1.50
MoPatch 45
53.55
79.82
88.92
96.29
1.36
54.54
79.68
89.35
96.11
1.31
Small-KTT
TriModal
44
58.10
86,34
93.08
96.47
1.08
60.23
86.44
93.22
95.87
1,20
LaMP 14
52.50
84.70
92.70
97.60
54.00
84.40
92.20
97.60
SGAR
62.12
87.88
94.25
98.00
1.04
63.00
87.25
94.38
98.38
1.03
SGAR++
62.50
88.12
94.75
98.75
1.04
63.38
87.75
95.00
98.
1.02
Experiments on Motion-Text Retrieval
4.1 Datasets and Implementation Details
HumanML3D 8 (HML3D) enriches the AMASS (2 and HumanAct12 [ motions by with the
textual descriptions. It consists of 23,384 and 4,380 motions for training and testing with a mirror augmentation, respectively. There are 3.0 different textual labels on average for each motion.
KIT-ML 26 (KIT) is small dataset with focus on locomotion motions, consisting of 3,911 sequences and 6,278 text descriptions. We obtain 4,888, 300, and 830 motions for training, validation and testing, respectively, each of which is annotated 2.1 times on average.
Motion-X [5] a large-scale dataset with more than 80K sequences from different motion domains. The texts are generated by LLMs. We further process the motions into adopted representation. Since there are few previous works evaluated on this dataset, it is mainly utilized to support cross- dataset evaluation of transfer capacity.
For implementation, we utilize the same data pre-processing methods with ViT-B and DBST as motion and text encoders following previous works (25 45). The model trained using Adam optimizer [12] for 50 epochs, with learning rates of 10 5, 10 and 10-3 for the text encoder, the motion encoder, and the projection heads, respectively. The embedding dimension after projection 256 for contrastive learning. Amar and A, are set to 0.5 and 0.1. The temperature coefficients 7 and r' are 0.07 and 0.05. The batch size is 128. For the input data, the patch size N is 224. We pad or crop the motions to a fixed length of 224 following VIT. For evaluation, we adopt the metrics and protocol in (25]. Specifically, the standard Recall various ranks, i.e., R@k, and median rank, MedR, are employed to evaluate the performance. Meanwhile, we vary the scale and data similarity of the target gallery sets, yielding All Data?) Small protocols following 255 45) More details in Appendix D.2
4.2 Performance Comparison
In this section, we compare our method with the latest works with experimental settings aligned. As shown in Table I our method achieves the best scores with significant performance improvement
Following 45.. we consider misjudgments due to mirror augmentation for evaluation correction.

---

Aperaon walks di steoe to
The righttomiss raxeng Q. lowening
to sis with left oro
and mokkss wavinng
stade towerds thai ant
right raed then switcher risling
ot sometiing by miing therright
Forward etoulde: beightann
personis
lop stops then stonddoct
ploving
pnitapp goll Aponan pwtoor tg
person priiging duubmmttio.
persen catap and Nitheh
brwaad
Appyonn objechwith right
Figure 3: Qualitative T2M retrieval with top-3 results on HML3D testing set. The left column is full-hody retrieval, with test set query (the first sample) and the customized text query (second) Green text is the perfect match. The right column part-based retrieval, using only part motion queries.
Table Ablation studies on our designs, part contrastive learning (Part CL), part-mixture learning (Part Mix), and directional relation regularization (Relation Reg.)
Part
Part
Relation Reg.
T-to-M Retrieval
M-to-T Retrieval
CL
Mix
R@I
R@5
MedR
R@1
R@5
MedR
9.87
26.36
19.50
10.19
26.56
21.50
11.27
28.44
16.50
12.57
27.87
19.00
11.90
28.56
16.00
12.95
28.47
18.00
4
12.09
30.48
15.50
13.41
30.13
17.00
12.86
30.75
15.00
13.82
30.09
16.00
compared with the state-of-the-art methods on both HML3D and KIT benchmarks. Previous works mainly focus how enable the availability of pre-trained vision/language models while ignoring the structural nature of human motion data. In contrast, our method starts from the characteristics of the human body and models the part kinematics a GAR manner, achieving remarkable performance improvement and renewing the scores. Mcanwhile, compared with Lex [19] which requires 4 training stages, our method more wraiing-frriendly (~ 8 times faster) and clear with better performance.
Besides, benefited from our GAR paradigm, SGAR can be boosted with further generated (or user- provided) part information discussed in Sec. 3.5| yielding
Berinn-augmmnt version denoted as "SGAR++". Although it can be unfair to directly compare SGAR++ with previous works without part information testing, show this intended verify our motivation and demonstrate remarkable testing-augment capacity. As we can see Table[] additional part information as supplementary can significantly benefit retrieval performance. More discussions and visualizations on this part-based testing GAR can be found in Appendix B.2 and C.I
4.3 Qualitative Results
We present the text-to-nne sot-to-mooiion retrieval results with full- full-body and part motion text queries, respectively, Figure For the full-body retrieval, our model successfully retrieves the ground-truth/semanne aligned motion samples, whether by the original text annotations or the ussr-customized text queries. For the part-based retrieval, we obtain the candidate motions by calculating the similarity of their part motion embeddings and the given part texts, achieving the partial match retrieval (29). Meanwhile, these two techniques can combined achieve more fine-grained motion retrieval. As we can see, although the results possess different global motion concepts, the motions of the target local regions align with the part text descriptions well, demonstrating our superior structure-aware modeling capacity. More visualization results can be found in Appendix B
4.4 Ablation Study
In this section, we present the ablation study results the HumanML3D dataset.

---

Table 3: Comparison for the different LLMs for part motion knowledge generation.
Text-to-motion retrieval
Motion-to-text retrieval
Knowledge Experts
R@ 1 f R@51 1
R@ 10
MedR
R@] 1 R@5 + R@ 10
MedR !
No Part Text
9.87
26.36
37.38
19.50
10.19
26.56
36.63
21.50
ILaVAI 6-6B44
17
11.82
28.36
41.89
16.00
12.18
28.62
39.93
17.00
1JLMMA-70BB 37
12.86
30.75
43.00
15.00
13.82
30.09
41.83
16.00
Table 4: Comparison of the computational complexity and runtime using one NVIDIA 4090 GPU.
Method
ChronRet (6]
MoPatch 45
SGAR
SGAR+
FLOPss
5.48G
6.46G
6,89G
6,89G+1.000P555122222
Runtime
0.0072s
0.0078s
0.0083s
0,0086s
1) Effectiveness of Part Motion Modeling. The crucial idea of our method is the integration of the structural part motion learning. Specifically, it consists of two parts, i.e., the basic part contrastive learning for subtler semantic alignment and the part-mixture learning for better local motion decorre- lation. As shown in Table it significantly boosts the model performance, demonstrating stronger human motion understanding capacity due to the tne-grained action unit modeling.
2) Effectiveness of Directional Relation Regularization. Meanwhile, the proposed relational consistency further regularizes the feature manifold and pursues the corresponding relation knowledge consistency, which improves the model representations shown in Table
3) Different LLMs for Generation. Table shows the results using different LLMs for part descrip- tion generate. First, it can seen the above LLM choices can generally be bring about performance improvement, which indicates the promising future of our GAR paradigm. Besides, we find the gen- eral reasoning capacity of LLMS matters more than the visual capacity in our agent task. Furthermore, employing stronger LLMs could further improve part text quality and performance, which, however, is not the focus of this work. We kindly refer the readers to Appendix A for more discussions.
4) Complexity Analysis. As shown in Table 4 our SGAR has similar complexity and runtime to other methods. When testing GAR is enabled (SGAR++). the main additional cost is the encoding of the extra part texts (1.05G for once forward of text encoder, here take 5 parts for example). In terms of the runtime, it should be noted that the encoding of part texts is highly parallel and does not incur much time cost.
5 Applications
In the following, we demonstrate the transfer capacity and application value of our model in terms of the motion encoder and text encoder.
5.1 Skeleton-Baser Human Motion Understanding
1) Transfer Learning for Action Recognition. We adopt the BABEL 60-classes benchmark [2 consisting of 10892 sequences from AMASS. It is challenging due to the extremely short clips and fine-grained action categories. In our transfer learning setting, the motion encoder is pre-trained on HumanML3DD/Motions recognition. A new fully-connected (FC) layer is attached to the encoder to predict final scores. As shown in Table 5 our method can surpass the state-of-the-art supervised models by fine-tuning the last FC layer solely. When we utilize the larger dataset, i.e., Motion-X, a stronger transfer learning capacity be achieved. Finally, we demonstrate the best performance by fine-tuning the whole model, indicating the promising benefits for Ikeseton-based action recognition.
first, and then transferred to the BABEL dataset for 60-class action
2) Cross-Datasee Motion Retrieval. In this part, we implement the cross-dataset retrieval, i.e. training the model on HumanML33D first and then testing on Motion-X test set, which covers various domains and noise more comprehensive benchmark.

---

Table 5: Transfer learning results for action recognition on BABEL benchmark. "I." indicates linear evaluation and "F" uses fully finetuning for evaluation.
Table 6: Cross-datasct motion retrieval
results. All models are trained on Hu-
manML3D dataset.
Method
Pretrain
Finetune
Top-1
Top-S
Method
R@I
R@s
R@10 10
MedR
2s-AGCN
F
41.14
73.18
T-tn-M Retrieval
MotionCLIP
40.90
57.71
TMR [25
20.23
46.18
60.95
MoPatch
24.00
48.81
64.32
7.50
TMR 25
HML3D
I.
40.16
70.42
SGAR
31.05
54.77
69.22
6.81
MoPatch 45
HML3D
L
41.05
71.97
SGAR
HML3D
L
42.59
73.09
M-to-T T Retrieval
SGAR
Motion-X
L
43.05
73.82
TMR [25
20.21
43.82
59.05
9.24
MoPatch 45
22.68
46.45
61.74
8.25
SGAR
Motion-X
F
46.21
77.09
SGAR
30.13
52.92
66.77
7.46
Table 7: Results (Topl -aceuracy)) of using different text encoders for zero-shot skeleton-based action recognition.
Table 8: Quantitative results of motion genera- tion with different text encoders.
Precision
Method
FID
MM Dist
NTC 60 split
NTU 120 split
Top-1
Top-2
Top-3
Method
55/5
48/12
110/10
96/24
MDM
36
SA-DVAF
R
TMR
0.459
0.663
0.770
0.528
3,259
Sentence- BERTT
30
82.37
41.38
68.77 78.77
46.12
19
CLIP-B 28
83.28
39.72
73.88
48.38
SGAR
0.515
0,710
0.806
0.221
3.024
CLIP-L 28
79.35
36,39
70.21
49.99
T2M-GPT
47
Distill-BERe BERT
83.78
39.72
55.51
47.94
0.680
ChronRet (DistillBERTT) 6
31.49
11.47
TMR
0.513
0.705
0.803
0.103
2,964
LaMP (BERT) 14
38.05
14.92
LaMP
0.540
0.732
SGAR (Distill-BERC
83.92
45.66
69.97
52.27
ChronRet 6
0.528
0.717
0.810
0.074
2.915
SGAR
0.535
0.728
0.081
Specifically, retrieve the counterparts from batch following Small protocol. The results shown Table 6 As we can see, our method demonstrates a remarkable improvement over previous works, verifying the stronger generalization capacity. Meanwhile, also present the model trained on Motion-X as in-domain reference, which is much better than the HumanML.3D model. This is partly due the limited domains in HumanMI.3D. resulting in limited generalization capacity. On the other hand, HumanML3D uses SMPL-format data and lacks hand motion supervising, leading to some motions Motion-X being indistinguishabl.
5.2 Stronger Text Encoder for Human Motion
1) Application to Zero-Shot Action Recognition. Existing zero-shot skeleton action recognition models are trained on the seen categories to generalize to the unseen categories. They necessitate a pre-trained text encoder to derive the category label embeddings, capturing the potential semantic relations between the seen and unseen action categories. To verify the effectiveness of our text representations, we vary the choice of text encoder from general trained to Human--otion-eee text encoders, and adopt the recent work 13 as the zero-shot algorithm for comparison.
The accuracy of the unseen categories is reported in Table Our method can achieve better performance than other language models which are trained on general domains. This shows the advantage of our text encoder in human motion text representation.
2) Application to Text-Guided Motion Generation. For motion generation, the text encoder is responsible for obtaining textual embeddings accurate guidance. However, previous works directly utilize the text encoder derived from lasonnlanguuae pre-training. e.g., CLIP. Here we study the motion generation from the perspective of text embedding quality which few previous works consider. Similarly, we utilize the text embeddings output by different text encoders, and adopt two popular generation methods as baseline, the diffusion-basee MDM 36 and the auto-regressive model T2M- GPT 47.. Following the previous works 36), the evaluation metries include Frechet Inception Distance (FID), Top 1-3 retrieval precision (R- RR-Prccision), and Multi-mode Distance (MM-Dise

---

person runs waving the right hand to someone.
A person unleashes strikes with his fists while dodging.
This person stumbles left and right while moving forward.
Text Prompt
MDM w CCLIP / Ours
Figure 4: Comparison of motion generation results guided by different text encoders. Left: MDM model with default CLIP text encoder. Right: MDM model with our text encoder.
Specifically, we compare different text encoders for motion generation. As shown in Table S we can find existing motion-language pre-trained text encoders perform better than traditional CLIP baseline for motion generation tasks due to human-centrie motion modeling, and our text encoder achieves competitive scores compared with other latest methods (Note that LaMP utilizes larger text encoder, i.e., BERT). Meanwhile, evidenced by the cases in Figure our text encoder can output more accurate text embeddings conditions due the decoupled structural learning, effectively alleviating the difficulty of composite motion generation.
Conclusion
This paper presents novel structural generative augmented retrieval paradigm for motion-text data, which exploits part motion concepts as more generalizable knowledge to boost retrieval in a GAR manner. By generating part linguistic knowledge through LLMs. SGAR effectively aligns part motions with part-mixture decorrelation learning and directional relation consistency techniques. Remarkably, the adopted GAR paradigm enables model with optional testing-augment capacity for fine-grat ane-grained and improved retrieval. Extensive experiments on retrieval as well as recognition, and generation applications, demonstrate our superior performance and promising transferring capacity.
7
Acknowledgements
This work was supported part by the National Key R&D Program of China (No. 2024YFB2809101, and in part by the National Natural Science Foundation of China under Contract No.62172020.
References
[1] Kevin Bergamin, Simon Clavet, Danicl Holden, and James Richard Forbes. DReCon: Data-driven responsive control of physics-based characters. ACM TOG, -81) 2019.
[2] Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Ncelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot leamers. In NeurlPS, 2020.
[3] Min Cao, Shiping Li, Juntao Li, Liqiang Nie, and Min Zhang. Image-text retrieval: A survey on recent research and development. 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for image recognition at scale. ICLR, 2021.
[5] Ke Fan, Jiangning Zhang, Ran Yi, Jingyu Gong, Yabiao Wang. Yating Wang. Xin Tan, Chengjie Wang, and Lizhuang Ma. Textual decomposition then sub-motion-space scattering for open-vocabulary motion generation. 2024.
[6] Kent Fujiwara, Mikihiro Tanaka, and Qing Yu. Chronologically accurate retrieval for temporal grounding of mation- models. ECCV, 2024.
[7] Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. Retrieval-angmented generation for large language models: A survey. 2024.
[8] Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang. Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3D human motions from text. In JEEE CVPR, 2022.
10

---

[9] Chuan Guo. Xinxin Zuo, Sen Wang. Shihao Zou, Qingyao Sun, Annan Deng. Minglun Gong, and Li Cheng. Action2motion: Conditioned generation of 3D human motions. In ACM MM, 2020.
[10] Shailja Gupta, Rajesh Ranjan, and Surya Narayan Singh. comprehensive survey of -ugmeette generation (rag): Evolution, current landscape and future directions. 2024.
[11] Yiming Huang, Weilin Wan, Yue Yang, Chris Callison- Burch,, Mark Yatskar, and Lingjie Liu. Como: Controllable motion generation through language guided pose code editing, In ECCV, 2024.
[12] Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint 1412.v:412.69900 2014.
[13] Sheng- Wei Li, Zi-Xiang Wei, Wei-Jie Chen, Yi-Hsin Yu, Chih Yuan Yang, and Jane Yung-jen Hsu. SA- DVAE: Improving zero-shot skeleton- based action recognition by disentangled variational autoencoders. In ECCV, 2024.
[14] Zhe Li, Weihao Yuan, Yisheng He, Lingteng Qiu, Shenhao Zhu, Xiaodong Gu. Weichao Shen, Yuan Dong, Zilong Dong, and Laurence T Yang. LaMP: Language-motion pretraining for motion generation, retrieval, and captioning. ICLR, 2025.
[15] Jing Lin, Ailing Zeng, Shunlin Lu, Yuanhao Cai, Ruimao Zhang, Haoqian Wang, and Lei Zhang. Motion-X. X: A large-scale 3D expressive whole-body human motion dataset. In NeurIPS, 2023.
[16] Lilang Lin, Jiahang Zhang, and Jiaying Liu. Actionlet-dependent contrastive learning for unsupervised Akeleton-basee action recognition. In JEEE CVPR, 2023.
[17] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurlPS, 2023.
[18] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. SMPL: skinned multi-person linear model. ACM TOG, 34(6), 2015.
[19] Guangtao Lyu, Chenghao Xu, Jiexi Yan, Muli Yang, and Cheng Deng. Towards unified human motion- language understanding via sparse interpretable characterization. In ICLR, 2025.
[20] Naureen Mahmood, Nima Ghorbani, Nikolaus F Troje, Gerard Pons-Moll, and Michael Black. AMASS: Archive of motion capture surface shapes. In ICCV, 2019.
[21] Yuning Mao, Pengcheng He, Xiaodong Liu, Yelong Shen, Jianfeng Gao, Jiawei Han, and Weizhu Chen. ceneratioo-augmeenn retrieval for open-domain question answering. 2021.
[22] Markus Mühling, Nikolaus Korfhage, Eric Müller, Christian Otto, Matthias Springstein, Thomas Langelage, Uli Veith, Ralph Ewerth, and Bernd Freisleben. Deep learning for content- content-based video retrieval in film and television production. Multimedia Tools and Applications, 76:22169-22194, 2017.
[23] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, 2018
[24] Mathis Petrovich, Michael Black, and Güi Varol. TEMOS: Generating diverse human motions from textual descriptions. In ECCV, 2022.
[25] Mathis Petrovich, Michael Black, and Güi Varol. TMR: Text-to-motion retrieval using contrastive 3D human motion synthesis. In ICCV, 2023.
[26] Matthias Plappert, Christian Mandery, and Tamim Asfour. The kit motion-language dataset. Big data, 4(4):236-25,, 2016.
[27] Abhinanda R Punnakkal, Arjun Chandrasckaran, Nikos Athanasiou, Alejandra Cairros and Michael Black, BABEL: Bodies, action and behavior with english labels. In IEEE CVPR, 2021.
[28] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In ICML, 2021.
[29] Kotagiri Ramamohanarao, James Thom, and John W Lloyd. Partial-match retrieval using hashing and descriptons. ACM Trans. on Database Systems, 8(4):552-576, 1983.
[30] N Reimers. Sentence-BeiN BERT: Sentence embeddings using siamese
arXiv preprint
190.v:1908.0000 2019.

---

[31] Olga Russakovsky. Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. IJCV, 115:211-252, 2015.
[32] V Sanh. DistilBERT, distilled version of BERT: Smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019.
[33] Haoyu Shi and Huaiwen Zhang. Modal-enhanced semantic modeling for fine-grained 3D human motion retrieval. ACM MM, 2024.
[34] Lei Shi, Yifan Zhang, Jian Cheng, and Hanqing Lu. Two-stream adaptive graph convolutional networks for skeleton- based action recognition. JEEE CVPR, 2019.
[35] Guy Tevet, Brian Gordon, Amir Hertz, Amit H Bermano, and Daniel Cohen-Or. MotionCLIP: Exposing human motion generation CLIP space. In ECCV, 2022.
[36] Guy Tevet, Sigal Raab, Brian Gordon, Yoni Shafir, Daniel Cohen-or, and Amit Haim Bermano. Human motion difftusion model. In ICLR, 2023.
[37] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie- Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and efficient foundation language models. arXiv preprint 10w.22302.1391..0000
[38] Ekaterina Vylomova, Laura Rimell, Trevor Cohn, and Timothy Baldwin. Take and took, gaggle and goose, book and read: Evaluating the utility of vector differences for lexical relation learning. arXiv preprint 2015.
[39] Bizhu Wu, Jinheng Xie, Meidan Ding, Zhe Kong, Jianfeng Ren, Ruibin Bai, Rong Qu, and Linlin Shen. Finemotion: A dataset and benchmark with both spatial and temporal annotation for fineggriined motion generation and editing, arXiv preprint arXiv:2507. 2025.
[40] Wangmeng Xiang. Chao Li. Yuxuan Zhou, Biao Wang, and Lei Zhang. Generative action description prompts for skeleton-based action recognition. In ICCV, 2023.
[41] Hu Xu, Gargi Ghosh, Po-Yas Yao Huang. Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, Christoph Feichtenhofer. VideoCLIP: Contrastive pre-training for zero-shot video-text understanding. arXiv preprint arXiv:2109.1400 2021.
[42] Yang Yang, Haoyu Shi, and Huaiwen Zhang. Hierarchical semantics alignment for 3D human motion retrieval. 2024.
[43] Jun Yin. Jun Han, Chenghao Wang. Bingyi Zhang, and Xiaoyang Zeng. A skeleton- based action recognition system for medical condition detection. In Proc. IEEE Biomedical Circuits and Systems Conference, 2019.
[44] Kangning Yin, Shihao Zou, Yuxuan Ge, and Zheng Tian. Tri-modal motion retrieval by learning a joint embedding space. In IEEE CVPR, 2024.
[45] Qing Yu. Mikihiro Tanaka, and Kent Fujiwara. Exploring vision transformers for 3D human motion- language models with motion patches. In IEEE CVPR, 2024.
[46] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. CutMix: Regularization strategy to train strong classifiers with localizable features. In ICCV, 2019.
[47] Jianrong Zhang, Yangsong Zhang, Xiaodong Cun. Shaoli Huang, Yong Zhang, Hongwei Zhao, Hongtao Lu, and Xi Shen. T2M-GPT: Generating human motion from textual descriptions with discrete representations. In IEEE CVPR, 2023.
[48] Angi Zhu, Qiuhong Ke, Mingming Gong, and James Bailey. Part-aware unified representation of language and skeleton for zero-shot action recognition. In IEEE CVPR, 2024.
[49] Cunjuan Zhu, Qi Jia, Wei Chen, Yanming Guo, and Yu Liu. Deep learning for video-text retrieval: A review. 2023.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims in the abstract and introduction accurately reflect contri- bution.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Please the Limitations and Future Work section (Appendix E
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
13

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Please see the Experiments on Motion- Text Retrieval and Applications sections (Sec. 4 & (5 Main Paper and Experiment Settings and Implementation Details section in
Justification:
Appendix D.2
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
14

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: The code and data have not been released yet. We plan to release them uponpublication and we provided extensive implementattinndttals in the main paper and appendix to facilitate reproducibility.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https: cc/
public/ public/quidss/ / Cooee www.mnssssoooooooooooooooooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https?
//nips httpp/cccppblccccgocc.cec. gedgeeiiieeeee des/ /ooeee/bmmm ss onPol oollcyyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please see the Experiments Motion-Tere Retrieval and Applications sections (Sec. 4 & 55) in Main Paper and Experiment Settings and Implementation Details section in
Appendix D.2
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Due to limited resources and following the practice of previous works, we do not report the error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
15

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative eπτor rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please see this in Experiment Settings and Implementation Details section in Appendix D.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The research conducted in the paper conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., - if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please see the Broader Impact section (Appendix F in the Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
16

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: This work properly credits and respects the assets used.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
17

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, (papersw haperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Please see the Method, Experiments, and Applications (Sec. 5 in Main Paper.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
18

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: The LLMs are used as the knowledge experts for generation query-relevant in- formation to boost retrieval. Details can be found in Method section (Sec. and Appendix A and D.2 for usage.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non standard components.
Please refer to our LLM policy (https: /tttu::///eurpps.. http:iiss..ccooeeeeeeee/////2
LLM) for what should should not be described.
19

---

Discussion on Structural Knowledge Generation by LLMs
20
A.1
Comparison of Different LLMss for Generation,
20
A.2 Discussion on the Quality of Generated Texts and Model Robustness,
22
B
Visualization Results
23
Motion-Texx Retrieval Comparison with Other Methods
23
B.2
Motion-Text Retrieval Results with/without Testing GAR
23
Additional Experimental Results
25
Retrieval Based on Only Part Queries
25
Different ViT Backhones)
26
More Ablation Study Results
27
D Approach and Implementation Details
27
D.I Part-Mixture De-correlation Learning
27
D.2 Experiment Settings and Implementation Details
28
I Limitations and Future Work
29
Broader Impact
29
We present more details of our method, experiments and implementation below. Code and models will be available upon publication.
A Discussion on Structural Knowledge Generation by LLMs
A.1 Comparison of Different LLMs for Generation.
Table 9: Comparison for the different knowledge experts for part motion knowledge generation.
Text-to-motion retrieval
Motion-to-text retrieval
Knowledge Experts
R@11 t R@5 R@ 10 MedR
R@ 1 f R@5 R@ 10
MedR
No Part Text
9.87
26.36
37.38
19.50
10.19
26.56
36.63
21.50
LLaVA 1.6-34B 17
11.82
28.36
41.89
16.00
12.18
28.62
39.93
17.00
I.IaMA3-70R
37
12.86
30.75
43.00
15.00
13.82
30.09
41.83
16.00
ChatGPT-3.5 Turbo 2
12.47
30.84
42.09
15.00
13.48
29.89
41.07
16.00
Hybrid
12.61
29.14
43.92
15.00
13.75
28.96
40.29
16.50
We compare the widely used MLLM/LLM model, LLaVA model (v1.6-34B) [7], LLaMA3-70B B7 and GPT-3.5 Turbo [2] model. For LLaVA, render 3D motions into sequence of images, which are then stitched together as additional visual prompts. our experiments, we observe that LLaVA tends to over-rely on the image prompt, leading confusion between the left and right sides. To mitigate this, we use distinct colors for the left and right sides to aid the model's differentiation. However, the hallucination problem still exists. Some examples LLaVA still exhibits hallucinations with limited output diversity. We also present the diversity results quantitatively in Table by calculating the type-token ratio of the generated part descriptions in the HumanML3D dataset to support our claim.
shown in Figure 5 where
We provide the quantitative results of using different LLMs as shown in Table In addition to relying on a single LLM, we have explored refining the LLaVA results by LLaMA to combine their merits, denoted as "Hybrid"'. Specifically, given the global texts and the part texts generated by LLaVA, we prompt the LLAMA to improve the accuracy and diversity of these part texts. However, do not observe non-trivial improvement as shown in Table 9
20

---

A man grabs his right log with right hand and extends left arm into an okward stretch.
LLLVA Output:
The right arm extended outward and upward, then bent at the elbow and brought toward the
body, then extended outward ogain.
The left arm is extended outward and upward, then bent at the elbow and brought toward the body, then extended autward again.
The right leg extended straight out the side, then bends at the knee and brings the foot toward the body, then extends straight out again.
The left leg is extended straight out to the side, then bends at the knee and brings the foot toward the body, then extends straight out again.
The torso is upright and facing forward, then leans the side as the arms are brought in, then straightens as the arms extend outward again.
LLaMA Output:
The right arm bends downward from straight to bent when grabbing the right leg.
The left arm extends from bent to straight when stretching outward.
The right leg is lifted upward by the right hand.
The left leg remains stationary, providing balance.
The torso leans forward slightly, aligning with the direction the stretch.
A person picks something with his right hand, then raises it slightly several times.
# LLavA Output:
The right arm extended downward and then retracted upward.
The left arm is extended downward and then retracted upward.
-
The right leg is extended forward and then retracted backward.
The left leg is extended forward and then retracted backward.
The torso is bent forward and then straightened back
# LLaMA Output:
The right arm bends from straight to bent when reaching downward and then extends from bent to straight when lifting the object.
The left arm remains relaxed under the shoulder by the side of the body.
The right leg is stationary, providing balance.
The left leg is stationary, providing balance.
- The torso leans forward slightly when reaching downward and then returns to upright when lifting the object.
Figure 5: Some examples of the part motion text generation results of LLaVA and LLaMA. The output deseriptions that differ greatly from the actual motions are highlighted in red.
Finally, select LLaMA3-70B for linguistic knowledge generation, reasoning capacity is more critical than additional visual prompts for our task. Another reason is that in the context of GAR, paired motion-tex. data is often unavailable during testing. Therefore, jointly utilizing the motions and texts as prompts, such as LLaVA, is not preferred. Although this joint strategy be considered enhance the text quality during training, we find that general text-only generation strategy can already achieve satisfactory performance.
find that its superior
Overall, paradigm is compatible with different generation strategies, which is not the very focus of this work. We believe that as the capabilities of future LLMs improve, the proposed GaR-based solutions promising.
Next, we further discuss the generation quality and model robustness to support our motivation.
21

---

Input Motion Text: person holds both hands in front of them, punching forward with the left hand, then punching to left hand side before stopping.
LLaMA Output:
The right arm is holding straight in front, then moving to the right (when punching to
the left hand side) and stopping.
The left is holding straight front, bending the elbow when punching forward,
then extending when punching to the left.
The right leg [udf].
-
The left leg is [udf]
The torso is [udf]
Input Motion Text: person breakdances while kicking feet and then flips backwards
while spinning on their hands, landing on their feet.
# LLaMA Output:
The right arm is bending from straight to bent when placing hands on the ground,
and then extending from bent straight when pushing off ground to flip backwards.
arm bending from straight to bent when placing hands on the ground, and then extending from bent to straight when pushing off the ground to flip backwards.
- The right leg is kicking outward, lifting off the ground, and then tucking inward when
flipping backwards, and finally landing the ground, #missing walking forward
The left leg is kicking outward, lifting off the ground, and then tucking inward when flipping backwards, and finally landing on the ground. #missing walking forward The torso is twisting rapidly when spinning on hands, and then aligning with the direction of the flip when landing on feet.
Figure 6: Some examples of the imperfect generated part descriptions. For example, the right arm "holding straight in front" is accurate in the first case. And in the second case, the man walks forward first as shown in the rendered images while the leg motion texts miss this movement. However, these imperfect part texts, due the lack visual information incomplete global motion, are still closely related to the part motions to benefit the model training.
Table 10: Type-Token Ratio analysis of the generated part motion descriptions LLaVA and LLaMA in HumanML.31 dataset. A higher metric indicates more diverse corpus.
Knowledge Experts
Type-Tokee Ratio f
LLaVA I. .6-34B LLaMA3-70B
0.360
0.416
A.2 Discussion on the Quality of Generated Texts and Model Robustness.
Based on our observation, most L.M--generted part descriptions effectively capture local motion patterns with clear kinematic phrasing. Although subset of results (particularly for complex motions, as illustrated in Figure (6) exhibit minor deviations from ground truth, they are acceptable and still closely associated with the actual movements, which usually deviate only in some small movements. With the supervision the accurate ground-trutt global motion descriptions, the additional part texts introduce structurally decomposed knowledge regularization. Therefore, the generated part texts can still serve as beneficial knowledge for model training. as evident Table 9
On the other hand, we study the model robustness on the quality of the generated part texts. We artificially perturb the quality imperfections in the part texts generated by LLMs.. Specifically, we randomly select and exchange the part texts of different data samples, or randomly delete certain proportion of words, or randomly add some predefined motion descriptions of part movements noise. In implementation, to eliminate
part texts for portion of the training data to roughly simulate
22

---

Table 11: Retrieval results using artificially perturbed part texts for training. Noise ratio means the
part texts of ratio of training data are perturbed through human intervention roughly simulate the imperfections in the part texts generated by LLMs.
Text-fo-motion retrieval
Motion-to-fext retrieval
Noise Ratio
R@s
R@
MedR
R@5
MedR
No Part Training
9.87
26.36
37.38
19.50
10.19
26.56
36.63
21.50
0%
11.27
28.44
41.21
16.50
12.57
27.87
39.74
19.00
10%
11.06
28.34
40.80
16.50
12.87
27.80
38.92
19.50
20%
10.45
27.07
39.05
18.00
12.21
27.04
37.49
19.50
30%
10.62
26.41
38.24
19.00
11.12
26.38
36.92
21.00
the influence of other proposed alignment strategiee, we only use global and local contrastive learning objectives for training. The results are shown Table As we can see, our paradigm is robust the part text quality. When the noise ratio is less than 30%, can always show non-trivial performance improvement compared to training without part involvement. This is mainly because under the dominant global supervision, slight noise from body parts will not make the model to learn "wrong" representations. Meanwhile, due to the relatively low diversity of body part motions, this adverse effect is further weakened. Instead, the knowledge of body parts can be used an effective regularization to guide the precise alignment learning of the model, thereby achieving performance improvement.
B
Visualization Results
B.1 Motion-Text Retrieval Comparison with Other Methods
We present more motion-to-text and text-to-motion retrieval results in Figure 8 and Figure for comparison. As we can see, owing to the part motion knowledge modeling, our method can retrieve results that are more consistent at fine- ane-ggained semantics, e-g., "at knee height" (the third case in Figure 7), "in circle" (the third case in Figure (8. This is not observed in the bascline method MoPatch [45], which solely applies the global alignment, leading to the potential over-fitting with limited data.
B.2 Motion-Text Retrieval Results with/without Testing GAR
To better illustrate the effect our proposed testing-augment policy, i.e., GAR using additional part information generated by LLMss, we present some qualitative results of text-to-motion retrieval results with/without Testing GAR in Figure 9
As we can see, providing additional part information is generally beneficial for retrieval during testing. More precisely, the "generation" of part information more like kind of information reformulation to explicitly associate the motion patterns with the linguistic verbs. Although part motion information derived from full-body motion descriptions, exploiting can still bring about improvement. We believe that this improvement mainly comes from a more precise local concept matching constraint. This leads to the explicit retrieval of local motion concepts. For example, the model without testing GAR can generally retrieve the roughly similar motions according to the queries in Figure However, is still insufficient for the precise matching some local motion patterns. Especially when dealing with some long text queries, some movement details can be ignored by the text encoder. At this time, the decoupling structural parts can effectively enhance the loyalty of retrieval results to this query information.
23

---

MoPatch
Ours
Rank #1
Rank #2
Rank #1
Rank #2
person jumps side with legs and arms then squats few times with arms raised.
squats knees facing inwards then stands up and quickly jogs before stopping.
brings right elbow to left knee, then left albow right knee, stands straight then bends at the knees few times.
person jumps side with legs and then squatts few times with arms raised.
squats the way to the ground, then leaps up all the way, raising both hands above the head.
person walks forward, and repeatedly reaches a down then shakes something.
A person walks forward and appears to pour two items into bowl.
The person watch forward and is leaning over some thing.
person walks forward, and repeatedly reaches down then shokes something.
A person walks forward and picks things up and puts them down with their hands.
A steps forward, then brushes something at knee height with his left arm, then steps back and stretches his arm out to either side.
persan step and waves his left back forth.
person forward and is leoning over some thing.
person stops for ward and picks something up his left. hand, before stepping bock and holding sides.
man stops forward, then brushes something ot knee height with leftt then steps back and stratches out to either side.
Figure 7: Comparison results of lext-to-motion retrieval between MoPatch [45 and our proposed SGAR. The t retrieval motion results with their textual annotations shown here. Green texts indicate the perfect match of the query text. All motions in the gallery are from the test set and were unseen during training.
24

---

MoPatch
Ours
Rank #1: man bends over and puts his hands the ground and fours.
Rank #1; A man does a push up and then his bock fect.
balance himself
Rank #2: The figure rises from a laying position and wolks
Rank #2: person doing tarsa-bwists, side-bos bendds, and a lowar
counterclochwwse circle, and then lays back down the ground.
bock/hamstring stretch.
Q
Rank #3. person who is prone pushes up off ground using his arms and propping himself with his knees before standing owkwordly ready to wrestle position.
Rank #3: figure laying the ground then rolls into sit, then into being fours, then standing.
#4: The person laying and then they got
Rank #4: The person was lcying down and they got up.
Ground-Truth Text: man does push up and then uses his arms to balance himself back his feet.
Rank #1: person walks forward and appears to pour two items into bowl. Rank #2: A man steps forward, then picks something up with his right hand, then with his right hand, brings them close together, and them back down in the some order.
Rank #1: This
forward
and grabs an item then moves left up and down.
Rank #2: Person walks forword with left hand extanded to side, trying to feel something.
Rank This person steps forword and grabs an item then moves his right up and forward, picks oloject, and raises towards their
Rank #3: man steps forward, then picks something up with his hand, then with his left hand, brings them close together, and them back down in the same order.
person gracefully
Rank #4 A person walks forward and moves something
right
Ground-Truth Text: This person steps forward and grabs an item then moves his left arm up and down.
Rank #1: person crouched down and walking around sneakily. Rank #2: A the ground with both knees banded moving from side to anather, trying hide or something.
Rank #1: person limping with leftt leg hurt and going around circle.
with bath feet
Rank #2: person limping with right leg hurt and going around in circle. Rank #3: person walks in counterclockwise circle while bent forward at the waist, and holding their left leg both hands.
Q
Rank #3: A man slowly leaiss forward around and object.
heavy
person circle while bent forward at the waist, halding right log hands.
clockwise
Rank #4. A person is sneaking around.
Ground-Truth Text: A person walks in counterclockwise circle while bent forward at the waist, and holding their left leg with both hands.
Figure 8: Comparison results of motion-to-text retrieval using our method with and without Testing GAR.
C Additional Experimental Results
C.1 Retrieval Based Only Part Queries
In the previous sections, primarily utilize the part information as the auxiliary information of the full-body motion. Next, we study the motion retrieval performance using only the part-based queries, i.e., full-body motion descriptions.
As shown in Table 12 retrieval only relying on part queries confusing due to the absence of other part motion descriptions. As shown in Figure 3 our model can return reasonable results that differ in
25

---

Ours w/o Testing GAR
Ours w Testing GAR
Rank #1
Rank #2
Rank #1
Rank #2
This person put both arms up front of themselves, while standing still.
Additional Part Info.
The right/left arm is lifting from down to up front themselves
A roises both arms to his face, then drops them bock his sicees.
raises their hands and brings them closer their face then lowers them.
This put both arms up in front of themselves, while standing still.
person roises both arms his face, then draps them back sides.
A person jumps into wide stance with his arms making a shope and then jumps back into normal, relaxed stance
Additional Part Info.
takes
A who jumping around with leg but alternating legs every
person jumps wide stance with arms making shope and then jumps book normal, relexxed stance.
takes
right//t//ett leg jumping outword (from to out) and londing wide stonoe, (from jumsing
big jump forwords.
big jump forwards.
normal position.
Figure 9: Text-to-Motion retrieval results in terms of using not using testing GAR. i.e., jointly considering the full-body motion-text similarity and the part motion-text similarity to derive the final retrieval results.
Table 12: Text-to-motion retrieval results by only part queries on HumanML3D benchmark.
Text-to-motion retrieval
Query Type
R@1
R@2
R@3
R@5
@@ 10
MedR
Left Arm
1.92
3.06
4.63
7.17
12.12
171.00
Right Arm
2.46
3.81
5.18
8,06
12.87
150.00
Torso
3.54
5.27
7.12
10.20
16.91
89.00
Left Leg
2.58
3.63
5.39
8.31
13.90
154.50
Right Leg
2.81
3.90
5.91
9.01
14.67
127.50
All Parts
6.21
9.63
13.16
18.18
27.18
37.00
the global motion but match the query for local motion patterns, which can support partial match retrieval (2)] We also study the retrieval by averaging several types of part information to jointly perform retrieval as shown in Table 12 However, the performance is still lower than using full-body motion descriptions. This is because when decoupled to local part motion descriptions, for the query to contain global motion semantics and the temporal correspondence, resulting in degraded retrieval results. Therefore, our key idea is to use part information as auxiliary information for retrieval or to enable the possibility of partial match retrieval.
difficult
C.2 Different VIT Backbones
Table shows the results of different sizes of the ViT backbone. As the size of the model increases, the performance improves. Note that this is not observed the previous works [25] 45), where the model still suffers from severe over-fitting problem, resulting in poorer performance of VIT-L than ViT-B. In contrast, our model well exploits the more general part knowledge enable better generalization, alleviating the over-fitting problem with limited training data.
26

---

Table 13: Ablation studies of different VIT backbone sizes on HumanML3D benchmark.
Text-to-motion retrieval
Motion-to-text retrieval
VIT Size
R@1
R@5
R 10
MedR
R@1
R@5
R@ 10
MedR
Small
11.79
28.85
41.61
16.00
12.64
29.13
39.94
17.00
Base
12.86
30.75
43.00
15.00
13.82
30.09
41.83
16.00
Large
13.05
31.84
44.32
14.00
13.73
31.18
42.01
16.00
C.3
More Ablation Study Results
We present the ablation results of the hyper-parameters, Amir and A the loss weight of the mixing objectives in Table 15 and the relational regularization in Table [4 We choose the best settings for implementation.
D Approach and Implementation Details
D.1 Part-Mixture De-correlation Learning
Here provide detailed introduction of the part-mixture learning method. First, the mixed motion sequence m is constructed. Specifically, giving two randomly sampled motions mi my, we randomly swap the patches of 2 parts (5 total) for the mixing operation. Then the mixed data are fed into the motion encoder to obtain the full-body and part motion embeddings, Am,mi and mi (recall that j indicates the juh part). Meanwhile, the text encoder takes the full-body and part textual descriptions inputs, and outputs the corresponding embeddings žr.; and {=ī,});- Then, we calculate the global and local alignment objectives of the mixed motions for optimization. Although the implementation is not complicated, its formalization is somewhat cumbersome, and try to present it as clearly as possible.
For global aligmment, the positive text embedding of mm,mir is the linear blending of 2 and 2t, iee, Ztmmiz Am 2t,; + (1 Am 2t, where the Am is the mixing ratio determined by the number of replaced parts. Then, the global mixed alignment can be formulated as
exp(2m mam. "t.mii T)
emin.m2t full
log
Eg exp( eexp(mm.n Zmi 2t.j/T)
(5)
Cmian22m
log
exp(Zmmmake Pt.mke/)
full
E,
exp(2mm. Zmm. 2t,min/
This objective introduces more diverse motion patterns for motion-language representation learning, improving the model performance.
On the other hand, we rearrange the part text embeddings according to the sampling process in mixing operation and match the part motion embeddings of the ith mixed motion the corresponding part text embeddings. Denoted the rearranged part text embeddings {z] } that matches the part
Table 14: Ablation study on the loss weight of directional relation regularization.
Table 15: Ablation study on the loss weight of part-mixture de-correlation learning.
T2M @@
M2T R@
Amiz
T2M R@
M2T R R@)
0.007
12.73
13.75
0.3
12.93
13.60
0.1
12.86
13.82
0.5
12.86
13.82
0.3
12.61
13.66
0.7
12.48
13.63
0.5
12.25
13.54
1.0
11.93
13.37
27

---

The multiple text descriptions of the same person's motion will be provided. Please decouple the specific pose and movement descriptions of different body parts from the total body mation descriptions. Default that the performer's dominant hand is the right hand. Body ports should include [right left right leg, left leg, torzo], If the specific movement of body cannot be determined, denote its motion as "undefined!:
Output the description ench body port in the following format The (body part] ... For the output, use general words aligning with common sense and keep the sentence os simple as possible.
Q: la man bowling with right hand [a man swung his right hand backward, then threw something forward]]
A
The right aim bending from stroight to bent under the shoulder when preporing release the ball and then extending from bant straight when raleasing the bail.
The left omm remoins reloxed under the shoulder by the side of the body.
The right leg is undefinad.
The left leg is undefined. torso the throw.
slightly during the the bowling ball, aligning with direction
O: [xox]]
P 0.0
The right arm - The left amm.... The right leg... The left The torso...
Figure 10: The text prompt of the part motion description generation with LLaMA3-70B.
motion embeddings ( Jj i the part-mixture alignment objective is
P
2 m ((.: logp (2m ((
part
P
P
j=1
Cmix..21 part
Σ j=1
P
( ( "m log (zt
(zlmiiz of =mrr:
(6)
exp(z, zi
min TT
exp(z, min,
where Pk 2, ((::7)
Σ exp(2 m x/T) Pk (z, 2) 7)
Σ exp(2 m/
m is the kih mixed sample and we have omitted k in some cases where there is no ambiguity to reduce clutter.
D.2 Experiment Settings and Implementation Details
For the motion-language alignment pre-training, we conduct the experiments on a single NVIDIA A40 GPU. We present the details of the downstream tasks as follows:
1) Motion-Text Retrieval. In the paper, the adopted protocols include: a) All: The entire test set used the gallery set without any modifications. This is challenging setting because the recall scores can be directly affected by several texts with very conceptually similar meanings but different words, e-g., "walk" or "walking". b) Small Batches (Small): This protocol is designed by Guo al. [ It involves randomly selecting batches of 32 motion-text pairs and then reporting the average performance. While this approach introduces randomness, serves as a benchmark for comparison. Note that the batch size of 32 is relatively manageable, making it less challenging scenario.
For our texting-time augmentation version, SGAR++, we first calculate the similarity maps of the full-body motions and each part motions. Then, we do a weighted sum over them, and the weight ratio between the full body and any part is 10 1. The final calculated similarity map is used to determine the ranks for retrieval.
2) Transfer Learning for Action Recognition. We process the BABEL dataset using the Hu- manML style methods normalize the orientation, foot contact, etc. The pre-trained motion encoder with new fully connected (FC) layer is fine-tuned for transfer learning. For the linear
28

---

evaluation, we only train the FC layer with the encoder fixed. The learning rate is 0. with a batch size of 512. For the full fine-tuning, the learning rate is 0.01 with batch size of 384.
Motion-Text Retrieval. This is the same as Motion-Text retrieval, except for the difference of the training dataset and testing dataset. We utilize the HumanML33D dataset for training and Motion-X for testing.
3)
4) Zero-Shot sealeton-baseie Action Recognition & Text-guided Motion Generation. This part mainly to evaluate the text encoder. Therefore, we simply replace the original text encoder in SA-DVAE [3), MDM [36], and T2M-GPT [47] baseline algorithms with other text encoders to conduct the experiments. Other implementations are not changed the baseline algorithms.
5) Prompts for LLM Generation. A few-shot prompting strategy is employed, with the detailed prompt shown in Figure TO
E Limitations and Future Work
As discussed before, proposed paradigm relies the generated part texts by LLMs, which may exhibit minor imperfections for highly complex motions. As extensively analyzed Sec. Scc. Appendix and Table TT experiments demonstrate that despite these challenges, existing LLMs can almost always be expected to boost performance, indicating the promising potential of the GAR paradigm for motion-text retrieval. Meanwhile, future work could explore temporal information generation, which remains challenging due to the lack of fine- nee-grained temporal correspondence (a.k.a weakly-supervised temporal grounding problem). This further highlights the scalability of the GAR paradigm to inspire future research.
Broader Impact
Accurate human motion-tex. retrieval can effectively improve technologies such as virtual assistants and rehabilitation robots, and enhance the
experience. Mcanwhile, it is also expected to be applied to the action demonstration in physical education teaching, dance teaching, etc.. Furthermore, the storage of traditional dance, martial arts, and other movements in the form of texts may help digitally preserve cultural heritage and have positive impact. The other side of the coin is the issue of privacy protection, which needs attention to avoid being used for monitoring specific groups of people.
29


Reviews:
[{"review_id": "qxeQOSesWI", "reviewer": "Reviewer_5qdg", "summary": "This paper presents SGAR, a novel framework for 3D human motion-text retrieval that leverages structural generative augmentation from large language models. The key innovation is decomposing holistic human motions into fine-grained semantic units (body parts) to enable more precise cross-modal alignment. The method demonstrates strong performance on motion-text retrieval benchmarks, showing significant improvements over previous state-of-the-art methods.", "questions": "Q1: How can we ensure that the query generated by LLM is correct or meets the search intent?\nQ2: How do we explain the fact that the test is better without LLM? Is it essentially because more data (generated by LLM) is used?\nQ3: This method does not seem to be suitable for action-to-text retrieval? For text-to-action, LLM can enhance text queries, but for action-to-text retrieval, LLM needs to enhance every sentence in the database, and the real-time retrieval is very poor.", "limitations": "Potential error propagation from LLM hallucinations unquantified\nThere seems to be a limitation in the real-time nature of retrieval", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "S1: Well-designed ablation studies validate each component's contribution\nS2: Its novel part-mixture learning and directional relation alignment effectively enhance motion representation. \n\nW1: The discussion on computational efficiency is limited. Providing details on runtime performance and resource requirements would help assess the practical trade-offs involved.\nW2: The novelty of the approach could be better contextualized by comparing more explicitly with prior part-based motion retrieval methods.\nW3: How can we ensure that the query generated by LLM is correct or meets the search intent?", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback and constructive suggestions! We hope our responses adequately address the following questions raised about our work. Please let us know if there is anything we can clarify further.\n\n---\n\n**W1: Computional Efficiency**\n\nAs suggested, we study the computational efficiency with FLOPs and Runtime. All experiments are conducted on one NVIDIA 4090 GPU and Intel(R) Xeon(R) Platinum 8358P CPU @ 2.60GHz. \n\nAs shown in the following table, our SGAR has similar complexity and runtime to other methods. When testing GAR is enabled (SGAR++), the main additional cost is the encoding of the extra part texts (1.05G for once forward of text encoder, here we take 5 parts for example). In terms of the runtime, it should be noted that the encoding of part texts is highly parallel and does not incur much time cost.\n\nTable 1: Comparison of computational complexity and runtime.\n| Method       | FLOPs                           | Runtime  |\n|--------------|---------------------------------|----------|\n| ChronRet     | 5.48G                           | 0.0072s  |\n| MoPatch      | 6.46G                           | 0.0078s  |\n| SGAR (Ours)  | 6.89G                           | 0.0083s  |\n| SGAR++ (Ours)| 6.89G+1.05G*5=12.12G            | 0.0086s  |\n\n---\n\n**W2: Comparing with other part-based motion retrieval methods**\n\nThanks for your advice. We have studied the previous works for 3D motion-text retrieval, and take the following works that are relevant to body parts as discussions. We distinguish our method and novelty from the following aspects:\n\n- First, in terms of the paradigm, we argue that our method explores the part motion modeling under a new paradigm of generation-augmented retrieval (GAR), which is different from [1] and [2]. This enables the testing augmentation option for performance boosting. \n- For detailed methodology, [1] considers the word-level features with a gating mechanism, to align with fine-grained motion features. However, it fails to model separate part motions on the motion encoder side with the coupled encoding of different parts. Meanwhile, it is difficult to learn this fine-grained motion-text correspondence due to the lack of explicit decoupling of linguistic knowledge as guidance. Besides, [2] explicitly considers the part motion description generation as extra knowledge. However, a naive solution for part alignment is adopted, i.e., directly using a contrastive learning objective. This ignores the difference of part motions compared with full-body motions, which are more coupled with each other due to the concurrence of different part motion patterns. In contrast, we apply LLM-based knowledge decomposition and propose part mixture decorrelation learning with directional relation regularization design, presenting a comprehensive part motion modeling framework. \n- Finally, we compare the performance of the above works in the following table, where we can see that our method has a desirable performance advantage.\n\nTable 2: Comparison of motion-text retrieval on HumanML3D dataset.\n| Method | M-to-T   |          |          |          |          | T-to-M    |          |          |          |          |\n|--------|--------------------------------|----------|----------|----------|----------|--------------------------------|----------|----------|----------|----------|\n|        | R@1                            | R@3      | R@5      | R@10     | MedR     | R@1                            | R@3      | R@5      | R@10     | MedR     |\n| [1]    | 11.36                          | 19.69    | 25.75    | 36.02    | 23.25    | 7.14                           | 17.4     | 24.02    | 34.67    | 24.00    |\n| [2]    | 9.29                           | 20.61    | 28.29    | 40.25    | 18.00    | 8.13                           | 19.69    | 27.07    | 39.18    | 18.00    |\n| **SGAR**   | **12.32**                          | **23.36**    | **30.47**    | **41.45**    | **16.00**    | **8.71**                           | **20.79**    | **27.92**    | **40.94**    | **17.00**    |\n| **SGAR++** | **11.76**                          | **23.47**    | **30.56**    | **43.18**    | **15.00**    | **9.38**                           | **22.23**    | **30.08**    | **42.58**    | **15.00**    |\n\n**The codes of [1,2] are not available. Therefore, we follow their test protocol that ignores the misjudgments caused by mirror augmentation for evaluation correction, and report our performance under this setting for fairness.*\n\n[1] Hierarchical Semantics Alignment for 3D Human Motion Retrieval, SIGIR 2024.  \n[2] KinMo: Kinematic-aware Human Motion Understanding and Generation, arXiv 2024.\n\n---\n\n**W3 & Q1: How to ensure LLM generated query is correct?**\n\nThanks for the insightful question. This is an important problem that we have made efforts to study the corresponding design. We explain our consideration from the following aspects:\n- First, the generation of part motion texts based on the given full-body motion texts can be viewed as a process of knowledge decomposition or refinement from coarse to fine. From the perspective of users, the provided queries should include all the content that is intended to be emphasized. Otherwise, any other unrestricted associated content should be regarded as reasonable. Therefore, at this point, the LLM only needs to perform relatively simple part-wise decoupling of the provided coupled full-body motion texts. Many recent studies [1-5] have demonstrated that LLMs possess such capabilities, even in tasks that are more complex than human motion text data, which provides support for our design.\n- Technically, we carefully design the LLM prompts, i.e., a few-shot strategy with [udf] indicator for uncertainty, to improve the quality of generation.\n- On the other hand, due to the current limitations of LLMs and the existence of hallucination problems, this automated method cannot achieve 100% correct output results. Therefore, we randomly selected 200 samples and conducted manual checks, with an error rate of approximately 9%. Fortunately, our method has demonstrated a certain degree of robustness to the quality of part texts, as shown in Table 10 in the Appendix, which enables this design to bring expected performance improvement to motion-language models. \n\n[1] Large language models are versatile decomposers: Decomposing evidence and questions for table-based reasoning, SIGIR 2023  \n[2] Least-to-Most Prompting Enables Complex Reasoning in Large Language Models, ICLR 2023  \n[3] Mldt: Multi-level decomposition for complex long-horizon robotic task planning with open-source large language model, arXiv 2024  \n[4] Reasoning Abilities of Large Language Models: In-Depth Analysis on the Abstraction and Reasoning Corpus, ACM TIST 2025  \n[5] Tree of Thoughts: Deliberate Problem Solving with Large Language Models, NeurIPS 2023\n\n---\n\n**Q2: Explain why testing is better without LLM**\n\nThis can be explained from three aspects:\n- The key motivation is to decompose and model the part motion consistency to benefit the general motion-text alignment learning. Taking \"walks and waves hand\" as an example, with single global alignment learning, the model may incorrectly correspond the motion pattern of arm swinging to the verb \"walk\" rather than \"wave hand\", especially when training with a limited data scale. This means the difficulty of motion-text retrieval arises not only from the cross-modal alignment but also from the reorganization and decomposition of semantic concepts within the same modality. In this context, learning the correspondence of local concepts as primitives can lead to more precise, generalized local alignment, and ultimately enhance global understanding.\n- It is also beneficial to introduce more diverse part texts generated by LLMs, which alleviates the over-fitting problem with more data.\n- Moreover, as shown in the ablation study Table 2, the proposed part mixture learning and directional relation regularization further boost the performance with no LLM during testing. These designs introduce additional knowledge, e.g., novel mixed motion patterns and relational knowledge, to boost the overall performance and generalization.\n\n---\n\n**Q3: Efficiency on the motion-to-text retrieval with LLM**\n\nThanks for this valuable question! Actually, in motion-to-text retrieval, numerous inferences of LLM can degrade the real-time efficiency. Fortunately, we can alleviate this problem through some simple strategies. For example, the following two strategies can be considered\n- **Pre-computed Text Augmentation**: During the database construction stage, LLM can be used in advance to generate part motion descriptions as supplementary (or the database can use LLM to generate it offline during idle time). The augmented part texts are vectorized with the index established. When querying, we can directly retrieve the pre-augmented database to avoid the overhead of real-time generation.\n- **Hierarchical Query**: Another solution is to use hierarchical query. For instance, we can first use only the full-body motion texts to retrieve the Top-k candidates in the database, and then perform more fine-grained LLM augmentation on these Top-k texts with the GAR strategy. This can effectively reduce the number of LLM inference. To support this scheme, we provide the results of the hierarchical query when using different k (Top-k candidates). We can see that a small k (10~20) can also boost the performance significantly.\n\nTable 3: Results using hierarchical query with different k on motion-to-text retrieval on HumanML3D. \n| Top-k    | R@1   | R@3   | R@5   | R@10  | MedR  |\n|-------------|-------|-------|-------|-------|-------|\n| 0 (no LLM)  | 13.82 | 23.38 | 30.09 | 41.83 | 16.00 |\n| All test data (4380)  | 14.64 | 25.52 | 32.55 | 44.50 | 14.00 |\n| 20          | 14.65 | 25.60 | 32.59 | 44.39 | 14.00 |\n| 10          | 14.67 | 25.63 | 32.43 | 41.85 | 16.00 |"}, {"review_id": "0ckbehi86g", "reviewer": "Reviewer_JXyx", "summary": "This paper proposes SGAR, a framework to improve motion-text retrieval performance by LLM-based augmentation. Specifically, it utilizes LLMs to decompose human motion descriptions into body-parts motion descriptions. It further proposes a part-mixture learning to make full use of the partitioned data, and augment training data pairs. Extensive experiment demonstrate the proposed method achieves better performance compared to previous SOTA methods. It also provides promising results in improving related downstream tasks by using the trained text/motion encoder.", "questions": "1. Can you explain how only using Part CL could boost the performance, which seems to be not directly related to the global semantics?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "## Strengths\n1. The LLM-based motion decomposition is very reasonable. The proposed part-mixture learning for data augmentation is interesting and shows good improvement.\n2. The Directional relation alignment strategy borrow from language modeling is also interesting and novel for the motion task.\n3. The proposed augmentation naturally come up with test-time augmentation strategy that is able to boot the performance effectively as demonstrated by the quantitative experiments.\n\n## Weaknesses\n1. One major issue is the concern around reproducibility. The good performance could be better demonstrated by any kinds of video demo, interactive web demo or code that reviewers can see or test. Otherwise, it could be a bit skeptical. Another alternative is, for the major experiments, train the models with different seeds and report the average performance and error bar, since retrieval results could be sensitive to random seed at both training and testing.\n2. I’m not fully convinced by the part-mixture learning design.  When mixing data sample by exchanging body parts, what is the reason to require the latent to be the interpolation of their corresponding latent with the corresponding ratio. For example, why replacing the right arm or left leg corresponds to the same semantic latent, according to the formula, if i understand correctly? Are there any related work to support this design decision?", "was_revised": true, "final_justification": "This work is clearly presented and meets the general quality standards of NeurIPS. However, due to the simplicity and narrow scope of the motion retrieval task, I have reservations about the broader impact and originality of the proposed LLM-based augmentation method. While there are no major flaws, the contribution is relatively modest and borders on incremental—especially in light of the declining bar for acceptance.", "rebuttal": "We thank the reviewer for the valuable feedback and constructive suggestions! We hope our responses adequately address the following questions raised about our work. Please let us know if there is anything we can clarify further.\n\n---\n\n**W1: About reproducibility, i.e., code, performance error bars**\n\nThanks for your helpful comments. First, the code would be available upon publication. Due to the author's policy of NeurIPS, we are sorry that we can not provide any link to a demo or code during rebuttal. Alternatively, we report the average performance with error bar to better demonstrate the performance fluctuations in the experiment under 5 runs. As mentioned, retrieval results could be sensitive to randomness, while we can see that our method can consistently achieve the SOTA performance, verifying the effectiveness of our method.\n\nTable 1: Average performance with the error bar.\n| Method | (T2M) R@1           | R@3           | R@5            | R@10          | MedR         | (M2T) R@1           | R@3           | R@5            | R@10          | MedR         |\n|--------|---------------|---------------|----------------|---------------|--------------|---------------|---------------|----------------|---------------|--------------|\n|HumanML3D |        |               |               |                |               |              |               |               |                |              |\n| MoPatch | 10.80         | 20.00         | 26.72          | 38.02         | 19.00        | 11.25         | 19.98         | 26.86          | 37.40         | 20.50        |\n| ChronRet | 8.30          | 16.70         | 23.65          | 35.31         | 22.00        | 9.08          | 11.22         | 23.68          | 34.31         | 23.00        |\n| **SGAR** | **12.82 ±0.448** | **23.08 ±0.724** | **30.72 ±0.231** | **43.45 ±0.656** | **14.20 ±0.447** | **13.71 ±0.334** | **23.41 ±0.399** | **30.45 ±0.509** | **41.57 ±0.509** | **16.00 ±0.000** |\n| **SGAR++** | **14.12 ±0.301** | **25.12 ±0.337** | **32.76 ±0.415** | **45.48 ±0.627** | **13.00 ±0.000** | **14.55 ±0.399** | **25.33 ±0.383** | **32.67 ±0.206** | **44.72 ±0.214** | **14.00 ±0.000** |\n| KIT-ML    |        |               |               |                |               |              |               |               |                |              |\n| MoPatch | 14.02         | 28.91         | 34.10          | 50.00         | 10.50        | 13.61         | 27.54         | 33.33          | 44.77         | 13.00        |\n| **SGAR** | **16.02 ±0.629** | **30.08 ±0.693** | **40.15 ±1.236** | **54.30 ±1.848** | **9.00 ±0.707** | **14.82 ±0.360** | **26.22 ±0.805** | **35.63 ±1.317** | **49.40 ±1.987** | **10.20 ±1.095** |\n| **SGAR++** | **16.27 ±0.641** | **31.49 ±0.546** | **41.04 ±1.754** | **56.83 ±2.294** | **8.00 ±1.000** | **16.71 ±0.686** | **29.72 ±0.386** | **38.83 ±0.739** | **52.73 ±1.094** | **9.00 ±0.000** |\n\n---\n\n**W2: Further explanation on part-mixture learning design**\n\nThanks for your valuable question. We design the part mixture learning inspired from both the success of previous image-based methods but also the characteristics of skeleton data.\n\nSpecifically, \n- On the one hand, this can be referred in the image-based mixing contrastive learning works [1-3]. Similarly, they copy-and-replace the image patches like Cut-Mix, using linear interpolation of the embeddings of the images before mixing as positives for contrastive learning. Similarly, mixing ratio is adopted as the interpolation coefficient. In this case, the mentioned problems can still occur, that is, replacing image patches of the same size in the upper left corner or lower right corner may result in the same interpolated latents for contrastive learning. This seems to be an inherent defect of random mixing operations, whether it is for supervised learning or self-supervised contrastive learning. However, despite this, the success of these works has proved the effectiveness of the design. This can mainly be interpreted as an effective regularization of the feature space *in statistical sense* by applying linear equivariant transformation constraints during training [3]. To help understand, interpolation in the InfoNCE-objective-applied feature space can be viewed similar to interpolation in the prediction label space of supervised learning, (like original Cut-Mix for classification), which both perform interpolation / smoothing in the deep feature space of models.\n- In addition, previous work [4] has revealed the essence of local motion pattern modeling for skeleton data, which means a subtler meaningful semantic unit as body parts. This makes different body parts all potentially contribute to the extraction of a certain global motion concept. Therefore, the mixture at the input data level can be expected to achieve interpolated features in the high-level semantic latent space. Meanwhile, as you understand, although we do not restrict which parts to be mixed (the importance of different parts to certain global motion may vary), this design is still statistically reasonable and empirically effective as discussed above. Therefore, we adopt this simple part-wise replacement strategy, retaining the meaningful motion semantic unit while encouraging a smoother representation space as regularization. Furthermore, from the authors' personal experience, mixing augmentation improves performance mainly through the representation smoothing constraint as regularization rather than obtaining strictly precise linear interpolation coefficients.\n\n[1]i-Mix: A Domain-Agnostic Strategy for Contrastive Representation Learning, ICLR 2021.  \n[2]MixSiam: A Mixture-based Approach to Self-supervised Representation Learning. arXiv 2021.  \n[3]Un-Mix: Rethinking Image Mixtures for Unsupervised Visual Representation Learning, AAAI 2022.  \n[4]Actionlet-Dependent Contrastive Learning for Unsupervised Skeleton-Based Action Recognition, CVPR 2023.\n\n---\n\n**Q1: Explain why only Part CL is better**\n\nOnly part CL (no LLM is used under testing) still boosts global-motion-based retrieval, which can be explained as follows:\n- The key motivation is to decompose and model the part motion consistency to benefit the general motion-text alignment learning. Taking \"walks and waves hand\" as an example, with single global alignment learning, the model may incorrectly correspond the motion pattern of arm swinging to the verb \"walk\" rather than \"wave hand\", especially when training with a limited data scale. This means the difficulty of motion-text retrieval arises not only from the cross-modal alignment but also from the reorganization and decomposition of semantic concepts within the same modality. In this context, learning the correspondence of local concepts as primitives can lead to more precise, generalized local alignment, and ultimately enhance global understanding.\n- It is also beneficial to introduce more diverse part texts generated by LLMs, which alleviates the over-fitting problem with more data."}, {"review_id": "Tuje1EAaRF", "reviewer": "Reviewer_pgZA", "summary": "This paper presents SGAR, a novel framework for 3D human motion-text retrieval that leverages Structural Generative Augmentation. Rather than treating a motion as a single holistic entity, SGAR decomposes it into part-based components and uses LLMs to generate textual descriptions for each part, enabling further alignment of text and motion representations.", "questions": "As shown in the Weaknesses section, A reasonable explanation will make me raise the score for this work.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths:\n- Part-based Motion Decomposition: The method breaks down human motion into smaller, more precise components (like arm movements, leg steps, etc.), providing a more granular way of aligning motion and text, while the text can be further generated by the LLMs with their prior knowledge.\n- Part-Mixture Learning: A novel strategy to prevent correlation between different body parts’ motions during training by mixing parts from different motions. This encourages the model to learn more diverse and robust representations.\n- Directional Relation Alignment: This approach introduces a consistency regularization that aligns full-body and part-level motion representations by modeling the directional relations between them, improving the consistency and accuracy of alignment.\n# Weaknesses:\n- Line 133, how much the texts with this indicator are excluded? How do the authors require the LLMs to output this indicator for the parts with high uncertainty?\n- Line 166, I didn't see the Euclidean distance in formulas 1 and 2. So  how does the author ensure vector consistency while the inner product of vectors cannot guarantee that?\n- Line 203, If the text encoder of SGAR include LLMs ? (IF NO: How can the human pose be split into parts during the testing?)  (IF YES: Such a huge text encoder may lead to unfairness and application.)\n- In Section 5 (Application), SOTA methods such as ChronRet are not included in the comparison, even though they report relevant results in their respective papers. A more comprehensive comparison is therefore needed.\n- What is k in Formula 2?", "was_revised": true, "final_justification": "The author's response has effectively addressed my concerns. Taking into account the feedback from other reviewers, I would like to give a score of 4.", "rebuttal": "We thank the reviewer for the valuable feedback and constructive suggestions! We hope our responses adequately address the following questions raised about our work. Please let us know if there is anything we can clarify further.\n\n---\n\n**W1:About [udf] indicator and implementation**\n\nThanks for your helpful question. We have calculated that ~15.9% of samples include this indicator and the corresponding texts are excluded in HumanML 3D, while ~18.2% in KIT-ML. \n\nIn implementation, we utilize the designed prompt to guide the LLM to output the [udf] indicator for the parts with high uncertainty. This is inspired by previous research findings [1-5] that LLMs can effectively decompose knowledge and model its uncertainties, thereby perceiving potentially more fine-grained concepts within it. Specifically, we utilize the few-shot prompting strategy to better guide the LLM to perform this uncertainty modeling. The related prompts are (full prompts can be referred to Figure 11 in Appendix):\n\n*...\nIf the specific movement of a body part cannot be determined, denote its motion as \"undefined\".  \nQ: [a man is bowling with right hand.] [a man swung his right hand backward, then threw something forward.]  \nA: {  \nThe right arm is bending from straight to bent under the shoulder when preparing to release the ball and then extending from bent to straight when releasing the ball.   \nThe left arm remains relaxed under the shoulder by the side of the body.   \nThe right leg is **undefined**.   \nThe left leg is **undefined**.   \nThe torso leans forward slightly during the release of the bowling ball, aligning with the direction of the throw.   \n}  \nQ: [xxx] [xxx] ...   \nA:*\n\n[1]Large language models are versatile decomposers: Decomposing evidence and questions for table-based reasoning, SIGIR 2023  \n[2]Least-to-Most Prompting Enables Complex Reasoning in Large Language Models, ICLR 2023.  \n[3]Mldt: Multi-level decomposition for complex long-horizon robotic task planning with open-source large language model, arXiv 2024.  \n[4]Reasoning Abilities of Large Language Models: In-Depth Analysis on the Abstraction and Reasoning Corpus, ACM TIST 2025.  \n[5]A Survey of Uncertainty Estimation in LLMs: Theory Meets Practice. arXiv 2024\n\n---\n\n**W2: Explaination on Euclidean distance in formulas 1 and 2**\n\nSorry for the confusion. In fact, we calculate the inner product of the feature vectors in the **normalized space**. In this case, maximizing this inner product is equivalent to minimizing the l2 norm in the Euclidean space. Specifically, given two vectors x and y that are l2-normalized ($||x|| = ||y|| = 1$)\n$$ \\text{argmin} \\|\\|\\mathbf{x} - \\mathbf{y}\\|\\|_2^2 = \\text{argmin} (\\mathbf{x}^\\top \\mathbf{x} + \\mathbf{y}^\\top \\mathbf{y} - 2\\mathbf{x}^\\top \\mathbf{y}) = \\text{argmin} (1 + 1 - 2\\mathbf{x}^\\top \\mathbf{y}) = \\text{argmax} \\ \\mathbf{x}^\\top \\mathbf{y}.$$\nTherefore, the formulas 1 and 2 can encourage the consistency learning of positive motion-text pairs. We will clarify this in paper.\n\n---\n\n**W3: If the text encoder of SGAR include LLMs?**\n\nOur text encoder **does not** involve an LLM. Instead, we keep the text encoder implementation the same as previous works [1-3], i.e., a pre-trained Distill-BERT, for fairness. LLMs are only for knowledge generation/augmentation as an additional external expert, which is an optional choice and independent of the text encoder (Distill-BERT). Therefore, when the LLMs are available or accessible during testing, they can be directly used for augmentation. If not, part texts can be provided by users (text-to-motion retrieval scenario), or just turn off this part-based augmentation option.\n\n[1]Exploring vision transformers for 3D human motion language models with motion patches, CVPR 2024.  \n[2]TMR: Text-to-motion retrieval using contrastive 3D human motion synthesis, ICCV 2023.  \n[3]Chronologically accurate retrieval for temporal grounding of motion-language models, ECCV 2024.\n\n---\n\n**W4: More comparison results in application experiments**\n\nThanks for your advice. We try our best during rebuttal to conduct additional experiments to include recent methods for comparison, ChronRet (ECCV 24) and LaMP (ICLR 25), following the experimental settings of Section 5. Note that ChronRet has reported the motion generation results with its text encoder and we directly report their original results. Other results are reproduced using the officially released model weights under our experimental setting.\n\nSpecifically, as shown in the following tables,   \n1) ChronRet is mainly designed for temporally aware retrieval task, which does not demonstrate advantages for single action recognition. LaMP introduces a new motion-text matching task based on naive contrastive learning paradigm. However, this simple design can still not alleviate the over-fitting problem effectively, leading to sub-optimal alignment learning and generation capacity. Therefore, our method can achieve better perormance on generalized motion understanding, e.g., transfer learning or cross-dataset retrieval.  \n2) Meanwhile, we find that Lamp and ChronRet present poor performance in zero-shot action recognition with their text encoders. This indicates that these methods struggle to learn generalized text representations with limited data. In contrast, our method involves a more fine-grained part motion text alignment and regularization designs, to achieve more meaningful and generalized text representations.   \n3) For text-guided motion generation on HumanML3D dataset, we find these methods can well boost the performance compared with CLIP baseline, and our method has more advantages on R-Precision metrics compared with ChronNet (*Note LaMP adopts and finetunes a larger text encoder, BERT, which can lead to unfair comparison under a in-domain setting*).\n4) For the main comparsion on motion-text retrieval, our method can significantly surpass ChronNet and LaMP as shown in Table 1 in the main paper.\n\nTable 1: Transfer learning results for action recognition on BABEL.\n| Method       | Finetune    | BABEL Top-1 |\n|--------------|-------------|-------------|\n| ChronRet     | Linear Head | 40.58       |\n| LaMP         | Linear Head | 38.93       |\n| **SGAR** | Linear Head | **42.59**    |\n\nTable 2: Cross-dataset motion retrieval results.\n| Method   | (T2M) R@1 | (T2M) R@5 | (T2M) R@10 | (T2M) MedR | (M2T) R@1 | (M2T) R@5 | (M2T) R@10 | (M2T) MedR |\n|----------|-----------|-----------|------------|------------|-----------|-----------|------------|------------|\n| ChronRet | 22.58     | 47.80     | 64.45      | 7.67       | 21.94     | 47.25     | 62.62      | 8.10       |\n| LaMP     | 19.48     | 46.82     | 63.28      | 7.99       | 19.46     | 42.61     | 59.49      | 9.31       |\n| **SGAR** | **31.05** | **54.77** | **69.22**  | **6.81**   | **30.13** | **52.92** | **66.77**  | **7.46**   |\n\nTable 3: Text encoder for zero-shot action recognition.\n| Method       | NTU 60 55/5 split | NTU 60 48/12 split |\n|--------------|-------------|--------------|\n| ChronRet     | 31.49       | 11.47        |\n| LaMP         | 38.05       | 14.92        |\n| **SGAR** | **83.92**   | **45.66**    | \n\n**LaMP utilizes a larger text encoder, i.e., BERT, while ChronNet and our method SGAR use the smaller DistillBERT.*\n\nTable 4: Text encoder for motion generation on HumanML3D.\n| Method       | R@1   | R@2   | R@3   | FID   | MM-Dist |\n|--------------|-------|-------|-------|-------|---------|\n| T2M-GPT      |      |      |      |      |       |\n| Baseline (CLIP) | 0.491 | 0.680 | 0.775 | 0.116 | 3.118   |\n| LaMP (BERT)  | **0.540** | **0.732** | **0.825** | 0.084 | **2.783**   |\n| ChronNet (DistillBERT)    | 0.528 | 0.717 | 0.810 | **0.074** | 2.915   |\n| SGAR (DistillBERT)    | 0.535 | 0.728 | 0.822 | 0.081 | 2.862 |\n\n**LaMP utilizes a larger text encoder, i.e., BERT, while ChronNet and our method SGAR use the smaller DistillBERT.*\n\n---\n\n**W5: Exaplain k in Formula 2**\n\nSorry for the confusion. In Formula 2, $p^\\tau$ is a similarity distribution that consists of multiple similarity scores between $z$ and each embedding anchor, i.e., $p^\\tau = \\{p^\\tau_1, p^\\tau_2 ..., p^\\tau_N\\}$ where N is the batch size in our implementation. Therefore, $k$ is a subscript in Eq. (2) and $ p^\\tau_k (z,z^j_{t,*}) $ denotes the similarity score between $z$  and $z^{j}_{t,k}$ after softmax with temperature $\\tau$. We will polish this presentation in the final version."}, {"review_id": "AZ8AU8HPnR", "reviewer": "Reviewer_MCsr", "summary": "The paper handles the task of 3D human motion-to-text and text-to-motion retrieval. The main idea of the proposed solution is to leverage LLM to decompose text into detailed descriptions for different body parts. The proposed solution then aligns the text and motion representations at both global and local body part levels, which is achieved by considering a part-mixture decorrelation learning and directional relation alignment strategy. \n\nThe proposed solution outperforms existing methods regarding text-to-motion and motion-to-text retrieval.  Furthermore, the authors extend their discussion to such as action recognition, cross-dataset motion retrieval, and text-guided motion generation.", "questions": "1. In Tab. 1, the authors only compare the proposed solution with Lex [19] on All-HML3D protocol. What about comparisons with Lex on other protocols? Can the proposed solution consistently outperform Lex [19] across different protocols?\n2. Tab. 7 discusses text-guided motion generation. Still, how does the proposed method perform against state-of-the-art text-guided motion generation methods? Furthermore, I noticed that LaMP and ChronNet also study text-motion retrieval and discuss motion generation. What are the results when comparing the proposed method with LaMP [14] and ChronNet [7] on motion generation?\n3. The t-SNE visualization in Fig. 10 is confusing. Why is the proposed method considered better in terms of visualization? Is it because the line segments are generally shorter? Still, for the figure of \"Our-right arm part,\" the paired data points present significant distances.", "limitations": "Yes the authors have addressed the limitations and potential negative social impact.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strength**\n- The paper provides comprehensive experiments, demonstrating competitive performance in text-to-motion and motion-to-text retrieval. Extension to diverse application tasks (Sec. 5) strengthens the submission.\n- The proposed part-mixture decorrelation learning and directional relation alignment strategy effectively aligns the text and motion representation at both global and part levels. \n\n**Weakness**\n- The idea of leveraging LLM to decompose text descriptions into body parts has been previously discussed in motion-text literature [1], where the submission fails to discuss this literature.\n- I still have some concerns related to the comparisons, as listed in the question session. I hope the authors could help address these questions.\n\n[1] CoMo: Controllable Motion Generation through Language Guided Pose Code Editing. ECCV'24\n\n\nI vote for a \"borderline accept\" for the current version. Although decomposing motion descriptions with LLMs is not novel, the proposed part-mixture decorrelation learning and directional relation alignment strategy is interesting. \nMoreover, in the main tasks of motion-text retrieval, the proposed solution generally demonstrates competitive performance, with ablation studies verifying the key designs.", "was_revised": true, "final_justification": "I appreciate the feedback from the authors. I consider that this feedback could address most of my concerns raised in my original comments. As a result, I will keep the original score.", "rebuttal": "We thank the reviewer for the valuable and positive feedback with constructive suggestions! We hope our responses adequately address the following questions raised about our work. Please let us know if there is anything we can clarify further.\n\n---\n\n**W1: Discussion on the previous work [1]**\n\nThanks for the helpful suggestions. While [1] also uses LLM to decouple body motion into part motions, we emphasize the following differences:\n\n- The motivations and objectives are different. The work [1] focuses on the motion generation task, aiming to enhance model generation performance through part motion conditioned generation, which can serve for fine-grained controllable generation. In contrast, we target motion-text retrieval and cross-modal alignment learning, aiming to learn a shared semantic space through explicit part decoupling to mitigate potential over-fitting problems. \n- In terms of the LLM knowledge generation, [1] adopts a complex prompting strategy, i.e., PoseScript+LLM, which we have thoroughly considered and discussed (Line 118-123). To our knowledge, PoseScript, as a non-deep learning method, is highly sensitive to hyperparameters and exhibits fragile performance across different motion domain data, significantly limiting its practical application potential. On the other hand, the PoseScript+LLM generation strategy requires the simultaneous availability of motion and text data, which is only possible during the training stage and contradicts our proposed GAR paradigm. In comparison, we employ a more universal and generalized strategy of few-shot + special token generation only dependent on texts, enabling our model to perform GAR augmentation during the testing stage.\n- The utilization of part text data differs. [1] employs the traditional generative condition injection learning approach, i.e., auto-regressive generation, whereas we propose part mixture decorrelation learning and directional alignment to fully leverage part information for alignment.\n\nWe will add this discussion in the final version.\n\n[1] CoMo: Controllable Motion Generation through Language Guided Pose Code Editing, ECCV 2024.\n\n---\n\n**Q1: More comparison with Lex**\n\nThanks for your advice. Lex [19] is the recent work, while the authors do not make the code open-sourced. Due to its complex training stages and designs, it is difficult to reproduce it well. Therefore, we have to directly copy the reported results of Lex for comparison (**only All protocol is reported in the Lex paper**). We have provided the additional comparison of All protocol on the KIT-ML dataset in the following table, from which we can see he effectiveness of our method, especially for the motion-to-text retrieval. *Meanwhile, from an empirical perspective, the All protocol is well representative of the results of other protocols.*\n\nTable 1: Comparion with Lex on motion-text retrieval.\n| Dataset   | Method | (T2M) R@1 | (T2M) R@5 | (T2M) R@10 | (T2M) MedR | (M2T) R@1 | (M2T) R@5 | (M2T) R@10 | (M2T) MedR |\n|-----------|--------|-----------|-----------|------------|------------|-----------|-----------|------------|------------|\n| HumanML3D | Lex    | 11.80     | 30.81     | 43.36      | 14.00      | 12.39     | 29.25     | 40.34      | 17.00      |\n|           | **SGAR** | **13.51** | **31.75** | **44.75**  | **14.00**  | **14.33** | **30.92** | **42.23**  | **15.00**  |\n| KIT-ML    | Lex    | 15.13     | 36.81     | 54.12      | 8.00       | **15.01**     | 35.63     | 47.53      | 10.50      |\n|           | **SGAR** | **15.42** | **41.33** | **55.90**  | **8.00**   | 14.70 | **37.11** | **50.60**  | **9.00**   |\n\n---\n\n**Q2: More comparison on motion generation with LaMP and ChronNet**\n\nThanks for your question. We have provided additional results on motion generation, including LaMP and ChronNet with one of the recent SOTA methods, MoMask [1], as baseline. As shown in the following tables, our method can benefit the current motion generation methods significantly. Moreover, compared to ChronNet, our method has overall better performance. While compred to LaMP (*Note LaMP adopts the larger text encoder, which can lead to unfair comparison*), our method can achieve better FID scores with smaller DistillBERT text encoder.\n\nTable 2: Comparison with MoMask and T2MGPT as the backbones.\n| Method          | R@1   | R@2   | R@3   | FID   | MM-Dist |\n|-----------------|-------|-------|-------|-------|---------|\n| MoMask          |      |      |      |      |        |\n| Baseline (CLIP) | 0.524 | 0.715 | 0.809 | 0.073 | 2.965   |\n| LaMP (BERT)     | 0.546 | **0.750** | **0.843** | 0.058 | **2.792**   |\n| ChronNet (DistillBERT)       | 0.541 | 0.737 | 0.838 | 0.056 | 2.885   |\n| SGAR (DistillBERT)       | **0.550** | 0.748 | 0.842 | **0.049** | 2.864 |\n| T2M-GPT         |      |      |      |      |        |\n| Baseline (CLIP) | 0.491 | 0.680 | 0.775 | 0.116 | 3.118   |\n| LaMP (BERT)  | **0.540** | **0.732** | **0.825** | 0.084 | **2.783**   |\n| ChronNet (DistillBERT)     | 0.528 | 0.717 | 0.810 | **0.074** | 2.915   |\n| SGAR (DistillBERT)   | 0.535 | 0.728 | 0.822 | 0.081 | 2.862 |\n\n**1) For MoMask, we made efforts to reproduce MoMask during rebuttal with the official code. However, we cannot reproduce the performance reported in the original paper in terms of FID metric (0.073 vs. 0.045). We have found that a few researches also encountered similar problems posted in the github issue. Therefore, during rebuttal, we report the currently reproduced results for reference.*    \n*2) LaMP utilizes a larger text encoder, i.e., BERT, while ChronNet and our method SGAR use the smaller DistillBERT.*\n\nWe also provide more comprehensive comparison of LaMP and ChronNet on motion understanding and zero-shot action recognition in the **W4 of Reviewer pgZA** to demonstrate the advantages of our method.\n\n\n[1]MoMask: Generative Masked Modeling of 3D Human Motions, CVPR 2024.\n\n---\n\n**Q3: Explain t-SNE visualization**\n\nSorry for the confusion. We make efforts for cross-modal feature visualization to provide more intuitive comparison results, although this might not be easy. As explainations, for the left two figures, it is to demonstrate our method has a more aligned motion-text representation space because the line segments (connecting paired motion-text data) are shorter. Then, for the third column figure, it can be viewed as an ablation study to demonstrate the difficulty in performing retrieval only based on part information caused by the less diverse texts and false negative problem in contrastive learning. Therefore, we can conclude that only part-level alignment without global information is insufficient to achieve desirable retrieval performance, as verified in Table 11. Based on this, our idea is to use the part information as a beneficial supplement to enhance the global alignment during testing.\n\nSorry for the confusion again. We are considering some better presentations, e.g., replacing this figure with the feature space of full body+right-arm in the final version to better demonstrate the improvement effect of part-level alignment on the feature space. We will refine this in our final version."}]

Author rebuttal:

Meta-review: This paper introduces SGAR, a framework for 3D human motion–text retrieval that leverages LLM-based motion decomposition, part-mixture decorrelation learning, and directional relation alignment. The method achieves consistent improvements over prior state-of-the-art approaches (e.g., ChronRet, LaMP, Lex) and demonstrates generalization to related tasks such as cross-dataset retrieval and text-guided motion generation.

Across four reviewers, the paper is viewed positively: reviewers highlighted the novelty of the part-mixture and alignment strategies, the comprehensive ablation studies, and the strong empirical results. Concerns raised include: (i) limited discussion of computational efficiency, (ii) the need for clearer positioning relative to prior part-based and LLM-augmented motion retrieval works (e.g., CoMo, ECCV’24), (iii) missing comparisons with some recent baselines in applications such as motion generation, and (iv) reproducibility issues (lack of demos or multiple-seed results). The authors provided clarifications, additional experiments, and committed to addressing these points in the final version.

For the camera-ready, the authors should: (1) expand related work discussion to better position the novelty of LLM-based augmentation and part-level alignment, (2) provide runtime and resource usage details, (3) add or clarify comparisons with recent strong baselines in retrieval and motion generation, and (4) strengthen reproducibility by releasing code and, if possible, demos or averaged results.
Decision: Accept (poster)