A Cross-Dataset Study for Text-based 3D Human Motion Retrieval

Venue: thecvf_com_CVPR_2024_Workshop_HuMoGen
Published: 2024-03-22 16:49:42

Abstract: We provide results of our study on text-based 3D human motion retrieval and particularly focus on cross-dataset generalization. Due to practical reasons such as dataset-specific human body representations, existing works typically benchmark by training and testing on partitions from the same dataset. Here, we employ a unified SMPL body format for all datasets, which allows us to perform training on one dataset, testing on the other, as well as training on a combination of datasets. Our results suggest that there exist dataset biases in standard text-motion benchmarks such as HumanML3D, KIT Motion-Language, and BABEL. We show that text augmentations help close the domain gap to some extent, but the gap remains. We further provide the first zero-shot action recognition results on BABEL, without using categorical action labels during training, opening up a new avenue for future research.

Main paper: 
 introduction: 
 Dataset bias is a known phenomenon in machine learning research. The pioneering work of Torralba and Efros [26] shows that given a sample from an object recognition dataset, both a human researcher and a computer (SVM classifier) can guess which dataset the image comes from, known as the 'Name That Dataset' task. In a similar spirit, we observe that 3D human motion description datasets typically have a language style that distinguishes them from each other. KIT Motion-Language (KITML) [17] is dominated by locomotive motions and often starts by 'A person is...'. HumanML3D [8] similarly contains such full-sentence descriptions, but tends to be more verbose, and covers a larger vocabulary of motions. BABEL [18] language style is distinct, concisely describing with a single verb such as 'sit'. The t-SNE [27] visualization in Figure 1 confirms this observation, where we plot MPNet [22] text embeddings of random subset of 400 labels from each dataset. BABEL textual labels appear clearly distinct from HumanML3D and KITML.In this work, we perform cross-dataset evaluations to quantify these gaps, and attempt reducing them via text augmentations.We instantiate our study with the text-to-motion retrieval task. While there is a large literature on text-to-motion synthesis [1, 2, 5, 7, 9, 10, 25, 30 ], text-to-motion retrieval is 038 relatively new [8, 16, 29] . TMR [16] employs a contrastive 039 training, similar to CLIP [20] , to learn a cross-modal embedding 040 space. In this work, we train TMR models and show several 041 improvements. Similar to ActionGPT [11] which improves 042 text-to-motion synthesis with text augmentations, we leverage 043 large language models (LLMs) to increase robustness of 044 retrieval models via label augmentations such as paraphrasing 045 (see Table 1 ). Furthermore, we study the ability of a model 046 trained with free-form text labels to generalize to the zero-shot1 047 action recognition task, by performing motion-to-text retrieval.Our contributions are the following: (i) We report Table 1 . Example LLM paraphrasing: We prompt Llama-2 as described in Section 3 in order to augment the original motion descriptions on the left. Middle column shows results of instructing the LLM to paraphrase. The right column is the result of instructing to convert into the style of action labels. The three example input labels are taken from HumanML3D, KITML, and BABEL datasets from top to bottom. cross-dataset retrieval performance using TMR on a unified SMPL [12]  
related work: 
 We briefly describe few works on (i) 3D human motions and language, with a particular emphasis on datasets in this domain, and (ii) zero-shot classification with natural language supervision in other domains of computer vision. For a broader overview, we refer to the survey of [31] .3D human motions and language. Following advances in natural language processing, there has been an increased interest in building models to control 3D human motion generation with language inputs [1, 2, 5, 7, 9, 10, 25, 30] , and more recently on text-based motion search [8, 16, 29] . The performance of these models naturally depend on the datasets they are trained on. KITML [17] is one of the first 3D human motion description datasets, collecting annotations for a relatively small amount of motions, with a relatively small vocabulary of words, thus limiting its generalization to out-of-distribution samples.More recently, two concurrent works HumanML3D [8] and BABEL [18] collected manual labels for the large AMASS [13] motion collection. Since these efforts were in parallel, the resulting annotations differ in style, incurring a domain gap. As mentioned in Section 1, HumanML3D follows KITML-style verbose full sentence descriptions, while BABEL introduces concise labels, typically with verbs in an imperative form (e.g.,M MD z T z M par1: A person is raising their arms and then lowering themselves to the ground. par2: The man raises both arms and then bends down to his knees.GT: A man raises both arms, then kneels down.Action: Arms raise and kneel. We simply employ TMR [16] for text-motion retrieval, but unify several text augmentation approaches to increase its robustness across domains. For each ground truth (GT) textual label, we generate n paraphrased versions, as well as a short action-style description using Llama-2 prompting. During training, we randomly sample either of these augmented labels with probabilities defined by pgt,ppar,pavg,pact.With probability pavg, we also randomly subsample paraphrased versions and average their text embeddings. The selected text embedding z T is then matched to the motion embedding z M using contrastive loss. Note that we do not visualize the motion decoder for simplicity, but we keep the original architecture as in [16] . 
methodology: 
 We build on the recent method of TMR [16] , and make several improvements: mainly the use of text augmentations and using a hard-negative variant (HN-NCE [19] ) of the InfoNCE [14] loss. We also train on a combination of datasets (instead of a single dataset) using motion representation of Guo et al. [8] computed on the SMPL [12] body skeletons (instead of dataset-specific skeletons). When training jointly on multiple datasets, we simply append training sets and sample disproportional to training set size to balance the distributions.In the following, we detail our text augmentation procedure.We perform text augmentation by paraphrasing each motion text label several times. First, given a motion, for each of its text annotations, we use Llama-2 [6] to generate paraphrases of this text. We prompt Llama-2 by instructing to paraphase a given motion description with the paraphrasing style defined by few-shot examples that we provide in the form of text pairs. This procedure applies to HumanML3D and KITML sentences.When paraphrasing concise BABEL text annotations, we alter the prompt by instructing to describe the motion, and providing few-shot examples in the form of "Sentence: 'Point. ' Paraphrased: 'A person motions forward with their hand. ' ".For HumanML3D and KITML, that are annotated with full sentences, we additionally generate action-style annotations.For example, an action-style annotation for "The person sprints down the track, their feet pounding against the ground" is "Sprint". We refer to Table 1 for more text augmentation examples.We have two sources for providing few-shot examples in the prompts. First, we generate example pairs using GPT-3.5 [15] .Second, we leverage the multiple annotations corresponding to the same motion segment (either within or across datasets), and assume that such annotations may be paraphrases of each other.As a final augmentation strategy, we sample uniformly at random, among a set including all the annotations (ground truth and its augmentations). We then encode all the texts in this set We experiment with HumanML3D [8] and KITML [17] 163 standard text-motion datasets. We also benchmark this task 164 on BABEL [18] raw textual labels, and report on its action 165 recognition benchmarks, BABEL-60 and BABEL-120 for 166 60 and 120 action labels, respectively. The source of these 167 captioned motions largely overlap with the AMASS [13] 168 collection that unifies motions from multiple MoCap sources 169 into SMPL body format [12] . We therefore simply extract 170 motion representation from Guo et al. [8] on SMPL skeletons 171 for each of these datasets, alleviating the issue of dataset-specific 172 skeleton definitions, e.g., for KITML [17] . HumanML3D KITML BABEL Training data Augm HN-NCE R@1 R@3 R@10 R@1 R@3 R@10 R@1 R@3 R@ 
evaluation protocol: 
 We report recall at several ranks as in [16] for both text-tomotion retrieval and action recognition (i.e., motion-to-text retrieval). Given an input modality, rank k recall corresponds to the percentage of inputs whose label has been retrieved among the top k results. For action recognition, we additionally report class-balanced accuracy (Top-1 CB), by averaging the Top-1 accuracies over action categories.For the text-to-motion retrieval task, we report metrics using the 'All with threshold' protocol described in [16] . Within the test set, we compute the similarity across texts using their MPNet [23] embeddings. The rank of a sample is taken as the highest rank among the ranks of all its similar samples. We con- In Table 2 , we report rank R@1, R@3 and R@10 metricsfor text-to-motion retrieval, using protocol 'All with threshold' 228 as described in [16] . We evaluate on HumanML3D, KITML if the text labels are different).Cross-dataset evaluations. In the first three rows of 3 . In the first block, we list the previous works reporting on this benchmark [18, 24] , using the BABEL action labels for training (B-actions). We first check that TMR reaches their performance on this fully-supervised setting. We then provide intermediate results by using BABEL motions, but their free-form textual labels, instead of the categorical action labels.Both 'raw' and 'proc' (processed) labels provided by this dataset match the performance of action labels (perhaps due to action labels being derived from those). In the last block, we report the zero-shot setting by training on HumanML3D texts. Here, we observe significant improvements via text augmentations (e.g., 22 .44 vs 26.30). We also ablate our average embedding strategy described in Section 3 (p gt = 0.4, p par = 0.3, p act = 0.3, p avg =0) and see its benefits (last two rows).We first study the impact of the choice of probabilities used in our augmentation strategy, p par , p sum and p act . Next, we compare our text augmentation approach to the one of Action-GPT [11] , the method we find to be most related to ours.We conduct these ablations by training on the combination of HumanML3D + KITML training, and by evaluating on HumanML3D.Augmentation probabilities. 2 , where the mean across 3 runs is reported as 14.47 R@1.subset of labels (p avg ). Rows 2-4 experiment only with the para-307 phrasing approach, rows 5-7 only with the averaging approach, 308 and rows 8-9 studies combinations of both, without including 309 the action-type labeling approach. Finally, last two rows report 310 combinations of these 3 approaches. While the model does 311 not seem sensitive to the choice of the probability values, its 312 performance increases when using a combination of all the 313 augmentation approaches. We also observe that giving more 314 weight to the averaging protocol further boosts the performance.Comparison to Action-GPT. We compare our text augmen-tation to an approach we implement similar to Action-GPT [11] .Although used with a different training dataset, BABEL, on 318 a different task (text-to-motion synthesis), this is the method we 319 find to be most related to ours. More specifically, we compare 320 both our ways of leveraging the use of several paraphrases for 321 one text. Results are summarized in Table 5 .There are three main differences between our approach and (3) They average the paraphrase tokens at the entrance of the text 329 encoder, while we average the sentence embeddings obtained af-330 ter passing them through the text encoder (see Figure 2 ). Table 5 331 ablates each of these combinations, contrasting the approach of 332 [11] that corresponds to the first row, with that of ours (last row).HumanML3D Averaging p gt p avg k R@1 R@3 R@10 Table 5 . Comparison to token averaging as in Action-GPT [11] : We systematically analyze the impact of averaging multiple paraphrases of the textual label. Action-GPT performs token averaging before passing through the text encoder using a fixed number of k = 4 paraphrases, and does not use the original ground truth (GT) label. In our setting, averaging the sentence embeddings after the text encoder, for a random subset of a larger set of 30 paraphrases, using both GT and this average, outperforms significantly over this baseline (green vs red rows).Averaging the sentence embeddings performs clearly better than averaging the token embeddings for every parameter combination. We also validate our random sampling strategy,showing both the benefits of also including the ground truth labels, as well as not fixing the number of paraphrases.In this section, we provide visual illustrations of results for both text-to-motion retrieval (Figure 3 ) and action recognition (Figure 4 ). We further analyze action recognition results, in particular investigating per-action performances with/without text augmentations (Figure 5 ) and the confusions between actions (Figure 6 ).Figure 3 shows qualitative results for text-to-motion retrieval on the HumanML3D test set, using the model trained on HumanML3D + KITML. We display two text queries, and top-5 ranked motions for each of them both with and without text augmentations. We notice that our model allows the retrieved motions to capture more elements and details of the input text.For instance in the above example, while the baseline captures the rough information that the query text targets the legs, the model with text augmentation captures the more specific interaction between knee and elbow in 4 motions out of 5.In Figure 4 , we illustrate several examples for the action recognition results on BABEL-60. We notice that while the correct action class is not always at the top rank, it often appears within the top 5 retrieved action labels. We observe that all top retrieved predictions are often related to the ground-truth action (e.g., 'Place something' vs 'Interact with/use object').  
conclusion and limitations: 
 We presented our work analyzing the generalization performance of text-motion retrieval models. Specifically, we perform cross-dataset experiments using standard benchmarks. Our results suggest that significant gains are observed when applying text augmentations to overcome the domain gap across datasets.  
 
 Appendix: Similar to contemporary literature[20], we abuse the term zero-shot, meaning training on a separate dataset than the downstream dataset used for evaluation. 


Decision: The paper received positive reviews. The authors appreciated the comprehensive evaluations presented in the draft and the interesting use of LLMs in the context of motion synthesis. The AC agrees with the reviewers and believes the paper would be highly relevant to the motion synthesis community. The authors are encouraged to release the data.

Reviews: review: title: title: This paper introduces a novel cross-dataset evaluation for the task of text-to-motion retrieval.
main_review: main_review: **Overview**

This work examines the generalization capability of a text-to-motion retrieval approach on three popular and publicly available 3D text-to-motion datasets. This work also attempts to unify these datasets by augmenting the input text, allowing their model to generalize beyond the original dataset they’re trained on. This study suggests that biases exist between the datasets. Furthermore, this work also demonstrates the zero-shot generalization capability of their proposed approach when tested on a dataset that is not used as a part of the training data.

**Strengths**

- The paper offers an extensive and interesting analysis of the popular 3D text-to-motion datasets which will be useful to the motion synthesis community.
- Once released, the cross-dataset data augmentation will be appreciated by the 3D human motion synthesis community, as it enables a larger training set for their model. This also opens a wider range of labeling options, enabling synthesis from either free-form text descriptions or shorter action-based labels across different datasets.
- It introduces a unified representation of the 3D motion sequences across different datasets.
- Their experimental analysis is quite extensive. The proposed modifications to the TMR text-to-motion retrieval model show positive improvements over the original design in general. These proposed components are also well ablated in their evaluation.

**Weaknesses**

- The proposed improvements over the TMR model appear minimal, as they mainly come from adjusting a contrastive loss function as well as the additional augmentation labels.
- The evaluation indicates that the proposed improvements for the text-to-motion retrieval task do not consistently guarantee better performance on datasets other than HumanML3D. However, this is also understandable due to the potentially high label-to-motion variance of certain datasets.
- The paper does not provide further discussion regarding dataset bias beyond what is presented in the introduction.

**Justification**

This paper presents a fresh perspective on how to analyze, examine, and dissect some of the overlooked aspects of the field of 3D human motion generation. In addition to being able to improve upon a state-of-the-art method of text-to-motion retrieval, it also offers a new upgrade on the popular 3D text-to-motion datasets, which will certainly be appreciated by the community.
title: title: accept
main_review: main_review: The authors point out the domain gap between the textual annotations of the three common text-motion datasets - BABEL, KIT, and HumanML3D. They suggest bridging the gap using textual augmentations with LLMs (namely, Llama-2 and GPT3.5). They use the augmented datasets to improve the motion retrieval task, retrain the recent TMR, and show improved results.

Pros
- Analyzing an important gap in the common text-motion datasets.
- Use it to improve motion retrieval and classification tasks.
- Comprehensive experiments - demonstrating the usefulness of the augmentation for single and multi-dataset training.

Cons
- The technical novelty is marginal.


Questions and suggestions for the next revision
- While you showed the domain gap in the textual annotations between the different datasets, you didn’t address the domain gap in the motion modality. Since both BABEL and HumanML are composed of subsets of AMASS and KIT is included in AMASS, there should be an overlap between the datasets. Measuring this overlap can be beneficial to understanding the usefulness of multi-dataset experiments.
- Please publish the textual augmentations for the datasets. That can be super useful for other text-to-motion tasks.

Overall, the paper highlights an important issue in text-motion research and shows it can be overcome with simple textual augmentations. This can be useful for the community, hence I would recommend accepting the paper.