Modal-Enhanced Semantic Modeling for Fine-Grained 3D Human Motion Retrieval

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-09 02:09:20

Abstract: Text to Motion Retrieval (TMR) is an emerging task to retrieval relevant motion sequence  with the nature language description. The existing dominant approach is to learn a joint embedding space to measure global-level similarities. However, simple global embeddings are insufficient to represent complicated motion and textual details, such as the movement of specific body parts and the coordination among these body parts. In addition, most of the motion variations occur subtly and locally,  resulting in semantic vagueness amony these motions, which further presents considerable challenges in precisely aligning motion sequences with texts. To address these challenges, we propose a novel Modal-Enhanced Semantic Modeling (MESM)  method, focusing on fine-grained alignment through enhanced modal semantics. Specifically, we develop a prompt-enhanced textual module (PTM) to generate detailed descriptions of specific body part movements, which comprehensively captures the fine-grained textual semantics for precise matching. We employ a skeleton-enhanced motion module (SMM) to effectively enhance the model's capability to represent intricate motions. This module leverages a graph convolutional network to meticulously model the intricate spatial dependencies among relevant body parts. To improve the sensitivity to the subtle motions, we further propose a text-driven semantics interaction module (TSIM). The TSIM first assigns motion features into a set of aggregated descriptors, then employs a cross-attention to aggregate discriminative motion embeddings guided by textual, enabling precise semantic alignment between subtle motions and corresponding texts. Extensive experiments conducted on two widely used benchmark datasets, HumanML3D and KIT-ML, demonstrate the effectiveness of our proposed method. Our approach outperforms existing state-of-the-art retrieval methods, achieving significant Rsum improvements  of 24.28\% on HumanML3D and 25.80\% on KIT-ML.

Main paper: 
 introduction: 
 Text-to-Motion Retrieval (TMR) is a significant cross-modal retrieval task, aiming at retrieving semantically similar motion sequences by the given natural language description. Recently, there has been a notable increase in research and exploration within the field of TMR. Mathis 𝑒𝑡 𝑎𝑙. [28] were the first to introduce the task of text to 3D human motion retrieval, establishing various evaluation benchmarks. They proposed combining motion synthesis with text-to-motion retrieval (TMR) to supervise the shared semantic space between textual descriptions and motion sequences. Nicola 𝑒𝑡 𝑎𝑙. [25] developed a two-stream pipeline framework for text-tomotion retrieval. Their framework incorporates a motion transformer that captures intricate motion information from skeleton data, representing motion sequences as embeddings. Additionally, they employ a pre-trained CLIP [31] text encoder to encode text as sentence embeddings. Yan 𝑒𝑡 𝑎𝑙. [39] introduce a method by constructing a concise yet effective model, and design a droptriple loss to reduce semantic discrepancies in triplet training. However, these methods may have the following problems that may lead to sub-optimal matching: • Limitation of coarse-text description In Fig. 1a , illustrates a retrieval results using a coarse textual description. Specifically, the Motion1 is the correct retrieval result with a similarity of 0.87, while the Motion2 and other motions (Motionk) not semantic similarity to given text, also have high similarities. These motions fail to accurately retrieval the intricate motion "thrust a javelin", simply considering it as a jump movement. Since these coarse textual descriptions may correspond to various motion sequences, training with coarse-grained texts may leads to alignment difficulties, thus affecting the accuracy of retrieval results.• Coordination of multiple-body parts within a motion: In TMR, an motion is typically performed by the simultaneous movement of multiple relevant body parts. For instance, a motion sequence "A person jogs and throws an object." includes the submotion "throws", which may involve the movement of feet, arms, and multiple body parts working together to complete this motion. Previous methods [25, 28, 39] , focusing solely on the temporal dependencies of motion sequences, falling short in capturing the essence of intricate motions. This limitation poses a challenge in obtaining a comprehensive understanding of these intricate motions.• Semantic vagueness between subtle motions: In the TMR, there exists semantic vagueness between subtle motions, making it challenging to capture the semantic variations in continuous motions. As shown in Fig. 1b , the motions1 and motion2 occur successively, involving only the variation on the hands without any changes in other body parts. By aggregating these subtle motions to the semantic spaces by previous methods, we observe that these subtle motions cluster closely. Existing work [25, 28, 39] have achieved promising retrieval performance by leveraging the global embeddings, overlooking the significance of effectively capturing discriminative embeddings for a precise matching.To address the above problems, we propose a novel solution named modal-enhanced semantic modeling method (MESM), which contain three components: the prompt-enhanced textual module (PTM), the skeleton-enhanced motion module (SMM), and the textdriven semantics interaction module (TSIM). Specifically, given a text query, we propose a prompt-enhanced textual module that utilizes a large language model to generate fine-grained text descriptions. The descriptions maintain strict chronological order while accurately specifying movements of related body parts, which comprehensively captures the fine-grained textual semantics for precise matching. Moreover, considering the human body as a graph structure, we implemented a skeleton-enhanced motion module to effectively enhance the model's capability to represent intricate motions. In this module, we employ a multi-layer graph conventional network to meticulously model the intricate spatial dependencies among relevant body parts within a motion. Finally, to effectively solve semantic vagueness between subtle motions, we introduce a text-driven semantics interaction module to obtain discriminative motion embeddings to achieve precise matching between subtle motions and text. The TSIM assigns motion features into a set of local aggregated descriptors, then employ a cross-attention mechanisms to aggregate discriminative motion embeddings under the guidance of textual description, enabling precise semantic alignment between subtle motions and corresponding texts. The main contributions of this work can be summarized as follows:• We present a novel modal-enhanced semantic modeling method to achieve fine-grained alignment by modeling the enhanced semantics information from motion and text level. • We propose a prompt-enhanced textual module to obtain fine-grained descriptions by utilizing a large language model, which comprehensively captures the textual semantics for a precise matching. • We design a skeleton-enhanced motion module to capture spatial dependencies among relevant body parts within a motion, which effectively enhances the model's capability to model intricate motion sequences. • We introduce a text-driven semantics interaction module to aggregate discriminative motion embeddings conditioned by textual descriptions, which enables a precise alignment between subtle motions with corresponding texts. 
related work 2.1 text-motion retrieval: 
 Recent advancements in research on text to 3D human motion retrieval [25, 28, 39] (TMR) have been increasingly emphasized.Compared with image retrieval [4, 30, 42] , TMR is a challenging task that aims to construct a common space between text descriptions and 3D human motions by aligning the sentences and motion sequences. Mathis 𝑒𝑡 𝑎𝑙 . [28] firstly define text to 3D human motion retrieval as an independent task. They present a method to jointly train text-to-motion retrieval and text-to-motion synthesis,by employing the contrastive learning with a special attention to the definition of negatives samples. Nicola 𝑒𝑡 𝑎𝑙 . [25] propose a transformer-based motion encoder capture spatio-temporal informations in skeleton data and a CLIP-based [31] text encoder to obtain text embeddings. Yan 𝑒𝑡 𝑎𝑙. [39] propose a method by constructing a concise yet effective model, and introduce droptriple loss to minimize semantic discrepancies during triplet training. However, existing methods focus on aligning the global representations , which is inadequate to achieve precise semantics matching between intricate motion sequence and their corresponding text.Prompt learning is initially introduced in the field of natural language processing (NLP) [9, 16, 19, 21, 26, 34] . Using prompt learning approaches, pretrained language models can be adapted to specific few-shot tasks by incorporating extra hand-crafted or dynamically learnable prompt tokens. Kalakonda 𝑒𝑡 𝑎𝑙 . [13] utilize GPT-3 [3] to enhance coarse-grained descriptions. However, they apply simplistic zero-shot prompts that generate overly detailed, fine-grained texts filled with excessive, non-essential information. Athanasiou 𝑒𝑡 𝑎𝑙 . [2] employ GPT-3 [3] to extract information about the body parts involved in various motion descriptions and then integrate these details into the corresponding motion sequences. However, these two works focus solely on single motion description and do not fully explore the extent to which large language models can comprehend the detailed of time, space, and human bodies from a coarse-grained motion description.Cross-modal Semantic alignment [5, 6, 18, 23, 36-38, 40, 41, 44, 45] is a significant task in multi-modality scenarios, which aims to bring together instances from different modalities that share similar semantics. Wang 𝑒𝑡 𝑎𝑙 . [37] present a effctive approach for the local alignment of textual and video inputs, inspired by the success of NetVLAD encoding [1] , Zhu 𝑒𝑡 𝑎𝑙 . [45] introduce the FIMA framework, which combines dense contrastive learning, foreground sampling, and a motion decoder to address weak alignment between modalities at the pixel and frame levels. Dong 𝑒𝑡 𝑎𝑙 . [6] introduce a region-to-patch framework that contains a coarse-to-fine encoding branch to extract different granularities for a comprehensive crossmodal alignment. Wang 𝑒𝑡 𝑎𝑙 . [38] present a hierarchical alignment framework that addresses cross-modal correspondence at various levels of granularity, achieving unified multi-grained alignment. Despite significant advancements, existing methods are typically designed for the image or video. Due to the semantic vagueness among subtle actions and complexity of the motions, it may be difficult to directly incorporate it into 3D motion scenarios. 
methodology: 
 In this section, we provide a comprehensive overview of each component of our proposed MESM, depicted in Fig. 3 . We first introduce the overall network architecture. Then, we elaborate on our proposed Prompt-Enhanced Textual Module (PTM), Skeleton-Enhanced Motion Module (SMM) and Text-Driven Semantics Interaction Module (TSIM). Finally, we detail the similarity calculation and objective function for text to 3D motion retrieval.Unlike the previous text-motion retrieval methods [25, 28] , which solely aligning the global embeddings, we introduce a novel framework capturing the enhanced fine-grained semantics to achieve a precise fine-granied alignment. Specifically, our modal consists of there modules. (1) The PTM utilizes a large language model provided appropriate prompts to generate detailed descriptions that maintain strict chronological order while specifying movements of related body parts with appropriate granularity. (2) The SMM utilizes a graph convolutional network to learn the spatial relationships between skeleton joints as complement to piror motion fetures [10] and then employs a transformer encoder to capture the temporal dependencies within motion sequence . (3) The TSIM adopts multi-head attention to encode discriminative features conditioned by language description for precise alignment. Next, we will describe the above modules in details.In real scenarios, there exists numerous coarse textual descriptions, such as "A man squats, " making it difficult to effectively retrieval motions using these text description. Additionally, motion sequences consists of specific motions at relevant body parts, posing challenges in precisely aligning these intricate motion sequences with coarse-grained text. To address this issue, we utilizes a large language model, provided appropriate prompts, to generate detailed descriptions that maintain a strict chronological order while specifying movements of related body parts.Prompt-guided Text Learning. We expect GPT-4.0 to properly expand text descriptions from coarse-grained to fine-grained ones, specifying movements of relevant body parts. To be more specific, an optimal fine-grained description should be in time order and specify spatial changes of relevant body parts, omit unnecessary details. After multiple empirical trials, we determine the following 𝑃𝑟𝑜𝑚𝑝𝑡 : "Provide a fine-grained textual description based on our provided coarse-textual decription, the new description should be in chronological order, step by step, and specify the movement of relevant body parts.[example1],[example2]". The example is a coarse textual description and its fine-grained version. By feeding prompt into LLM, we can obtain fine-grained step description 𝑆𝑇 𝐸𝑃 𝑘 corresponding to the original coarse-grained ones, which describe the movement of relevant body parts step by step. As illustrated in Fig. 2 , the coarse-text is expanded into a fine-grained description by the large langnage model with our designed prompt, that precisely aligns specific body parts with description, thus ensuring a precise matching between the text and the motion sequence.Step-aware Self-Attention. Initially, we utilize pre-trained Distill-BERT to extract step features for each motion step in fine-grained description. Subsequently, we aggregate the tokens of each step using an average pooling layer to generate step embeddings.EQUATIONwhere step [𝑘 ] 𝜖R 𝑑 𝑠 𝑡 , 𝑑 𝑠 𝑡 is the dimension of step embeddings. After obtaining the step embeddings, our step-aware self-attention mechanism incorporates hard positional embeddings (pos) through sine and cosine functions, and feeds these position-encoded step embeddings into a multi-head self-attention. we obtain the fine-grained textual features t 𝑖 by t 𝑠 = MH-Attn (step + pos)(2)EQUATIONwhere 𝐹 𝐹 𝑁 (•) is a feed-forward network that consists of two linear projection layers with ReLU activations. This step-aware selfattention improves the capture of temporal relationships among steps and enhances the quality of fine-grained features through interactions among step embeddings. In TMR, most motions are completed through the coordinated movement of various relevant body parts. To effectively capture the spatial relationship between these specify body parts, we introduce the Skeleton-Enhanced Motion Module(SMM) to utilizes a multi-layer Graph Convolutional Network for spatial modeling among body parts, by capturing intricate dependencies between body joints. We first employ the model from previous work [10] as the feature extractor. This feature extractor is utilized to extract frame features m 𝑡 = m 1 𝑡 , m 2 𝑡 , . . . , m 𝐿 𝑡 ∈ R L×d 𝑟 𝑚 . However, due to the coordination of multiple body parts involved in intricate motions, the motion sequences extracted by using the SMPL parameter model may potentially lose some spatial information about the relationships between relevant body parts. Human-Aware GCN Network. Considering the human body can be interpreted as a nature graph structure. Inspired by previous work action-recognition [14, 22, 32, 33] and pose-estimation [8, 35, 43] , we can represent the human body's joints and bones as a graph 𝐺 = {𝑁 ; 𝐸}. Nodes are represented by 𝐽 = { 𝑗 1 , 𝑗 2 , ..., 𝑗 𝑛 }, with 𝑛 indicating the number of joints. The adjacency matrix 𝐴 = (𝑎 𝑖𝑘 ) ∈ R 𝐽 ×𝐽 denotes the correlation between joint 𝑗 𝑖 and 𝑗 𝑘 , where 𝑎 𝑖𝑘 is defined as:EQUATIONSpecially, we employ a multi-layer Graph Convolutional Network inside each frame to extract spatial feature embedding and compute correlations between each pair of joints in single frame independently. Hence,the output of l-layer GCN is computed by :EQUATIONHere, Ã is the adjacency matrix of the undirected graph G with added self-connections. D = 𝑗 Ã𝑖 𝑗 and 𝑊 (𝑙 ) is a trainable weight matrix. 𝜎 (•) is an activation function. 𝐻 (𝑙 ) ∈ R 𝐽 ×𝐷 is the matrix of activations in the 𝑙 𝑡ℎ layer. 𝑏 (𝑙 ) is a bias term. After obtaining interjoint dependencies through our multi-layer Graph Convolutional network, we obtain motion representation m 𝑠 ∈ R F×d 𝑝 𝑚 by a flatten operation.Lastly, based on the features extracted by feature extractor and the spatial motion features m 𝑠 , we employ a transformer encoder to obtain the final motion representation:EQUATIONHere, 𝐹𝐶 (•) is a linear layer, mapping m 𝑠 from dimension d 𝑝 𝑚 to d 𝑠 𝑚 . A transformer encoder applied to the sum of m 𝑡 and m 𝑠 enhances the motion embeddings, providing more comprehensive semantic for intricate motion.In real-world scenarios, most motion variations occur locally and subtly. There exists semantic vagueness between these local and subtle motions, making it challenging to achieve precise alignment between these motions and text. To effectively capture the discriminative motion embeddings, we introduce a Text-Driven Modal-Enhanced Semantic Modeling for Fine-Grained 3D Human Motion Retrieval MM'24, October 28-November 1, 2024, Melbourne, AustraliaSemantics Interaction Module (TSIM) to aggregate motion represatations under the guidance of textual embeddings, enabling a precise semantics alignment. Motion Descriptor Aggregation. We utilize a motion-wise descriptors aggregation method to capture local motion features. We first build a descriptors set with 𝑘 motion-wise cluster descriptors {𝑝 1 , . . . , 𝑝 𝐾 } ∈ R 𝑑 𝑝 . Then, the motion frame features m 𝑙 could be assigned to the motion-wise descriptors , the 𝑗-th motion-wise descriptors is obtained as followed:EQUATIONwhere 𝒑 ′ 𝑘 has same size with 𝒑 𝑘 as a trainable weight, 𝑏 𝑗 is a learnable bias, and ∥ • ∥ 2 indicates the L2 normalization. The motion information within the motion sequence is described using the motion-wise descriptors. After assigning each frame feature to the descriptors, we could obtain the descriptors-based motion local features m 𝑝 = m 1 𝑝 , m 2 𝑝 , . . . , m 𝑘 𝑝 , where m 𝑝 ∈ R 𝑘 ×𝑑 𝑐 . Text-Guided Cross-Modal Interaction. As shown in Fig. 3 , to enable the interaction between two modalities, we first apply a cross-modal attention to obtain the linguistic representations t 𝑝 corresponding to the motion local representations m 𝑝 .EQUATIONEQUATIONwhere 𝑄 𝑚 , 𝐾 𝑡 , 𝑉 𝑡 represent the query, key, and value obtained by projecting the motion and textual features. Norm denotes L2 normalization. The parameter matrix 𝑊 𝑂 is a projection weight matrix.Based on 𝑡 𝑝 , we employ another multi-head attention module to perform language-conditioned self-attention for the discriminative motion features. Specifically, we take the sum of 𝑡 𝑝 and 𝑚 𝑝 as the query and key inputs for the multi-head attention module, facilitating the computation of feature correlations in both the motion and linguistic representations. The attention weight value from position i to position j is formulated as follows:EQUATIONwhere 𝑊 𝑄 and 𝑊 𝐾 are the linear projection weights for the query and key, and 𝑅(•) represents the positional encodings of relative positions. This text-guided cross-modal interaction enables the model to capture discriminative features for subtle motions based on the given textual descriptions. The multi-head attention outputs final motion representations as m 𝑖 = m 1 𝑖 , m 2 𝑖 , . . . , m 𝐿 𝑖 ∈ R L×d 𝑐 , 𝐿 is the number of final motion embeddings.Similarity Calculation. The similarity score 𝑆 (𝑚 𝑖 , 𝑡 𝑗 ) measures the semantic similarity between the text 𝑡 𝑗 and motion sequence 𝑚 𝑖 .It is computed as the mean of the maximum similarities between each motion embedding and all text embeddings in bi-directions.EQUATIONwhere 𝑀, 𝑁 denote visual embedding and text embedding number in the 𝑖 th sample pair. Loss Function. Many studies [7, 17, 20] have proved that not only do positive samples learn joint semantics from visual-textual alignment, but a sufficient number of negative samples better fill up the semantic inequality. In our study, we adopt contrastive learning along with a hinge-based triplet loss [12] , deliberately establishing distinct threshold based on relative similarity instead of absolute similarity. This formulation is expressed as follows:EQUATIONEQUATIONHere, [•] + = max(𝑥, 0). 𝑆 (𝑚, 𝑡) represents the similarity between positive samples and 𝑤 is a boundary factor. 𝐵 is the mini-batch. 𝑆 (𝑚, t) represents the similarity between 𝑚 and the hardest textual negative sample corresponding to 𝑚 (same as 𝑆 ( m, 𝑡) ). Additionally, we employ a logistic loss function to strengthen the correlations among positives samples, which is denoted as follows:EQUATIONFinally, our total loss function is designed with the linear combination of L pos and L neg :L = L pos + 𝛼 • L neg (15) 
experiments 4.1 datasets: 
 To validate the proposed methods, we employ two recently widely used 3D human motion datasets, HumanML3D [10] and KIT Motion Language [29] .HumanML3D [10] is currently the largest 3D human motion dataset accompanied by textual descriptions. The motion sequences are sourced from two well-established and widely-used motion-capture datasets: AMASS [24] and HumanAct12 [11] . Following the setup in the benchmark [28] All experiments are conducted on a workstation with NVIDIA A100 GPUs , by using the PyTorch-1.10 library. We set the mini-batch size as 64 and employ the Adam optimizer [15] to optimize our model Table 1 : Performance comparison with the state-of-the-art methods on HumanML3D [10] . The best results are shown in bold.Text-to-Motion Motion-to-Text Rsum ↑ R@1 ↑ R@2 ↑ R@3 ↑ R@5 ↑ R@10 ↑ MedR ↓ R@1 ↑ R@2 ↑ R@3 ↑ R@5 ↑ R@10 ↑ MedR ↓ (a)Methods Text-to-Motion Motion-to-TextRsum ↑ R@1 ↑ R@2 ↑ R@3 ↑ R@5 ↑ R@10 ↑ MedR ↓ R@1 ↑ R@2 ↑ R@3 ↑ R@5 ↑ R@10 ↑ MedR ↓ (a) NormalTSA [10] 3.37 . To prevent overfitting, we utilize a learning rate of 0.0002 and initiate decay of 15% every 10 epochs after epoch 30. For the both HumanML3D and KIT-ML datasets, we set the maximum motion sequence length and training epochs to 200 and 300, respectively. In addtion, the hyper-parameters 𝑤 in Eq. 12 is set to 0.2, 𝜏 in Eq. 14 is 10, and the trade-off weight 𝛼 in the overall loss function is set to 1.0. The representation dimension d 𝑐 of common space is set to 256. To ensure a consistent comparison with the baseline, we follow the experimental settings presented in previous work [28] , where a text is randomly selected as the matching text for training, and the first text in the test set is used to report the evaluation performance under both the "Normal" and "Threshold" setups. Under the "Normal" setup, all test sets were evaluated without modification. Under the "Threshold" setup, we searched all test sets, considering a motion correct if its text label closely matched the query text above a threshold (set to 0.95, following prior work [28] ). We adopt the common metrics to report retrieval performance, including Recall at K (R@K), Median Rank (MedR), and Rsum, following the benchmark [28] . The best evaluation results are highlighted in "bold".In this section, we compare our proposed method with the other state-of-the-arts methods on various text-to-motion retrieval, including TEMOS [27] , TSA [10] , TMR [28] , DTL [39] and MoT [25] . Considering that MoT [25] and DTL [39] differ from our experimental settings, we re-implement them on our settings and report the results based on official codes. Tab. 1 and Tab. 2 provide a comparative overview of our model against state-of-the-art methods on the HumanML3D and KIT datasets, respectively. We observe that our modal achieves better performance than existing methods on most of the metrics on both text-to-motion and motion-to-text retrieval settings. Specifically, under "Normal" setups, it outperforms the current state-of-the-art TMR [28] , achieving relative improvement of 1.48% and 2.06% on the R@1 metric for text-to-motion retrieval on both HumanML3D and KIT datasets . Furthermore, the overall retrieval quality, as indicated by the Rsum metric, experiences a significant boost (+24.28%, +25.80%) across both datasets. Under the "Threshold" setups, our proposed MESM continues to outperform all baselines, particularly the TMR [28] , which incorporates specific designs for negative filtering. The effectiveness of our methods demonstrates that enhanced visual and textual features provide richer semantic information for the retrieval process.The effectiveness of each component. In Tab. 3, we conduct a thorough ablation study to verify the effectiveness of proposed components in our methods. In the first group, only the single component is used for enhancing the retrieval process. We observe that 1) the first row of the first group shows our baseline that performs achieves R@1 of 5.48 slightly below TMR [28] . 2) These three components show a better retrieval performance than baseline. It indicates that PTM and SMM can enhance our obtained features to achieve fine-grained alignment between text and motion sequences.3) Only TSIM used in our methods achieves the best performance in comparison with other components. It proves that TSIM can learn discriminative motion embeddings for precise text-motion alignment to improve retrieval performance. For the second group of ablation studies, we pair different components to investigate their interrelated effects. We observe that 1) compared to using a single component , combining components leads to a notable improvement in retrieval performance. It indicates that the proposed PTM, SMM and TSIM can provide varying fine-grained semantics and are complementary to each other.2) The most significant improvement is achieved in the integration of PTM and SMM . It may be attributed that the SMM and PTM could provide more richer semantics to enhance representations.3) The integration of PTM, SMM, and TSIM results in a further boost to retrieval performance and achieves the state-of-the-art. This reveals that the proposed MESM framework captures the comprehensive semantics to achieve precise retrieval performance. Generalization Analysis of PTM and SMM. In this subsection, we aim to verify the generalization of our proposed promptenhanced textual module (PTM) and skeleton-enhanced motion module (SMM). We plug the proposed PTM and SMM into the different backbones TEMOS, MoT, and TMR (denoted as "+ PSM * ") on the HumanML3D dataset. The improvement performance by our PSM * are marked in green. The results in shown in Tab. 4. Specifically, R@1 of text-motion matching on MoT [25] obtains a significant boost of 3.26% . It may be attributed to the backbone of MoT [25] only focusing on the coarse-grained matching while our PSM * enhances the semantic information both on text and motion. However, the improvement in results for TMR is limited, with R@1 increasing by only 1.13%. This may be attributed to the complex distribution alignment and motion synthesis within TMR, both of which could offer supplementary information for auxiliary alignment between text and motion. All baselines show a satisfying improvement by equipping with our PTM and SMM, which demonstrate the generalization of our proposed components.The ablation study of descriptor number. We conduct ablation study on the number of motion descriptors as proposed in our text-driven semantics interaction module. Based on the average movement number within the sequence, we set the descriptor numbers from 3 to 6 to explore the optimal setting on HumanML3D and KIT-ML. We report the Rsum to measure the overall performance. As shown in Tab. 5, it demonstrates that a larger setup of descriptor numbers brings much retrieval performance. However, an increase beyond four descriptors tends to degrade performance.The ablation studies of the proposed components in our method. We report the experimental results on the text-to-motion retrieval on both HumanML3D and KIT-ML datasets. It may be attributed that excessively numerous descriptors can introduce redundant semantics and lead to dispersed semantics at the descriptors to damage the fine-grained alignment.EQUATIONThe Impact of hyper parameters. We conduct a group of experiments with different values of the boundary factor 𝑤 in Eq. 12 and weights 𝛼 in total loss function. As shown in Fig. 5 , we set the 𝑤 from 0.1 to 0.5, and 𝛼 from 1e-2 to 1e+2. In Fig. 5a , we get the best retrieval performance when 𝑤 = 0.2. We note that as 𝑤 increases to 0.5, there is a substantial decrease in retrieval performance across all metrics. This could be because that the model might overly penalize negative sample pairs that are closely similar to positive ones,if the margin value is too large. Furthermore, in Fig. 5b , the overall retrieval performance initially increased before reaching saturation at 𝛼 = 1.0 and then slightly declined. These results suggest that a larger value of 𝛼 may lead the model to assign excessive weight to hard negative pairs, while a smaller value of 𝛼 may underestimate the impact of hard negative pairs.In Fig. 4 , we present the visualization of the retrieval results obtained by our proposed MESM (Ours), as well as the baseline approaches TMR [28] and MoT [25] on the HumanML3D dataset. Successful retrieval results are highlighted with a green border. For the first We present the top-2 motion sequences retrieved by our proposed method MESM , as well as by TMR [28] and MoT [25] . The text below each motion sequence represents the ground truth. The successful retrieval results are highlighted by the green border. example, our proposed method successfully retrieves the motion sequence. The second retrieved motion sequence also includes the key motions mentioned in the query text, such as "walks" and "knocked". Notably, the "knocked" motion sequence contains multiple coordinated movements of relevant body parts, such as "arms" and "hand", demonstrating that our method can achieve fine-grained semantics alignment by capturing intricate details within the motion sequences. In contrast, the baselines TMR [28] and MoT [25] are able to capture simple motions like "walks". However, they struggle to effectively retrieve more complex motions like "knocked", which require coordinated movements of multiple body parts. This limitation can be attributed to their coarse-level modeling of text and motion representations, which makes it challenging to align intricate motions with textual descriptions. For the second example, our proposed method achieves successful retrieval of ground truth, despite the motions being subtle and indiscriminative. This demonstrates our method could effectively align these subtle motions with the text descriptions and achieves a accurate retrieval. However, both TMR and MoT fail to retrieve the ground truth motion sequences, particularly the second retrieved results. These results primarily contain minor motions that lack relevant semantics. This limitation arises from the the coarse global representation alignment that aligns the average semantics of the query with the motion sequence, which can not provide substantial meaningful semantic information. 
conclusion: 
 In this paper, we propose a modal-enhanced semantic modeling method to model and align the precise semantics for text to 3D human motion retrieval. We employ prompt-enhanced textual module and skeleton-enhanced motion module to enhance the semantic information of both the text and motion modalities. In addition, we introduce a Text-Driven Semantic Interaction Module to learn more discriminative motion embeddings for precise fine-grained semantic alignment between subtle motions and the corresponding texts. In the future, we aim to design a framework that progressively models semantic information at different levels to achieve hierarchical alignment. We believe that the the findings and solutions proposed in this paper have the potential to offer valuable insights to related domains, contributing to a deeper understanding of the field. 
: 
 Figure 1 : Illustration of our motivations. In subfigure (a), a example retrieval results is showcased, where motion-text pairs with a coarse-grained text description that matches multiple motion sequences. In subfigure (b), motion1 and motion2 in motion sequence occur subtly and locally. These subtle motions cluster closely in the semantic space by employing simple regression methods, contributing to semantic vagueness and posing a significant challenge in achieving precise alignment. 
 
 Appendix:  


Decision: There is a consensus to accept this paper.
The authors provided relevant and satisfactory responses to the reviewers' comments in their rebuttal.
The reviewer appreciated the precise alignment between subtle motions with corresponding texts, clarity, performance improvement margin, ablation studies, and richness of the experiments.

Reviews: review: paper_summary: paper_summary: This paper proposes a modal-enhanced semantic modeling method to model (MESM) and align the precise semantics for text to 3D human motion retrieval. The proposed MESM contains three components: the prompt-enhanced textual module (PTM), the skeleton-enhanced motion module (SMM), and the text-driven semantics interaction module (TSIM), enabling precise semantic alignment between subtle motions and corresponding texts.
strength_weakness: strength_weakness: 1. This paper proposes a prompt-enhanced textual module that utilizes a large language model (GPT-4) to generate fine-grained text descriptions, which maintain strict chronological order while accurately specifying movements of related body parts.
2. This paper introduces a text-driven semantics interaction module to aggregate discriminative motion embeddings conditioned by textual descriptions, which enables a precise alignment between subtle motions with corresponding texts.
limitations: limitations: 1. How can the model eliminate situations where the generated descriptions do not match the actual 3D actions, as the descriptions generated by LLM may not always fully match the images.
2. The description of "Motion Descriptor Aggregation" in the paper is insufficient, which raises doubts about the necessity of this module.
3. For the proposed $L_{pos}$, ablation experiments are required to verify its effectiveness.
4. Some expressions in the article are redundant and need to be streamlined.
paper_summary: paper_summary: The authors introduce a fine-grained approach to the emerging text-motion retrieval task. Specifically, they employ an LLM to obtain fine-grained motion descriptions, and design an architecture able to cope with fine-grained text-motion associations. The architecture is composed of a fine-grained motion encoder, a text encoder that processes the different motion description steps extracted from the LLM, and a text-driven semantic interaction module that combines the two modalities at a fine-grained level using cross-attention. The results are reported on two well-known datasets for text-motion retrieval, and the authors perform an ablation study to validate all the proposed components.
strength_weakness: strength_weakness: - The paper is clearly written and its motivation is sound. Many works tackled the fine-grained matching task in the text-to-image scenario. However, the role of fine-grained matching was underexplored in this emerging text-motion retrieval task.
- The results are sound and achieve a great improvement margin over the previous coarse-grained approaches.
- The ablation studies show the effectiveness of all the proposed components.
limitations: limitations: - It is not very clear how LLM can transform a coarse-grained description into a fine-grained one without adding some expected noise, as it adds some details that may not be present or incorrect for the corresponding motion. In this sense, I was expecting some ablation experiments where the authors used the original captions and not the augmented ones. In that case, as in many other fine-grained image-text matching works, the authors could have used the word embeddings in output from DistillBERT as fine-grained text tokens (so one token for each word instead of one token for each step description).
- Concerning the related work, it is not clear why the authors included a Prompt Learning section, given that it seems they are not learning additional prompt tokens in the end-to-end training (the LLM is prompted using a fixed human-written prompt).
- The employed loss function is greatly inspired by previous fine-grained alignment methods on text and images (e.g., [1, 2]), which should be carefully cited in the Related Work section or Section 3.5.
- There are some imprecisions in the employed formalisms. For example: i) it is not clear what p’_k is in Eq. 7; in the same equation, I believe there are some incorrect indexes being used, as i is defined in the summation but never used; ii) it seems the output from the text-driven semantic interaction module should be a set of K vectors, but in line 516 instead L (the initial number of motion features before the aggregation) is reported; iii) both N and B are used to indicate the batch size (for example, Eq. 12, the summation is over B, while in Eq. 14 it is over N). This should be uniform.
- The role of Lpos is not too clear, given that the triplet loss is usually already sufficient (here called Lneg). I would have expected an ablation experiment with only Lneg.

Minor issues:
- The text is full of annoying typos that may confuse the reader. For example:
  - “to accurately retrieval” -> “to accurately retrieve”
  - “graph conventional network” -> “graph convolutional network”
  - “our modal consists” -> “our model consists”
  - “piror motion features” -> ? 
- In the second paragraph of Section 4.4, the boosts are expressed in percentages. However, they seem to be absolute recall values rather than a relative increase. Please remove the “%” sign or convert them to a percentual rise/decrease.

[1] Karpathy, Andrej, and Li Fei-Fei. "Deep visual-semantic alignments for generating image descriptions." Proceedings of the IEEE conference on computer vision and pattern recognition. 2015.

[2] Lee, Kuang-Huei, et al. "Stacked cross attention for image-text matching." Proceedings of the European conference on computer vision (ECCV). 2018.
paper_summary: paper_summary: The authors propose a modal-enhanced semantic modeling method to model and align the precise semantics for text to 3D human motion retrieval. Specifically, the authors (1) employ a prompt-enhanced textual module and a skeleton-enhanced motion module to enhance the semantic information; and (2) introduce a Text-Driven Semantic Interaction Module to learn more discriminative motion embeddings.
strength_weakness: strength_weakness: The experiments of this paper are rich.
The motivation of this paper is clear.
limitations: limitations: (1) The overview of this paper is somewhat lacking novelty. Specifically, this paper utilizes LLM for fine-grained text expansion and designs a multimodal feature alignment module. This requires further explanation of the contributions of the authors.
(2) The caption of Figure 2 is redundant, such as "The TMR... However, we build ...". Therefore, it needs to be refined.
(3) In the abstract, the motivation of this paper needs to be refined. For example, in stating the task challenge, we can see these descriptions "However, simple global embeddings. In addition, ... ".
(4) Some paper lacks citations, such as GPT-4.
paper_summary: paper_summary: The authors proposed a framework for text to motion retrieval by designing a novel modal-enhanced semantic modeling method. The authors proposed a prompt-enhanced textual module to obtain fine-grained descriptions by utilizing a large language model. Extensive experiments and analyses on benchmark datasets demonstrate that the proposed methods achieve the state-of-the-art performance.
strength_weakness: strength_weakness: 1. Numerous experiments have been conducted to demonstrate the efficacy of the method.
2. LLM offers a novel perspective for the text to motion retrieval.
3. The writing style and structure of the paper make it easy for readers to comprehend the methodology and significance of the study.
limitations: limitations: 1. Your approach appears to be more intricate than the TMR[1].  Comparing the training and test times with other retrieval methods would be beneficial.  Providing the specific time measurements would be valuable.
2. The incorporation of LLM may entail the integration of external information, raising questions about its fairness when compared to other methods.
3. The experiment lacks part experiment compared with TMR[1] such as "dissimilar subset'' and "small batches'' in two datasets.
4. The extra visual feature extractor may raise questions about its fairness when compared to other methods.

[1]TMR: Text-to-Motion Retrieval Using Contrastive 3D Human Motion Synthesis

Further Comments

I hope the author takes the rebuttal seriously, and I will revise the review as appropriate based on the response provided in the rebuttal.