SATO: Stable Text-to-Motion Framework

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-05 10:06:56

Abstract: Is the Text to Motion model robust? Recent advancements in Text-to-Motion models primarily stem from more accurate predictions of specific actions. However, the text modality typically relies solely on pre-trained Contrastive Language-Image Pretraining (CLIP) models. Our research has uncovered a significant issue with the text-to-motion model: its predictions often exhibit inconsistent outputs, resulting in vastly different or even incorrect poses when presented with semantically similar or identical text inputs. In this paper, we undertake an analysis to elucidate the underlying causes of this instability, establishing a clear link between the unpredictability of model outputs and the erratic attention patterns of the text encoder module. Consequently, we introduce a formal framework aimed at addressing this issue, which we term the Stable Text-to-Motion Framework (SATO). SATO consists of three modules, each dedicated to stable attention, stable prediction, and maintaining a balance between accuracy and robustness trade-off. We present a methodology for constructing an SATO that satisfies the stability of attention and prediction. To verify the stability of the model, we introduced a new textual synonym perturbation dataset based on HumanML3D and KIT-ML. Results show that SATO is significantly more stable against synonym and other slight perturbations while keeps its high accuracy performance.

Main paper: 
 introduction: 
 The Text-to-Motion (T2M) model signifies a groundbreaking and swiftly advancing paradigm with immense potential across various domains, such as video games, the metaverse, and virtual/augmented Figure 1 : Comparisons on 𝐹 𝐼𝐷 𝐷 and 𝐹 𝐼𝐷 𝑃 . The closer the model is to the origin, the better. The arrow indicates the effect of our method on the model. Our SATO framework can make the text-to-motion model more stable. reality environments. This innovative approach, as evidenced by research contributions from [5, 6, 6, 8, 26, 31, 33] revolves around generating motion data directly from textual descriptions, thereby simplifying the overall process and mitigating associated time and cost overheads.However, a fundamental challenge inherent in text-to-motion tasks stems from the variability of textual inputs [32] . Even when conveying similar or the same meanings and intentions, texts can exhibit considerable variations in vocabulary and structure due to individual user preferences or linguistic nuances. Despite the considerable advancements made in these models, we find a notable weakness: all of them demonstrate instability in prediction when encountering minor textual perturbations, such as synonym substitutions (examples and comparisons are shown in Fig. 4 ). This is a serious issue. The instability of the model leads to inconsistent outputs, with errors in details or even entirely incorrect motion sequence, when users input synonymous or closely related sentences. This limitation confines our model research within a narrow range of expressions, hindering the future development and practical applications of text-to-motion models. This prompts us to inquire: What are the root causes of these issues? Are they rooted in inadequacies in textual modalities, language comprehension, or their harmonization? Through posing these questions, elucidating this problem, and striving for a robust textto-motion framework emerges as an urgent necessity.Most text-to-motion models build upon pre-trained text encoders, such as CLIP [20] . Previous works have shown discrepancies in downstream tasks utilizing CLIP text encoders despite similar semantic inputs [13] . Further investigation reveals that similar phenomena occur in the text-to-motion domain. Taking the T2M-GPT [31] model as an example, several experimental findings emerge. First, we observed a close correlation between instability attention and incorrect prediction outcomes (shown in Fig. 4 ). Differences in attention can lead to significant disparities in text feature representations during intermediate processes. Secondly, in many instances, When we correct the first erroneous token during the model prediction process, we obtain the correct motion sequence, as depicted in the bottom-right figure. The accuracy of the first token is crucial for the subsequent temporal predictions of the model. by rectifying the initial token of inaccurately predicted action sequences, subsequent accurate action sequences were obtained (see Fig. 2 ). Lastly, the initial motion sequence token was predicted based on the text feature. Significant differences in the text feature can lead to significant variations in the first motion sequence token. We further elucidate the aforementioned experimental findings: When perturbed text is inputted, the model exhibits unstable attention, often neglecting critical text elements necessary for accurate motion prediction. This instability further complicates the encoding of text into consistent embeddings, leading to a cascade of consecutive temporal motion generation errors. Notably, the stability of the model manifests in the consistency of textual attention, highlighting its pivotal role in mitigating such errors.For a more robust text-to-motion framework, we must delve into what constitutes stability, meaning requiring us to define stability for the text-to-motion model. Intuitively, a stable attention and prediction text-to-motion model should possess the following three properties for any text input:• Considering from a bionics perspective, it should possess a stable attention mechanism, focusing on key motion descriptions without changing with synonym perturbation. • Its prediction distribution should exhibit stability, i.e., robustness to synonym or near-synonym substitution replacement perturbations during training and testing. • Its prediction distribution closely resembles that of the original model in inputs without perturbation, ensuring outstanding performance. For the first two criteria, as discussed earlier, we work on stabilizing the model's attention and predictions, both indirectly and directly, to stabilize the overall results. As for the last criterion, we emphasize the trade-off between model stability and accuracy. We aim for the model to maintain its excellent performance as much as possible. Based on these criteria, this paper presents a formal definition of a stable attention and robust prediction framework called SATO (Stable Text-to-Motion Framework).To assess better robustness, we construct a large dataset of synonym perturbations based on two widely used datasets: KIT-ML [19] and HumanML3D [7] . It is noteworthy that even when not utilized specifically for stability tasks, our perturbed text dataset can still serve as valuable data augmentation to enhance model performance. Empirically, SATO achieves comparable performance to state-of-the-art models while demonstrating superior stability, as illustrated in Fig. 1 . Extensive experimentation on these benchmark datasets, employing T2M-GPT and Momask models for verification, underscores the effectiveness of our approach. Our results reveal that we achieve optimal stability while maintaining accuracy (e.g., on T2M-GPT, HumanML3D dataset original text FID 0.157 vs 0.141, perturbed text FID 0.155 vs. 1.754). Moreover, human evaluation results indicate a significantly reduced catastrophic error rate postperturbation in contrast to the SOTA models, while also suggesting a subjective preference for the outputs generated by our model. In conclusion, our contributions can be summarized as follows:• To the best of our knowledge, this is the first work to discover the instability issue in text-to-motion models. Our work formulates a formal and mathematical definition for a stable text-to-motion framework named SATO, proposes a dataset for measuring stability, and establishes relevant evaluation metrics, laying the foundation for improving the stability of text-to-motion models. • Through extensive experimentation, we validate the effectiveness of our approach, showcasing its superiority in handling textual perturbations with comparable performance and higher stability. Additionally, we successfully strike a balance between accuracy and stability, ensuring our model maintains high precision even in the face of perturbations. • Our work points to a novel direction for improving text-tomotion models, paving the way for the development of more robust models for real-world applications. 
related work 2.1 text-conditioned human motion generation: 
 Text-conditioned human action generation aims to generate 3D human actions based on textual descriptions. Recent mainstream work can be divided into two categories, namely VQ-VAE-based methods and diffusion models. VQ-VAE [1, 3, 4, 22, 27, 28] has achieved excellent performance in multi-modal generation tasks. ACTOR [17] proposes a Transformer-based VAE for generating motion from predefined action categories. TEMOS [18] introduces an additional text encoder based on ACTOR for generating different action sequences based on text descriptions, but mainly focusing on short sentences. Guo et al. [7] propose an autoregressive conditional VAE conditioned on the generated frame and text features, and proposed to predict actions based on the length of the text. TEACH [2] is based on TEMOS, which generates temporal motion combinations from a series of natural language descriptions and extends space for long action combinations. TM2T [8] considers not only text-to-motion tasks, but also motion-to-text tasks, and the joint training of these two tasks will be improved. T2M-GPT [31] quantizes motion clips into discrete markers and then uses a converter to generate subsequent markers. The emerging diffusion models are also changing the field of motion generation. MDM [26] uses a Transformer Encoder as the main body of prediction samples. MotionDiffuse [32] uses the DDPM architecture to generate realistic and diverse motion. However, whether it is the diffusion-based method the VQ-VAEbased method, or even previous work such as MotionCLIP [25] , the structure is based on the CLIP encoder. Although the work of TEMOS, TEACH, and Guo et al. considered the sequence length of the text and the time and space issues, they did not take into account the diversification of text raised by users. When the text is subject to slight perturbation, the model may exhibit inconsistent outputs, even leading to catastrophic errors in motion, which is a common and severe problem with these past methods. Therefore, based on these issues, this paper is the first work to consider the diversity of user-proposed texts and the first work based on the stable framework in the field of text-conditional human action generation. In this paper, we propose SATO so that the text generation results can still show strong robustness when encountering synonyms or other slight replacements or interference.For the stabilization of input vector perturbations, some work has been done on stabilizing the output pattern of the model from various perspectives. Reconstructing the perturbed text with the actual input text can improve the the robustness of the text model [23] , but does not guarantee the model's attention similarity before and after the perturbation. Cansu et al. [24] analyze human and machine attention to the text. However, they fail to analyze the consistency of the descriptions before and after perturbation. Compared to the text after the perturbation, the model is more inclined to use "unfamiliar vocabulary" in human comprehension. As the "unfamiliar vocabulary" increases in a description, it interferes with the comprehension of the text. Shunsuke et al. [14] enhance the text embedding stability using adversarial learning but do not analyze the consistency between old and new attention, making it difficult to ensure that textual attention can have consistent results for different descriptive scenarios under the same semantics. Yin et al. [29] employ an adversarial robustness approach to enhance the stability of NLP models. However, these techniques are designed for ex-post interpretation of model predictions and thus cannot be applied to enhance attentional stability in the prediction phase.Different from text or visual stable attention, for the multimedia text-to-motion domain, we not only need to consider the distribution of attention weights during text embedding but also focus on the coordination performance between text and motion generation. Therefore, we need to pay attention to the consistency of textual attention before and after the description perturbation. And we also pay attention to the local importance and overall compatibility of semantic weights, to avoid the repeated generation of the emphasized part of the description, thus ignoring the coherence of the whole action. 
method 3.1 preliminaries: 
 Vanilla Attention. For text embedding, Text-to-Motion mainly uses CLIP or other text vector models [5] to encode the action description text. For the original description text, tokenization is first performed to obtain the token index vector, i.e., t ∈ R 𝑛×1 , where 𝑛 represents the number of tokens. Next, the token will be embedded by the embedding weights, i.e. e ∈ R 𝑉 ×𝑑 , where 𝑉 is the size of vocab, 𝑑 is the dimension of the embedding vectors. Therefore, when t goes through the embedding layer, it can obtain the corresponding embedded expression based on the token index, which is notated as t e ∈ R 𝑛×𝑑 . Attention weight is to express the relationship between the query and the key, here we use Scaled Dot-Product to calculate its correlation, which is 𝑎 (q, k) = softmaxqk 𝑇 √ 𝑑 ∈ R 𝑛×𝑚 ,where k ∈ R 𝑚×𝑑 and q ∈ R 𝑛×𝑑 . Finally, the correlation weight is multiplied by v ∈ R 𝑚×𝑣 to get the output R 𝑛×𝑣 . Additionally, the tensors are divided into multi-heads, thus the corresponding final attention weights obtained are also the average between the individual heads, i.e.𝜔 t = 1 ℎ ℎ 𝑖=1 𝑎 (q, k) 𝑖 ∈ R 𝑛×𝑚 . VQ-VAE Based Text-to-Motion Model. Our objective is to pro- duce a 3D human pose sequence X = [x 1 , x 2 , ..., x 𝑇 ], where x 𝑡 ∈ R 𝑑 , guided by a textual description C = [𝑐 1 , 𝑐 2 , ..., 𝑐 𝑙 ],where 𝑇 represents the number of frames and 𝑑 denotes the dimension of the motion feature. Here, c 𝑖 represents the 𝑖 𝑡ℎ word in the sentence, and 𝑙 is the length of the sentence. The process begins with extracting a text embedding c e from input text using CLIP. Subsequently, a transformer model predicts the distribution of possible next indices 𝑝 (𝑆 𝑖 |c e , 𝑆 <𝑖 ) based on the text embedding c e and previous indices 𝑆 <𝑖 , where 𝑆 𝑖 represents the index of the next element at position 𝑖 in the sequence. These predicted indices are then mapped to corresponding entries in the learned codebook, yielding latent code representations ẑi [27] . Finally, the decoder network decodes these codebook entries into motion sequences X pred . The optimization objective aims to maximize the log-likelihood of the data distribution. This is achieved by denoting the likelihood of the full sequence as 𝑝 (𝑆 |c e ) = |𝑆 | 𝑖=1 𝑝 (𝑆 𝑖 |c e , 𝑆 <𝑖 ) and directly maximizing it: [6, 31] , facilitating the generation of motion sequences from input text. Perturbations for Texts. To introduce effective perturbation methods for text, we consider a scenario where a perturbation 𝐶 = [𝑐 1 , 𝑐 2 , ..., 𝑐 𝑙 ] is applied to transform the original text intoL trans = E 𝑆∼𝑝 (𝑆 ) [-log 𝑝 (𝑆 |c e )]𝐶 ′ = [𝑐 ′ 1 , 𝑐 ′ 2 , ...𝑐 ′ 𝑙 ]or perturbing the text embedding 𝑐 to 𝑐 ′ . Several strategies have been shown to be effective in prior works, such as Greedy Coordinate Gradient (GCG) [34] , Projected Gradient Descent [16] (PGD). However, due to the inherent diversity of user inputs and the presence of noise in sentences, we incorporate two distinct perturbation techniques in this study: Projected Gradient Descent (PGD) and Random Synonym Replacement (RSR). PGD finds the perturbation direction along the steepest ascent in the loss landscape, while RSR is done manually through human-designed synonym perturbations. These approaches are chosen to address the variability in user inputs and to tackle the challenges posed by noisy sentences. By employing PGD or RSR perturbations, we aim to enhance the robustness of our text-processing techniques against diverse inputs and noise.The Stability Issue in Text-to-motion Models. The pre-trained CLIP model used as a text encoder for text-to-motion tasks has inherent limitations in maintaining stable attention for semantically similar or identical sentences, while minor perturbations are inevitable during user input. Furthermore, text-to-motion models generally lack the stable capability to handle perturbed text embeddings, leading to inconsistent predictions despite similar or identical semantic inputs. This instability renders it unsuitable for real-world applications where robustness and reliability are crucial. Addressing these issues requires us to analyze them from different perspectives. Attention Stability. We first present the definition of the top-k overlap ratio for two vectors [10] . For vector x ∈ R 𝑛 , we define the set of top-𝑘 components 𝑇 𝑘 (•) as:𝑇 𝑘 (x) = {𝑖 : 𝑖 ∈ [𝑑] and |{x 𝑗 ≥ x 𝑖 : 𝑗 ∈ [𝑛]}| ≤ 𝑘 }.For two vectors x, x ′ , their top-𝑘 overlap ratio 𝑉 𝑘 (x, x ′ ) is denoted as:EQUATION)For the original text input, we can easily observe the model's attention vector for the text. This attention vector reflects the model's attentional ranking of the text, indicating the importance of each word to the text encoder's prediction. We hope a stable attention vector maintains a consistent ranking even after perturbations. For a piece of text, demanding all attention magnitudes to be similar is overly strict. For instance, in "Walking forward in an even pace", the words "Walking", "forward", and "even" should have the most significant impact on the motion sequence. Therefore, we relax the requirement and only demand that the top-k indices remain unchanged.Prediction Robustness. Even with stable attention, we still cannot achieve stable results due to the change in text embeddings when facing perturbations, even with similar attention vectors. This requires us to impose further restrictions on the model's predictions. Specifically, in the face of perturbations, the model's prediction should remain consistent with the original distribution, meaning the model's output should be robust to perturbations.Balancing Accuracy and Robustness. Accuracy and robustness are naturally in a trade-off relationship [21, 30] . Our objective is to bolster stability while minimizing the decline in model accuracy, thereby mitigating catastrophic errors arising from input perturbations. Consequently, we require a mechanism to uphold the model's performance concerning the original input.Let 𝑦 (x) denote the prediction of the original text-to-motion model, and 𝜔 denote the attention vector. Based on the discussion above, we introduce the Stable Text-to-Motion Framework (SATO) with modified prediction ỹ and attention vector ω as follows:(1)EQUATION) (Top-𝑘 Attention Robustness) 𝑉 𝑘 ( ω, ω + 𝝆 2 ) ≥ 𝛽 ,for some 1 ≥ 𝛽 ≥ 0, ∥𝝆 2 ∥ ≤ 𝑅 2 ; Specifically, 𝝆 1 and 𝝆 2 represent perturbations; 𝑅 1 and 𝑅 2 is the robust radius, which measures the robust region; 𝐷 1 and 𝐷 2 are metrics of the similarity between two distributions, which could be a distance or a divergence; 𝛾 1 measures the robustness of prediction while 𝛾 2 measures the closeness of the two prediction distributions; 0 < 𝛽 < 1 is the robustness of top-𝑘 indices. When 𝛽 is larger, then the attention module will be more robust;∥ • ∥ is L1 or L2 norm.It is worth noting that the roles of Prediction Robustness and Top-𝑘 Robustness are not redundant. For instance, consider the vectors v 1 = (0.2, 0.1, 0.4, 0.7) and v 2 = (0.3, 0.5, 0.8, 1.0), which have the same top indices. However, the difference in their magnitudes can significantly affect the final prediction. The former affects the robustness of the prediction, while the latter emphasizes the stability of the attention vector. From a bionics perspective, the latter facilitates the model to more stably focus on crucial motion information.We have already proposed a rigorous definition of SATO. To build our framework (shown in Fig. 3 ), we use the representative T2M-GPT as the basis to provide a more concrete demonstration of SATO. And we have also verified the wide applicability of our method in MoMask [6] . To obtain a text encoder module with more stable attention, we unfreeze the CLIP module, which was originally frozen in most of the work [6, 31] , and derive a minimum-maximum optimization problem with three conditions from the above three mathematical formulas, as shown in the following formula.EQUATIONwhere 𝜆 1 , 𝜆 2 , 𝜆 3 are hyperparameters, W represents the weight of the model. Here, we employ a maximum perturbation 𝝆 that acts simultaneously on both factors. We need to point out that there are two challenges in the optimization: (1) How to handle the non-differentiable function -𝑉 𝑘 ( ω, ω + 𝝆), and (2) how to find 𝝆 that maximizes the perturbation on 𝜔 within a certain range. Stable Attention Module. For the first issue, we need to seek an equivalent L Topk to replace -𝑉 𝑘 ( ω, ω + 𝝆). The previous discussion highlighted the necessity of considering the overlap of the previous k indices for the stability of our attention mechanism. This implies that solely relying on L 1 -norm or L 2 -norm is insufficient [10] . We need a method that is both differentiable and ensures attention to the top-k indices. One approach is to introduce the cross-distance of the values associated with the top-k indices for computation.Here, we introduce a loose surrogate loss:EQUATIONwhere 𝜁 𝜔 𝑘 represents the top-k indices set of vector 𝜔, and ∥ • ∥ denotes a norm. In this paper, the L 1 -norm is used, which yields the best experimental results. This definition serves two purposes: it ensures the stability of the top-k indices of attention and cleverly resolves the non-differentiability issue. We have 𝜔 = (0.1, 0.3, 0.7) and ω = (0.5, 0.1, 0.2). We use the top-2 indices, denoted as𝜁 𝜔 2 = [1, 2] and 𝜁 ω 2 = [0, 1]. Using the L1-norm, we obtain L Top2 = 1 4 (|[0.3, 0.7] -[0.1, 0.2]| + |[0.1, 0.3] -[0.5, 0.1]|) = 0.325.Perturbation Module. Regarding the second issue, one approach is to introduce artificially generated high-quality synonym perturbation datasets, thereby obtaining the maximum perturbation for ω. And for another approach, we interpret it as maximizing its susceptibility to attack for seeking the maximum perturbation for ω. We transform this problem into solving the minimal max-attack within a certain range. Through the process of PGD [16] with n iterations, we search for the maximum attack 𝝆.EQUATION𝝆 * 𝑘 = argmin ∥𝝆 ∥ ≤𝑅 ∥𝝆 -𝝆 𝑘 ∥where |B 𝑛 | represents the batch size, and 𝑟 𝑘 is the step size. We utilize the gradient descent algorithm, leveraging the gradient operator ∇, to iteratively update the parameters. By scaling the gradient with the step size 𝑟 𝑘 and averaging it over the batch size |B 𝑛 |, we calculate the perturbation at each iteration. Through 𝑛 iterations, we aim to find the maximum perturbation we desire. Pretrained Teacher Module. After solving two challenging issues, we can easily interpret the first term of Equation (1). We employ a frozen pretrained T2M-GPT model as a teacher module. We aim to ensure consistency between SATO and the teacher model in predicting the original text. This is done to maintain the superior predictive performance of the original model while other modules enhance model stability during training, preventing the model from becoming overly stable and resulting in poor performance. SATO Optimization Goal. we present our goal of SATO stable loss optimization as follows:min W E 𝑥 [𝜆 1 (𝐷 2 ( ỹ (x, w), 𝑦 (x, 𝑤))) L 1 +𝜆 2 L Topk ( ω, ω) L 2(5)+ 𝜆 3 (𝐷 1 ( ỹ (x, ω), ỹ (x, ω)) L 3 ]Here ω represents the attention vector after perturbation (PGD or RSR). In T2M-GPT (and likewise for other models), we incorporate the three mentioned losses as auxiliary attention stability losses into the original model transformer loss L 𝑡𝑟𝑎𝑛𝑠 for fine-tuning. Eventually, we obtain:L = L trans + 𝜆 1 • L 1 + 𝜆 2 • L 2 + 𝜆 3 • L 3 (6) 
experiments: 
 Datasets. We selected the mainstream text-to-motion datasets Hu-manML3D [7] and KIT-ML [19] for the evaluation of perturbed text generation and human pose generation. The purpose of the perturbation is to simulate the diversity of motion descriptions by different dimensions in the real scene. Therefore, we first define the perturbation of the motion descriptions as follows: (1) All substitutions should be randomized (different parts and number of sentences); (2) Ensure that the semantics of the motion description are consistent before and after the replacement, avoid the distraction of polysemous words; (3) There should be a clear perturbation before and after the description replacement.In detail, we randomly select 20% of the data for each of the training, validation, and test sets of the HumanML3D dataset, respectively, analyze the replaceable words in them, and generalize the replaceable words based on their lexical properties, where the generalized categories are: nouns, adjectives, adverbs, and verbs. Then, We combine them with the context to construct a thesaurus of synonymous substitutions for each word in each lexical property and batch replacements by rules.Our replacement rule is to traverse the word list for each motion description, randomly replacing words or verb phrases in the lexicon until two types of lexical properties have been replaced, resulting in a perturb motion description sentence. Here are some examples: (1) "A man flaps his arms like a chicken while bending up and down. " is replaced with: "A human flaps his arms like a chicken while stooping up and down. " (2) "A person walks forward on an angle to the right." is replaced with: "A man walks ahead on an angle to the right." The examples in the lexicon are: "finally, ultimately, eventually, " "clap, applaud, handclap, " and so on.We also performed a quantitative analysis of substitution as shown in Table 4 , in the HumanML3D dataset, 99.13% of the motion descriptions were perturbed, and the frequency of perturbation (number of perturbed words compared to the total number of words on that description) per description amounted to 25.08%. Similarly, 97.74% of the descriptions in the KIT-ML dataset were perturbed, and the average perturbation rate reached 31.73%, which shows that the perturbation level of this perturbation strategy is fully reflected in both datasets. While keeping high perturbation, the average cosine similarity of descriptions before and after perturbation reaches 94.57% in the HumanML3D dataset, and 93.97% in the KIT-ML dataset, which indicates that the semantics before and after perturbation have strong consistency. Evaluation Metrics. In addition to the commonly utilized metrics such as Frechet Inception Distance (FID), R-Precision, Multimodal Distance (MM-Dist), and Diversity, which are employed by T2M-GPT [31] , we have introduced two additional metrics based on Frechet Inception Distance to further assess the stability of the model. Additionally, we utilize Jensen-Shannon Divergence to evaluate the stability of the model's attention. Furthermore, human evaluation is employed to obtain accuracy and human preference results for the outputs generated by the model.• Frechet Inception Distance [9] (FID): We can evaluate the overall motion quality by measuring the distributional difference between the high-level features of the motions. • Human Evaluation: We conducted evaluations of each model's generated results in the form of a Google Form. We collected user ratings on motion prediction, which encompassed both the quality and correctness of the generated motions. Additionally, we analyzed user preferences for pose prediction. Further details will be discussed in section 4.2. • Jensen-Shannon Divergence [11] (JSD): We use JSD (Jensen-Shannon Divergence) to calculate the difference in attention vectors before and after perturbation to assess the stability of attention.Notely, to measure the stability of the model, we will use three different 𝐹 𝐼𝐷 input calculation methods: (1) 𝐹 𝐼𝐷: the distribution distance between the motion generated from the original text and real motion. (2) 𝐹 𝐼𝐷 𝑃 : the distribution distance between the motion generated from text after paraphrasing and real motion. (3) 𝐹 𝐼 𝐷 𝐷 : the distribution distance between the motion generated from the original text and the motion generated from the text after paraphrasing. Moreover, we also employ human evaluation for cross-dataset evaluation to further analyze the performance and stability of the model. For assessing the stability of model attention, we propose using JSD. A smaller JSD value indicates greater stability of attention under perturbation. More details about the evaluation metrics are provided in Supplementary Material Section 2.We adopt nearly identical settings for model architecture parameters as T2M-GPT or MoMask. Additionally, We set the batch size to 64 and utilize the AdamW [15] optimizer with hyperparameters [𝛽 1 , 𝛽 2 ] = [0.9,0.99]. The total iteration is set to 100000 and the learning rate is 1e-4, employing a linear warm-up schedule for training all models. We respectively set 𝜆 1 , 𝜆 2 , 𝜆 3 to 0.1, 0.2, and 0.05. For perturbation, we set 𝑟 𝑘 to 0.01, the PGD step as 10, and 𝑅 to 0.05 when we use text embedding perturbation. Training can be conducted on a single RTX4090-24G GPU. It is worth mentioning that our method is based on fine-tuning the original model to make it more stable, without incurring any additional computation cost during the inference process.As with previous experiments, each experiment was repeated twenty times, and we report the mean with a 95% statistical confidence interval. Tables 1 and 2 respectively present the results of models HumanML3D TM2T [8] ECCV2022 1.501 ±.017 3.909 ±.039 1.418 ±.035 0.424 ±.003 0.618 ±.003 0.729 ±.002 3.467 ±.011 8.589 ±.076 T2M [7] CVPR2022 1.087 ±.021 3.990 ±.064 2.110 ±.039 0.455 ±.003 0.636 ±.003 0.736 ±.002 3.347 ±.008 9.175 ±.083 MotionDiffuse [32] arXiv2022 0.630 ±.001 2.435 ±.067 1.549 ±.032 0.491 ±.001 0.681 ±.001 0.782 ±.001 3.113 ±. 001 9.410 ±.049 MDM [26] ICLR2023 0.544 ±.044 3.251 ±.071 2.015 ±.027 --0.611 ±.007 5.566 ±.027 9.559 ±.086 T2M-GPT [31] CVPR2023 0.141 ±.005 1.754 ±.004 1.443 ±.004 0.492 ±.003 0.679 ±.002 0.775 ±.002 3.121 ±.009 9.722 ±.082 MoMask [6] CVPR2024 0.045 ±.002 0.969 ±.030 1.068 ±.029 0.521 ±.002 0.713 ±.002 0.807 ±.002 2.962 ±.008 9.962 ±.008 SATO (T2M-GPT) -0.157 ±.006 0.155 ±.007 0.021 ±.006 0.454 ±.003 0.637 ±.003 0.738 ±.003 3.338 ±.013 9.651 ±.050 SATO(MoMask) -0.065 ±.003 0.070 ±.002 0.010 ±.001 0.501 ±.002 0.697 ±.003 0.801 ±.003 3.024 ±.010 9.599 ±.075 3 : Human evaluation and cross-dataset results on the original or perturbed text. "Excellent" means completely meets the semantic, with smooth and correct expression; "Good" means generally generates well with minor details; "Fair" means contains errors in details but is overall correct; "Poor" means overall incorrect; "Very poor" means motions and text cannot be matched at all. We believe that Excellent, Good, and Fair represent correctly generated postures, while the other two represent errors. Preference indicates human preference for the compared motions. The cross-dataset evaluation result is that the model is trained on the HumanML3D dataset, with text from KIT-ML used for testing. 4 : Dataset analysis. We analyzed the replacement rates (sentences, vocabulary). Additionally, we calculated the cosine similarity (Co-Sim) before and after replacement to ensure the validity of our substitutions.on the HumanML3D and KIT-ML datasets. We compare our results with six state-of-the-art (SOTA) methods. Stability. It is worth noting that all other models perform poorly on the perturbed dataset, with 𝐹 𝐼𝐷 𝑃 significantly greater than 𝐹 𝐼𝐷, indicating that diverse representations of perturbations are fatal to the performance of these models. On 𝐹 𝐼𝐷 𝑃 , SATO(T2M-GPT) significantly reduced by 1.599 and 2.175 on HumanML3D and KIT-ML respectively, while SATO(Momask) decreased by 0.899 and 2.311 respectively. Similarly, there was a significant increase in 𝐹 𝐼 𝐷 𝐷 , with SATO(T2M-GPT) decreasing by 1.422 and 2.757 respectively, and SATO(MoMask) decreasing by 1.058 and 2.178 respectively. This suggests that SATO yields similar predictive results on both perturbed and original datasets, indicating stronger stability. We can also observe a significant reduction in the fluctuation of our model on the 𝐹 𝐼𝐷, 𝐹 𝐼𝐷 𝑃 , 𝐹 𝐼𝐷 𝐷 metrics, which also reflects the stability of our approach in predictions. We further investigated the impact of our approach on the attention JSD metric. Our method exhibits stability in attention, as evidenced by experiments and visualizations provided in the supplementary material. Accuracy. Although our model experiences a slight decrease in 𝐹 𝐼𝐷 and R-precision, we would like to point out that previous work has shown that a small decrease in 𝐹 𝐼𝐷 does not necessarily imply a decrease in generation quality [12] . Our visualization results corroborate this point, and we have additional examples from supplementary material and anonymous website to further substantiate this perspective. Furthermore, compared to the significant improvement in stability measured by the 𝐹 𝐼𝐷 metric, the slight decrease in 𝐹 𝐼𝐷 on the original text can be almost disregarded. Our visualizations and additional human evaluation experiments also demonstrate that the quality of text generated by our model on both original and perturbed text is superior to the original model. This means that our model can generate higher-quality motion sequence outputs in practical applications, with a lower likelihood of catastrophic errors occurring when presented with a broader range of textual inputs. Human Evaluation on the Original or Perturbed Text. Our work's motivation is to address the catastrophic errors users encounter when using perturbed text by implementing a stable attention model. We further conduct a user study on Google Forms to validate the correctness of the model's generation. We generated 200 motions for each method using the same text pool from the HumanML3D test set as the input for all baseline models and SATO. We set up questions for users to rate the motions. Table 3 shows that SATO not only maintains or even achieves better accuracy on the original text but also ensures stability when the text is perturbed. User preferences indicate that SATO performs slightly better than the original model on the original text and significantly outperforms the original model on the perturbed text dataset. More details can be found in the supplementary material.Cross dataset evaluation. To further test and evaluate the robustness and applicability of our model, we compare SATO (T2M-GPT) with T2M-GPT as examples. We conducted training on the Hu-manML3D dataset, using 200 original and perturbed texts from the sample kit dataset as inputs for evaluation. Table 3 illustrates that SATO (T2M-GPT) achieves higher accuracy than the baseline by 7% on original texts and by 35.5% on perturbed datasets. Evaluators also tend to favor the quality of outputs generated by SATO. Both metrics indicate that our model demonstrates strong robustness to dataset variations. This also suggests that our approach can enhance the generalization performance of the model, enabling it to be applied in a wider range of domains.Overall, SATO achieves state-of-the-art stability, balancing accuracy and robustness, resolving catastrophic errors caused by synonymous perturbations. based on T2M-GPT. In Table 5 , we observe that both the Stable Attention Module(L 2 ) and Perturbation Module(L 3 ) contribute to improving stability, as evidenced by the enhancements in 𝐹 𝐼𝐷 𝑃 by 1.521, 1.533 respectively, and 𝐹 𝐼𝐷 𝐷 by 1.417, 1.422 respectively. The combined effect of these modules achieves optimal stability performance.The inclusion of the pre-trained Teacher Module (L 1 ) enhances the model's 𝐹 𝐼𝐷 performance, preventing excessive stability at the expense of accuracy, albeit with a potential slight decrease in stability metrics. Moreover, this module plays a crucial role by automating the selection of the best training iterations, striking a balance between robustness and accuracy, and keeping the model more evenly poised between stability and accuracy. Resistance to synonymous perturbation. Based on the varying numbers of synonymous word substitutions in the test set, we categorize perturbations as mild (1 word), moderate (2-3 words), and severe (>3 words). Visualizing the results in Fig. 5 , it's apparent that our model demonstrates superior stability compared to T2M-GPT across different levels of perturbation. Even when faced with severe perturbations, SATO consistently maintains excellent stability. Our model's stability metrics significantly outperform those of the original model on datasets with mild perturbations, underscoring its robustness across various degrees of perturbation. More ablation results are provided in Supplementary Material Section 4. 
conclusion: 
 We identified instability issues in the text-to-motion task and introduced a novel framework to address them. In the process of building SATO, we tackled two key challenges. We also proposed evaluation metrics for this task and constructed a dataset of 55k perturbed text pairs. Our experiments demonstrate that SATO is an attention-stable and prediction-robust framework, exhibiting broad applicability across various baselines and datasets. We aim to encourage more researchers to delve into this issue, further enhancing the performance and stability of text-to-motion systems.  
 
 Appendix:  


Decision: This paper addresses instability issues in text-to-motion models, where semantically similar text inputs lead to inconsistent outputs. The authors introduce the Stable Text-to-Motion Framework (SATO), which incorporates perturbations to text inputs to enhance output consistency in terms of the attention vector and predictions. Experimental validation on the HumanML3D and KIT-ML datasets shows that SATO improves stability while maintaining competitive accuracy. The paper is well-written and provides a clear motivation for its approach, which includes a new dataset for evaluating the model's performance against text synonym perturbations. The comprehensive experiments and the inclusion of comparative videos further support the effectiveness of the SATO framework. All reviewers agreed to accept the paper.

Reviews: review: paper_summary: paper_summary: This paper addresses the instability issues prevalent in current text-to-motion models. Specifically, the authors highlight that existing models exhibit inconsistent outputs when provided with semantically similar text inputs. To mitigate this issue, they propose the Stable Text-to-Motion Framework (SATO). The SATO framework first introduces perturbations to text inputs and then ensures the consistency of the output attention vector and predictions. Experimental results on the HumanML3D and KIT-ML datasets demonstrate improved stability while maintaining competitive accuracy.
strength_weakness: strength_weakness: 1. The identification of stability issues in current text-to-motion models is crucial for ensuring reliable and predictable behavior in practical applications.

2.  The results demonstrate that the proposed method yields improved performance on two widely used datasets.
limitations: limitations: 1. Lack of automatic metrics for evaluating the "stability" of final predictions. The proposed stability is only evaluated with user study and the JSD of attetion vectors. 

2. The paper may not fully capture the range of possible perturbations encountered in real-world applications. Text perturbations can include misspellings, abbreviations, which are not comprehensively addressed in this study.

3. The experimental analysis does not discuss certain notable behaviors. For example, there is a lack of explanation for why diversity decreases on the HumanML3D dataset while showing significant improvement on KIT-ML with the MoMask baseline.
paper_summary: paper_summary: This paper introduces a formal framework named Stable Text-to-Motion Framework (SATO) which improves the stability of text-to-motion models and contributes a new dataset.
strength_weakness: strength_weakness: 1. The writing is clear and easy to understand.
2. The experimental setup is comprehensive and sufficient.
3. The motivation behind the work is novel and well-supported.
4. The performance of the SATO model is impressive.
limitations: limitations: The captions in Figure 4 have some errors:

1. The captions in Lines 3 and 4 should be consistent in their formatting.
2. The words in red should be different between Lines 3 and 4.
paper_summary: paper_summary: This study identified instability issues in the text-to-motion task, wherein providing semantically similar text inputs leads to significant motion deviations. The paper introduces SATO to address this problem, consisting of three modules: stable attention, stable prediction, and a balance between maintaining accuracy and robustness. Additionally, a text synonym perturbation dataset based on HumanML3D and KIT-ML is introduced to validate the model's stability.
strength_weakness: strength_weakness: 1. This paper is well-written, and the motivation behind the proposed method is clear and reasonable.

2. The paper identifies instability issues in the text-to-motion task, which is an interesting new direction.

3. Good performance is achieved by evaluating on both the original dataset and the perturbed dataset.

4. Comparative videos are provided to validate the effectiveness of the method.
limitations: limitations: 1. There is an issue with the $FID_P$ and $FID_D$ in experimental comparison. SATO's method is trained on both the original and perturbed datasets, whereas other methods measure  $FID_P$ and $FID_D$ using the models trained only on the original data. In this scenario, the poor performance is expected. The fairest comparison would be: firstly training the T2M-GPT and MoMask on a combined dataset of the original and perturbed datasets, and then testing the $FID_P$ and $FID_D$.

2. There is a lack of discussion and citation of some text-to-motion papers related to robust text representations.

     [1] Jin et al. Act As You Wish: Fine-Grained Control of Motion Diffusion Model with Hierarchical Semantic Graphs. NeurIPS 2023.

     [2] Wang et al. Fg-t2m: Fine-grained text-driven human motion generation via diffusion model. ICCV 2023.

3. There are two errors in Table 1: the third "$FID$" should be "$FID_P$" and there is a citation error for MotionDiffuse[3].

     [3] Zhang et al. MotionDiffuse: Text-driven human motion generation with diffusion model. TPAMI 2024.