Towards Emotion-enriched Text-to-Motion Generation via LLM-guided Limb-level Emotion Manipulating

Authors: tan  yu
jingjing  wang
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 10:34:39

Abstract: In the literature, existing studies on text-to-motion generation (TMG) routinely focus on exploring the objective alignment of text and motion, which largely ignore the subjective emotion information, especially the limb-level emotion information. With this in mind, this paper proposes a new Emotion-enriched Text-to-Motion Generation (ETMG) task, aiming to generate motions with the subjective emotion information. Further, this paper believes that injecting emotions into limbs (named intra-limb emotion injection) and ensuring the coordination and coherence of emotional motions after injecting emotion information (named inter-limb emotion disturbance) is rather important and challenging in this ETMG task. To this end, this paper proposes an LLM-guided Limb-level Emotion Manipulating (${\rm L^{3}EM}$)  approach to ETMG. Specifically, this approach designs an LLM-guided intra-limb emotion modeling block to inject emotion into limbs, followed by a graph-structured inter-limb relation modeling block to ensure the coordination and coherence of emotional motions. Particularly, this paper constructs a coarse-grained Emotional Text-to-Motion (EmotionalT2M) dataset and a fine-grained Limb-level Emotional Text-to-Motion (Limb-ET2M) dataset to justify the effectiveness of the proposed ${\rm L^{3}EM}$ approach. Detailed evaluation demonstrates the significant advantage of our ${\rm L^{3}EM}$ approach to ETMG over the state-of-the-art baselines. This justifies the importance of the limb-level emotion information for ETMG and the effectiveness of our ${\rm L^{3}EM}$ approach in coherently manipulating such information.

Main paper: 
 introduction: 
 Human motion generation leverages various control signals (e.g., music [13, 14] , action categories [11, 21] and text descriptions [10, 35, 37] ) to generate vivid and realistic motions, which has numerous practical applications in fields such as game production, film and virtual reality. Among these control signals, text is extensively applied as a control condition for motion generation due to its rich semantic details and convenient user interaction, i.e., text-tomotion generation (TMG). Existing studies in TMG focus on using various generative models such as VAEs [10, 22] and diffusion models [5, 30] for the objective alignment of text and motion. However, these studies largely ignore the diverse and subjective emotion information, leading to existing approaches struggling to generate emotion-enriched motions.Inspired by the above observations, this paper proposes a novel Emotion-enriched Text-to-Motion Generation (ETMG) task, aiming to generate motions with rich emotional expressions, which can significantly enhance audience resonance and emotional connection in the application of many fields, such as virtual avatar [3, 20] and emotional robots [6, 27] . Specifically, the ETMG task generates emotion-enriched motions via given text descriptions. Taking Figure 1 (b) as an example, given the text "A person, filled with sadness, walks forward. ", the model requires to generate a more realistic and expressive motion sequence with emotion sad, compared to the generated motion sequence in Figure 1 (a) with text "A man walks forward". Especially, we explore two kinds of challenges inside the ETMG task. For one thing, how to inject emotion into the human limbs is challenging, namely the intra-limb emotion injection challenge. Factually, emotion-enriched motion sequences entail rich emotional semantics, while text description usually contain only one emotion word (e.g., sadness). Also as shown in Figure 1 (b), based on the text description "A man, filled with sadness, walks forward.", we obtain the human limbs Head: lowered, looking downwards; Torso: slightly bent; etc. with emotion sad, where human limbs associated with the motions could correspond to emotional expression in the generated motion sequence. Therefore, a well-designed approach should consider injecting emotion into the human limbs, for precisely matching the human limbs to corresponding emotions and effectively generating emotion-enriched motions.For another, how to ensure the coordination and coherence of emotional motions after injecting emotion information is challenging, namely the inter-limb emotion disturbance challenge 1 . We assume that the injection of emotion would limit the generation of motions, and affect the coordination and coherence of the generated motions. As shown in Figure 1 (a) , the text description "a man walks forward" without any emotion generates the motion sequences with no limitations on the human limbs, while after injecting emotion (e.g., sadness) into limbs, limbs related to the motion should have corresponding emotional expression, such as Head: lower; Torso: bent; etc. Thus, emotion information could cause some disturbance and interference to the motions of the human limbs, which may lead to incoordination and incoherence of the generated emotion-enriched motions. Therefore, a better-designed approach should consider capturing spatial position and relation information between human limbs to tackle emotional disturbance and ensure the coordination and coherence of emotional motions.In this paper, we propose an LLM-guided Limb-level Emotion Manipulating (L 3 EM) approach to tackle the above two challenges. Specifically, this approach first designs an LLM-guided Intra-limb Emotion Modeling (LEM) block via Large Language Models (LLM) to inject emotion into the human limbs to address the intra-limb emotion injection challenge. Then, this approach designs a Graphstructured Inter-limb Relation Modeling (GRM) block via the limb relation graph to capture limb spatial position and relation information for addressing the inter-limb emotion disturbance challenge. Furthermore, a coarse-grained Emotional Text-to-Motion 1 Comprehensive analysis is described in Section 5.3 "Is emotional disturbance really addressed?" (EmotionalT2M) dataset and a fine-grained Limb-level Emotional Text-to-Motion (Limb-ET2M) dataset are constructed to evaluate the effectiveness of the proposed L 3 EM approach. Detailed experiments demonstrate that our L 3 EM approach achieves significant improvements compared to the current state-of-the-art baselines. 
related work 2.1 text-to-motion generation: 
 Text-to-motion generation has always been a popular research topic in generative AI. There is a large body of work proposing different generative neural networks to handle this task. Early studies [1, 9, 22, 29] focus on using joint-latent models to learn a joint embedding space of motion and text. For example, MotionCLIP [29] aligns the human motion manifold to CLIP space implicitly for generating motion sequences with text descriptions. TEMOS [22] introduces a VAE architecture to learn a joint latent space of human motions and text descriptions and can generate different motion sequences given one text description. Inspired by the significant success of diffusion models in the field of image generation [2, 25, 26] , some recent works employ diffusion models to handle text-to-motion tasks. For example, MotionDiffuse [36] firstly introduces Denoising Diffusion Probabilistic Models (DDPM) for versatile and controllable human motion generation. MDM [30] predicts the sample, rather than the noise in each diffusion step, thus improving the quality of generated motions. Furthermore, MLD [5] performs a diffusion process on the motion latent space and substantially reduces the computational overhead. Most recently, some works [12, 35, 38] quantize motion sequences into discrete motion tokens and then a GPT-like structure to generate subsequent tokens. For example, MotionGPT [12] uses a Vector Quantised Variational AutoEncoder (VQ-VAE) to quantize motion sequence and use a transformer to automatically generate later tokens.Although the aforementioned works have made significant progress in the field of text-to-motion, they have always ignored subjective emotion information. Unlike prior studies, this paper first proposes a new Emotion-enriched Text-to-Motion Generation (ETMG) task and proposes two intra-limb emotion injection challenge and interlimb emotion disturbance challenge in tackling this task.Large language models(LLM) have demonstrated impressive capabilities for text generation and semantic understanding [28, 39] . Recently, some works [4, 7, 8, 23, 41] have begun to explore leveraging the powerful abilities of LLM to enhance the generation performance of diffusion models conditioned on text in the fields of Text-to-Image and Text-to-Video. Specifically, in the field of Textto-Image, Qu et al. [23] use LLM to generate the layout of images and then employ a diffusion model to synthesize high-fidelity images conditioned on both the prompt and the generated layout, which is inspirational to our approach. Zhong et al. [41] propose a Semantic Understanding and Reasoning Adapter (SUR-adapter) for pre-training diffusion models. By transferring knowledge from a large-scale language model (LLM) to the SUR-adapter, the diffusion model is able to understand and reason concise natural language without image quality degradation. In the field of Text-to-Video, Fei et al. [7] take advantage of the existing powerful LLM (e.g., ChatGPT) to generate video dynamic scene graphs for tackling the intricate video temporal dynamics issue in video generation. Different from all the above studies, this paper first designs an RL-based Emotional ICL module to assist LLM in generating high-quality limb emotional guidance for addressing the intra-limb emotion injection challenge, and then utilizes a limb relation graph to capture limb spatial position and relation information to tackle the emotion disturbance challenge. 
approach: 
 In this section, we formulate the ETMG task as follows. Given a text description 𝑦 (e.g., a man walks forward sadly), which describes a motion and conveys an emotion 𝑒 (e.g., sad), the goal of ETMG is to generate a 3D human motion sequence x 1:𝐿 with the length 𝐿 that matches the text description 𝑦 and the emotion 𝑒. In this paper, we propose an L 3 EM approach for generating motions with rich emotional expressions. The overall framework of L 3 EM is shown in Figure 2 , which mainly comprises the LLM-guided Intra-limb Emotion Modeling (LEM) Block (Sec. 3.2) for addressing the intralimb emotion injection challenge and the Graph-structured Interlimb Relation Modeling (GRM) Block (Sec. 3.3) for addressing the inter-limb emotion disturbance challenge.In this paper, we adopt the open-sourced Motion Latent-based Diffusion model (MLD) [5] as our backbone due to its high performance and low computation overhead. MLD first employs a Variational AutoEncoder (VAE) to obtain a representative and low-dimensional latent code for a motion sequence and then performs a diffusion process on the motion latent space, which substantially improves the computational efficiency. Specifically, a motion sequence 𝑥 1:𝐿 with the length 𝐿 is first mapped to a motion latent space through motion encoder E of a pre-trained VAE, i.e., 𝑧 0 = E (𝑥 1:𝐿 ). The diffusion process on latent space is modeled as a Markov nosing process using the:EQUATIONwhere 𝑞(𝑧 𝑡 |𝑧 𝑡 -1 ) denotes the posterior distribution. 𝛼 𝑡 ∈ (0, 1) is constant hyper-parameters. N represents the normal distribution and 𝐼 represents the identity covariance matrix. {𝑧 𝑡 } 𝑇 𝑡 =0 denotes the noising sequence in the motion latent space, where 𝑇 represents the time step. Then, at the denoising process, a conditional denoiser 𝜖 𝜃 (𝑧 𝑡 , 𝑡, 𝑐) is used to predict the noise 𝜖 added to 𝑧 𝑡 -1 in the diffusion process.i.e.,𝜖 = 𝜖 𝜃 (𝑧 𝑡 , 𝑡, 𝑐), where 𝑐 denotes the condition input. In our ETMG task, the condition 𝑐 is the text embedding encoded by the CLIP [24] text encoder. Subsequently, the denoised version 𝑧 𝑡 -1 of 𝑧 𝑡 at each time step 𝑡 is obtained by 𝑧 𝑡 -1 = 𝑧 𝑡 -𝜖 𝜃 (𝑧 𝑡 , 𝑡, 𝑐). During the inference stage, 𝑧 0 is obtained with 𝑇 iterative denoising steps given a noisy latent 𝑧 𝑡 ∈ N (0, 1), timestep 𝑡, and the conditioning text embedding 𝑐. Then, motion decoder 𝐷 of the pre-trained VAE maps 𝑧 0 to the original motion sequence.As shown in Figure 2 (b), the LLM-guided Intra-limb Emotion Modeling (LEM) Block is designed to generate limb emotional guidance from LLM. Specifically, in this paper, we use in-context learning (ICL) [31] to generate emotional guidance for ten major limbs (i.e., head, left shoulder, right shoulder, left arm, right arm, torso, left leg, right leg, left foot, right foot) of humans from LLM (i.e., Chat-GLM [34] ). The ICL prompt we use consists of three parts, a task description (Instruction), a few input-output examples (in-context examples), and a test input (Test). Previous studies [18, 19, 40] indicate that the efficiency of ICL is significantly affected by how in-context examples are structured. Therefore, it is crucial to choose appropriate in-context examples from the training set to activate the ICL capability of LLM. In this paper, inspired by [23] , we design an RL-based Emotional ICL module in Figure 2 (b) to tackle this issue, where RL and ICL refer to Reinforcement Learning and In-Context Learning, respectively. The RL-based Emotional ICL module mainly consists of two parts, i.e., policy network and emotion-enhanced reward.• Policy Network. Given a text 𝑦 (e.g., a man walks forward sadly.) and the emotion 𝑒 (e.g., sad) in the text, we use the policy network𝜋 𝜃 to select K in-context examples 𝐶 𝐼 = {𝑐 𝑘 |𝑘 = 1, 2, ..., 𝐾 } from a candidate set 𝐶, i.e., 𝑐 𝑘 ∼ 𝜋 𝜃 (𝑐 |𝑦), where we randomly select 𝑁 samples from training set and annotate them with limb emotional guidance, forming the candidate set 𝐶. 𝑐 𝑘 ∈ 𝐶 is independently sampled from the candidate set 𝐶. Specifically, the policy network 𝜋 𝜃 is implemented as follows:EQUATIONwhere 𝑦 [𝑐] and 𝑒 [𝑐] denote the text (e.g., A man steps forward, looking very happy.) and the emotion (e.g., happy) concerning the candidate 𝑐. 𝑓 (•) represents a mapping function that transforms a text and emotion into a latent embedding. In latent space, sentences with similar semantics and close emotions are to be mapped close to each other.• Emotion-enhanced Reward. In this paper, our goal is on the one hand to generate motions that are of high quality and match the text, and on the other hand, we also want to make sure that the generated motions are emotion-enriched. Therefore, we design an Emotion-enhanced reward 𝑅, which consists of three main parts, detailed as follows:EQUATIONwhere 𝛼, 𝛽, 𝛾 are hyper-parameters. 𝑅 𝑞 , 𝑅 𝑚 , and 𝑅 𝑒 are used to measure the quality of generated motions, the matching with the text, and how much the generated motions correspond to the intended emotions, respectively. Specifically, they are calculated by:𝑅 𝑞 = Cos(MotionEncoder( x1:𝐿 ), MotionEncoder(x 1:𝐿 ))𝑅 𝑚 = Cos(MotionEncoder( x1:𝐿 ), TextEncoder(𝑦))EQUATIONwhere Cos denotes the cosine similarity between two vectors. The larger the value, the closer the distance between the two vectors. MotionEncoder and TextEncoder denote the motion encoder and text encoder in our trained evaluator (Detailed in Sec 4.2), which aims at encoding the paired text and motion sequences into a joint embedding space. x1:𝐿 and x 1:𝐿 represent the generated motion and the ground-truth motion, respectively. 𝑦 denotes the text description corresponding to the motion. WF1 and NN-WF1 (Detailed in Sec 4.2)is the overall classification weighted average f1-score and the nonneutral classification weighted average f1-score, which reflects how much the generated motions correspond to the intended emotion in all emotion classes and non-neutral emotion classes, respectively. Through the RL-based Emotional ICL module, we select the K in-context examples 𝐶 𝐼 . Then, we combine it with instruction and 𝑦, inputting them into ChatGLM [34] to obtain the limb emotional guidance 𝐺 = {𝑔 1 , 𝑔 2 , ..., 𝑔 10 }, i.e., 𝐺 = ChatCLM(Instruction, 𝐶 𝐼 , 𝑦), where 𝑔 𝑖 , 𝑖 ∈ {1, 2, .., 10} denotes the emotional guidance for each limb (i.e., head, left shoulder, right shoulder, left arm, right arm, torso, left leg, right leg, left foot, right foot).In the above LEM Block, we use LLM (i.e., ChatGLM [34] ) to generate emotional guidance for each limb. However, this emotional guidance lacks spatial position and relation information about the limbs, which is crucial for guiding diffusion models in generating coordinated and coherent emotional motions. In this paper, as shown in Figure 2 (c), we integrate a limb relation graph into a Graph-Transformer(namely the Limb-relation Aware GraphTransformer) to capture this information. Specifically, for each limb (i.e., head, left shoulder, right shoulder, left arm, right arm, torso, left leg, right leg, left foot, right foot) emotional guidance 𝑔 𝑖 , where 𝑖 ∈ {1, 2, .., 10}, we first use the CLIP text encoder C 𝑇 to convert it into a vector representation f = {C 𝑇 (𝑔 1 ), C 𝑇 (𝑔 2 ), ..., C 𝑇 (𝑔 10 )}. Then, we use the skeleton structure of SMPL [16] with 22 joints as our limb relation graph. In addition, also as shown in Figure 2 (c), we use linear mapping to project these vector representations f onto the corresponding joints as the initialization for graph nodes(i.e., h = Linear (f), where h denotes the initial vector representation of the graph nodes. Linear refers to the linear mapping.). Subsequently, the limb relation graph is fed into a GraphTransformer to capture spatial position and relation information about the limbs based on the emotional guidance of limbs.EQUATIONwhere GraphTransformer denotes a graph transformer network [33] .A refers to the adjacency matrix of the limb relation graph. h ′ represents the output representation vector of the GraphTransformer, which integrates both spatial position and relation information and emotional guidance information of limbs.In order to integrate the output representation vector h ′ into the transformer layer in denoiser, we introduce a Limb-relation Aware CrossAttention. Specifically, as shown in Figure 2 (c), the output motion feature m ′′ of the transformer layer is obtained by:EQUATIONwhere CrossAttn denote the Cross-Attention. LayerNorm denotes the layer normalization. m denotes the intermediate motion feature.To ensure the stability of reinforcement learning training, we employ a two-stage training strategy. Specifically, in the first stage, we optimize our model by the simple objective used in MLD [5] :L 𝜖∼N (0,1),𝑡 ∈ [1,𝑇 ] = E 𝜖,𝑡,𝑐 ∥𝜖 -𝜖 𝜃 (𝑧 𝑡 , 𝑡, 𝑐)∥ 2 2 (7)This optimization objective is designed to enable the denoiser 𝜖 𝜃 (𝑧 𝑡 , 𝑡, 𝑐) to accurately predict the noise 𝜖 added during the diffusion stage.In the second stage, we use reinforcement learning to further improve the performance of our L3 EM model. We optimize the policy network 𝜋 𝜃 with policy gradient algorithm [32] . In detail, we first obtain the reward 𝑅 according to Eq.( 3). Then, the policy gradient is computed by differentiating the maximized expected reward ∇ 𝜃 𝐽 (𝜃 ) as follows:EQUATION4 Experimental Settings 
dataset construction: 
 To evaluate the effectiveness of our approach to the ETMG task, we construct a coarse-grained Emotional Text-to-Motion (Emo-tionalT2M) dataset and a fine-grained Limb-level Emotional Textto-Motion (Limb-ET2M) dataset.• Coarse-grained EmotionalT2M dataset. The coarse-grained EmotionalT2M dataset is constructed based on the HumanML3D dataset [10] and IDEA400 motion dataset [15] . Specifically, 1) for the Humanml3D dataset, it consists of 14616 motions and 44970 texts. However, it lacks emotion labels, and only a small portion of the data contains emotion information. Therefore, in order to construct our EmotionalT2M dataset, we manually select data containing emotions and annotate the emotion categories. 2) For the IDEA400 motion dataset, it comprises 12513 motions and texts. Each motion is labeled with an emotional label, but the text descriptions in the IDEA400 dataset exhibit low quality and lack emotion information. Such as, the text description Lifting Weights During Walking lacks a subject and does not convey any emotion. Therefore, in order to construct our high-quality EmotionalT2M dataset, we use ChatGLM [34] to integrate emotion labels(i.e., sad) into the text descriptions, resulting in text descriptions that are grammatically complete and fluent (i.e., A woman is lifting weights while walking, appearing sadly despite her efforts.). Furthermore, in the IDEA400 motion dataset, motion is represented using tomato representation [17] , which is different from the motion representation used in HumanML3D. To merge the two datasets, we convert the tomato representation used in the IDEA400 motion dataset to the motion representation used in HumanML3D. 3) In the end, we obtain our EmotionalT2M dataset, which comprises 14553 motions and 14792 text descriptions, with 8 emotion categories (i.e., Happy, Surprise, Neutral, Sad, Angry, Fear, Disgust, Contempt). Same as the HumanML3D dataset, we randomly split the dataset into training, validation, and test sets in a ratio of 0.8: 0.15 : 0.05. Detailed statistics of EmotionalT2M dataset are shown in Table 1 .• Fine-grained Limb-ET2M dataset. In order to validate that our approach can generate emotion-enriched motions for all limbs (i.e., head, shoulder, torso, arm, leg, and foot), we additionally annotate a fine-grained Limb-level Emotional Text-to-Motion (Limb-ET2M) dataset in the test set of our EmotionalT2M dataset. Specifically, we first spend about a month rendering a total of 500 emotional motions using Blender2 . Then, we assign two annotators to annotate three emotions for each limb of every motion sequence. Taking the motion sequence shown in Figure 1 (b) as an example, all the limbs in this motion exhibit negative emotion expressions. Therefore, we label all his limbs as "negative". The Kappa consistency check value of the annotation is 0.88. When two annotators cannot reach an agreement, an expert will make the final decision, ensuring the quality of data annotations. The Limb-ET2M dataset comprises 500 motions and 500 text descriptions, with each motion corresponding to a single text description. In addition, we randomly split the dataset into training, validation, and test sets in a ratio of 0.7: 0.1: 0.2, and the statistics of the dataset is shown in Figure 3 . From Figure 3 , we can observe that head and arm have the least neutral emotion. This aligns with our intuition, i.e., people are more prone to expressing emotions using them compared to other limbs 3 .Evaluation Metrics are summarized in four parts. 1) Motion Quality: Frechet Inception Distance (FID) reports quality of the generated motion by calculating the distance between features extracted from real and generated motion sequences. It can also to some extent reflect the coordination and coherence of the generated motions. 2) Condition Matching: R Precision calculates the text and motion Top 1/2/3 matching accuracy, which reflects the similarity between the text description and the generated motion sequence. Multi-modal Distance (MM Dist) measures the average Euclidean distance between the motion feature and its corresponding text description feature. 3) Generation Diversity: Diversity(DIV) evaluates the variability of the generated motion sequences, while Multi-Modality (MM) measures the average variance of generated motion sequences within the same text description. 4) Emotion Performance: We use the overall (e.g., positive, neutral, and negative) Table 2 : Results on the EmotionalT2M dataset. All methods use the real motion length from the ground truth for a fair comparison. '↑'('↓') indicates that the values are better if the metric is larger (smaller). We run all the evaluations 20 times. 𝑥 ±𝑦 indicates that the average metric is 𝑥 and the the 95% confidence interval is 𝑦. 3 , we separately train six emotion classifiers for six limb parts (i.e., head, should, torso, arm, leg, foot) on the Limb-ET2M dataset. Then, we input the generated motion sequences of each limb (e.g., arm motion sequences) into the corresponding emotional classifier (e.g., arm emotion classifier) and obtain the WF1 score and the NN-WF1 score, which reflects how much the generated motions of each limb correspond to the intended emotions in all emotion classes (i.e., positive, neutral and negative) and how much the generated motions of each limb correspond to the intended emotions in the non-neutral emotion classes (i.e., positive and negative).For the Motion Quality, Generation Diversity, and Condition Matching, following Guo et al. [10] , we train our contrastive model on our EmotionalT2M dataset as the evaluator. Specifically, this contrastive model consists of a motion encoder and a text encoder, which aims to map the paired text descriptions and motion sequences into a joint embedding space. For the motion encoder, we use the same motion encoder as employed by Guo et al. [10] . For the text encoder, we employ the text encoder used in our approach (i.e., CLIP text encoder). We train the contrastive learning model with the same loss in Guo et al. [10] . For the Emotion Performance, previous studies [10, 37] on motion generation don't evaluate the emotional performance of generated motions. For the first time, we use emotion classification metrics (i.e., weighted average f1-score) to assess the emotional performance of generated motions and train seven emotion classifiers(i.e., the overall motion emotion classifier and the six motion emotion classifier for six limbs) as evaluators.We choose several advanced baselines in Text-to-Motion Generation (TMG) task to compare performance with our approach, described as follows. T2M [10] proposes a two-stage approach: text2length sampling and text2motion generation to tackle the TMG task. Mo-tionDiffuse [36] is the first diffusion model-based text-driven motion generation framework. MDM [30] predicts sample rather than noise in each diffusion step to facilitate the use of geometric losses. MLD [5] designs a powerful VAE to get the low-dimensional latent codes for human motion sequences and then performs a diffusion process on the motion latent space. MotionGPT [38] can generate consecutive human motions by interpreting multimodal signals as unique input tokens within LLM. T2M-GPT [35] uses VQ-VAE and GPT for the TMG task. ReMoDiffuse [37] integrates a retrieval mechanism to refine the denoising process and enhances the generalizability and diversity of text-driven motion generation. This model is the state-of-the-art model in text-to-motion tasks.In our experiments, we re-implement all the baselines on our ETMG datasets according to their open-source codes. Besides, the hyperparameters of these baselines reported by their public papers are still adopting the same setting. The others and the hyper-parameters of our L 3 EM approach are tuned according to the validation set. In the first stage, we employ a frozen CLIP-ViT-L-14 4 model as the text encoder. Adam is adapted as the optimizer to train the model with a learning rate equal to 6e-5. The batch size and the epoch are 32 and 5k, respectively. We use ChatGLM (130B5 ) [34] to generate limb emotional guidance. In the second stage, The 𝛼, 𝛽 and 𝛾 in Eq.( 3) is 0.5, 0.5 and 0.7, respectively. The size of candidate 𝐶 is 32 (i.e., 𝑁 = 32), and we set 3 as the shot number (i.e., 𝐾 = 3) Table 3 : Results on our Limb-ET2M Dataset to evaluate the emotional performance of each limb.Head Shoulder Torso Arm Leg FootWF1 NN-WF1 WF1 NN-WF1 WF1 NN-WF1 WF1 NN-WF1 WF1 NN-WF1 WF1 NN-WF1Real motions 0.727 ±.000 0.786 ±.000 0.712 ±.000 0.773 ±.000 0.721 ±.000 0.775 ±.000 0.731 ±.000 0.789 ±.000 0.714 ±.000 0.772 ±.000 0.709 ±.000 0.761 ±.000 T2M 0.630 ±.002 0.688 ±.003 0.601 ±.003 0.647 ±.004 0.573 ±.001 0.635 ±.003 0.603 ±.002 0.661 ±.002 0.614 ±.003 0.653 ±.002 0.598 ±.003 0.644 ±.005MotionDiffuse 0.655 ±.003 0.700 ±.002 0.642 ±.003 0.685 ±.003 0.609 ±.005 0.669 ±.003 0.627 ±.002 0.696 ±.006 0.648 ±.001 0.692 ±.004 0.611 ±.002 0.669 ±.003 MDM 0.641 ±.003 0.693 ±.004 0.630 ±.002 0.679 ±.003 0.621 ±.004 0.683 ±.002 0.643 ±.003 0.685 ±.005 0.647 ±.002 0.684 ±.003 0.627 ±.001 0.662 ±.006 MotionGPT 0.631 ±.002 0.682 ±.004 0.622 ±.001 0.673 ±.002 0.613 ±.003 0.675 ±.003 0.629 ±.005 0.676 ±.004 0.637 ±.006 0.680 ±.005 0.629 ±.003 0.667 ±.007T2M-GPT 0.639 ±.004 0.699 ±.002 0.635 ±.002 0.682 ±.003 0.625 ±.002 0.686 ±.002 0.645 ±.003 0.692 ±.001 0.641 ±.002 0.691 ±.004 0.634 ±.002 0.683 ±.003 MLD 0.658 ±.002 0.711 ±.005 0.651 ±.003 0.699 ±.002 0.655 ±.003 0.706 ±.003 0.658 ±.004 0.714 ±.006 0.652 ±.003 0.696 ±.003 0.656 ±.002 0.698 ±.002 ReMoDiffuse 0.674 ±.003 0.733 ±.001 0.666 ±.002 0.721 ±.003 0.672 ±.002 0.725 ±.002 0.673 ±.001 0.731 ±.004 0.665 ±.004 0.719 ±.005 0.671 ±.003 0.720 ±.002L 3 EM (ours)0.700 ±.003 0.757 ±.002 0.691 ±.003 0.744 ±.003 0.696 ±.002 0.751 ±.004 0.708 ±.003 0.762 ±.003 0.692 ±.001 0.740 ±.002 0.692 ±.003 0.743 We believe this could be attributed to the introduction of emotion information. After introducing emotion information, each limb associated with the motions needs to exhibit a corresponding emotional expression. This, to a certain extent, limits the diversity of generated motions. And since our L 3 EM approach generates motions with better emotional performance, it consequently results in lower diversity compared to some baselines. 4) Emotion Performance. Our approach achieves WF1 score of 44.5% and NN-WF1 of 46.3%, surpassing all current baselines. More importantly, compared to the state-of-the-art baseline ReMoDiffuse, our L 3 EM approach improves WF1 and NN-WF1 by 2.8% and 3.5%, respectively. This indicates that our approach, integrating limb emotional guidance from LLM and limb relation graph, can generate emotion-enriched motions compared to other baselines. 
are emotions really injected into limbs?: 
 To validate our L 3 EM approach's ability to inject emotion into human limbs (i.e., the intra-limb emotion injection challenge), we conduct experiments on our Limb-ET2M dataset. Table 3 shows the comparative results. From Table 3 , we can see that: 1) WF1: Our L 3 EM approach surpasses all the baselines in WF1 score, indicating that our approach could enable all limb to have corresponding emotional expressions compared to the baselines. This justifies our L 3 EM approach could inject emotions into limbs. 2) NN-WF1: Our L 3 EM approach also achieve the SOTA performance in NN-WF1 score. Especially, compared to ReMoDiffuse, our L 3 EM approach improves the NN-WF1 score of head, shoulder, body, arm, leg, and foot by 2.4%, 2.3%, 2.6%, 3.1%, 2.1%, and 2.3%, respectively. This indicates that, aside from neutral cases, our approach indeed enables limb to exhibit corresponding positive or negative emotional expressions. This further justifies that our L 3 EM approach could inject emotions into limbs.To validate our approach's ability to mitigate emotional disturbances and produce coordinate and coherent emotional motions (i.e., the inter-limb emotion disturbance challenge), we conduct qualitative and quantitative experiments. 1) The qualitative results are shown in Table 2 . From this table, we can see that our L 3 EM approach achieves a comparable FID of 0.488, indicating that the motions generated by our approach are close to real motions, coordinated and coherent. This justifies that our approach addresses the challenge of emotion disturbance.2) The quantitative analysis is shown in Figure 4 , from Figure 4 (b), we can observe that compared to (a), after introducing emotion information, the generated motion sequence exhibits instances of limb incoordination and incoherence (such as the sudden high leg raise highlighted in the red box in Figure 4 (b)). This validates our argument that emotion information can disturb motion generation. Furthermore, from Figure 4 (c), we can observe that after incorporating the limb relationship graph in GRM, compared to Figure 4 (b), our approach is capable of generating coordinated and coherent emotional motions. This further justifies that our approach addresses the challenge of emotional disturbance.To further investigate the influence of key components within our L 3 EM approach, we conduct a series of ablation studies as shown in Table 2 and Table 3 . From the tables, we can see that: 1) w/o LEM exhibits inferior performance compared to L 3 EM with an increase in FID, and a decrease in R-Precsion Top 1 2 3, WF1, and NN-WF1. This further justifies the effectiveness of integrating limb emotional guidance from LLM in aiding emotional motion generation. Furthermore, the WF1 and NN-WF1 of six limb parts also show inferior performance compared to L 3 EM. This also justifies that limb emotional guidance indeed facilitates the generation of emotional motions for various limbs, ensuring that each limb of generated motions has corresponding emotional expressions. 2) w/o RL also display inferior results compared to L 3 EM in both Table 2 and Table 3 . This justifies that our designed RL-based Emotion Modeling module can assist in selecting in-context examples for LLM, thus enhancing the quality of LLM-generated limb emotional guidance and consequently improving the model's generation performance. 3) w/o GRM yields an inferior performance compared to L 3 EM with a increase in FID by 0.124. This demonstrates that our proposed limb relation graph can effectively capture spatial position and relation information among limbs, thereby aiding in generating more coordinated and coherent motion sequences.To further justify the effectiveness of our approach for the ETMG task, we provide a visualization and qualitative analysis as shown in Figure 5 . From Figure 5 , we can see that MLD [5] and RemoDiffuse [37] are both able to understand the motion description (e.g., dance and jump in the example 1) in the text and generate corresponding motions. However, they struggle to have a deep understanding of the emotion information (e.g., spreading joy and happiness all around) within the text. For example, in the first example, the motions generated by MLD exhibit smaller movement amplitudes and lack coordination in limbs, resulting in a weaker expression of happy emotion. In the second example, the motions generated by Remodiffuse, with a lower intensity and speed of arm movement outward and upward, fail to convey the anger emotion effectively. In contrast, our approach, which leverages limb emotional guidance from LLM and limb relation graph, generates emotional motions that not only consist with the text descriptions but also exhibit rich emotional expressions. 
conclusion: 
 In this paper, we propose an LLM-guided Limb-level Emotion manipulating (L 3 EM) approach to handle the ETMG task. Additionally, the LEM Block and GRM Block are designed to address the intra-limb emotion injection challenge and the inter-limb emotion disturbance challenges in the ETMG task, respectively. To comprehensively evaluate L 3 EM, we construct a coarse-grained Emotional Text-to-Motion (EmotionalT2M) dataset and a fine-grained Limblevel Emotional Text-to-Motion (Limb-ET2M) dataset. Experimental results on these datasets demonstrate the superior performance of L 3 EM over several state-of-the-art baselines. In our future work, we would like to incorporate emotion information into more generative tasks, such as text-to-image and text-to-video tasks. In these tasks, subjective emotion information is also important and has still been underachieving. In addition, we would like to integrate LLM and diffusion more closely, in order to endow our approach with LLM's interactive capability for better emotion manipulating. 
 
 Appendix: https://www.blender.org/download/releases/2-93/Code and dataset are released at https://github.com/aekx/L3EMhttps://huggingface.co/openai/clip-vit-large-patch14https://www.zhipuai.cn/ 


Decision: The paper introduces an innovative approach to text-to-motion generation by incorporating emotion as a condition for generating motions. This novel task, termed Emotion-enriched Text-to-Motion Generation, is supported by a newly constructed dataset. The authors propose two key components: a LLM-guided Intra-limb Emotion Modeling Block and a Graph-structured Inter-limb Relation Modeling Block, which enhance the embedding of emotion into the motion generation process. Reviewers praised the originality and significance of considering emotion in this domain, noting the effectiveness demonstrated through extensive experiments and demos. While the dataset construction process could be more clearly described, and further experimental interpretation is needed, the overall contribution to the field is substantial. Given the strengths and the innovative nature of this work, I recommend acceptance.

Reviews: review: paper_summary: paper_summary: This paper first proposes injecting emotion as a condition for motion generation. The authors formulate the emotion-enriched text-to-motion generation task and construct the corresponding dataset. Additionally, they propose a LLM-guided Intra-limb Emotion Modeling Block and a Graph-structured Inter-limb Relation Modeling Block to better embed emotion into the motion generation process. The experimental results demonstrate the effectiveness of the proposed methods.
strength_weakness: strength_weakness: 1. The idea of considering emotion in the text-to-motion generation process is interesting.
2. The methods effectively utilize LLM to guide limb generation and Graph-structured Inter-limb Relation Modeling.
3. The experiments and demos show the superiority of the proposed model.
limitations: limitations: 1. The dataset is newly constructed. There should be more details regarding the dataset construction process. However, the authors describe the details on pages 5 and 6 with large paragraphs, making comprehension difficult.

2. A question: Does emotion really influence the motion generation results? It is true that different emotional descriptions cause different motions or limb actions. However, the emotion factor is similar to other factors such as walk/run, forward/backward, slowly/quickly, which all influence the final action. The only difference is that the prompt of ChatGLM explicitly emphasizes the emotion factor: "Now you are an **emotional** text-to-motion generation assistant...". If "emotional" is removed from the prompt, does the generation still work? Or if emotion is replaced with another aspect, such as "Now you are a speed-sensitive text-to-motion generation assistant...", will the results also be enhanced?
paper_summary: paper_summary: This paper focuses on text-to-motion generation. Their contribution includes incorporating emotion into this task, curating a dataset, LLM-guided emotion manipulating. The task is overall innovative to my knowledge. Incorporating emotion is an interesting direction for text-to-motion generation.
strength_weakness: strength_weakness: (1) This paper introduces a novel task and has dataset contribution.
(2) The methods of LLM-guided intra-limb emotion modeling and graph-structured inter-limb relation modeling are technically reasonable with certain novelty.
(3) The authors performed extensive experiments to show the superiority of their method of incorporating emotion for text-to-motion generation.
limitations: limitations: There are no obvious limitations in this paper.
paper_summary: paper_summary: This paper proposes a new Emotion-enriched Text-to-Motion Generation (ETMG) task, aiming to generate motions with the subjective emotion information. First, few works consider subjective emotion information, especially limb-level emotion information.
Second, injecting emotions into limbs can ensure the coordination and coherence of emotional motions after injecting emotional information.

The motivation is interesting to explore.
strength_weakness: strength_weakness: 1. It is an interesting and valuable topic for Text-to-Motion Generation.
2. The writing is good and the authors present the work well.
3. This topic has significant applications in the real world.
4. The authors conducted an extensive experiments.
limitations: limitations: 1. The interpretation of experimental analysis needs to be further improved.