MotionRL: Align Text-to-Motion Generation to Human Preferences with Multi-Reward Reinforcement Learning

## Abstract

Abstract We introduce MotionRL , the first approach to utilize Multi-Reward Reinforcement Learning (RL) for optimizing text-to-motion generation tasks and aligning them with human preferences. Previous works focused on improving numerical performance metrics on the given datasets, often neglecting the variability and subjectivity of human feedback. In contrast, our novel approach uses reinforcement learning to fine-tune the motion generator based on human preferences prior knowledge of the human perception model, allowing it to generate motions that better align human preferences. In addition, MotionRL introduces a novel multi-objective optimization strategy to approximate Pareto optimality between text adherence, motion quality, and human preferences. Extensive experiments and user studies demonstrate that MotionRL not only allows control over the generated results across different objectives but also significantly enhances performance across these metrics compared to other algorithms.

## 1 Introduction

High-quality human motion is in high demand across various fields, such as animation, robotics, and gaming. Due to the simplicity and user-friendliness of text input, text-driven human motion generation (Zhu et al., ; Aliakbarian et al., ; Bouazizi et al., ; Barsoum et al., ; Mao et al., ; Habibie et al., ; Yan et al., ; Aristidou et al., ; Lee et al., )has become a prominent research topic in recent years.
However, generating motion from text is a highly challenging multimodal task, and current methods face several issues.
First, there is often a semantic mismatch between text and motion, where the algorithm must understand diverse textual descriptions and accurately generate corresponding motions.
Second, the quality of generated motion needs to be as close to the ground truth as possible.
Additionally, as a branch of generative tasks, human perception of motion is also highly important (Voas et al., ).

Figure: Figure 1: Examples generated by MoMask and Ours. Our method significantly outperforms the previous state-of-the-art MoMask in text adherence, motion quality and human preferences.
Refer to caption: x1.png

To address this task, traditional generative frameworks, such as Variational Autoencoders (VAEs) (Van Den Oord et al., ), have been used to align textual descriptions with motion sequences.
Recently, more advanced methods like generative masked modeling (Guo et al., ) have aimed to model conditional masked motion to achieve better alignment between motion and text.
Another approach involves Diffusion Models Dabral et al. (); Zhang et al. (); Tevet et al. (); Chen et al. (), which predict motion sequences through a gradual denoising process.
While these methods have shown promising results on traditional metrics, they suffer non-trivial limitations.
On one hand, they require the motion length as input to guide generation, which may lead to motion quality degradation, since the length is closely tied to the content of the motion sequence (Pinyoanuntapong et al., ).
On the other hand, predicting entire motion sequences works well for short texts, but results in inaccuracies for more complex textual inputs, particularly in the fine-grained details of the motion.

On the other hand, generative transformer-based methods (Jiang et al., ; Zhang et al., ; Mao et al., ) alleviate some of the above issues by following the GPT-type training process, where the motion sequence is generated through autoregressive next-token prediction.

Regardless of the generation method, almost all mainstream research has largely ignored the role of human perception in evaluating generated motions.
Generally, generating realistic human motion, including smooth and natural movement, is more important than fitting existing error-based metrics, such as FID and R-Precision (Heusel et al., ).
Current methods tend to overfit to small datasets and focus on optimizing for FID scores.
In fact, experiments in Pinyoanuntapong et al. () have shown that a better FID score does not necessarily correlate with human perception, as issues like visual artifacts and foot sliding are not well captured by these metrics.
Since such artifacts are difficult to measure using existing metrics (Zhang et al., ), human perception of generated motions becomes crucial.

Despite this, few studies have taken human perception into account.
While some motion perception models (Voas et al., ; Wang et al., ) do incorporate human perceptual priors, these priors are often more subtle and complex than traditional metrics.
Directly incorporating these priors (Wang et al., ) into the training process poses two challenges.
Firstly, it is difficult for the model to capture the fine nuances of human perception.
Secondly, it may significantly reduce the performance on other metrics, such as semantic alignment between motion and text.
Therefore, optimizing for motion quality, text adherence, and human preferences jointly has been quite difficult in prior approaches.

In this work, we propose MotionRL, a reinforcement learning (RL)-based framework for multi-objective optimization in human motion generation.
Unlike previous methods that often ignore human perception, our approach uses an RL framework (Schulman et al., ) to capture subtle and complex human perceptual priors from perception models.
To avoid degradation of other metrics, we also incorporate motion quality and text adherence as part of the reward. However, as demonstrated in other fields (Lee et al., ), simple weighted combinations of rewards often lead to unstable training, especially when optimizing for complex human perception.
To this end, we introduce a multi-reward optimization strategy to approximate Pareto optimality.
Intuitively, each generated sample in a batch embodies a distinct trade-off among the three rewards, with some samples exhibiting better trade-offs than others.
Instead of updating gradients using all batch samples, our approach selects non-dominated points that have better trade-offs.
This allows the model to automatically learn the optimal balance among different rewards.
Furthermore, our method learns reward-specific preference prompts, which can be used individually or in combination to control the trade-offs between rewards during inference, solving the problem of manually adjusting reward weights and ensuring more stable training.

In summary, our contributions are three-fold as follows:

- •
We propose MotionRL, a novel reinforcement learning-based multi-reward optimization algorithm for fine-tuning human motion generation. It effectively optimizes multiple rewards, including text adherence, motion quality, and human preferences, and allows the use of reward-specific tokens during inference to control the output.
- •
We make the first attempt to introduce reinforcement learning for incorporating and improving human perception in the text-to-motion domain. Unlike previous methods that either ignore human perception or directly incorporate perceptual priors into training, our RL-based approach is more effective at capturing complex human perceptions and produces results that are more convincing than those evaluated by error-based metrics.
- •
We evaluate our approach with experimental results, which demonstrate the superiority of our approach across traditional metrics like FID and R-Precision, as well as human perceptual model scores and user studies.

## 2 Related Work

### 2.1 Human Motion Generation

Many works have attempted to generate motion from text, primarily because language input is very convenient for users. Recent mainstream research has employed stochastic models for motion generation. For instance, the diffusion method (Zhang et al., ; Chen et al., ; Tevet et al., ) describes motion generation as a denoising process conditioned on text. Some BERT-type masked generative models (Guo et al., ) have also successfully tackled this task. However, these approaches face a significant limitation: they require the motion length as input to guide generation, which can lead to significant motion quality degradation (Pinyoanuntapong et al., ). In contrast, GPT-type generative models (Zhang et al., ; Guo et al., ) solve this issue by first learning a VQVAE (Van Den Oord et al., ) to map continuous motion into discrete motion tokens, and models like Zhang et al. () and Mao et al. () employ a transformer decoder to perform next token prediction.

### 2.2 Human Feedbacks of Generated Motion

Human evaluation of generated motion should be highly correlated with motion quality and is arguably more important than traditional metrics such as FID or R-Precision(Heusel et al., ). However, in the text-to-motion domain, few studies incorporate human perceptual priors. Recent works, such as Voas et al. (), have established datasets of human motion ratings, and others, like Wang et al. (), use contrastive methods to improve the robustness of human evaluations. However, these approaches directly integrate ratings into the training process, which can negatively impact other metrics and fail to capture subtle human perceptual differences.

### 2.3 Reinforcement Learning with Human Feedbacks

Reinforcement Learning from Human Feedback (RLHF) (Nguyen et al., ) has achieved tremendous success in fine-tuning generative models, whether for text generation(OpenAI, ) or image generation(Fan et al., ). RL effectively aligns model outputs with human evaluations and offers significant advantages over explicit alignment methods for complex optimization objectives. OpenAI’s technical report (OpenAI, ) noted that fine-tuning pretrained GPT models with RL helps models better understand human perceptions. However, in the text-to-motion domain, few studies use RL to align with human perceptions. For example, Mao et al. () uses RL to guide generation, but without incorporating any human perceptual priors, focusing only on enhancing text-to-motion alignment.

## 3 Preliminary

### 3.1 Autoregressive Model for Motion Generation

A common method (Zhang et al., ) to design a motion generator is using vector quantized variational autoencoder (VQ-VAE) (Van Den Oord et al., ) and generative pre-trained transformer (GPT). In T2M-GPT, the trained VQ-VAE maps motion sequences $m=[x_{1},x_{2},\dots,x_{T}]$ to a sequence of indices $S=[s_{1},s_{2},\dots,s_{T/l},End]$ which are indices from the learned codebook $C$. Note that a special $End$ token is added to indicate the end of a motion. The VQ-VAE uses a standard CNN-based architecture and is trained following the details in T2M-GPT (Zhang et al., ).

With a trained VQ-VAE, the motion generation task can be described as a next-token prediction task. In this stage a transformer is designed to autoregressively predict the next index $s_{i}$ based on previous indices set $S_{<i}$ and text condition $c$.
The optimization goal in this stage is to maximize the distribution $p(S|c)=\prod_{i=1}^{|S|}p(s_{i}|c,S_{<i})$.
We train the GPT with cross-entropy loss $\mathcal{L}_{CE}$.

Some relevant works introduce a sequence-level semantic supervision to help motion-text alignment. Specificlly, pre-trained motion encoders and text encoders like TMR encoders (Lu et al., ) can measure the similarity between the input text embedding and output motion embedding. The text embedding from the encoder aligns better with human motions than that from CLIP (Radford et al., ) or other LLMs. It also provides a way to directly supervise the text-motion alignment. So we calculate an additional loss $\mathcal{L}_{align}$ for supervising the alignment.

### 3.2 Reward-specific Token Design and Sampling

Lin et al. () introduced the use of preference information in multi-objective optimization. In MotionRL, reward-specific identifiers are employed, which allow the model to differentiate between various types of rewards during multi-objective optimization. The details about sampling and token design and are in the Appdix B.

## 4 Methodology

Most existing methods for text-to-motion generation are based on supervised training on pre-existing datasets, followed by performance evaluation on test sets. However, human perception of the quality of motion generation is often more meaningful than the performance metrics obtained from test sets. Human feedback signals, however, are more complex compared to standard evaluation metrics like FID (Heusel et al., ), and incorporating them directly into training poses challenges such as overfitting and the inability to dynamically optimize model outputs (Fan et al., ). To address these issues, we propose the MotionRL method, which utilizes reinforcement learning (RL) to fine-tune the model using human perception data. Furthermore, since incorporating perception introduces multiple optimization objectives, rather than using traditional methods of reward averaging, we adopt a Pareto-based multi-reward optimization approach as shown in Figure 2.

Figure: Figure 2: The overall pipeline of MotionRL. Given a text input, the Transformer serves as a motion generator, first producing multiple motions as a batch. Various rewards are then computed for these motions. Within this batch of motions, the Pareto set is identified. Finally, using the rewards from the Pareto set, along with the outputs of the critic model and the prediction logits, the motion generator is optimized using the PPO algorithm (note that the critic model is omitted in the diagram).
Refer to caption: x2.png

In Section 3.1, we first explain how MotionRL reformulates the text-to-motion task as an autoregressive process, which serves as the foundation for applying RL fine-tuning. We then proceed to the design of multi-objective rewards for RL, detailed in Section 4.1. For a given text input, multiple samples can be generated, each associated with several rewards. To address this multi-objective optimization challenge, MotionRL introduces a batch-wise Pareto-optimal selection strategy, discussed in Section 4.2. Finally, Section 4.3 provides a detailed explanation of our Pareto-based policy gradient optimization method.

### 4.1 Multi-Reward Design

Regarding the design of the reward models, we define three specific objectives to fine-tune the model’s output: text adherence, motion quality, and human preferences. The first two rewards are derived from encoders trained on paired motion and text data, while the latter is obtained from a model trained on real human preference data.

For text adherence and motion quality, we use the paired text encoder and motion encoder from Guo et al. () and Lu et al. (). The training objective of the encoder in Guo et al. () is:

$$ $\mathcal{L}_{CL}=y\left(\|\mathbf{f}_{t}-\mathbf{f}_{m}\|^{2}\right)+(1-y) \left(\max(0,m-\|\mathbf{f}_{t}-\mathbf{f}_{m}\|)\right)^{2},$ (1) $$

where $\mathbf{f}_{t}$ and $\mathbf{f}_{m}$ denotes the embeddings extracted by text and motion encoders, respectively. $y$ is a binary label indicating the matched text-motion pairs and $m$ is the manually set margin.

The training objective of the encoder in Lu et al. () is:

$$ $\mathcal{L}_{InfoNCE}=-\frac{1}{B}\sum_{i=1}^{B}\left[\log\frac{\exp\left( \mathbf{f}_{t}^{i}\cdot\mathbf{f}_{m}^{i}/\tau\right)}{\sum_{j=1}^{B}\exp\left (\mathbf{f}_{t}^{i}\cdot\mathbf{f}_{m}^{j}/\tau\right)}+\log\frac{\exp\left( \mathbf{f}_{m}^{i}\cdot\mathbf{f}_{t}^{i}/\tau\right)}{\sum_{j=1}^{B}\exp\left (\mathbf{f}_{m}^{i}\cdot\mathbf{f}_{t}^{j}/\tau\right)}\right],$ (2) $$

where $\tau$ is learnable temperature parameter and $B$ is the batch size.

Now we have pretrained motion encoders and text encoders, we can define the text adherence reward as:

$$ $r_{t}=-\sum^{i}\lambda_{i}\|\mathbf{f}_{t,i}-\mathbf{f}_{m_{\mathrm{pred}},i} \|^{2},$ (3) $$

where $\mathbf{f}_{m_{\mathrm{pred}}}$ is the generated motion contioned on $t$ , $i$ represents different encoders and $\lambda$ represents their respective weights, used to constrain the values to a similar magnitude. Higher rewards indicating a better match between the generated motion and the input text.

Similarly, the motion quality reward is defined as:

$$ $r_{m}=-\sum^{i}\lambda_{i}\|\mathbf{f}_{m_{\mathrm{gt}},i}-\mathbf{f}_{m_{ \mathrm{pred}},i}\|^{2},$ (4) $$

where $m_{\mathrm{gt}}$ is the ground truth motion sequence. Higher rewards signify that the generated motion is closer to the ground truth.

We also need a model to align the human preferences. Giving an input motion sequence $m$, an implicit perception model $\mathcal{P}$ is assumed, where a higher rate indicates that the motion has better quality. To align the generated motions with human perception, we need a computational perception model $\mathcal{C}$ that best aligns $\mathcal{P}$. We use the model from Wang et al. () as our human perception model. The pairwise comparison annotations from the collected dataset $\mathcal{D}$ can be used to calculate the training loss:

$$ $\mathcal{L}_{perception}=-\mathbb{E}_{(m^{(h)},m^{(l)})\sim\mathcal{D}}[ \operatorname{log}\sigma(\mathcal{C}(m^{(h)}),\mathcal{C}(m^{(l)}))],$ (5) $$

where $m^{(h)}$ is the better motion and $m^{(l)}$ is the worse one. After that, the model $\mathcal{C}$ is able to map the high-dimension motion to a reward $r_{p}$ as the motion rating. Therefore the human preference reward is defined as:

$$ $r_{p}=\mathcal{C}(g(m_{\mathrm{pred}})),$ (6) $$

where $g$ is a function that converts human motion from a 3D coordinate parameterized form to an SMPL (Loper et al., ) parameterized form. Here we train a simple neural network to achieve this goal rather than traditional methods. Please refer the Appendix A.

We normalized three different types of rewards to constrain them within the same order of magnitude, ensuring that the numerical differences between the various rewards do not vary significantly. For specific normalization methods, please refer to Appendix C.

### 4.2 Batch-wise Pareto-optimal Selection

Lin et al. () pointed out that using batchwise Pareto-set learning and selecting good samples in a batch can approximate Pareto-optimality across multiple objectives. In the preliminary stage, for $k$ types of rewards, we generated $k$ texts with special tokens, and each text sampled $N$ motions respectively. Therefore, MotionRL will find the corresponding Pareto-optimal set in this $N$ batch denoted as $\mathcal{M}$. For the different sampled motions $m_{i}$, the dominance relationship is defined as follows:

Let $m_{a},m_{b}\in\mathcal{M}$, $m_{a}$ is considered to dominate $m_{b}$, denoted as $m_{a}\succ m_{b}$, if and only if $r_{k}(m_{a})\geq r_{k}(m_{b}),\forall k\in\{1,\dots,K\}$ and $\exists j\in\{1,\dots,m\}$ such that $r_{j}(m_{a})>r_{j}(m_{b})$.

In order to enable the model to approximate Pareto optimality during the training process, we designed the algorithm as Algorithm 1.

Figure: Algorithm 1 MotionRL: Pareto-optimal Multi-Reward RL for Motion Generation

### 4.3 Pareto-based Policy Gradient Optimization

We employ reinforcement learning to optimize the alignment between motion sequences and textual descriptions, human perception preferences, and motion quality. A Pareto-based multi-reward objective guides this process, balancing the different rewards.

The actor model $\pi_{\theta}$ generates motion sequences by selecting motion tokens at each time step based on the input text, while the critic model $V_{\phi}(s_{t})$ estimates the value of the current state. The objective function is:

$$ $\mathcal{J}_{r}(\pi_{\theta})=\sum_{k=1}^{K}\frac{1}{n(\mathcal{P})}\sum_{i=1, m_{i}\in\mathcal{P}}^{N}\sum_{t=1}^{T}\left[r(t_{k},m_{k})-\beta\log\frac{\pi_ {\theta}(m_{k}\mid t_{k})}{\pi_{\text{ref}}(m_{k}\mid t_{k})}\right].$ (7) $$

In the above equation, $r(t_{k},m_{k})$ represents the multi-objective reward function evaluating the alignment between the textual description $t_{k}$ and the generated motion sequence $m_{k}$, taking into account factors like semantic consistency, human preference, and motion quality. $\pi_{\theta}(m_{k}\mid t_{k})$ is the actor model that predicts the motion $m_{k}$ based on the input text $t_{k}$, and $\pi_{\text{ref}}(m_{k}\mid t_{k})$ is the reference model serving as a constraint to regulate how much the updated policy can deviate from the pre-trained model. The term $\beta$ controls the strength of this regularization.

We employ Proximal Policy Optimization (PPO) (Schulman et al., ) for training, where the advantage function is computed as:

$$ $A_{t}=G_{t}-V_{\phi}(s_{t}).$ (8) $$

Here, $A_{t}$ measures the relative improvement of the selected action at time $t$ compared to the expected value. $G_{t}$ denotes the return, or the total reward accumulated from time step $t$, and $V_{\phi}(s_{t})$ is the value function estimated by the critic, representing the expected return at state $s_{t}$.

The actor’s loss function is given by:

$$ $\mathcal{L}_{\text{actor}}(\theta)=\mathbb{E}_{t}\left[\min\left(r_{t}(\theta) A_{t},\text{clip}(r_{t}(\theta),1-\epsilon,1+\epsilon)A_{t}\right)\right],$ (9) $$

where the clipping function ensures that the policy update is constrained to prevent large deviations from the previous policy. The critic model is updated by minimizing the squared error between the predicted value and the actual return:

$$ $\mathcal{L}_{\text{critic}}(\phi)=\mathbb{E}_{t}\left[(V_{\phi}(s_{t})-G_{t})^ {2}\right].$ (10) $$

This framework allows both models to effectively learn from multiple reward signals, balancing semantic alignment, human preferences, and motion quality in the generated sequences.

## 5 Experiments

### 5.1 Experiment Setting

We select InstructMotion as our baseline model. The text-to-motion transformer is composed of 18 layers, each with a hidden size of 1,024 and 16 attention heads. For the PPO algorithm, we use a mini-batch size of 32 during training. We run the PPO algorithm for 2 epochs, using the AdamW (Loshchilov & Hutter, ) optimizer with $\beta_{1}=0.9$ and $\beta_{2}=0.99$. The pre-trained text-to-motion generator is fine-tuned over 40k iterations, with a learning rate of 5e-6. All experiments are performed on 4 NVIDIA RTX 3090 GPUs. The specific method for reward normalization is detailed in Appendix C, while the process and results of the user study are elaborated in Appendix D.

### 5.2 Quantitative Evaluation

Tables 1 provides an evaluation of our MotionRL framework on the widely used datasets: HumanML3D. The results indicate that MotionRL outperforms the baseline models, T2M-GPT and InstructMotion, by a significant margin in key quantitative metrics such as R-Precision and FID. This demonstrates that the motion sequences generated by our model exhibit stronger alignment with the corresponding textual descriptions and better quality of motion.

**Table 1: Quantitative comparison on HumanML3D test set. The evaluation metrics are computed following Guo et al. (). ^§ indicates reliance on ground-truth sequence length for generation. Underline indicates the second best.**
| Methods | R-Precision $\uparrow$ | FID $\downarrow$ | MM-Dist $\downarrow$ | Diversity $\uparrow$ |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Top-1 | Top-2 | Top-3 |  |  |  |  |
| TEMOS^§ (Petrovich et al., ) | 0.424 | 0.612 | 0.722 | 3.734 | 3.703 | 8.973 |
| MLD^§ (Chen et al., ) | 0.481 | 0.673 | 0.772 | 0.473 | 3.196 | 9.724 |
| MDM^§ (Tevet et al., ) | - | - | 0.611 | 0.544 | 5.566 | 9.559 |
| MotionDiffuse^§ (Zhang et al., ) | 0.491 | 0.681 | 0.782 | 0.630 | 3.113 | 9.410 |
| GraphMotion^§ (Jin et al., ) | 0.504 | 0.699 | 0.785 | 0.116 | 3.070 | 9.692 |
| ReMoDiffuse^§ (Zhang et al., ) | 0.510 | 0.698 | 0.795 | 0.103 | 2.974 | 9.018 |
| MoMask^§ (Guo et al., ) | 0.521 | 0.713 | 0.807 | 0.045 | 2.958 | - |
| Language2Pose (Ahuja & Morency, ) | 0.246 | 0.387 | 0.486 | 11.02 | 5.296 |  |
| Ghosh et al. () | 0.301 | 0.425 | 0.552 | 6.532 | 5.012 |  |
| TM2T (Guo et al., ) | 0.424 | 0.618 | 0.729 | 1.501 | 3.467 | 8.589 |
| Guo et al. () | 0.455 | 0.636 | 0.736 | 1.087 | 3.347 | 9.175 |
| T2M-GPT (Zhang et al., ) | 0.491 | 0.680 | 0.775 | 0.116 | 3.118 | 9.761 |
| Fg-T2M (Wang et al., ) | 0.492 | 0.683 | 0.783 | 0.243 | 3.109 | 9.278 |
| MotionGPT (Jiang et al., ) | 0.492 | 0.681 | 0.778 | 0.232 | 3.096 | 9.528 |
| InstructMotion (Mao et al., ) | 0.505 | 0.694 | 0.790 | 0.099 | 3.028 | 9.741 |
| Ours | 0.531 | 0.721 | 0.811 | 0.066 | 2.898 | 9.653 |

It is important to note that ^§ indicates reliance on the ground-truth sequence length for generation. All methods that depend on the ground-truth motion length tend to perform better on the FID metric because they ensure that the motion lengths are consistent with the test set prior to inference. However, Pinyoanuntapong et al. () points out that superior FID scores for these methods do not necessarily imply higher motion quality. To demonstrate the superiority of MotionRL, we utilize the scores from the motion perception model provided in Wang et al. () alongside real human user study ratings.

Figure: (a) Perception model scores of different methods
Refer to caption: extracted/5912029/figures/perception.png

Figure 3(a) illustrates the output scores from the motion perception model in Wang et al. (). It is evident that our model achieves higher perceptual scores compared to other models, indicating that it effectively captures human preference information embedded in the perceptual model, leading to motions that align more closely with human perception. Figure 3(b) presents the scores given by real human evaluators for the model-generated motions. We compared the success rates of MotionRL against other models, further demonstrating the overall superiority of our model in terms of motion quality.

### 5.3 Qualitative Evaluation

To validate the superiority of MotionRL in generating high-quality actions, we compared MotionRL with other well-performing models, including T2M-GPT, InstructMotion, and Momask, based on several prompts from the test set. As shown in Figure 4, our method generates actions that align with the text, whereas the other methods fail to produce accurate actions. For example, the other methods did not generate the motion of pushing the arms forward and could not comprehend the motion of walking diagonally two steps to the right. In contrast, our method produces high-quality motions that correspond to the text. More details can be found in the supplementary materials.

Figure: Figure 4: Qualitative comparisons with top-performing methods. Our MotionRL exhibits better motion generation quality.
Refer to caption: x3.png

### 5.4 Ablation

Reward Design: Our framework employs a multi-objective optimization approach with three distinct rewards. To verify that our method effectively captures human perceptual preferences, we conducted numerical experiments on the HumanML3D test set. By using different reward combinations, we assessed our method’s performance in terms of both traditional metrics and human perceptual model scores. We observed that the perception reward we designed effectively enhanced the output of the human perception model. Additionally, the rewards we developed for motion quality and text adherence significantly improved performance on both FID and Top-1 Precision metrics.

**Table 2: Ablation of Reward Design $R_{p}$ is the human preferences reward, $R_{m}$ is the motion quality reward, and $R_{t}$ is the text adherence reward. The results indicate that each of the designed rewards, along with our multi-objective optimization approach, successfully improves the generated motions in different aspects. Notably, when all three rewards are combined, the model achieves the best overall performance.**
| $R_{p}$ | $R_{m}$ | $R_{t}$ | Top-1 $\uparrow$ | FID $\downarrow$ | Perception $\uparrow$ |
| --- | --- | --- | --- | --- | --- |
| ✓ |  |  | 0.519 | 0.090 | 0.495 |
|  | ✓ | ✓ | 0.528 | 0.064 | 0.465 |
| ✓ | ✓ | ✓ | 0.531 | 0.064 | 0.494 |

Pareto Optimality: Instead of transforming the multi-objective optimization into a single reward through weighted summation, MotionRL approximates Pareto optimality within each batch of samples. To verify the superiority of our method, we compared the effects of using batch-wise Pareto selection and different reward-specific tokens on model rewards. As shown in Figure 3, the use of Pareto optimization effectively improved the overall reward values of the model. Additionally, employing different reward-specific tokens allowed effective control over the model’s output. It is important to note that comparisons of reward values across different optimization objectives are meaningless because our normalization process during training constrains the rewards to the same scale. However, the physical meaning of the values for different objectives varies, and only the relative magnitudes of rewards within the same optimization objective are meaningful.

Figure: Figure 5: Impact of Pareto Selection and Reward-Specific Tokens. It illustrates the effectiveness of our proposed Pareto selection in enhancing the model’s overall reward value. It also demonstrates how using different reward-specific tokens allows for trade-offs between various optimization goals, improving the balance between motion quality, text adherence, and human preferences in the generated outputs.
Refer to caption: extracted/5912029/figures/ablation_plot.png

## 6 Conclusion and Future Work

We propose MotionRL, an algorithmic framework for generating human motions based on GPT and reinforcement learning. Addressing the complexities and challenges of capturing human perception, we draw inspiration from other fields by integrating existing human motion perception models with reinforcement learning. This innovative approach uniquely tackles the alignment of human perception with generated motions, an area where no other methods currently exist. To enhance motion generation quality and textual alignment, we introduce two additional rewards in our reinforcement learning framework. Rather than employing traditional reward-weighted averaging, we propose a batch-wise Pareto sample selection optimization method. Evaluations of both quantity and quality demonstrate significant success in human perception, motion quality, and text alignment. The discussion about limitation and future work is shown in Appendix F.

## Appendix A Convert Joints Data to SMPL

To efficiently convert joint-based motions to SMPL format without the computational overhead of iterative methods, we developed a lightweight neural network. The traditional approach required over 20 iterations per sequence, making it impractical for real-time training in our framework. Our goal was to significantly reduce conversion time while maintaining the quality of the generated SMPL motions.

The network consists of a combination of 1D convolutional layers (Conv1D) and Long Short-Term Memory (LSTM) units:

Conv1D Layers: These layers capture spatial dependencies between joints within each frame.

LSTM Layers: LSTMs are used to model the temporal dynamics of the motion sequences, allowing the network to understand how motions evolve over time.

Fully Connected Output Layer: Finally, a fully connected layer converts the processed features into SMPL format, where each frame consists of 25 joints with 6 parameters (rotation data).
This architecture allows the network to efficiently handle the transformation from joint-based data to SMPL format, leveraging the strong temporal and spatial relationships in the data.

Figure: (a) Joint-based motion
Refer to caption: extracted/5912029/figures/joints.png

After training on the HumanML3D dataset, using SMPL motions generated through iterative methods as ground truth, the model demonstrated both speed and accuracy in converting joint-based motions into SMPL format. The neural network performed the conversion much faster than traditional methods, making it feasible for real-time use during model training. This enabled us to integrate SMPL format conversion directly into the training loop, optimizing both the efficiency and the overall quality of the generated motions.

## Appendix B Details of Sampling and Token Design

This section provides a detailed explanation of our sampling strategy and the design of reward-specific tokens in the training process.

In our implementation of the Proximal Policy Optimization (PPO) algorithm, the actor model is a fine-tuned Transformer that generates the probability distribution for predicting the next motion index. The selection of the next index from this probability distribution follows a sampling strategy.

Sampling Strategy
During training, for a given text prompt $t$, multiple samples $n$ are generated. Common approaches include probability-based sampling or beam search (greedy sampling). However, in practice, we found that greedy sampling tends to cause the critic model to overfit, meaning the actor model lacks sufficient exploration. To address this, we adopted probability-based sampling with a temperature coefficient set to 1.5. This encourages broader exploration by the actor model, leading to more diverse and varied motion outputs during training.

Reward-Specific Token Design
The text inputs to the model are not simply raw text but include a special token at the end of each sentence. For example, the prompt ”a person is running forward” is modified to ”a person is running forward <mm>”, where <mm> signifies that the motion quality reward is being calculated for this sample. Similarly, the token <mt> indicates a text adherence reward, and <perception> represents the human preference reward.

This token-based approach allows the model to differentiate between different reward types, guiding the model towards optimizing multiple objectives. However a key challenge in the text-to-motion domain is that the dataset size is significantly smaller compared to fields like image or text generation. This makes the model more sensitive to small changes in input text. Directly introducing these reward-specific tokens into the input text led to a noticeable drop in performance during initial training, as the pre-trained model had no prior knowledge of these tokens and thus struggled to interpret them.

To address this issue, we employed weighted guidance by combining both the original text and the modified reward-specific text during training.

$$ $\hat{\mathbf{f}}_{t_{k}}=(1-\alpha)\mathbf{f}_{t}+\alpha\mathbf{f}_{t_{k}},$ (11) $$

where $\alpha$ is a weight parameter, $t$ is original texts , $t_{k}$ is the texts with reward-specific tokens and $\mathbf{f}$ is the text encoder. By adjusting the weight of the features corresponding to these special tokens, we ensure that the model’s output is not overly influenced by the tokens themselves. At the same time, the model retains the ability to distinguish between different reward types, something that traditional single-objective optimization approaches are unable to achieve.

This approach effectively balances exploration and reward differentiation, allowing the model to generate high-quality motions while accounting for multiple optimization goals.

## Appendix C Reward Normalization

In this section, we explain how we normalize different reward values for stable training.

Since we already employ reward-specific tokens, different rewards produce slightly different text features even when the input text remains the same. This prevents the model from confusing different reward inputs, even without directly weighting or summing the rewards. However, the token’s ability to control the output might be limited. To ensure more stable training, we normalize all rewards to the same scale.

We use an extended min-max normalization method:

$$ $r_{k,\text{normalized}}=\begin{cases}\frac{r_{k}-\text{min_val}_{k}}{\text{ max_val}_{k}-\text{min_val}_{k}},&\text{if }\text{min_val}_{k}\leq r_{k} \leq\text{max_val}_{k}\\ \frac{r_{k}-\text{min_val}_{k}}{\text{max_val}_{k}-\text{min_val}_{k}},& \text{if }r_{k}<\text{min_val}_{k}\\ \frac{r_{k}-\text{max_val}_{k}}{\text{max_val}_{k}-\text{min_val}_{k}}+1,& \text{if }r_{k}>\text{max_val}_{k}\end{cases}$ (12) $$

In this equation,
$\text{min_val}_{k}$ and $\text{max_val}_{k}$
represent the estimated minimum and maximum values for each reward type $k$. These estimates do not need to be highly precise because, even if the reward values exceed the expected range slightly—whether a bit over 1 or below 0—it does not affect the stability of training.

It is important to note that the normalized rewards across different reward types are not directly comparable. Our goal is simply to bring all rewards to the same scale, without needing to precisely control the normalization range or fine-tune weight parameters as required by traditional weighted-sum methods. This flexibility is enabled by our use of reward-specific tokens and the Pareto-based policy gradient.

## Appendix D Details of User Study

Data Preparation:
We randomly selected 30 prompts from the HumanML3D test set. Each prompt describes a specific human motion.

Model Inference:
Using various models, we generated corresponding human motions based on the same prompts, and the generated motions were rendered as GIF images.

Evaluation Method:
To compare the performance of our model with other models, we presented the GIFs created by different models to volunteers for evaluation. For each set of GIFs, we recruited 4-6 volunteers to assess the overall quality of the motions. The volunteers were asked to choose which of the two motions better matched the text description, exhibited higher quality, and appeared smoother and more natural. Figure 7 shows an example of the interface we provided to the volunteers, where the order of the GIFs was randomized.

Figure: Figure 7: An example page displayed to volunteers. These GIFs are randomly shuffled.
Refer to caption: x4.png

Volunteers could select one of three options for each comparison: which motion was better (win or loss), or indicate if it was too difficult to decide (draw). We then analyzed these responses to determine the performance of each model.

## Appendix E More Qualitative Visualizations

In our supplementary materials, we provide additional motion examples that showcase the superiority of our method in terms of text adherence, motion quality, and human preference. These examples demonstrate how our approach outperforms others, not only based on quantitative metrics but also in real human perception.

## Appendix F Limitation and Future Work

While our model effectively captures complex human perceptual information using existing perception model output scores and reinforcement learning, we have not introduced additional human annotation costs prior to training, relying solely on pre-trained motion perception models. This fine-tuning approach is heavily dependent on the quality of the perception models themselves. To further enhance the quality of human perception, more human annotations would be needed as training data for reinforcement learning.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. “Aligning Motion Generation with Human Perceptions” by Wang et al., 2024, utilizes their proposed motion perception model as an effective supervision signal for training a motion generator. This paper similarly uses the motion perception model from Wang et al.  as a supervision signal to aid motion generation. What are the distinction and novelty of this paper's method compared to the approach of Wang et al.?
2. The use of **skeleton** rendering in the user study section for evaluation could be more reasonable, as there is a gap between **skeleton** and **SMPL** rendering. Some unnatural details are more noticeable in SMPL, using SMPL rendering for the user study might be more convincible.
soundness: 2
strengths: This paper introduces human preference into the text-to-motion generation task by applying a motion perception model from the motion generation field. Additionally, two other rewards (text adherence and motion quality) are used to prevent potential degradation caused by the human preference reward. Experimental results demonstrate the superiority of the proposed approach.
summary: The paper proposes incorporating human preference priors into the text-to-motion task to enhance the quality of generated motions. Through the use of reinforcement learning and a motion perception model, the paper construct a human preference reward, enabling the generation model to learn human perception. To prevent degradation in other performance metrics resulted by human preference reward, the paper introduces a motion quality reward and a text adherence reward, forming a proposed multi-reward system. To mitigate potential training instability caused by multiple rewards, Pareto optimality is employed to balance the different rewards. The experimental results demonstrates superior performance in FID, R-Precision, human perceptual model scores, and user studies.
weaknesses: The most concerning limitation of this paper lies in its novelty. Introducing human perception into the field of motion generation is not novel, as also mentioned by this paper in line#86-87 (Voas et al., 2023; Wang et al., 2024). Wang et al. utilizes their proposed motion perception model as an effective supervision signal to finetune the motion generator. This paper similarly uses the motion perception model from Wang et al. as a supervision signal to train model for motion generation. Given these, the technical novelty is limited.

There are also insufficiencies in the experimental section: **1)** The main insufficiency is that the ablation study does not adequately demonstrate the effectiveness of the proposed multiple rewards. Table 2 lacks a baseline ablation result with no rewards applied, making it difficult to confirm the method’s effectiveness. **2)** In text-to-motion generation tasks, methods are typically validated on both the HumanML3D and KIT-ML datasets, but this paper only provides results for HumanML3D. **3)** Additionally, on the HumanML3D dataset, the MModality metric is usually reported; however, this paper neither provides MModality results nor explains why it was omitted. **4)** For the Diversity metric, Diversity closer to real test data is preferable, yet this paper labels higher values as better.

Minor comments:
1. The paper does not explain the metrics reported in the tables, such as Diversity.
2. The “Conclusion and Future Work” section does not include future work but instead links to the appendix.
3. Descriptions for figures should not be placed in the appendix; for example, <mm> and <mt> in Fig. 5 lack explanation in the main text.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: How does MotionRL handle the trade-offs between text adherence, motion quality, and human preferences during the training process?

What are the limitations of relying on pre-trained human perception models for aligning generated motions with human preferences, and how can these limitations be addressed in future work?
soundness: 3
strengths: Human-Centric Optimization: MotionRL uniquely incorporates human preferences into the optimization process, ensuring that the generated motions align better with human perception. This focus on human feedback addresses the limitations of traditional metrics that may not fully capture the nuances of human motion quality.
Multi-Objective Optimization: The use of a multi-reward reinforcement learning framework allows MotionRL to balance multiple objectives simultaneously, such as text adherence, motion quality, and human preferences. This approach ensures that the generated motions are not only accurate and high-quality but also meet the subjective preferences of users.
Pareto Optimality: MotionRL introduces a novel multi-objective optimization strategy to approximate Pareto optimality. By selecting non-dominated points within each batch, the model learns to balance different rewards effectively, leading to more stable and optimal training outcomes. This method enhances the overall performance across various metrics compared to other algorithms.
summary: The paper introduces a novel approach that leverages Multi-Reward Reinforcement Learning to optimize text-to-motion generation tasks. Unlike previous methods that primarily focus on numerical performance metrics, MotionRL incorporates human preferences to enhance the alignment of generated motions with human perception. The approach uses a multi-objective optimization strategy to balance text adherence, motion quality, and human preferences, aiming for Pareto optimality. Extensive experiments and user studies demonstrate that MotionRL significantly outperforms existing methods in generating high-quality, realistic motions that align well with textual descriptions and human feedback. This work represents a significant advancement in the field of text-driven human motion generation, offering a more nuanced and human-centric approach to evaluating and improving motion quality.
weaknesses: Dependence on Pre-trained Perception Models: MotionRL relies on pre-trained human perception models to capture complex human perceptual information. If these models are not of high quality or do not accurately reflect human preferences, the performance of MotionRL could be significantly impacted. This dependency limits the flexibility and robustness of the approach.

Limited Dataset Size: The text-to-motion domain typically has smaller datasets compared to other fields like image or text generation. This limitation makes the model more sensitive to small changes in input text and can lead to overfitting. The smaller dataset size also poses challenges in effectively training the model to generalize well across diverse motion scenarios.

Complexity of Multi-Objective Optimization: While the multi-reward optimization strategy aims to balance text adherence, motion quality, and human preferences, it introduces significant complexity into the training process. Managing and fine-tuning multiple rewards can lead to unstable training and requires careful calibration to ensure that the model does not prioritize one objective at the expense of others. This complexity can make the approach less accessible and harder to implement effectively.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 2
questions: Could the authors elaborate on the potential for dynamically adjusting the weighting of rewards based on real-time feedback?

How does the model handle scenarios where text descriptions are ambiguous or open to interpretation? Is there a mechanism to weigh certain rewards more heavily in such cases?

For practical applications, are there plans to develop more user-friendly interfaces for non-technical users to fine-tune MotionRL’s generated motions?

Please also provide some details about the dataset you created
soundness: 2
strengths: The paper introduces a novel approach by combining reinforcement learning with human preference alignment for text-to-motion tasks, which is an underexplored area in motion generation. The application of multi-reward RL to text-to-motion represents a good contribution, especially as it integrates human perceptual feedback.
summary: The paper "MotionRL: Align Text-to-Motion Generation to Human Preferences with Multi-Reward Reinforcement Learning" introduces MotionRL, a framework that leverages Multi-Reward Reinforcement Learning to enhance text-to-motion generation by aligning outputs with human preferences. Unlike prior models, MotionRL incorporates human perceptual data in its reward structure to address limitations in previous approaches focused on numerical metrics alone. MotionRL employs a multi-objective optimization strategy to achieve a balance between text adherence, motion quality, and human preferences, approximating Pareto optimality for optimal trade-offs. The authors demonstrate the model's effectiveness through extensive experiments and user studies, showing MotionRL's superior performance in terms of perceptual quality and alignment with human expectations.
weaknesses: The model’s reliance on pre-trained perception models, as mentioned in the limitations, could restrict generalizability. Fine-tuning without additional human annotations might limit the model’s adaptability to new or unique datasets, where the pre-trained perception model may not fully capture the nuances of human preferences.

While the multi-reward RL framework is effective, there is limited discussion on dynamically adjusting the weight of each reward in real-time based on specific user feedback. A more adaptive reward weighting mechanism could further enhance user-centered customization.

Although the human preference model provides valuable perceptual data, additional insights from continuous human interaction could help refine the model further. The paper would benefit from exploring how human feedback could be iteratively incorporated to improve long-term model performance, potentially through ongoing human-in-the-loop adjustments.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: * How did you come up with the balance between the three different rewards?

* How is the pareto-based policy gradient optimization different from simple PPO?

* What is the demographic data for the user study, did you do any formal survey? 

Minor things:
- You introduce the abbreviation "RL" multiple times in the text
- Line 200: the Appdix -> the Appendix
soundness: 3
strengths: * Introducing RL with the set of reward functions to optimize for multiple objectivesin text-to-motion generation seems to be a novel approach to that specific domain.

* The idea is simple and easy to understand which is also further improved by a good presentation and writing. The images helps me understand what is the problem you're trying to solve.

* I find the ablation in table 2 important as it shows that human perception does not always align with the metricses typically used.
summary: The paper offers MotionRL, an approach for optimizing text-to-motion generation tasks using Multi-Reward RL. This method aims to align the generated motions with human preferences more effectively than traditional models by incorporating human perceptual models into the RL training process. This is done by employing a multi-objective optimization strategy to balance text adherence, motion quality, and human preferences, approximating Pareto optimality. The paper provides experiments, demonstrating that the proposed model outperforms baselines on traditional metrics and in user studies assessing human perception.
weaknesses: * The overall novelty of the method is modest at best, but the domain application seem to be novel.

* There is a great lack of details about the user study and "volunteers for evaluation" could mean anyone. Given that the participants was called "volunteers" (and not paid?) it seems like it's not randomly recruited people which makes it hard to evaluate the soundness of the user study. 

* The paper uses up all 10 pages but given that it's a simple idea with limited technical contribution I think it's excessive, especially since the limitation was pushed to the appendix.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 1
questions: 1. In line 95, it seems that the purpose of introducing RL is merely to supervise human preference, and the rationale for using RL isn’t well explained.

2. The concept of Pareto optimality lacks a citation—what specific issue does this?

3. There are factual inaccuracies in the stated contributions, as ReinDiffuse appears to have introduced RL into motion before this paper.

4. The supplementary materials don’t clearly explain their purpose, which only provides some generated samples. 

5. The paper contains several grammatical errors. The title should change "Align" to "Aligning". A layout issue in line 292.
soundness: 2
strengths: The experimental metrics show good results, and a motion generation framework based on RL has been designed.
summary: This paper introduces MotionRL, a reinforcement learning (RL)-based framework for multi-objective optimization in human motion generation. Recognizing that prior methods often overlook human perceptual nuances, MotionRL integrates human perceptual priors into the generation process through a reward structure. The framework balances three objectives: human perception, motion quality, and text adherence. To address the challenges of optimizing multiple objectives, MotionRL incorporates a multi-reward optimization strategy that approximates Pareto optimality by selecting non-dominated points in each batch, ensuring better trade-offs across objectives.
weaknesses: The reason for introducing RL is not well explained.