Aligning Human Motion Generation with Human Perceptions

## Abstract

Abstract Human motion generation is a critical task with a wide range of applications. Achieving high realism in generated motions requires naturalness, smoothness, and plausibility. Despite rapid advancements in the field, current generation methods often fall short of these goals. Furthermore, existing evaluation metrics typically rely on ground-truth-based errors, simple heuristics, or distribution distances, which do not align well with human perceptions of motion quality.
In this work, we propose a data-driven approach to bridge this gap by introducing a large-scale human perceptual evaluation dataset, MotionPercept , and a human motion critic model, MotionCritic , that capture human perceptual preferences. Our critic model offers a more accurate metric for assessing motion quality and could be readily integrated into the motion generation pipeline to enhance generation quality. Extensive experiments demonstrate the effectiveness of our approach in both evaluating and improving the quality of generated human motions by aligning with human perceptions. Code and data are publicly available at https://motioncritic.github.io/ .

### 1 Introduction

Human motion generation is an important emerging task [] with wide-ranging applications, including augmented and virtual reality (AR/VR) [, ], human-robot interaction [, ], and digital humans [, ]. Achieving high realism in generated human motions is crucial, necessitating naturalness, smoothness, and plausibility.
However, current generation methods still fall short of these goals, often producing subpar results. Meanwhile, designing appropriate evaluation metrics that accurately reflect these qualities remains a significant challenge. This complexity stems from the highly non-linear and articulated nature of human motion, which must adhere to physical and bio-mechanical constraints while also avoiding visual artifacts. Effective metrics would not only facilitate the objective comparison of generated results but also have the potential to enhance generation models by addressing their shortcomings.

Existing evaluation metrics typically rely on error with pairing ground truth (GT) motion, simple heuristics, or on distribution distance with real motion manifold. The error-based metrics cannot fully reflect the performance because GT is only one reasonable possibility.
The heuristics fall short in comprehensively representing motion quality. For instance, foot-ground contact metrics [, ] fail to penalize twisting arm motions that violate bio-mechanical constraints. It is also infeasible to manually define all the human motion rules in a handcrafted manner.
Meanwhile, distribution distance metrics like Fréchet Inception Distance (FID) [] do not operate on an instance level but rather assess overall distribution similarity. Consequently, they cannot identify implausible motions or provide direct supervision signals to guide the generation of higher-quality motions. Some studies [, ] also indicate that FID correlates poorly with user studies due to the misalignment between its distance measurement and human perception of motion quality. Consequently, existing automatic evaluation metrics cannot effectively reflect or replace subjective user studies, hindering objective evaluation and comparison.

In light of this, we advocate the need for *automatic evaluation aligned with human perceptions*. Firstly, humans are the primary audience and interaction partners for motion generation, making their perception crucial for evaluating motion quality. Secondly, the human brain possesses specialized neural mechanisms for processing biological motion [, ] and is sensitive to even slightly unnatural motions [, ].
Therefore, we explore the possibility of directly learning perceptual evaluations from humans using a data-driven approach. This method could bridge the gap between objective metrics and subjective human judgments, providing a more accurate assessment of motion quality.

First, we carefully curate a human perceptual evaluation dataset named MotionPercept, which contains $52590$ pairs of human preference annotations on generated motions.
Next, we train a human motion critic model, MotionCritic, that learns motion quality ratings from the collected dataset. Our critic model significantly outperforms previous metrics in terms of alignment with human perceptions. Notably, it generalizes well across different data distributions.
In addition to motion evaluation, we further propose to utilize the critic model as a direct supervision signal. We demonstrate that MotionCritic can be seamlessly integrated into the generation training pipeline, effectively improving motion generation quality by increasing alignment with human perceptions with few steps of finetuning.

We summarize our contributions as follows: 1) We contribute MotionPercept, a large-scale motion perceptual evaluation dataset with manual annotations. 2) We develop MotionCritic which models human perceptions of motions through a data-driven approach. Extensive experiments demonstrate its superiority as an automatic human-aligned metric of motion quality. 3) We show that the proposed motion critic model could effectively serve as a supervision signal to enhance motion generation quality. Remarkably, it requires only a small number of fine-tuning steps and can be easily integrated into existing generator training pipeline in a plug-and-play manner.

Figure: Figure 1: Framework Overview. We collect MotionPercept, a large-scale, human-annotated dataset for motion perceptual evaluation, where human subjects select the best quality motion in multiple-choice questions. Using this dataset, we train MotionCritic to automatically judge motion quality in alignment with human perceptions, offering better quality metrics. Additionally, we show that MotionCritic can enhance existing motion generators with minimal fine-tuning.
Refer to caption: x1.png

### 2 Related Work

#### 2.1 Human Motion Generation

Human motion generation is a pivotal task in computer vision, computer graphics, and artificial intelligence, aiming to produce natural and realistic human pose sequences []. This field has seen substantial advancements with the rise of deep generative models [, , , ]. Previous works have explored text-conditioned motion generation that transform narrative descriptions into coherent pose sequences [, , , , ], audio-conditioned methods that synchronize movements with rhythmic cues [, , ], and scene-conditioned generation that integrates environmental contexts to produce contextually appropriate motions [, , ].
Despite significant progress, current mainstream data-driven kinematic motion generation methods sometimes produce unnatural motions that are jittery, distorted, or violate physiological and physical constraints. These issues could be attributed to the inherent uncertainty of the task, limitations of supervision signals, and dataset noises.
Furthermore, evaluating generated human motions presents additional challenges. Conventional metrics such as error and FID fall short in capturing the nuanced details essential for producing lifelike and visually appealing movements [, ]. These measures can overlook critical aspects like the fluidity and biomechanical plausibility that are fundamental to human perceptual judgments.
Given these challenges, it is imperative to develop metrics that are more closely aligned with human perception to more accurately evaluate and enhance the motion generation results.

#### 2.2 Human Perception Modeling

Pioneer work [] collect human perceptual similarity dataset and propose to utilize distance in deep features as perceptual metrics.
Some works [, , , , ] in language models to explore aligning model performance with human intent by first training a reward model, then performing reinforcement learning with the reward model. Recent works  [, , ] also explore utilizing human feedback to improve visual generation results. For example, ImageReward [] propose a reward feedback learning method (ReFL) to to align text-to-image generative models with human judgements. In human motion generation, however, few studies have explored modeling human feedbacks, even though the generated motion quality is highly relevant to human perceptions.
One recent work, MoBERT [], constructs a dataset of human ratings for generated motions. Our work differs from MoBERT in that we collect real human data on a scale tens of times larger (52.6K vs 1.4K) and use comparisons instead of ratings, which is more robust. We design the critic model to learn ratings from these comparisons automatically. Additionally, our approach could not only evaluate motion quality but also effectively improve motion generation results.

### 3 MotionPercept : A Large-scale Dataset of Motion Perceptual Evaluation

We build MotionPercept to capture real-human perceptual evaluations with large-scale and diverse human motion sequences. Hence, we implement a rigorous and efficient pipeline for data collection and data annotation. We also design a concensus experiment in order to examine the perceptual consistency across various human subjects.

#### 3.1 Motion Data Collection

We first collect generated human motion sequence pairs for subsequent perceptual evaluation. We utilize state-of-the-art diffusion-based motion generation method MDM [] and FLAME [] to generate human motion sequences parameterized by SMPL [].
For MDM [], we utilize the action-to-motion model trained on HumanAct12 [] and UESTC [] respectively. For FLAME [], we utilize the text-to-motion model trained on HumanML3D [].
For each group of $4$ motion sequences to be annotated, we use the same condition (text prompt or action labels) while sampling different random noises. This makes the motions similar in content while still having distinguishable differences, thereby making it easier to annotate the choices.

#### 3.2 Human Perceptual Evaluation

Human perceptual evaluation is the core component of MotionPercept, therefore we implement a rigorous pipeline to ensure annotation quality. We first introduce the question design of the perceptual evaluation, then describe the protocol for conducting the evaluation. Finally, we present a statistical analysis of the evaluation results.

##### 3.2.1 Question Design

Our perceptual evaluation is designed in the form of multiple-choice questions as selection is generally easier and more robust than directly rating [, , ]. Given a group of four motion sequence options, we instruct the annotators to select the best candidate that is most natural, visually pleasing, and free of artifacts. Specifically, we summarize the typical failure modes of the generated motions (*e.g*., jittering, foot skating, limb distortion, penetration, *etc*.) and explicitly require the annotators to exclude these options. We provide detailed guidance with task descriptions and representative video examples to better communicate the goal to the annotators. The full guidance is presented in the supplementary materials.
While the optimal choice can be decided unambiguously in most cases, there are situations where the decision can be challenging. Therefore, we add two additional options, “all good” and “all bad”, so that the annotator is not required to pick one of the motions in these cases, thereby improving overall annotation quality.
Results indicate that these cases account for a small portion of the total data. We exclude these cases from our subsequent experiments.
In total, we set six options for each entry: four motion candidates plus “all good” and “all bad”.

##### 3.2.2 Protocols

To ensure the quality of perceptual evaluation results, our annotation process consists of annotator training, annotation, and quality control.
We recruit 10 annotators to perform the perceptual evaluation. Before the evaluation begins, we provide annotation guidelines to help the annotators understand the task and maintain consistent criteria. The annotators must pass a pilot test before starting the formal annotation to ensure they correctly understand the annotation requirements. Additionally, we conduct a perceptual consensus experiment to assess whether the annotation pipeline is suitable for our dataset, as discussed in Section 3.3. Finally, we implement a quality control process where the annotated data is reviewed by an expert quality inspector. During the annotation process, we continuously monitor the quality of each batch of data. For each batch, we randomly sample 10% of the data for quality inspection. The consistency between the sampled data and the expert’s annotations must exceed 90%; otherwise, the entire batch will be re-annotated.

Figure: ((A))
Refer to caption: x2.png

#### 3.3 Analysis

In total, we collect annotations for 18260 multiple-choice questions covering 73040 unique motions, significantly surpassing previous work [] (1400 motions).
We further investigate the following two questions:

- 1.
Based on our experimental setup, can the subjects confidently select the suitable options from the choices provided?
- 2.
Is there a significant difference in perceptual preferences among different subjects, or are they well-aligned?

For the first question, we calculate the proportion of cases where a choice could not be made (including “all good” and “all bad”), and find a total of 418 such groups (2.29%). The result indicates that most of the time subjects can make a definite judgment, demonstrating the validity of our protocol design.

For the second question, we conduct a perceptual consensus experiment where all 10 subjects perform perceptual evaluation independently on 312 groups of randomly selected data. We calculate their pairwise and overall consistency in choices. Figures 2(A) and 2(B) show that for most questions (82.37%), all 10 subjects make the unanimous decision. Figure 2(C) reveals that all 10 subjects exhibit high pairwise agreement (90%). These results indicate a high level of consistency in perceptual judgments of human motion among different human subjects. This not only validates the rationality of our perceptual evaluation pipeline but also inspires us to train machine learning models to emulate this consistent judgment capability.

### 4 MotionCritic : Advancing Motion Generation with Perceptual Alignment

Based on MotionPercept, we develop a human motion critic model, MotionCritic, to emulate the perceptual judgment capabilities of human subjects regarding human motion. We first present the problem formulation and training approach of the critic model, and then explain how to use the critic model for optimizing motion generation.

#### 4.1 Problem Formulation

We formulate the problem as follows: given an input human motion sequence $\mathbf{x}$, we assume there is an implicit human perception model $\mathcal{H}$ that rates the motion quality $\mathcal{H}(\mathbf{x})$, where a higher rate indicates better quality. We aim to build a computational critic model $\mathcal{C}$ that best aligns with $\mathcal{H}$.
Since $\mathcal{H}$ is not explicitly available, we take a data-driven approach. We obtain the human perceptual evaluation dataset $\mathcal{D}$ containing multiple pairs of samples $(\mathbf{x}^{(i)},\mathbf{x}^{(j)})$.
Our training objective is to train the model $\mathcal{C}$ using the dataset $\mathcal{D}$ so that it approximates the human perception model $\mathcal{H}$ as closely as possible. Specifically, we want the model prediction $\mathcal{C}(\mathbf{x}^{(i)})>\mathcal{C}(\mathbf{x}^{(j)})$ if and only if $\mathcal{H}(\mathbf{x}^{(i)})>\mathcal{H}(\mathbf{x}^{(j)})$.
Based on the Bradley-Terry model [, ], the overall training objective could be written as maximizing the joint probabilities that the model $\mathcal{C}$ makes judgments consistent with $\mathcal{H}$ for each pair of samples in the dataset $\mathcal{D}$:

$$ $\displaystyle\arg\max_{\mathcal{C}}\,\mathbb{E}_{(\mathbf{x}^{(i)},\mathbf{x}^ {(j)})\sim\mathcal{D}}\left[\log\sigma\left((\mathcal{C}(\mathbf{x}^{(i)})- \mathcal{C}(\mathbf{x}^{(j)}))\cdot(\mathcal{H}(\mathbf{x}^{(i)})-\mathcal{H}( \mathbf{x}^{(j)}))\right)\right],$ (1) $$

where $\sigma$ is the sigmoid function.

Figure: Figure 3: (I) Critic model training process. We sample human motion pairs $\mathbf{x}^{(h)},\mathbf{x}^{(l)}$ annotated with human preferences, upon which the critic model produces score pairs. We use perceptual alignment loss $L_{\text{Percept}}$ to learn from the human perceptions. (II) Motion generation with critic model supervision. We intercept MDM sampling process at random timestep $t$ and perform single-step prediction. Critic model computes the score $s$ based on the generated motion $\mathbf{x}_{0}^{\prime}$, which is further used to calculate motion critic loss $L_{\text{Critic}}$. KL loss $L_{\text{KL}}$ is introduced between $\mathbf{x}_{0}^{\prime}$ and last-time generation result $\widetilde{\mathbf{x}_{0}}^{\prime}$.
Refer to caption: x5.png

#### 4.2 Human Motion Critic Model

In practice, we represent human motion by $\mathbf{x}\in\mathbb{R}^{L\times J\times D}$ where $L$ denotes the sequence length, $J$ denotes the number of body joints, and $D$ denotes parameter dimensions. We implement the critic model $\mathcal{C}$ as a neural network that maps the high-dimensional motion parameters to a scalar $s$. We draw pairwise comparison annotations from the collected dataset, where $\mathbf{x}^{(h)}$ is the better instance and $\mathbf{x}^{(l)}$ is the worse. The perceptual alignment loss is thus given by:

$$ $\mathcal{L}_{\text{Percept}}=-\mathbb{E}_{(\mathbf{x}^{(h)},\mathbf{x}^{(l)}) \sim\mathcal{D}}\left[\log\sigma\left(\mathcal{C}(\mathbf{x}^{(h)})-\mathcal{C }(\mathbf{x}^{(l)})\right)\right].$ (2) $$

#### 4.3 Motion Generation with Critic Model Supervision

Additionally, we explore to utilize the learned human perceptual prior of $\mathcal{C}$ not only for evaluating generated motions, but also improving them. We demonstrate that our motion critic model could be integrated into state-of-the-art diffusion-based motion generation approaches with ease by using MDM [] as an example.
The forward diffusion is modeled as a Markov noising process $\{\mathbf{x}_{t}\}_{t=0}^{T}$ where $\mathbf{x}_{0}$ is drawn from the data distribution, and

$$ ${q(\mathbf{x}_{t}|\mathbf{x}_{t-1})=\mathcal{N}(\sqrt{\alpha_{t}}\mathbf{x}_{t -1},(1-\alpha_{t})I),}$ (3) $$

where $\alpha_{t}\in(0,1)$ are constant hyper-parameters. When $\alpha_{t}$ is small enough, it’s reasonable to approximate $\mathbf{x}_{T}\sim\mathcal{N}(0,I)$, allow sampling $\mathbf{x}_{T}$ from random noise to begin our denoising process.

Figure: Algorithm 1 Fine-tuning Motion Generation with MotionCritic

Given an MDM model $\mathcal{M}$ with pre-trained parameters $\theta_{0}$, we fine-tune to improve its alignment with a pre-trained critic model $\mathcal{C}$. We develop a lightweight perceptual-aligned fine-tuning approach based on ReFL []. Notably, in order to utilize the critic model in a plug-and-play manner, we keep the MDM training step and objective $\mathcal{L}_{\text{MDM}}$ unchanged.
Instead, we simply add one optimization step with critic model supervision in each training iteration as shown in Figure 1.

Specifically, we sample a Gaussian noise $\mathbf{x}_{T}$ and inference until $\mathbf{x}_{t}$ where $t\in[T_{1},T_{2}]$ is randomly selected in later denoising steps. Then, a single-step denoising is performed to predict $\mathbf{x}_{0}\textquoteright$ from $\mathbf{x}_{t}$. Based on the predicted motion $\mathbf{x}_{0}\textquoteright$, we compute the critic score $s=\mathcal{C}(\mathbf{x}_{0}\textquoteright)$, which is used to compute the motion critic loss:

$$ ${\mathcal{L}_{\text{Critic}}=\mathbb{E}_{y_{i}\sim{\mathcal{Y}}}\left[\phi( \mathcal{C}(\mathbf{x}_{0}^{\prime})\right],\\ }$ (4) $$

where $\phi(s)=-\sigma(\tau-s))$ is a critic-to-loss mapping function, $\tau$ is a constant for shifting the critic value, $\sigma$ is the sigmoid function.
We further introduce a Kullback-Leibler (KL) divergence regularization to prevent $\mathcal{M}$ from moving substantially away from the conditional motion generation task:

$$ $\displaystyle\mathcal{L}_{\text{KL}}$ $\displaystyle=\mathbb{E}_{y_{i}\sim{\mathcal{Y}}}\left[{D_{\text{KL}}}\left(p( \mathbf{x}_{0}^{\prime})\|p(\widetilde{\mathbf{x}_{0}^{\prime}})\right)\right].$ (5) $$

The overall fine-tuning loss is given by

$$ $\displaystyle\mathcal{L}_{\text{FT}}$ $\displaystyle=\mathcal{L}_{\text{MDM}}+\lambda\mathcal{L}_{\text{Critic}}+\mu \mathcal{L}_{\text{KL}}.$ (6) $$

where $\lambda$ and $\mu$ are constants for loss balancing. The detailed algorithm workflow is shown in Algorithm 1.

### 5 Experiment

#### 5.1 Implementation Details

###### Critic Model.

We train our critic model using the MDM subset in MotionPercept. We convert each multiple-choice question into three ordered preference pairs, which results in 46761 pairs for training and 5829 pairs for testing. We parameterize motion sequences with SMPL [], including 24 axis-angle rotations, and global root translation.
We implement the critic model with DSTformer [] backbone with 3 layers and 8 attention heads. We apply temporal average pooling on encoded motion embeddings followed by an MLP with a hidden layer of 1024 channels to predict a single scalar score.
We train the critic model for 150 epochs with a batch size of 64 and a learning rate starting at 2e-3, decreasing with a $0.995$ exponential learning rate decay.

###### Fine-tuning.

We use MDM [] model trained on HumanAct12 [] as our baseline, which utilizes 1000 DDPM denoising steps. We load the checkpoint trained for 350000 iterations and fine-tune for 800 iterations, with a batch size of 64 and learning rate 1e-5. We fine-tune with critic clipping threshold $\tau=12.0$, critic re-weight scale $\lambda=$1e-3, and KL loss re-weight scale $\mu=1.0$. We set the step sampling range $[T_{1},T_{2}]=[700,900]$.

#### 5.2 MotionCritic as Motion Quality Metric

We first evaluate whether the proposed critic model could serve as an effective motion quality metric. Specifically, we are interested in the following research questions:

- 1.
How does MotionCritic align with human perceptual evaluations?
- 2.
Could MotionCritic generalize to different data distributions?

To investigate the first question, we evaluate the performance of our critic model on a held-out test set and compare it with existing motion quality metrics as follows:

- •
Error-based metrics, including Root Average Error (Root AVE), Root Absolute Error (Root AE), Joint Average Error (Joint AVE), and Joint Absolute Error (Joint AE). These metrics involve directly computing the distance between the generated motion and a pairing GT with the same condition.
- •
Heuristic metrics, including acceleration [, ], Person-Ground Contact [], Foot-Floor Penetration [], and Physical Foot Contact (PFC) []. These metrics does not compare against GT; instead, they implement intuitive rule-based evluations. For example, PFC models the relationship between center of mass acceleration and foot-ground contact.
- •
Learning-based metrics. Prior work MoBERT [] proposes to evaluate motion quality with a motion feature extractor and SVR Regression.

**Table 1: Quantitative comparison of motion evaluation metrics on MDM and FLAME subsets of MotionPercept.**
| Metric | MDM | FLAME |  |  |
| --- | --- | --- | --- | --- |
| Accuracy (%) $\uparrow$ | Log Loss $\downarrow$ | Accuracy (%) $\uparrow$ | Log Loss $\downarrow$ |  |
| Root AVE | 59.47 | 0.6891 | 48.42 | 0.6984 |
| Root AE | 61.79 | 0.6798 | 59.54 | 0.6711 |
| Joint AVE | 56.77 | 0.6889 | 44.61 | 0.6973 |
| Joint AE | 62.73 | 0.6794 | 58.37 | 0.6891 |
| Acceleration [] | 64.26 | 0.7792 | 66.67 | 0.6919 |
| Person-Ground Contact [] | 71.78 | 0.7260 | 69.82 | 0.7243 |
| Foot-Floor Penetration [] | 53.61 | 0.6939 | 55.56 | 0.6906 |
| Physical Foot Contact [] | 64.79 | 0.6926 | 66.00 | 0.6930 |
| MoBERT [] | 49.40 | 0.6931 | 52.40 | 0.6932 |
| MotionCritic (Ours) | 85.07 | 0.5486 | 81.43 | 0.5758 |

Note that distribution-based metrics (*e.g*. FID) could not compare quality of individual motion sequences, and the comparison can be found in subsequent experiments. For each metric, we calculate the percentage they align with GT annotations (accuracy) and also their probabilistic distribution distance with GT annotations (log loss). We use the softmax function to convert the scores to probabilities (taking the opposite before softmax for metrics where smaller is better).
Table 4 demonstrates that our critic model significantly outperforms previous metrics. These results not only validate the effectiveness of learning from large-scale human perceptual evaluations but also prove that our critic model can serve as a more comprehensive and robust metric for assessing motion quality.

Figure: ((A))
Refer to caption: x6.png

Furthermore, to investigate the second question, we test the critic model on data outside of the training distributions. We collect a standalone test set with a different motion generation algorithm, FLAME [], and perform perceptual evaluation with a different human subject. Note that this model is trained on a different dataset [] with the model used to generate critic model training data, which means the action categories have large variations. The results in Table 4 further shows that our critic model could well generalize to the new test set, indicating its efficacy in evaluating different generation algorithms and unseen motion contents.

Additionally, we test the generalization of our critic model on the real GT motion distribution. Figure 4(A) illustrates the critic score distribution of HumanAct12 [] test set. We group the 1190 GT motions into 5 groups based on their critic scores, evenly distributed from highest to lowest. We compare the average critic score between the groups with distribution-based metric FID and user study. The user study is conducted by comparing motion pairs sampled from each groups and then computing Elo rating [, ] for each group. Figure 4(B) clearly indicates that the critic score aligns well with human preferences, while FID does not. Notably, we discover that the outliers with small critic values (group V) are indeed artifacts within the dataset. Please refer to the supplementary materials for video results. The results indicate that our critic model can also generalize to the GT motion manifold, even though the model has never been trained on it. It also highlights the potential of using our critic model as a tool for dataset diagnosis (*e.g*., discover failure modes).

#### 5.3 MotionCritic as Training Supervision

Figure: ((A))
Refer to caption: x8.png

Figure: Figure 6: Motion generation results from different fine-tuning steps.
Refer to caption: x10.png

Furthermore, we investigate whether our critic model can also serve as an effective supervision signal. Specifically, we fine-tune a pre-trained motion generator [] with the proposed framework, and evaluate on HumanAct12 [] test set every 200 steps. Additionally, we conduct a standalone user study by comparing motion pairs generated at different fine-tuning steps and compute the Elo Rating [, ].
Figure 5 reveals that as fine-tuning progresses, the motion quality consistently improves according to the user study, in line with the training objective of increasing the critic score. We also present a visualization comparison in Figure 6. We discover that as fine-tuning progresses, unreasonable human motions such as jittering, twisting, and floating significantly decrease. Please refer to the supplementary materials for video comparisons.
The results also demonstrate that our fine-tuning process requires only hundreds of iterations to take effect, significantly improving the perceptual quality of the model. Compared to the 350K pre-training steps, this accounts for only 0.23% of the training cost. This further demonstrates the advantages of our proposed framework in using a perceptually-aligned critic model to fine-tune the motion generation model, not only improving quality but also being lightweight and efficient.

### 6 Conclusion

In conclusion, our work bridges the important gap in human motion generation between objective metrics and human perceptual evaluations by introducing a data-driven framework with MotionPercept and MotionCritic. This paradigm not only offers a more comprehensive metrics of motion quality but could also improve the generation results by aligning with human preferences.
We hope this work could contribute to more objective evaluations of motion generation methods and results.
One limitation of our approach is its primary focus on perceptual metrics without explicitly simulating biomechanical plausibility, which could be explored in future work. Future research could also investigate more fine-grained perceptual evaluation methods to obtain rich human feedback on motion quality like [].

## Part 2 Appendix

### Appendix A Details on MotionPercept

#### A.1 Prompt Selection

We utilize the prompts from HumanAct12 [], UESTC [] and HumanML3D [] for generating the motion candidates.
Specifically, we use the 12 action labels from HumanAct12 [] (shown in  Table 2) and the 40 categories of aerobic exercise description from UESTC [] (shown in Table 3) for the MDM [] model.

**Table 2: 12 action labels from HumanAct12 [].**
| HumanAct12 [] Action Labels |  |
| --- | --- |
| warm up | walk |
| run | jump |
| drink | lift dumbbell |
| sit | eat |
| turn steering wheel | phone |
| boxing | throw |

**Table 3: 40 action labels from UESTC [].**
| UESTC [] Action Labels |  |
| --- | --- |
| punching and knee lifting | marking time and knee lifting |
| jumping-jack | squatting |
| forward-lunging | left-lunging |
| left-stretching | raising-hand-and-jumping |
| left-kicking | rotation-clapping |
| front-raising | pulling-chest-expanders |
| punching | wrist-circling |
| single-dumbbell-raising | shoulder-raising |
| elbow-circling | dumbbell-one-arm-shoulder-pressing |
| arm-circling | dumbbell-shrugging |
| pinching-back | head-anticlockwise-circling |
| shoulder-abduction | deltoid-muscle-stretching |
| straight-forward-flexion | spinal-stretching |
| dumbbell-side-bend | standing-opposite-elbow-to-knee-crunch |
| standing-rotation | overhead-stretching |
| upper-back-stretching | knee-to-chest |
| knee-circling | alternate-knee-lifting |
| bent-over-twist | rope-skipping |
| standing-toe-touches | standing-gastrocnemius-calf |
| single-leg-lateral-hopping | high-knees-running |

We randomly select texts from HumanML3D [] test set as prompts for the FLAME [] model.

#### A.2 Annotation Management

We recruit 10 annotators for this task, and data entries are randomly allocated to them. We provide detailed guidelines to annotators.
We evaluate the annotation result by spot check.
We randomly select 10% of all data to inspect the annotation results according to guidelines and calculate the proportion of unqualified data entries. If the unqualified proportion is less than 10%, the results are considered to be acceptable. All the unqualified data entries will be re-annotated. We will update the guidelines during annotation based on spot check feedback, and annotators will study the new guidelines.

#### A.3 Annotation Design

Figure: Figure 7: An example of raw data entry before annotation.
Refer to caption: extracted/5700629/figs/data_entry.jpg

We generate four motions from the same prompt for each data entry, as shown in Fig 7. The prompts are hidden during the annotation process.
Annotators are required to select either the best or the worst motion for data entries generated by MDM [] and FLAME []. MDM [] exhibits better motion diversity but lacks stability, so annotators are instructed to select the best motion. Conversely, FLAME [] demonstrates better stability but lacks diversity, so annotators are instructed to select the worst motion for these entries.

Figure: Figure 8: Our annotation platform.
Refer to caption: x11.png

#### A.4 Annotation Guidance Documentation

We provide a detailed annotation document to explain the annotation process. The annotation platform is shown in Fig 8.

###### Introduction

Each data entry to be annotated consists of four videos, as shown in Fig 7. Each video is approximately three seconds long, with all four videos playing simultaneously and concatenated into one video.

###### Requirements

Each set of videos has six options: A, B, C, D, "all are good," and "all are bad." Annotators should select the most natural and reasonable video for each data entry. If one option stands out as the best, select that option. If all actions seem equally good or equally bad, choose "all are good" or "all are bad." Text prompts will be hidden during annotation.

###### Video Examples

We provide annotators with examples if what kinds of motions are unnatural and unaccepetable:

- 1.
Body pose is unnatural, including hands, feet and so on.
- 2.
Human motion violates physiological constraints.
- 3.
Human motion is erratic or severely stutters.
- 4.
Human body collides, such as hands fully embedded into leg.
- 5.
Human body is severely tilted, to the point of losing balance.
- 6.
Human body appears to be drifting instead of walking.

Examples of these problems are shown in Fig 9.

Figure: Figure 9: Representative examples of options to be excluded.
Refer to caption: x12.png

### Appendix B Data Documentation

We follow the datasheet proposed in [] for documenting our MotionPercept:

- 1.
Motivation
(a)
For what purpose was the dataset created?
This dataset was created to collect human perceptual data on whetner human motions seem natural, and ultimately advance our study of perceptual-aligned metric and finetuning human motion generation model.
(b)
Who created the dataset and on behalf of which entity?
This dataset was created by Haoru Wang, Yishu Xu, Luyi Miao, Wentao Zhu, Feng Gao and Yizhou Wang with Peking University.
(c)
Who funded the creation of the dataset?
The creation of this dataset was funded by Peking University.
(d)
Any other Comments?
None.
- 2.
Composition
(a)
What do the instances that comprise the dataset represent?
Each instance contains 4 video of human motions generated from the same prompt by existing motion generation methods [, ].
(b)
How many instances are there in total?
In total, we collect annotations for 18260 multiple-choice questions covering 73K unique motions.
(c)
Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set?
No, this is a brand-new dataset.
(d)
What data does each instance consist of?
See appendix A for details.
(e)
Is there a label or target associated with each instance?
Yes. See appendix A.
(f)
Is any information missing from individual instances?
No.
(g)
Are relationships between individual instances made explicit?
Yes.
(h)
Are there recommended data splits?
Yes, we have separated the whole dataset into MDM-A (motions generated by MDM [] from prompts in HumanAct12 []), MDM-U (motions generated by MDM [] from prompts in UESTC [] and FLAME (motions generated by FLAME [] from prompts in HumanML3D []). We provide the recommended data splits by combining MDM-A and MDM-U and randomly splitting them into a training set and a test set at a ratio of 8:1. Data generated by FLAME [] is primarily used as test data for generalization.
(i)
Are there any errors, sources of noise, or redundancies in the dataset?
No.
(j)
Is the dataset self-contained, or does it link to or otherwise rely on external resources (*e.g*., websites, tweets, other datasets)?
The dataset is self-contained.
(k)
Does the dataset contain data that might be considered confidential (*e.g*., data that is protected by legal privilege or by doctor-patient confidentiality, data that includes the content of individuals’ non-public communications)?
No.
(l)
Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety?
No.
(m)
Does the dataset relate to people?
Yes. Our human motion data is generated as body model parameters [], not from real people, and therefore does not contain biometrics. These data are annotated by human annotators.
(n)
Does the dataset identify any subpopulations (*e.g*., by age, gender)?
No. Our human motion data are generated as body model parameters [] with no explicit gender or age.
(o)
Is it possible to identify individuals (*i.e*., one or more natural persons), either directly or indirectly (*i.e*., in combination with other data) from the dataset?
No. Our human motion data are generated by algorithms with commonly used body models.
(p)
Does the dataset contain data that might be considered sensitive in any way (*e.g*., data that reveals racial or ethnic origins, sexual orientations, religious beliefs, political opinions or union memberships, or locations; financial or health data; biometric or genetic data; forms of government identification, such as social security numbers; criminal history)?
No.
(q)
Any other comments?
None.
- 3.
Collection Process
(a)
How was the data associated with each instance acquired?
See appendix A for details.
(b)
What mechanisms or procedures were used to collect the data (*e.g*., hardware apparatus or sensor, manual human curation, software program, software API)?
We use existing motion generation models to collect videos and require annotators to label them. See appendix A for details.
(c)
If the dataset is a sample from a larger set, what was the sampling strategy (*e.g*., deterministic, probabilistic with specific sampling probabilities)?
See appendix A and appendix B for details.
(d)
Who was involved in the data collection process (*e.g*., students, crowdworkers, contractors) and how were they compensated (*e.g*., how much were crowdworkers paid)?
The video data was collected by the authors. The annotations were performed by the workers in DATATANG TECHNOLOGY INC., and the workers were offered a fair wage as per the prearranged contract.
See appendix A and appendix B for details.
(e)
Over what timeframe was the data collected?
The data were collected from 2023 to 2024, and labeled in 2024.
(f)
Were any ethical review processes conducted (*e.g*., by an institutional review board)?
No. The MotionPercept dataset raises no ethical concerns regarding the privacy information of human subjects.
(g)
Does the dataset relate to people?
Yes. Our human motion data are generated as body model parameters [], not real people. The annotation is done by people.
(h)
Did you collect the data from the individuals in question directly, or obtain it via third parties or other sources (*e.g*., websites)?
We obtain raw data from motion generation model. Annotation data are collected by annotators.
(i)
Were the individuals in question notified about the data collection?
Yes.
(j)
Did the individuals in question consent to the collection and use of their data?
Yes.
(k)
If consent was obtained, were the consenting individuals provided with a mechanism to revoke their consent in the future or for certain uses?
Yes.
(l)
Has an analysis of the potential impact of the dataset and its use on data subjects (*e.g*., a data protection impact analysis) been conducted?
Not applicable.
(m)
Any other comments?
None.
- 4.
Preprocessing, Cleaning and Labeling
(a)
Was any preprocessing/cleaning/labeling of the data done (*e.g*., discretization or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)?
Yes, see appendix A.
(b)
Was the “raw” data saved in addition to the preprocessed/cleaned/labeled data (*e.g*., to support unanticipated future uses)?
Yes. We provide raw data entries and their annotations respectively.
(c)
Is the software used to preprocess/clean/label the instances available?
No. The annotation software is the private labeling platform provided by DATATANG TECHNOLOGY INC. .
(d)
Any other comments?
None.
- 5.
Uses
(a)
Has the dataset been used for any tasks already?
No, the dataset is newly proposed by us.
(b)
Is there a repository that links to any or all papers or systems that use the dataset?
Yes, we provide the link to all related information on our [project page](https://motiioncritic.github.io/).
(c)
What (other) tasks could the dataset be used for?
This dataset could be used for other research topics, including but not limited to human preference study, human motion study.
(d)
Is there anything about the composition of the dataset or the way it was collected and preprocessed/cleaned/labeled that might impact future uses?
See appendix A for details.
(e)
Are there tasks for which the dataset should not be used?
The usage of this dataset should be limited to the scope of human motion.
(f)
Any other comments?
None.
- 6.
Distribution
(a)
Will the dataset be distributed to third parties outside of the entity (*e.g*., company, institution, organization) on behalf of which the dataset was created?
Yes, the dataset will be made publicly available.
(b)
How will the dataset be distributed (*e.g*., tarball on website, API, GitHub)?
The dataset will be published on our [code website](https://github.com/ou524u/AlignHP) with its [metadata document](https://drive.google.com/file/d/1WnBI8UDCINnv1LHAtsNZJ6QY2tRehUdG/view?usp=drive_link).
(c)
Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)?
We release our benchmark under CC BY-NC 4.0 (^†^†[https://paperswithcode.com/datasets/license](https://paperswithcode.com/datasets/license)) license.
(d)
Have any third parties imposed IP-based or other restrictions on the data associated with the instances?
No.
(e)
Do any export controls or other regulatory restrictions apply to the dataset or to individual instances?
No.
(f)
Any other comments?
None.
- 7.
Maintenance
(a)
Who is supporting/hosting/maintaining the dataset?
Haoru Wang is maintaining.
(b)
How can the owner/curator/manager of the dataset be contacted (*e.g*., email address)?
ou524u@stu.pku.edu.cn
(c)
Is there an erratum?
Currently, no. As errors are encountered, future versions of the dataset may be released and updated on our website.
(d)
Will the dataset be updated (*e.g*., to correct labeling errors, add new instances, delete instances’)?
Yes, if applicable.
(e)
If the dataset relates to people, are there applicable limits on the retention of the data associated with the instances (*e.g*., were individuals in question told that their data would be retained for a fixed period of time and then deleted)?
Our human motion dataset is generated as body model parameters [], not real people. No applicable limits on retention of the data and the annotators are aware of the use of data.
(f)
Will older versions of the dataset continue to be supported/hosted/maintained?
Yes, older versions of the benchmark will be maintained on our website.
(g)
If others want to extend/augment/build on/contribute to the dataset, is there a mechanism for them to do so?
Yes, please get in touch with us by email.
(h)
Any other comments?
None.

### Appendix C Details on MotionCritic : as Motion Quality Metric

###### Data Pre-processing.

Each multiple-choice question is divided into three ordered preference pairs. Motion sequences are parameterized using SMPL [], which includes 24 axis-angle rotations and one global root translation.

###### Training and Evaluation.

We train the critic model from scratch using the DSTformer [] backbone with 3 layers and 8 attention heads on MotionPercept. To ensure robustness, we train our model for multiple times and report the error bars, considering variations such as the random seed across multiple runs. Evaluation results, detailing action-label splits, are presented in the following two tables. Our MotionCritic gets the best results and can robustly score different types of human motions.

**Table 4: Accuracy comparison of motion evaluation metrics on HumanAct12 action classes(%).**
| Metric | Warm. | Walk | Run | Jump | Drink | Lift. | Sit | Eat | Turn. | Phone | Box. | Throw | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Root AVE | 57.6 | 47.3 | 56.8 | 62.7 | 59.5 | 46.3 | 37.9 | 64.5 | 54.1 | 0.62 | 51.6 | 53.7 | 59.5 |
| Root AE | 70.1 | 70.0 | 69.7 | 57.2 | 70.0 | 49.8 | 52.5 | 61.2 | 63.2 | 61.7 | 52.7 | 55.2 | 61.8 |
| Joint AVE | 42.0 | 52.2 | 50.4 | 64.7 | 53.2 | 50.2 | 42.4 | 48.6 | 51.9 | 55.7 | 48.4 | 45.2 | 56.8 |
| Joint AE | 63.6 | 69.1 | 75.2 | 55.7 | 59.9 | 41.1 | 51.4 | 66.7 | 59.3 | 60.6 | 53.5 | 54.1 | 62.7 |
| Acceleration [] | 66.7 | 78.0 | 61.6 | 53.0 | 65.4 | 62.4 | 82.6 | 61.6 | 51.1 | 59.3 | 69.5 | 61.7 | 64.3 |
| Person-Ground Contact [] | 69.8 | 70.1 | 70.2 | 66.0 | 72.8 | 71.8 | 90.1 | 76.9 | 70.3 | 67.9 | 62.6 | 63.2 | 71.8 |
| Foot-Floor Penetration [] | 47.1 | 52.4 | 52.7 | 55.7 | 48.5 | 56.8 | 59.9 | 52.4 | 50.4 | 55.7 | 52.8 | 53.3 | 53.6 |
| Physical Foot Contact [] | 80.5 | 77.8 | 73.1 | 51.7 | 67.5 | 57.9 | 78.5 | 63.4 | 53.2 | 65.5 | 68.6 | 68.5 | 64.8 |
| MoBERT [] | 67.4 | 68.4 | 44.8 | 37.4 | 70.0 | 18.9 | 43.6 | 49.9 | 65.2 | 25.6 | 56.1 | 44.8 | 49.4 |
| MotionCritic (Ours) | 90.6${}_{\pm\text{0.2}}$ | 94.2${}_{\pm\text{0.4}}$ | 91.9${}_{\pm\text{0.3}}$ | 90.6${}_{\pm\text{0.1}}$ | 83.9${}_{\pm\text{1.3}}$ | 85.3${}_{\pm\text{0.7}}$ | 86.0${}_{\pm\text{0.7}}$ | 78.3${}_{\pm\text{1.4}}$ | 79.8${}_{\pm\text{0.9}}$ | 85.1${}_{\pm\text{0.6}}$ | 86.6${}_{\pm\text{0.4}}$ | 82.5${}_{\pm\text{0.3}}$ | 85.1${}_{\pm\text{0.5}}$ |

**Table 5: Log-loss comparison of motion evaluation metrics on HumanAct12 action classes.**
| Metric | Warm. | Walk | Run | Jump | Drink | Lift. | Sit | Eat | Turn. | Phone | Box. | Throw | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Root AVE | 0.69 | 0.69 | 0.69 | 0.69 | 0.69 | 0.69 | 0.69 | 0.69 | 0.69 | 0.69 | 0.69 | 0.69 | 0.69 |
| Root AE | 0.68 | 0.66 | 0.67 | 0.68 | 0.68 | 0.69 | 0.70 | 0.69 | 0.69 | 0.69 | 0.69 | 0.69 | 0.68 |
| Joint AVE | 0.69 | 0.69 | 0.69 | 0.69 | 0.69 | 0.69 | 0.69 | 0.69 | 0.69 | 0.69 | 0.69 | 0.69 | 0.69 |
| Joint AE | 0.68 | 0.67 | 0.67 | 0.69 | 0.68 | 0.70 | 0.70 | 0.69 | 0.69 | 0.68 | 0.68 | 0.69 | 0.68 |
| Acceleration [] | 0.70 | 0.59 | 0.88 | 1.5 | 0.60 | 0.69 | 0.60 | 0.64 | 0.88 | 0.71 | 0.61 | 0.76 | 0.78 |
| Person-Ground Contact [] | 0.71 | 0.68 | 0.68 | 0.71 | 0.69 | 0.70 | 0.73 | 0.68 | 0.74 | 0.70 | 0.73 | 0.72 | 0.73 |
| Foot-Floor Penetration [] | 0.70 | 0.70 | 0.69 | 0.70 | 0.69 | 0.70 | 0.71 | 0.70 | 0.69 | 0.70 | 0.69 | 0.69 | 0.69 |
| Physical Foot Contact [] | 0.69 | 0.69 | 0.69 | 0.69 | 0.69 | 0.69 | 0.69 | 0.69 | 0.69 | 0.69 | 0.69 | 0.69 | 0.69 |
| MoBERT [] | 0.69 | 0.69 | 0.69 | 0.69 | 0.69 | 0.69 | 0.69 | 0.69 | 0.69 | 0.70 | 0.69 | 0.69 | 0.69 |
| MotionCritic (Ours) | 0.51${}_{\pm\text{0.01}}$ | 0.52${}_{\pm\text{0.02}}$ | 0.50${}_{\pm\text{0.01}}$ | 0.51${}_{\pm\text{0.02}}$ | 0.56${}_{\pm\text{0.02}}$ | 0.54${}_{\pm\text{0.02}}$ | 0.54${}_{\pm\text{0.02}}$ | 0.59${}_{\pm\text{0.03}}$ | 0.59${}_{\pm\text{0.01}}$ | 0.57${}_{\pm\text{0.01}}$ | 0.53${}_{\pm\text{0.02}}$ | 0.55${}_{\pm\text{0.01}}$ | 0.55${}_{\pm\text{0.02}}$ |

### Appendix D Details on MotionCritic : as Training Supervision

#### D.1 Fine-tuning

###### Critic Score Clipping.

Generally, a higher MotionCritic score indicates better motion quality. However, this relationship has an upper limit. During our fine-tuning process, we clip motions with reward scores exceeding a threshold $\tau$ when computing gradients before back-propagation. This threshold, determined through a series of comparative experiments, is set at $\tau=12.0$, approximately the upper bound of ground-truth critic scores. We found that this setting yields the best results. Fine-tuned motion generation models without reward clipping tend to artificially inflate reward scores on a few specific motions, which increases the average MotionCritic score but degrades overall performance. Thus, reward clipping is essential to maintain the integrity and quality of the fine-tuning process.

###### Finetuning Details.

Inspired by [], we observe how the critic score changes over denoising steps to identify the optimal time window for ReFL intercept. As shown in Figure 10(A), we set the hyperparameter step sampling range to $[T_{1},T_{2}]=[700,900]$, where the critic score witnesses a rapid increase.

Figure 10(B) illustrates the variation in the average critic score of a training batch over the course of fine-tuning steps. The fine-tuning process is stable and quick to take effect.

Figure: ((A))
Refer to caption: x13.png

#### D.2 Results

###### Improved Critic Score.

As shown in Figure 11, the critic score increases after MotionCritic supervised fine-tuning. This scatter plot collects all data points from the test set, with the critic score of motions before fine-tuning on the x-axis and the critic score of the corresponding motions after fine-tuning on the y-axis.
As demonstrated in Figure 11(A), we first compare results with and without critic model supervision. In the latter case, the original MDM loss is used for continued training without our MotionCritic-based plug-and-play module. The scatter plot clearly indicates that the results with critic model supervised fine-tuning achieve significantly higher scores. The second experiment in Figure 11(B) examines different fine-tuning steps using 800 steps from the first set as a baseline. The results demonstrate that critic model supervised fine-tuning consistently improves the critic score throughout the fine-tuning process.

Figure: ((A))
Refer to caption: x15.png

###### Improved Motion Quality.

We conduct an independent user study to compare motion pairs generated at various fine-tuning stages and calculate the Elo Rating [, ]. Figure 12 demonstrates that the quality of motions consistently enhance as fine-tuning advances, as indicated by the user study. This improvement aligns with the training objective of elevating the critic score.

We further inspect the change of different metrics during the fine-tuning process in Figure 12(B).
PFC [] and FID are expected to be negatively correlated with motion quality (the smaller, the better), and MotionCritic and multimodality are expected to be positively correlated (the greater, the better).
The results indicate that existing motion quality metrics (*e.g*. FID, PFC) do not adequately reflect human preference, as they poorly correlate with Elo ratings from user studies. Meanwhile, improving the critic score does not necessarily conflict with the multimodality metric, which models the diversity of generated motions.

Figure: ((A))
Refer to caption: x17.png

### Appendix E Details on User Studies

###### Annotation

We conduct user studies on GT subsets grouped from HumanAct12 [] and motions generated during finetune steps as discussed in the main text. Our user study platform is shown in Fig 13. In user study, one motion pair of two motions are played simultaneously, with their critic scores and text prompts being hidden. Annotators should choose the better motion or choose "Almost the Same" if they can’t make a decision. We perform user study on 5 different finetune steps and 5 GT batches grouped from HumanAct12 [].

###### Win-rates

After annotation, we calculate win-rates of subsets pairs. In user study, each subset has the same amount of motions. Given subsets pair $(A,B)$, win-rates shows the percentage of motion pairs where motion of subset $A$ win over motion of subset $B$ in naturalness. Then we paint heatmaps of all subsets with their win rates. Since the result of one match maybe tie, the sum of win-rates of two subsets in a pair and data in symmetric positions of heatmap might be less than 1.

Figure: Figure 13: Our user study platform.
Refer to caption: x19.png

###### Elo Rating

[, ] After annotation, we calculate elo rating of each subsets as follows:
Suggest $R_{A},R_{B}$ are the initial ratings of two compared subsets $A$ and $B$.
The expectated win rate of subset9s $A$ and $B$, denoting as $E_{A},E_{B}$ can be calculated as follows:

$$ $\displaystyle E_{A}$ $\displaystyle=\frac{1}{1+10^{(R_{B}-R_{A})/400}}$ (7) $\displaystyle E_{B}$ $\displaystyle=\frac{1}{1+10^{(R_{A}-R_{B})/400}}$ (8) $$

The new ratings of subsets $A$ and $B$ are:

$$ $\displaystyle R_{A}^{{}^{\prime}}$ $\displaystyle=R_{A}+K(S_{A}-E_{A})$ (9) $\displaystyle R_{B}^{{}^{\prime}}$ $\displaystyle=R_{B}+K(S_{B}-E_{B})$ (10) $$

where K is rating coefficient, we choose 32; and $S$ is real score, which is 1 for winner, 0 for loser and 0.5 if the result is a tie. We set the initial rating of each subset as 1500.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: (1) Why not use the latest models, but MDM and FLAME released in 2022? 

(2)Since currently, another two paradigms Noise De-masking and GPT-like Auto-regressive model show their potentials for motion generation task, how could this method of motion generation with critic model supervision be general to these methods?

(3) Why the step sampling range set as [700,900] in Sec 5.1? Is there any experiment support this setting, because I think from 700 - 900 step to get x0 directly results in motion with more artifacts, it might be help for those motion with obvious errors, but if you want to improve the motion quality use critic supervision as post-training, maybe it doesn't make much sense. If I am wrong, please explain your thoughts with some proofs. 

BTW, I intially wanted to give 7 out of 10, but there is not this option. If authors can solve my concerns, I will raise the score.
soundness: 4
strengths: （1） The use of a large-scale human-annotated dataset and a critic model for automatic evaluation to align motion generation quality with human perceptions is a significant step forward.
（2） The experimental design is comprehensive. It evaluates the MotionCritic model on different data distributions and shows its generalization ability. Also, it tests the model as a training supervision signal and analyzes its impact on motion generation quality.
summary: This paper proposes a data-driven approach to align human motion generation with human perceptions. It introduces a large-scale dataset MotionPercept and a human motion critic model MotionCritic. The main contributions include providing a more accurate metric for motion quality assessment and showing that the critic model can enhance motion generation quality.
weaknesses: 1. Table 2 shows, after 700 steps finetuning, the MotionCrotic decrease at 800 step. I am not sure about the trends, could you provide results with much much more steps to let us see the potential of designed Critic Supervision. 

2. I think motion model trained on larger datasets HumanML3D and Motion-X is should be used to see Critic Supervision's potential.

3. Sec 4.3 could be with more detail and Algorithm 1 could be much simpler and clearer maintaining only necessary procedures.

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: -
soundness: 2
strengths: This paper addresses a common problem in generative motion synthesis: how to evaluate the motion quality of a generated sequence? In contrast to the often utilized FID score MotionCrititic operates on a per-sequence base, which allows for a much more granular evaluation.
summary: This work introduces MotionPercept, a large-scale synthetic human motion dataset which contains human-ranked synthetic human motion generations, and MotionCritic, a model learned on MotionPercept. MotionCritic can be utilized to evaluate motion quality but also as an additional loss for motion quality. 
The authors show that MotionCritic follows human perception closer than existing evaluation metrics and can even be used to find outliers in existing human motion datasets.
weaknesses: One major concern of the motion critic is its quality ceiling: MotionPercept is highly dependent on the generative models that produces the motion, so in turn MotionCritic is highly dependent on those generative models as well. That means that, for the model, the best possible motion is the best generative model of MotionPerecpt, potentially limiting the usefulness of the critic once generative models produce significantly better motion than has been produced for MotionPercept. Could the authors comment if there is a way to remedy this or if they have considered this a problem?

Another concern would be: how does MotionCritic behave when provided with unseen motion modalities? I.e. lets say MotionCritic was not trained with dancing motion but is now tasked to judge the motion quality of dancing: do the authors expect MotionCritic to continue being reliable or would retraining be required?

Why did the authors choose 2.3s sequence lengths (60 frames @24Hz)? This seems very short for complex motion, i.e. a person “sitting down”. Were complex motions discarded for the dataset generation? Otherwise, only the motion initiation would have been recorded.
Did the authors observe that the model tends to start from the same pose or position, i.e. a T-Pose? This might limit the application of MotionCiritic as it could favor motions and poses that are more similar to the “initialization” pose/motion.

How can MotionCritic be applied to sequences of different lengths, i.e. [A] How can MotionCritic deal with sequences shorter than 60 frames and how does it perform ?

How is the input motion to MotionCritic normalized? Is the first frame translation at the origin? How is the global rotation set up? Are the first frame rotations set to the same rotation for each generation?

[B] How can MotionCritic be applied to sequences far longer than 60 frames, i.e. 1000 frames?

Table 1: NDMS and NPSS are flipped.

The evaluation methods against which motion critic is compared should be discussed in the related work, and not just in Table 1.

The authors should describe in how they apply the other metrics that they compare against, i.e. how they normalize the data, what kind of data representation/hyperparameters they are using.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: 1- This aligns with my comment on sensitivity analysis. The paper addresses only fine-tuning but does not consider an end-to-end training scenario. Theoretically, the critic model could also be used to train a motion model from scratch. I mention this because, even in the MDM fine-tuning experiment, the critic term is tightly controlled (with a low learning rate, low critic weight, and high KL-term weight). Is this due to difficulties in training with the critic term, possibly because of a non-smooth energy landscape?

2- Locomotion sequences with significant foot sliding receive high critic scores. Could this be due to a bias in the annotated samples, given that MDM generates many sequences with foot sliding? Have the authors considered training the critic with ground-truth samples as well? This should be straightforward, as ground-truth samples could always be preferred over synthetic ones.

3- Are the critic scores for two different samples comparable? I assume it isn’t a distance function. Could the authors comment on this?

4- This is minor but it would be great to have a comparison against a log-likelihood metric such as a normalizing flow.
soundness: 3
strengths: The paper addresses a fundamental problem in human motion generation and is both well-written and well-motivated. I appreciate the authors' meticulous effort in their work.

Annotating a large sample set and training a critic model on this data is a clever idea. Additionally, the thorough analysis and comparisons effectively demonstrate the proposed method's impact. While the human motion modeling field has converged on a set of complementary metrics for quality assessment, these metrics still fall short of aligning with human perception, as shown in this paper. The proposed MotionCritic takes a more direct approach to addressing this issue. I believe this work is highly valuable for the community, contributing both to the research on evaluation and analysis of future human motion synthesis research. I also hope it will serve as an alternative to the counterintuitive FID score.
summary: The paper introduces a novel evaluation metric for 3D human motion that aligns more closely with high-level human perception. A large-scale data annotation effort is conducted by asking users to select the highest quality motion from a set of samples. To capture human preferences, the paper proposes training a critic model that learns to assign a quality score to each motion sample. A comprehensive analysis compares the proposed critic model with commonly used evaluation metrics from prior work, revealing deficiencies in these established metrics for aligning with human perception, while the proposed critic model shows much higher alignment. Additionally, the paper presents a fine-tuning approach for improving the motion quality of pre-trained motion diffusion models by leveraging the proposed critic model.
weaknesses: A sensitivity analysis could be included to assess the robustness and smoothness (e.g., Lipschitz continuity) of the critic model. For instance, how would the critic model respond if a pose is slightly perturbed, either randomly or in a structured way by rotating a joint beyond its limits?

By design, certain artifacts—such as foot-floor penetration, person-ground contact, etc.—might be missed, as the critic model lacks information needed to detect these issues (e.g., it only uses rotations and root translation as inputs). What are the authors' thoughts on this limitation?

Another potential concern is the reliance on a single model (e.g., MDM) for generating data used in the annotations. Although the paper presents results with an additional model (e.g., FLAME) to demonstrate generalization, it may be beneficial to curate an annotation set from various models and architectures to mitigate possible biases.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. There is inherent diversity in the motion generated from text-to-motion. Regarding the fine-tuning comparisons in the supplementary materials, I observe that the generated motions generally align with semantic accuracy, and the physical accuracy appears to improve over time. Could it be that the critic score primarily reflects physical correctness? What I’d like to see is a visualization that compares multiple generated motions for a single text prompt, showing the distribution of critic scores (with five or more samples), as well as how motion diversity changes before and after fine-tuning. This result would better demonstrate the effectiveness of your proposed method and its balance in generating diverse outputs. 

2. When the text is simple, such as "run" or "punching," the corresponding motions should naturally be diverse. However, based on the annotation process provided in the HTML document, it seems that only the best out of four generated motions is selected, which shows no particular consideration for diversity. This raises the question of how Multimodality would improve after fine-tuning. [1] aligns motion generation with human preference. They only select the best result, which significantly reduces diversity (MModality, the diversity of motion generated for one text description). A discussion with [1] would also be beneficial.

3. The annotation of human preference in this paper for motion follows the same process as for images. I see that many useful insights (as metrics and supervision) also resemble those used in image-based tasks. Since motion data is a unique form of 3D data, differing greatly from 2D image data, are there any distinct insights here?

4. The annotation of human preference in this paper for motion follows the same process as for images. I also see that many insights (as metrics and supervision) also the same as image-based tasks [2,3]. Since motion data is a unique form of 3D data, differing greatly from 2D image data, are there any distinct insights here?

5. In Table 1, where MDM’s generated results are evaluated with different metrics, why does the metric with higher Acc is better?

Reference:

[1] HuTuMotion: Human-Tuned Latent Motion Diffusion Models with Minimal Feedback.   
[2] ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation.  
[3] Better Aligning Text-to-Image Models with Human Preference.
soundness: 2
strengths: A large amount of human preference annotation work has been done for text-to-motion, and a sizable dataset has been provided, which is beneficial to the community.
summary: This paper addresses the challenges in evaluating and improving human motion generation by proposing a human-aligned evaluation approach. The authors introduce MotionPercept, a large-scale dataset containing over 52,000 human preference annotations for generated motions, which provides a foundation for understanding motion quality from a human perspective. They also develop MotionCritic, a model trained on MotionPercept, which accurately predicts human-perceived motion quality and surpasses traditional metrics in alignment with human preferences. MotionCritic can be easily integrated into existing generation pipelines, where it functions as a supervision signal to improve motion quality with minimal fine-tuning.
weaknesses: The human preference annotation lacks consideration for the diversity of motion.

Meta-review: The authors present a new data-driven framework for evaluating and improving human motion generation. They propose (1) a large-scale human perceptual dataset, MotionPercept, that contains human preference annotations of synthetic motions; and (2) a human motion critic model, MotionCritic, which is trained on this dataset to capture high-level, subjective aspects of motion plausibility, smoothness, and overall realism from a human perspective. The paper’s second claim is that MotionCritic can serve not only as an evaluation tool but also as a supervision signal to improve the quality of generated motions. By fine-tuning existing motion diffusion models (e.g., MDM, FLAME) with critic scores, the authors show that generated motions become more physically correct and perceptually appealing. 

Strengths of the Paper:

-- The authors construct a substantial dataset (MotionPercept) with more than 50k human preference labels, a valuable asset for the community and a strong foundation for learning perceptual scores.

-- MotionCritic provides sequence-level evaluation, enabling more nuanced assessments of motion plausibility compared to distribution-level metrics (e.g., FID).

-- The paper shows that using MotionCritic to fine-tune generative models yields improvements in motion realism, generalizing to different model architectures.

Weaknesses:

-- Since MotionCritic is trained on data from existing generators, it may overfit to their quality range and struggle if future models significantly surpass the baseline motions it was trained on.

-- Reviewers raise concerns about how the critic handles longer or more complex sequences (e.g., dancing, long-form actions). The dataset mostly uses shorter segments, potentially biasing the critic.

--The paper briefly mentions noise de-masking or auto-regressive approaches but does not provide extensive experiments showing how critic supervision might generalize to these newer techniques. Additional experiments on larger datasets and more training steps could strengthen claims.

Overall, the paper makes a valuable contribution by proposing an innovative perceptual metric and providing initial evidence of its benefits.  After carefully reading the paper, the reviews and rebuttal discussions, the AC agrees with the reviewers on recommending to accept the paper.

Decision: Accept (Poster)