Aligning Human Motion Generation with Human Perceptions
Authors: Haoru Wang
Wentao Zhu
Luyi Miao
Yishu Xu
Feng Gao
Qi Tian
Yizhou Wang
Venue: ICLR 2025 Poster (2025)
Keywords: Human Motion Generation
Primary area: datasets and benchmarks
TL;DR: Aligning Human Motion Generation with Human Perceptions

Abstract: Human motion generation is a critical task with a wide spectrum of applications. Achieving high realism in generated motions requires naturalness, smoothness, and plausibility. However, current evaluation metrics often rely on simple heuristics or distribution distances and do not align well with human perceptions. In this work, we propose a data-driven approach to bridge this gap by introducing a large-scale human perceptual evaluation dataset, MotionPercept, and a human motion critic model, MotionCritic, that capture human perceptual preferences. Our critic model offers a more accurate metric for assessing motion quality and could be readily integrated into the motion generation pipeline to enhance generation quality. Extensive experiments demonstrate the effectiveness of our approach in both evaluating and improving the quality of generated human motions by aligning with human perceptions. Code and data are publicly available at https://motioncritic.github.io/.

Published as a conference paper at ICLR 2025
ALIGNING HUMAN MOTION GENERATION WITH HUMAN PERCEPTIONS
Haoru Wang
Wentao Zhu
Luyi Miao
Yishu Xu
Feng Gao Qi Tian'
Yizhou Wang
1 CCette on Frontiers of Computing Studies, School of Compter Science, Peking University
School of Arts, Peking University
IInt for Artificial Intelligence, Peking University HHuawei Technologies, Ltd.
6 Naa'll Eng. Research Center of Visual Technology, Peking University
6 "ttate Key Laboratory of General Artificial Intelligence, Peking University
ABSTRACT
Human motion generation is critical task with a wide range of applications. Achieving high realism in generated motions requires naturalness, smoothness, and plausibility. Despite rapid advancements in the field, current generation methods often fall short of these goals. Furthermore, existing evaluation metries typically errors, simple heuristics, or distribution distances,
rely on ground-truth-basee which do not align well with human perceptions of motion quality, In this work, we propose a data-driven approach to bridge this gap by introducing large-scale human perceptual evaluation dataset, Mot ontiieeer,,e,,, and a human motion critic model, cootinCrriiin that capture human perceptual preferences. Our critic model offers more accurate metric for assessing motion quality and could be readily integrated into the motion generation pipeline to enhance generation quality. Extensive experiments demonstrate the effectiveness of our approach in both evaluating and improving the quality of generated human motions by aligning with human perceptions. Code and data are publicly available at thtpps //motioncrii wwwoonniiiiiiiiiooooooooo wwt.nrriiiii............... 1o/]
INTRODUCTION
Human motion generation is an important emerging task (Zhu et al. (024) with wide-ranging applica- tions, including augmented and virtual reality (AR/VR) ( Yin et al. 2022 Yang et al. 22200), human- robot interaction (Nishimura et al. 2020 Gulletta et al. 2020), and digital humans (Kucherenko et al., 2019 Yi et al. 2023). Achieving high realism in generated human motions is crucial, necessitating naturalness, smoothness, and plausibility. However, current generation methods still fall short of these goals, often producing subpar results. Meanwhile, designing appropriate evaluation metrics that accurately reflect these qualities remains a significant challenge. This complexity stems from the highly non-linear and articulated nature of human motion, which must adhere to physical and bio-mechanical constraints while also avoiding visual artifacts. Effective metrics would not only facil- itate the objective comparison of generated results but also have the potential to enhance generation models by addressing their shortcomings.
Existing evaluation metrics typically rely on error with pairing ground truth (GT) motion, simple heuristics, or on distribution distance with real motion manifold. The error-based metrics cannot fully reflect the performance because GT is only one reasonable possibility. The heuristics fall short in comprehensively representing motion quality. For instance, foot-ground contact metrics (Rempe) et al. 2021 Tseng et al.) (2023) fail to penalize twisting arm motions that violate bio-mechanical constraints. It also infeasible to manually define the human motion rules in a handcrafted manner. Meanwhile, distribution distance metries like Fréchet Inception Distance (FID) (Heusel al. (017)) do not operate on an instance level but rather assess overall distribution similarity. Consequently, they cannot identify implausible motions or provide direct supervision signals guide the generation of higher-quality motions. Some studies TTeng et al. 2023 Voas et al.) (023) also indicate that FID) correlates poorly with user studies due to the misalignmen between its distance measurement and
LLad Authors.

---

Published as a conference paper at ICLR 2025
Train
MotionCritic
Critic Score - -6.875
6
Frozen
"Throw"
Motion Generator
MetionPercept
Figure 1:
Framework Overview. We collect MotionPercepe large-scale, human-annotated dataset for motion perceptual evaluation, where human subjects select the best quality motion in multiple-choice questions. Using this dataset, train Moti SettionCrtiii to automatically judge motion quality in aligmment with human perceptions, offering better quality metrics. Additionally, we show that Mot iocoriCiiicc can enhance existing motion generators with minimal fine-tuning.
human perception of motion quality. Consequently, existing automatic evaluation metries cannot effectively reflect or replace subjective user studies, hindering objective evaluation and comparison.
In light of this, advocate the need for automatic evaluation aligned with human perceptions. Firstly, humans are the primary audience and interaction partners for motion generation, making their perception crucial for evaluating motion quality. Secondly, the human brain possesses specialized neural mechanisms for processing biological motion (Blakemore & Decety 200T] Grossman et al. 2000) and is sensitive even slightly unnatural motions (Troje) [2002] Shimada & Oki (012). Therefore, we explore the possibility of directly learning perceptual evaluations from humans using data-driven approach. This method could bridge the gap between objective metrics and subjective human judgments, providing a more accurate assessment of motion quality.
First, we carefully curate human perceptual evaluation dataset named Mot potieeeet which contains 52563 pairs of human preference annotations on generated motions. Next, we train human motion critic model, MotionCrit. that learns motion quality ratings from the collected dataset. Our critic model significantly outperforms previous metrics in terms of alignment with human perceptions. Notably, generalizes well across different data distributions. In addition to motion evaluation, demonstrate that Mot i onCri i i can be seamlessly integrated into the generation training pipeline, effectively improving motion generation quality by increasing alignment with human perceptions with few steps of finetuning.
further propose to utilize the critic model as direct supervision signal. We
We summarize our contributions follows: We contribute Mot Motion ionPeerepe,, large-scale motion perceptual evaluation dataset with manual annotations. 2) We develop Mot inttiinriit which models human perceptions of motions through data-driven approach. Extensive experiments demonstrate its superiority as an automatic human-alignu the proposed motion critic model could effectively serve as supervision signal enhance motion generation quality. Remarkably, it requires only small number of fine-tuning steps and can be easily integrated into existing generator training pipeline in a plug-and-pla manner.
metric of motion quality. 3) We show that
RELATED WORK
2.1 HUMAN MOTION GENERATION
Human motion generation is pivotal task in computer vision, computer graphics, and artificial intelligence, aiming to produce natural and realistic human pose sequences (Zhu et al., 2024). This field has seen substantial advancements with the rise of deep generative models (Kingma & Welling 2014 Rezende & Mohamed 2015; Goodfellow et al. 2014 Ho et al. 2020). Previous works have explored text-conditioned motion generation that transform narrative descriptions into coherent pose sequences (Tevet et al., 2023 2022 Li et al. 2017: et al. 2022) Lucas et al. 2022), audio-
conditioned methods that synchronize movements with rhythmic cues (Huang et al. 202 Siyao et al.
2

---

Published as a conference paper at ICLR 2025
2022.
Tseng et al. 2023), , and sceme-conditooned generation that integrates environmental contexts to produce contextually ly appropriate motions (Corona et al. 2020; Wang et al. 2021: Araújo et al. 2023).
Despite significant progress, current mainstream data-driven kinematic motion generation methods sometimes produce unnatural motions that are jittery, distorted, or violate physiological and physical constraints. These issues could be attributed to the inherent uncertainty of the task, limitations of supervision signals, and dataset noises. Furthermore, evaluating generated human motions presents additional challenges. Traditional metrics like error and FID fail capture key aspects such as fluidity and biomechanical plausibility. Some works incorporate handcrafted physical priors (Tseng et al. 2023 Rempe et al. 2021). while others propose improved distance metrics for human poses or motions (Gopalakrishnan et al. 2019 Tanke et al. 2021 Tiwari et al. 2022). However, they still face limitations in fully and accurately reflecting human evaluations of motion quality. These challenges highlight the need for metrics that better align with human perception more effectively evaluate and improve motion generation results.
2.2 HUMAN PERCEPTION MODELING
Pioneer work Zhang et al. 2018) collect human perceptual similarity dataset and propose to utilize distance in deep features as perceptual metrics. Some works (Ouyang et al.] 2022] Bai et al.)
2022 Yuan et al.) 2023b Hejna & Sadigh| 2024 Dong et al. 2023) in language models to explore aligning model performance with human intent by first. training a reward model, then performing
reinforcement learning with the reward model. Recent works Yuuan et al., 2023a Wu et al. 2023
Lee et al. 2023) also explore utilizing human feedback to improve visual generation results. For example, ImwgeReward (Xu et al., 2024) propose a reward feedback learning method (ReFL) to to align text-to- text-to-image generative models with human judgements. In human motion generation, however, few studies have explored modeling human feedbacks, even though the generated motion quality is highly relevant to human perceptions. HuTuMotion (Han et al. 2024) proposes to leverage few-shot human feedhack with a set of representative texts. One recent work, MOBERT (Voas et al. 2023), constructs dataset of human ratings for generated motions. Our work differs from MOBERT in that we collect real human data on a scale tens of times larger (52.6K vs 1.4K) and use comparisons instead of ratings, which is more robust. We design the critic model learn ratings from these comparisons automatically. Additionally, our approach could not only evaluate motion quality but also effectively improve motion generation results.
MOTIONPERCEPT: A LARGE-SCALE DATASET OF MOTION PERCEPTUAL EVALUATION
We build Mot MotionPercept to capture real-human perceptual evaluations with large-scale and diverse human motion sequences. Hence, we implement a rigorous and efficient pipeline for data collection and data annotation. We also design a concensus experiment in order to examine the perceptual consistency across various human subjects.
3.1 MOTION DATA COLLECTION
We first collect generated human motion sequence pairs for subsequent perceptual evaluation. We utilize state-of of the-hheaad art diffusion-based motion generation method MDM (Tevet al., 2023) and FLAME (Kim et al. 2023) to generate human motion sequences parameterized by SMPL (Loper ct al. 2015). For MDM (Tevet et al.] 2023), we utilize the action-to-motion model trained on HumanAcr12 (Guo et al. 2020) and UESTC (Ji et al. 2018) respectively. For FLAAME (Kim et al. 2023), we utilize the text-to-motion model trained on HumanMI.3D (Guo et al. 2022). For each group of 4 motion sequences to be annotated, we use the same condition (text prompt or action labels) while sampling different random noises, with a same length of 60 frames, 24 fps. This makes the motions similar in content while still having distinguishable differences, thereby making it easier to annotate the choices.
3.2 HUMAN PERCEPTUAL EVALUATION
Human perceptual evaluation is the core component of MotionnPee iertrioneerepppe therefore we implement a rigorous pipeline to ensure annotation quality. We first introduce the question design of the perceptual
3

---

Published as a conference paper at ICLR 2025
1.00
250
250
0.75
200
200
Annotator ID
Groups
150
Groups
150
0.50
100
100
6.0 61
6.00 6.2 000
10
0.25
50
50
0.00
Annotators
Options
Annotator
(A): Number of Annotators Select-
(B): Number of Options Selected Per Question
(C): Pairwise Agreement
Ratio
ing the Most Preferred Option
Among Annotators
We conduct perceptual consensus experiment with 10 subjects on 312 multiple-choice questions, each with 6 options. (A): The distribution of the number supporters for the most chosen option in each question. (B): Distribution of the number of options chosen by all subjects for each question. (C): Pairwise agreement ratio of all subjects.
Figure 2:
evaluation, then describe the protocol for conducting the evaluation. Finally, we present statistical analysis of the evaluation results.
3.2.1 QUESTION DESIGN
Our perceptual evaluation designed in the form of multiple-choice questions as selection is generally casier and more robust than directly rating (Kendall 1948 Stewarrt et al., 2005 Voas et al.. 2023) Given group of four motion sequence options, we instruct the annotators to select the hest candidate that is most natural, visually pleasing, and free of artifacts. Specifically, we summarize the typical failure modes of the generated motions (e.g., jittering, foot skating, limb distortion, penetration, etc.) and explicitly require the annotators to exclude these options. We provide detailed guidance with task descriptions and representative video examples to better communicate the goal to the annotators. The full guidance is presented Appendix A..
While the optimal choice can be decided unambiguously in most cases, there are situations where the decision can be challenging. Therefore, we add two additional options, "all good" and "all bad", so that the annotator is not required to pick one of the motions in these cases, thereby improving overall annotation quality. Results indicate that these cases account for a small portion of the total data. We exclude these cases from our subsequent experiments. In total, we set six options for each entry: four motion candidates plus "all good" and "all bad".
3.2.2 PROTOCOLS
To ensure the quality of perceptual evaluation results, our annotation process consists of annotator training, annotation, and quality control. We recruit 10 annotators to perform the perceptual evaluation. Before the evaluation begins, we provide annotation guidelines to help the annotators understand the task and maintain consistent criteria. The annotators must pass pilot test before starting the formal annotation to ensure they correctly understand the annotation requirements. Additionally, we conduct perceptual consensus experiment to assess whether the annotation pipeline is suitable for dataset, as discussed in Section 3.3 Finally, we implement a quality control process where the annotated data reviewed by an expert quality inspector. During the annotation process, we continuously monitor the quality of each batch of data. For each batch, we randomly sample 10% of the data for quality inspection. The consistency between the sampled data and the expert's annotations must exceed 90%; otherwise, the entire batch will be re-annotated. Complete protocols are detailed in Appendix A
3.3 ANALYSIS
In total, we collect annotations for 18260 multiple-choice questions covering 73040 unique motions, significantly surpassing previous work (Voas al. 2023) (1400 motions). We further investigate the following two questions:
Based on our experimental setup, can the subjects confidently select the suitable options from the choices provided?
4

---

Published as a conference paper at ICLR 2025
2. Is there significant difference in perceptual preferences among different subjects, they well-aligned?
For the first question, we calculate the proportion of cases where a choice could not be made (including "all good" and "all bad"), and find total of 418 such groups (2.29%). The result indicates that most of the time subjects can make a definite judgment, demonstrating the validity of our protocol design.
For the second question. we conduct perceptual consensus experiment where all 10 subjects perform perceptual evaluation independently 312 groups of randomly selected data, We calculate their pairwise and overall consistency choices. Figures 2(A) and 2(B) show that for most questions (82.37%), all 10 subjects make the unanimous decision. Figure 2(C) reveals that all 10 subjects exhibit high pairwise agreement (90%). These results indicate a high level consistency in perceptual judgments of human motion among different human subjects. This not only validates the rationality of our perceptual evaluation pipeline but also inspires us to train machine learning models to emulate this consistent judgment capability.
MOTIONCRITIC: ADVANCING MOTION GENERATION WITH PERCEPTUAL ALIGNMENT
Based on Mot i wetiinneeeeeppt we develop a human motion critic model, Mct MctionCritin cinnriitic to emu- late the perceptual judgment capabilities human subjects regarding human motion. We first present the problem formulation and training approach of the critic model, and then explain how to the critic model for optimizing motion generation.
4.1 PROBLEM FORMULATION
given an input human motion sequence x, we assume there implicit human perception model H that rates the motion quality H(x), where higher rate indicates better quality. We aim to build a computational critic model that best aligns with H. Since H is not explicitly available, take data-driven approach. We obtain the human perceptual evaluation dataset D containing multiple pairs of samples (x((), x())). Our training objective is to train the model C using the dataset D so that it approximates the human perception model H. as closely as possible. Specifically, we want the model prediction C(x(9) C(x ) if and only H(x(0)) > H))())). Based on the Bradley-Terry model (Bradley & Terry 1952, Hunter 2004), the overall training objective could be written maximizing the joint probabilities that the model C makes judgments consistent with H for each pair of samples in the dataset D:
We formulate the problem as follows:
arg max E x() [logo ((C(x()) -(((()) (H(x!)) H(x())))
(1)
where a is the sigmoid function.
4.2 HUMAN MOTION CRITIC MODEL
In practice, we represent human motion by LXXxD where L denotes the sequence length, J denotes the number of body joints, and denotes parameter dimensions. We implement the critic model C as neural network that maps the high-dimensional motion parameters to scalar We draw pairwise comparison annotations from the collected dataset, where xh) is the better instance and x(( is the worse. The perceptual alignment loss is thus given by:
Cpeccept = - (B xD) llo o (c(x(()) - c(x("))]
4.3 MOTION GENERATION WITH CRITIC MODEL SUPERVISION
Additionally, we explore to utilize the learned human perceptual prior of C not only for evaluating generated motions, but also improving them. We demonstrate that our motion critic model could
5

---

Published as a conference paper at ICLR 2025
MotionCritic
((xela)),C(xell ... Lpeeceps
x(k
x-0
C
eval, grad
train, grad
X7
X₁
X
MDM
x
MotionCritic
C(x)) ... CCritic
Predict
LKL.
C
x
Figure 3: (I) Critic model training process. We sample human motion pairs x(x), xx()) annotated with human preferences, upon which the critic model produces score pairs. use perceptual alignment loss Percep to learn from the human perceptions. (II) Motion generation with critic model supervision. We intercept MDM sampling process at random timestep and perform single-step prediction. Critic model computes the score based on the generated motion x/,, which is further used to calculate motion critic loss Lenie- KL loss LKL is introduced between x/ and last-time generation result x,y
be integrated into state-of-the-arr diffusion-basee motion generation approaches with ease by using MDM (Tevet et al. (003) as an example. The forward diffusion is modeled as a Markov noising process (x Yizo where is drawn from the data distribution, and
((x((x,1) = -N(a,, (1 az))),
where a E (0, 1) are constant hyper- ppper-parameters. When a is small enough, it's reasonable to approximate XT N00, f), allow sampling Xy from random noise to begin denoising process.
Given an MDM model M with pre- trained parameters dg fine-tune to improve its alignment with a pre- trained critic model C. We develop lightweight perceptual-aligned fine-
Algorithm Fine-tuning Motion Generation with Mithoooiiiiiiii
1: Dataset: Action-label set = (label, mot;)}
tuning approach based on ReFL (Xu) et al. 2024). Notably, in order to uti- Tize the critic model in a plug-and- play manner, we keep the MDM training step and objective LMDM unchanged. Of each fine-tuning iter-
2: Input: MDM model Mag Critic model C, Critic-to-los map function ø, Critic loss scale X, KI. loss scale μ
3: for (label;, mot;) E D do
4:
+
Update MDMo, with CMDM Pick random time step t [Ti, Ta > Sample noise
t 6- (Ti, T2)
6:
Xr ~ NO,,
7:
for jorj=1 T...., + do
ation, the original MDM loss is com-
8:
grad: x,-1 Ma; (xx} end for
puted using text labels and ground- 9:
truth motions as the initial step. This
10:
with grad: No + (o,
loss is first utilized to update the pre- trained model weights accordingly.
11:
xo is not None then
12:
LXL + HKL xx)
KL loss with previous x
13:
end
After that, we simply add another
14:
Lemie + Xp(C(xhh)
> Critic loss
optimization step with critic model
15:
O.t1 <- 0:
Update Mo, with Leai and Lxt.
supervision. Structure of this opti-
16:
Xo +- X6
Save xa for next-step Cx
mization step is shown in Figure
17:
end for
and detailed as follows:
We begin with sampling Gaussian noise XT and performing gradient-free denoising steps until X1, where E [T) is randomly selected in denoising steps. Gradient interception range (Ti, Ta]

---

Published as a conference paper at ICLR 2025
hyperparameters, with selection principles outlined in Appendix C.1 After reaching step t,
single-step denoising with gradient is performed directly to predict x from x,- Upon the predicted motion X/, we compute its critic score = C(x)), and use this score output to compute the motion critic loss. We formulate our critic loss as follows:
Leiti By (o(C(x)))
where (ss) = -r -5-))) s)) is critic-to-loss mapping function, being constant threshold for shifting the critic value, and a being sigmoid function.
We further introduce Kullback-Leibler (KL) divergence regularization prevent M from moving substantially away from the conditional motion generation task. We formulate our KL loss as follows:
LKL. = Ey,y DKLL (p(x))ll((x))]
where X/1 being X of the previous iteration. Overall, our fine-tuning loss is given by
LFT LMDM XCeriti pLKI.-
(6)
where À and j are re-scaling weights for loss balancing. Detailed algorithm workflow shown in Algorithm
5
EXPERIMENT
5.1
IMPLEMENTATION DETAILS
Critic Model. We train our critic model using the MDM subset in MotionPercept. We convert each multiple-choice question into three ordered preference pairs, which results in 46740 pairs for training and 5823 pairs for testing. We parameterize motion sequences with SMPL (Loper et al. 2013), including 24 axis-angle rotations, and global root translation. We implement the critic model with DSTformer (Zhu et al. 2023) backbone with 3 layers and 8 attention heads. We apply temporal average pooling on encoded motion embeddings followed by an MLP with hidden layer of 1024 channels to predict single scalar score. We train the critic model for 150 epochs with a batch size of 64 and a learning rate starting at 2e-3, decreasing with a 0.995 exponential learning rate decay.
Fine-tuning. We use MDM (Tevet et al. 2023) model trained on HumanAct12 (Guo et al. 2020 as our baseline, which utilizes 1000 DDPM denoising steps. We load the checkpoint trained for 350000 iterations and fine-tune for 800 iterations, with batch size of 64 and learning rate le-5, We fine-tune with critic clipping threshold - = 12.0, critic re-weight scale X = =1e-3, and KL loss re-weight scale 1.0. We set the step sampling range [Ti, TT] [700, 900]. Details of the setup shown at Appendix C.T.
5.2 MOTIONCRITED AS MOTION QUALITY METRIC
We first evaluate whether the proposed critic model could serve as effective motion quality metric. Specifically, we are interested in the following research questions:
1. How does MotionCritic align with human perceptual evaluations?
2. Could MctionCritic generalliieegg to different data distributions?
To investigate the first question, we evaluate the performance of our critic model on a held-out test set and compare it with existing motion quality metrics follows:
Distance-based metries, including Root Average Error (Root AVE), Root Absolute Error (Root AE), Joint Average Error (Joint AVE), and Joint Absolute Error (Joint AE). These metrics involve directly computing the distance between the generated motion and a pairing GT with the same condition. Furthermore, methods like PoseNDF (Twari al. 2022 Nor- malized Power Spectrum Similarity (oosasaGooallanrishnann et al. 2201)), and Normalized Directional Motion Similarity (NDMS) al. 2021) offer more advanced approaches measuring distances between motions.
7

---

Published as a conference paper at ICLR 2025
Table 1: Quantitative comparison of motion evaluation metrics on MDM and FLAME testsets of
Motion MationPercept.
MDM
FLAME
Metric
Acc. (%) 1
Log Loss
Acc. (%)
Log Loss
Root AVE
59.47
0.6891
48.
0.6984
Root AE
61.79
0.6798
59.54
0.6711
Joint AVE
56.77
0.6889
44.61
0.6973
Joint AE
62.73
0.6794
58.37
0.6891
Jerk
65.48
0.7516
65.84
0.5984
Acceleration Yang Person-Ground Contact [Kempe al
2023
64.26
0.7792
66.67
0.6919
2021
71.78
0.7260
69.82
0.7243
Foot-Floor Penetration [Rempe el
2021
53.61
0.6939
55.56
0.6906
Physical Foot Contact seng cl
202.3
64.79
0.6926
66.00
0.6930
PoseNDF (Tiwari al. 2002
55.13
0.6930
53.07
0.6931
NPSS (Gopalakrishaan ct 2019
52.37
0.6911
52.07
0.6944
NDMS (Tanke al.
202
63.92
0.6519
63.35
0.6301
MoBERT (Voas et al. 2023
49.40
0,6931
52.40
0.6932
MoticnCratic ( (Ours)
85.07
0.5486
81.43
0.5758
2000
MoBonnCiik
-2.0
44,0
1000
-6.0
-8.0
-20
-15
-10
5
10
(A)
(B)
Figure 4: We group HumanAet12 et al. 2020) GT test set into 5 subsets, and compare their qualities. (A): GT-I to GT-V subsets split based on critic scores from high to low. (B); Elo ratings from user study, FID and average critic scores of different GT subsets.
Heuristic metrics, including acceleration (Rempe al. 2021 Yang et al. 2023 Person-
Ground Contact (Rempe et al., 2021), Foot-Floor Penetration (Rempe et al.) 2021 and Physical Foot Contact (PFC) (Tseng et al. 2023). These metrics does compare against GT; instead, they implement intuitive rule-based evluations. For example, PFC models the relationship between center of mass acceleration and foot-ground contact.
Learning-based metrics. Prior work MoBERT (Voas et al. 2023 proposes evaluate motion quality with a motion feature extractor and SVR Regression.
Note that distribution-based metrics (e.g. FID) could not compare quality of individual motion sequences, and the comparison can be found in subsequent experiments. For each metric, calculate the percentage they align with GT annotations (accuracy) and also their probabilistic distribution distance with GT annotations (log loss). We use the softmax function convert the scores to probabilities (taking the opposite before softmax for metrics where smaller is better). Table T demonstrates that our critic model significantly outperforms previous metries. These results not only validate the effectiveness of learning from large-scale human perceptual evaluations but also prove that our critic model can serve as a more comprehensive and robust metric for assessing motion quality.
Furthermore, to investigate the second question, we test the critic model on data outside of the training distributions. We collect standalone test set from 804 motions with different motion generation algorithm, FLAME (Li et al. 2017). and perform perceptual evaluation with a different human subject. Note that this model is trained different dataset (Guo et al. 2022) with the model
8

---

Published as a conference paper at ICLR 2025
Figure 5: Model performance during fine-tuning process. (A): User study win rates (row vs column) with different fine-tuned model steps. (B): Elo ratings from user study, FID and average critic scores in the fine-tuning process.
step
0.50
0.38
0.38
0.25
0.17
3000
-3.0
step 200
0.88
0.50
0.38
0.88
0.22
2500
2.0
MationCrith
2000
step 400
0.58
0.30
0.17
0:14
1.0
step
0.38
0.75
0.0
-10
step
5.75
0:04
0.75
0.90
MD NatienCitt
-20
500
stapo
stop 200
step 400
2n0 9000
2000 108
200
400
600
800
Fine-tuning Steps
(A)
(B)
step400
sacy800
Figure 6: Motion generation results from different fine-tuning steps.
used to generate critic model training data, which means the action categories have large variations. further shows that our critic model could well generalize to the new test set,
The results in Table D indicating its efficacy in evaluating different generation algorithms and unseen motion contents.
Additionally, we test the generalization of our critic model on the real GT motion distribution. Figure (aaree4())llllllsl illluttrates the critic score distribution of HumanAct12 (Guo et al. 2020) test set. We group the 1190 GT motions into 5 groups based on their critic scores, evenly distributed from highest to lowest. We compare the average critic score between the groups with distribution based metric FID and user study. The user study is conducted by another 5 users different from previous annotators, comparing motion pairs sampled from each groups and then computing Elo rating (Elo 1978 Tseng let al. 2023) for each group. Figure ([a)) [BB]ccleary indicates that the critic score aligns well with human preferences, while FIDD does not. Notably,
discover that the outliers with small critic values (group V) are indeed artifacts within the dataset. Please refer to to Appendix for details in user study, and the supplementary materials for video results.
The results indicate that our critic model can also generalize to the GT motion manifold, even though the model has never been trained on it. It also highlights the potential of using our critic model as a tool for dataset diagnosis (e.g., discover failure modes). Please also refer to Appendix B for additional results, where we provide more discussions on Mot inctiiinrii t ic, including failure modes, out-of-distributtin tests, the impact of different motion lengths, and its relationship with other metrics.
5.3 MOTTONCRITIC AS TRAINING SUPERVISION
Furthermore, we investigate whether our critic model can also serve as an effective supervision signal. Specifically, fine-tune pre-trained motion generator (TTvet al. 2023) with the proposed

---

Published as a conference paper at ICLR 2025
Table 2: Comparison of motion generation metrics at different fine-tuning steps.
Step
FID
PFCL
MctionCritic
Accuracyt
Diversity?
Multimodality!
0.13
0.00095
-1.64
0.98
6.66
2.15
100
0.17
0.00086
1.83
0.98
6.62
2.32
200
0.14
0.0010
1.79
0.98
6.58
2.46
300
0.14
0.0010
2.35
0.98
6.56
2.36
400
0.11
0,0012
2.54
0.98
6.61
2.63
500
0.17
0.0013
2,61
0.98
6.62
2.36
600
0.18
0.0013
2.78
0.98
6.68
2.42
700
0.54
0.0017
3.45
0.92
6.53
2.43
800
0.50
0.0012
2.73
0.88
6.57
3.04
framework, and evaluate on HumanAct12 (Guo et al. 2020) test set every 200 steps. Additionally, we conduct standalone user study by comparing motion pairs generated at different fine-tuning steps and compute the Elo Rating (Elo 1978 Tseng et al., 2023). Figure reveals that as fine-tuning progresses, the motion quality consistently improves according to the user study, in line with the training objective of increasing the critic score. The user study conducted with new evaluators which not participated in dataset annotation. Plese refer Appendix (Door details of the user study. We also present a visualization comparison in Figure We discover that as fine-tuning progresses, unreasonable human motions such jittering, twisting, and floating significantly decrease. Please refer to the supplementary materials for video comparisons.
In addition, examine the impact of fine-tuning with Mot MotionCritic on other motion evaluation metrics, We track changes in various metrics, including Accuracy. Diversity, and Multimodality [Tevet al. 2023). ) over the course of fine-tuning from 0 800 steps. These metrics capture different aspects of motion generation, such as text alignment and richness. Table shows that our critic model aligns more closely with human judgment compared motion quality metrics like FID and PFC. Moreover, does not conflict with other key metrics. For example, at 600 steps, we observe improvements in Accuracy, Diversity, and Multimodality, along with significant increase in user preference compared to the baseline (step 0). In conclusion, Mot i onnri icc provides a more holistic evaluation of the intrinsic quality of the motion compared to other metrics in this domain. At the same time, it complements other metrics that focus different aspects, such as text-motion consistency and diversity, resulting in a more comprehensive and objective evaluation.
fine-tuning with
The results also demonstrate that fine-tune fine-tuning process requires only hundreds of iterations to take effect, significantly improving the perceptual quality of the model. Compared to the 350K pre-training steps, this accounts for only 0.23% of the training cost. This further demonstrates the advantages of our proposed framework in using perceptually-aliget motion generation model, not only improving quality but also being lightweight and efficient. Please refer to Appendix C for more details and analysis of fine-tuning.
critic model to fine-tune the
6 CONCLUSION
In conclusion, our work bridges the important gap in human motion generation between ob- jective metrics and human perceptual evaluations by introducing a data-driven framework with Mot MotionPed ecept and Moti oncritic.. This paradigm not only offers more comprehensive metrics motion quality but could also improve the generation results by aligning with human preferences. We hope this work could contribute more objective evaluations of motion generation methods and results.
To prevent potential misuse or misunderstanding, it is important to note that the critic model should be used either as an standalone evaluation metric or as a loss function for fine-tuning, but not for both simultaneously. It's also worth noticing that the energy landscape of Mot mroionCritic is not smooth, and Mot i onCrit t should not be interpreted Appendix B.3 Another limitation of our approach is its primary focus on perceptual metrics without explicitly simulating physical and biomechanical plausibility, which could be explored in future work. Future research could also investigate more fine- grained perceptual evaluation methods to obtain rich human feedback on motion quality like (Liang al. 2024
distance measure. Discussions are detailed in
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGMENTS
This work was supported by the National Key Research and Development Program of China (No. (022YFF0902002 and NSFCC6247070122. We extend our gratitude to the reviewers for their insightful comments and valuable discussions.

---

Published as a conference paper at ICLR 2025
BIBLIOGRAPHY
Joao Pedro Araújo, Jiaman Li, Karthik Vetrivel, Rishi Agarwal, Jiajun Wu, Deepak Gopinath, Alexander William Clegg, and Karen Liu. Circle: Capture in rich contextual environments. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit., pp. 21211-21221, 2023.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204. 05862, 2022.
Sarah-Jayne Blakemore and Jean Decety. From the perception of action to the understanding of intention. Nature reviews neuroscience, 2(8):561-565 -677 2001.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4)3324 345, 1952.
Kevin Clark, Paul Vicol, Kevin Swersky, and David Fleet. Directly fine-tuning diffusion models on differentiable rewards, 2024. URL
Enric Corona, Albert Pumarola, Guillem Alenya, and Francesc Moreno-Noguer. Context-aware human motion prediction. In Proc. IEEE Conf. Comput. Vis. Puttern Recognit, pp. 6992-7001, 2020.
Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06767 2023.
Arpad E. Elo. The Rating of Chessplayers, Past and Present. Arco Pub., 1978.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Proc. Adv. Neural Inform. Process. Syst., 2014.
Anand Gopalakrishnan, Ankur Mali, Dan Kifer, Lee Giles, and Alexander G Ororbia. A neural temporal model for human motion prediction. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit., pp. 12116-12125, 2019.
Emily Grossman, Michael Donnelly, R Price, D Pickens, V Morgan, G Neighbor, and Randolph Blake. Brain areas involved in perception of biological motion. Journal of cognitive neuroscience, 12(5):711-720, 2000.
Gianpaolo Gulletta, Wolfram Erlhagen, and Estela Bicho. Human-like arm motion generation: A review. Robotics, 9(4): 2020.
Chuan Guo, Xinxin Zuo, Sen Wang, Shihao Zou, Qingyao Sun, Annan Deng, Minglun Gong, and Li Cheng. Action2motion: Conditioned generation of 3d human motions. In Proc. ACM Int. Conf. Multimedia, pp. 2021-2029, 2020.
Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In Proc. IEEE Conf. Comput. Vis. Paltern Recognit., pp. 5152-5161, 2022.
Gaoge Han, Shaoli Huang, Mingming Gong, and Jinglei Tang. Hutumotion: Human-tuned navigation of latent motion diffusion models with minimal feedback. In Proceedings the AAAI Conference on Artificial Intelligence, volume 38, pp. 2031-2039, 2024.
Joey Hejna and Dorsa Sadigh. Inverse preference learning: Preference-bee baaerence-bbsedd rl without reward function. Advances in Neural Information Processing Systems, 36, 2024.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Proc. Adv. Neural Inform. Process. Syst., volume 30, 2017.
12

---

Published as a conference paper at ICLR 2025
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Proc. Adv. Neural Inform. Process. Syst., 2020.
Ruozi Huang, Huang Hu, Wei Wu, Kei Sawada, Mi Zhang, and Daxin Jiang. Dance revolution: Long- term dance generation with music via curriculum learning, In Proc. Int. Conf. Learn. Represent., 2021.
David R Hunter. Mm algorithms for generalized bradley-terry models. The annals of statistics, 32(1): 384-406 406, 2004.
Yanli Ji, Feixiang Xu, Yang Yang, Fumin Shen, Heng Tao Shen, and Wei-Shi Zheng. A large-scale rgb-d database for arbitrary-view human action recogmition. In Proc. ACM Int. Conf. Multimedia, pp. 1510-1518, 2018 8. ISBN 978 1450356657. doi: 10. 145/3240508.32406773
Gunnar Johansson. Visual perception of biological motion and a model for its analysis. Perception & Psychophysics, 14(2):201-21 221, 1973. doi: N0.3758/BF03212378.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit, pp. 4401-4410, June 2019.
Maurice George Kendall. Rank correlation methods. 1948.
Jihoon Kim, Jiseob Kim, and Sungjoon Choi. Flame: Free-form language-based motion synthesis & editing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 8255 8263, 2023.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In Proc. Int. Conf. Learn. Represent., 2014.
Taras Kucherenko, Dai Hasegawa, Gustav Eje Henter, Naoshi Kaneko, and Hedvig Kjellström. Analyzing input and output representations for speech-driven gesture generation. In Proc. Int. Conf. on Intelligent Virtual Agents, pp. 97-104, 2019.
Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback. arXiv preprint arXiv:2302. 12192, 2023.
Tianye Li, Timo Bolkart, Michael J Black, Hao Li, and Javier Romero. Learning a model of facial shape and expression from 4d scans. ACM Trans. Graph., 36(6): 194-1, 2017.
Youwei Liang, Junfeng He, Gang Li, Peizhao Li, Arseniy Klimovskiy, Nicholas Carolan, Jiao Sun, Jordi Pont-Tuset, Sarah Young, Feng Yang, Rich human feedback for text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19401-19411 2024.
Xiaoyang Liu, Yunyao Mao, Wengang Zhou, and Houqiang Li. Motionrl: Align text-to-motion generation to human preferences with multi-reward reinforcement learning, 2024. URL https
//azxiv.org/abs///111.. 06513
Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael Black. Smpl: A skinned multi-person linear model. ACM Trans. Graph., 34(6):1-16, 2015.
Thomas Lucas* Fabien Baradel*, Philippe Weinzaepfel, and Grégory Rogez. Posegpt: Quantization- based 3d human motion generation and forecasting. In Proc. Eur. Conf. Comput. Vis., pp. 417-435, 2022.
Yusuke Nishimura, Yutaka Nakamura, and Hiroshi Ishiguro. Long-term motion generation for interactive humanoid robots using gan with convolutional network. In Proc. ACM/IEEE Int. Conf. on Human-Robot Interaction, pp. 375-377, 2020.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Proc. Adv. Neural Inform. Process. Syst., 35:27730-27744, 2022.
13

---

Published as a conference paper at ICLR 2025
Mathis Petrovich, Michael J. Black, and Güi Varol. TEMOS: Generating diverse human motions from textual descriptions. In Proc. Eur. Conf. Comput. Vis., pp. 480-497, 2022.
Davis Rempe, Tolga Birdal, Aaron Hertzmann, Jimei Yang. Srinath Sridhar, and Leonidas J Guibas. Humor: 3d human motion model for robust pose estimation. In Proceedings of the IEEE/CVF international conference on computer vision, pp- 11888-1 1499, 2021.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Proc, Int. Conf. Mach. Learn., 2015.
Sotaro Shimada and Kazuma Oki. Modulation of motor area activity during observation of unnatural body movements. Brain and cognition, 80(1):1-6, 2012.
Li Siyao, Weijiang Yu, Tianpei Gu, Chunze Lin, Quan Wang, Chen Qian, Chen Change Loy, and Ziwei Liu. Bailando: 3d dance generation via actor-critic gpt with choreographic memory. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit., pp. 11050-11059, June 2022.
Neil Stewart, Gordon DA Brown, and Nick Chater. Absolute identification by relative judgment. Psychological review, 112(4):811, 2005.
Julian Tanke, Chintan Zaveri, and Juergen Gall. Intention -based long-term human motion anticipation. 202/ International Conference on 3D Vision (3DV), pp. 596-605, 2021. URL httphttps:// /api semant cscholar wttp:r//rooooooo/rooor
Guy Tevet, Brian Gordon, Amir Hertz, Amit H Bermano, and Daniel Cohen-Or. Motionclip: Exposing human motion generation to clip space. In Proc. Eur. Conf. Comput. Vis., pp. 358-374, 2022.
Guy Tevet, Sigal Raab, Brian Gordon, Yoni Shafir, Daniel Cohen-or, and Amit Haim Bermano. Human motion diffusion model. In Proc. Int. Conf. Learn. Represent., 2023.
Garvita Tiwari, Dimitrije Antić, Jan Eric Lenssen, Nikolaos Sarafianos, Tony Tung, and Gerard Pons-Moll. Pose-ndf: Modeling human pose manifolds with neural distance fields. In Proc. Eur. Conf. Comput. Vis., pp. 572-589, 2022.
Nikolaus Troje. Decomposing biological motion: A framework for analysis and synthesis of human gait patterns. Journal of vision, 2(5):2-2, 2002.
Jonathan Tseng, Rodrigo Castellon, and Karen Liu. Edge: Editable dance generation from music. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit., pp. 448-458, 2023.
Jordan Voas, Yili Wang, Qixing Huang, and Raymond Mooney. What is the best automated metric for text to motion generation? In SIGGRAPHI Asia 2023 Conference Papers, pp. 1-11, 2023.
Jiashun Wang, Huazhe Xu, Jingwei Xu, Sifei Liu, and Xiaolong Wang. Synthesizing long-term 3d human motion and interaction 3d scenes. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit., pp. 9401 1411, 2021.
Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint arXiv:2306.00 09341, 2023.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024.
Sicheng Yang, Zhiyong Wu, Minglei Li, Zhensong Zhang, Lei Hao, Weihong Bao, and Haolin Zhuang. Qpgesture: Quantization- and phase- guided motion matching for natural speech- driven gesture generation. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2023.
Zhuoqian Yang, Wentao Zhu, Wayne Wu, Chen Qian, Qiang Zhou, Bolei Zhou, and Chen Change Loy. Transmomo: Invariance-drive unsupervised video motion retargeting. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2020.
14

---

Published as a conference paper at ICLR 2025
Hongwei Yi, Hualin Liang, Yifei Liu, Qiong Cao, Yandong Wen, Timo Bolkart, Dacheng Tao, and Michael Black. Generating holistic 3d human motion from speech. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2023.
Tairan Yin, Ludovic Hoyet, Mare Christie, Marie-Paule Cani, and Julien Pettré. The one-man-crowd: Single user generation of crowd motions using virtual reality. IEEE Trans. Vis. Comput. Graph., 28(5):2245-222 2255,, 2022.
Hangjie Yuan, Shiwei Zhang, Xiang Wang, Yujie Wei, Tao Feng, Yining Pan, Yingya Zhang, Ziwei Liu, Samuel Albanie, and Dong Ni. Instructvideo: Instructing video diffusion models with human feedback. arXiv preprint arXiv:23 12490, 2023a.
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang. Songfang Huang, and Fei Huang. Rrhf: Rank responses align language models with human feedback without tears. arXiv preprint arXiv:2304.05302, 2023b.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and paitern recognition, pp. 586-595, 2018.
Wentao Zhu, Xiaoxuan Ma. Zhaoyang Liu, Libin Liu, Wayne Wu, and Yizhou Wang. Learning human motion representations: unified perspective. In Proc. Int. Conf. Comput. Vis., 2023.
Wentao Zhu, Xiaoxuan Ma, Dongwoo Ro, Hai Ci, Jinlu Zhang, Jiaxin Shi, Feng Gao, Qi Tian, and Yizhou Wang. Human motion generation: A survey. IEEE Transactions Pattern Analysis and Machine Intelligence, 2024.
15

---

Published as a conference paper at ICLR 2025
Part II
Appendix
A DETAILS ON MOTIONPERCERT
A.1 DATA GENERATION
We use synthetic data for annotating Mot dotionns iootineercep,, because we hypothesis that using options from the same synthetic distribution would be most effective for developing robust and generalizable critic model. Incorporating GT data, which may be of significantly higher quality, could make too easy for the model to differentiate between options, potentially reducing its discrimination power. This practice aligns with approaches training reward models for language models like GPT (Ouyang itt al., 2022), where reward models are trained using human feedback on model-generated outputs. note that the critic model trained subsequently produced reasonable results on out-of-distribtioon datasets including GT motion, demonstrating its ability generalize beyond the synthetic data.
We utilize the prompts from HumanAct12 (Guo et al. 2020), UESTC (Ji et al. 2018) and Hu- manML3D (Guo et al. 2022) for generating the motion candidates. Specifically,
the 12 action labels from HumanAct (Guo et al., 2020) (shown in Table 3) and the 40 categories of aerobic exercise description from UESTC (Ji et al., 2018) (shown in Table 4) for the MDDM (Tevet al.. 2023) model. We randomly select texts from HumanML3D (Guo et al. 2022) test set as prompts for the FLAME (Li al. 2017) model.
HumanAct12 (uo et al.
2020) Action Labels
warm up
walk jump
drink
lift dumbbell
eat
turn steering wheel boxing
phone throw
Table 3: 12 action labels from HumanAct12 (Guo et 2020
UESTC al. 2018) Action Labels
punching and knee lifting
marking time and knee lifting squatting
jumping-jack
forward-lunginn
left-lunging
left-stretching
raising-hande baand-and-jumping
left-kicking
rotation-clapping
front-raising
pulling-chest-expaader
punching
wrist-circling
iigle-dumbbeellraisiin
shoulder-raising
elbow-circling
arm-circling
dumbbell-shrugginn
pinching-back
head-aniicleckwiseccccchii
shoulder-abduction
delloiddmmuscle-sttechhhngg
straight-forward haarwadd-fixionn
dumbbell-side siel---de-bbeeee bend
coaddin--pposite-ee
wwwposstteeebo--eeeee seolow-t---eeeeeeee to-knne-crunch
standing- rotation
wverhead-stretching
upper-back-strentt stretchhnn
knee-to-chest
knce-circling
alternate-knco-lifiie
bent-over-twist
rope-skipping
moadiing-toe-oucchss
Maanding-gastrocnenntt
hungleleg-lateral-hopiiee
high- seggh-nnees-uuning
Table 4: 40 action labels from UESTC (Ji al. 2018
16

---

Published as a conference paper at ICLR 2025
A.2 ANNOTATION MANAGEMENT
We recruit 10 annotators for this task, and data entries are randomly allocated to them. We provide detailed guidelines to annotators. We evaluate the annotation result by spot check. We randomly select 10% of all data to inspect the annotation results according to guidelines and calculate the proportion of unqualificd data entries. the unqualified proportion is less than 10%, the results are considered to be acceptable. All the unqualified data entries will be re-annotated. We will update the guidelines during annotation based on spot check feedback, and annotators will study the new guidelines.
A.3 ANNOTATION DESIGN
A
B
C
D
Figure 7: An example of raw data entry before annotation.
We generate four motions from the same prompt for each data entry, shown in Fig The prompts are hidden during the annotation process. Annotators are required to select either the best or the worst motion for data entries generated by MDM (Tevet et al., 2023) and FLAME (Kim al., 2023). MDM (Tevet et al. 2023) exhibits better motion diversity but lacks stability, so annotators instructed to select the best motion. Conversely, FLAME (Kim al. 2023) demonstrates better stability but lacks diversity, so annotators are instructed to select the worst motion for these entries.
speed up sllow down
A
B
C
propecties
info
00:00000301
dunition
ypbex
O
aal good
whether this data entry valid
options
Figure Our annotation platform.
A.4 ANNOTATION GUIDANCE DOCUMENTATION
We provide a detailed annotation document to explain the annotation process. The annotation platform shown in Fig
17

---

Published as a conference paper at ICLR 2025
Introduction
Each data entry to be annotated consists of four videos, as shown in Fig 7 Each video approximately three seconds long, with all four videos playing simultaneously and concatenated into one video.
Requirements Each set of videos has six options: A, B, C, D, "all are good," and "all are bad." Annotators should select the most natural and reasonable video for each data entry. one option stands out as the best, select that option. If all actions seem equally good or equally bad, choose "all are good" or "all are bad." Text prompts will be hidden during annotation.
Video Examples We provide annotators with examples if what kinds of motions are unnatural and unaccepetable:
1. Body pose unnatural, including hands, feet and so on.
2. Human motion violates physiological constraints.
3. Human motion is erratic severely stutters.
4. Human body collides, such as hands fully embedded into leg.
5. Human body severely tilted, to the point of losing balance.
6. Human body appears be drifting instead of walking.
Examples of these problems shown in Fig
18

---

Published as a conference paper at ICLR 2025
feel unnatural
foet unnatural
hands unnatural
legs unnatural
human body levitatinn
hands movement erratic and stutters
left hand and left leg collision
lamma body tilted
human body drifting
Figure Representative examples of options be excluded.
19

---

Published as a conference paper at ICLR 2025
B DETAILS ON MCTIONCRITIC: AS MOTION QUALITY METRIC
B.1 IMPLEMENTATION DETAILS
Data Pre-processing. Each multiple-choice question is divided into three ordered preference pairs. Motion sequences parameterized using SMPL (Loper al. 2015), which includes 24 axis-angle rotations and one global root translation.
Training and Evaluation. We train the critic model from scratch using the DSTformer (Zhu et al. 2023) backbone with 3 layers and 8 attention heads on Mot MotionPere.pt. To ensure robustness, we train our model for multiple times and report the error bars, considering variations such as the random seed across multiple runs. Evaluation results, detailing action-label splits, are presented in the following two tables. Our Mot potionCritic gets the best results and can robustly score different types of human motions.
Metric
Warm.
Walk
Run
Jump
Drink
Lif:
Sit
Eat
Tum.
Phune
Bos.
Throw
Avg.
Root AVE
57.6
47.3
56.8
62.7
59.5
46.3
37.9
64.5
54.1
0.62
51.6
53.7
59.5
Root
57.2
70.0
49.8
52.5
61.8
AVE
50.4
53.2
50.2
42.4
45.2
Joun
59.9
51.4
Accelentionn
2023
53.0
65.4
$2.6
59.5
64.3
Person Ground Contet
Xemns Celle
72.8
90.1
71.8
Floor Penetration
Somie A094
55.7
52.4
53.6
Physical Foot Contact
[segg
51.7
67.5
57.9
McBERT Voss
37.4
70.0
18.9
43.6
49.4
SptionCritic (Ous)
90.6102 94.2 .4 91.9 90.6.a
83.
85.3
86.0.a7 78.3
79.8.00 85.1 86.6.na 82.5.13 85.1.03
Table 5: Accuracy comparison of motion evaluation metrics on HumanAct12 action classes(%).
Waxm.
Run
Jump
Sii
Tum.
Plust
Bus.
Threw
Avg.
Rust
0.70
0.60
0.69
0.68
0.69
0.69
0.59
0.89
0.69
0.69
0.69
01.99
0.69
Joint AE
0.68
0.67
0.67
0.59
0.68
0.70
0.70
0.69
0.69
0.66
0.68
0.69
0.68
Aaxlention
You as a 2023)
0.70
0.59
0.58
1.5
0.60
0.69
0,60
0.64
0.88
0.71
0.61
0.76
0.78
Person Grran Conter Keer 213 POT
0.21
0.68
0,68
0.71
0.69
0.20
0.73
0.68
0.24
0.70
0.75
0.22
0.73
Fout-Flace Pesctration
0.70
0.70
0.69
0.70
0.69
0.70
0.71
0.70
0.69
0.70
0.69
0.69
0.69
Physccal Foor Contact
Yenig 14 1r 112Y
0.69
0.69
0.00
0.59
0.60
0.69
0.89
0.09
0.69
0.69
0.69
0.69
0.69
MoBUKT (Vus
2023
0.69
0.69
0.69
0.69
0.69
0.89
0.89
0.69
0.69
0.20
0.69
0.69
0.69
Kes Lonceltieiii ie (Oax)
[0:51
0.52 0.50 0.51 0.56 0.54 0.54 0,30 0.59 son 0.57 0.53 0.55
0.58 pm
Table 6: Log-loss comparison of motion evaluation metrics on HumanAct12 action classes.
training and testing split is randomly divided in the paper. Here we also investigate into division based on annotator IDs. By adopting this approach, we ensure that each annotator's is data used exclusively in one of the sets, preventing overlap. Subsequently, we retrained and evaluated the model using this revised split. Disalignment between annotators as shown in Figure 2 results in slight decline of Mot Mottione ionCriitiii performance. Results are shown in the following table and align with our initial expectations.
Dataset Splitting and Model Retraining.
Metric
MDM Log
Acc. (3)
FLAME
Log
Root AVE
59.22
0.6875
48.42
0.6984
AE
62.10
0.6711
Joint AVE
57.01
Joint AE
63.17
0.6790
58.37
0.681
York
66. 33
0.7476
65.84
0.9944
Acceleration Yangetal al. ROOR
63.82
0.7611
66.67
0.6919
Penson-Groard Coutert
Keann ef
2021
68.08
0.6855
69.82
0.7245
Foot-Floor Penetzatin
$1.77
0.6072
55 56
0.696
Physical Fret Contect
66. 52
0.6927
66.00
0
PoseNDE (Tread
$6.00
0.6830
$3.07
0.6931
NPSS
$3.48
6064
NDMIS
0.6590
MoBERT
47.49
0.6933
$2.40
0.6932
Moniontea
[O65]
80.13
0.5402
0.5927
Table 7: Quantitative comparison of metrics on re-split testsets of MetionPercept.
20

---

Published as a conference paper at ICLR 2025
Inplementation Details of Other Metrics Here provide details of the implementation of the compared metrics. To ensure reproducibility, we will publish the code for all metries. Overall, we uti- lized official code and models whenever possible, making only minimal and reasonable modifications to meet our evaluation needs.
Root AVE, Root AE, Joint AVE, and Joint AE: These metries were implemented following the definitions provided by Voas al. (2023).
Acceleration and Jerk: We used the released official implementations and hyperparameters from Yang et al. (2023) without any modifications. Jerk was derived as the first derivative of
Acceleration.
Person-Ground Contact and Foot-Floor Penetration: We used the released official implementations and hyperparameters from Rempe et al. (022) without any modifications.
Physical Foot Contact: We followed the approach described in Tseng et al. (003)) with slight adaptations to rewrite the original NumPy implementation into Py TyTorch.
PoseNDF: For each motion sequence, extracted all frames poses and calculated neural distances using the official pretrained models provided by Tiwari et al. (2022) We implemented two methods: one using the average distances across all frames the motion evaluation metric, and the other using the maximum distance among all frames. The latter performed better and was reported in Table
NPSS:
We used the released official implementations and hyperparameters from Karras et al. 2019)) without any modifications. Motions were matched to corresponding fround-truth sequences in the HumanACTI2 action class, NPSS is uni-modal and compares motions to single ground-truth sequences.
NDMS: We used the released official implementations and hyperparameters from Tanke et al. (2021) without any modifications. We first transformed axis-angle SMPL into xyz coordinates and utilized code from official implementation for skeletons.
MoBERT: We utilized their pretrained model from Voas et (2023) and computed the Naturalness metric, which evaluates the intrinsic plausibility of motion itself and aligns with the theme of our work.
This setup ensures that our evaluation metrics are consistent with the official implementations and provides a clear methodology for reproducibility.
B.2 FAILURE MODES
In addition, we analyze the failure modes of MotionCritic when is used as a motion quality metric (Table R). To do this, we computed the critic score differences for all samples in the test set (user-annotated better vs. user-annotated worse) and summarized the statistics in the table below:
Metric
Mean
Std
Max
Min
All Scores
4.32
3.72
29.82
-11.73
Correct Cases (85.1 (
4.75
3.79
29.82
6.79e-5
Wrong Cases (14.9%)
-1.84
1.91
-1.91e-4
-11.73
Table 8: Critic score statistics for all cases, correct cases, and wrong cases.
We note that in correct cases, the critic score differences are larger, indicating higher confidence. Conversely, in the wrong cases, the score differences are smaller, suggesting that the errors occur when the model is less confident. We also provide the distributions critic scores for both correct and wrong predictions (see Fig. (O). On the left, the vertical axis represents the number of examples (with wrong cases comprising 14.9% and correct cases 85.1%) 16). On the right, the vertical axis shows the percentage within the corresponding part.
21

---

Published as a conference paper at ICLR 2025
Distr bution Ortic Scores (Frequency)
Ditribttion of Orhic Scores (Percentage)
Correct Predictions cridictions
Correct Predictions Pradictions
90
Frequenc
2:5
200
15
100
-10
10
20
30
-10
10
20
30
Differences
Differences
Figure 10: Distributions of critic scores' differences for correct and wrong predictions.
is clear that in cases where the model predicts incorrectly, the majority of the score differences are very small. For example, 35.10% of the wrong cases have critic score difference within 0.73 (one bin), whereas only 8. 8.74% of correct cases fall into the same range. This indicates that many of the incorrect predictions occur when the critic scores are close, rather than due large, obvious errors. In contrast, correct predictions tend to have larger score differences, reflecting higher confidence from the model. Additionally, some errors occur in particularly challenging examples where human annotators struggle to differentiate, and different individuals may make varying judgments.
B.3 SENSITIVITY ANALYSIS
We conducted preliminary sensitivity analysis by perturbing AMASS ground-truth (GT) motions with Gaussian noise of various scales. Assuming that GT motions should consistently receive higher scores than perturbed ones, evaluated the critic's ability distinguish between them.
Robustness. Mot ceriiinriiiin ic demonstrates robustness in ranking GT motions above perturbed ones across noise scales, as demonstrated in Figure 11(A) We also observed how the average and standard deviation of critic scores vary with different noise scales, as demonstrated in Figure 1B
Smoothness. By analyzing the 3D relationship between noise scale, critic scores, and perturbed critic scores, observed steep score changes the natural motion distribution, demon- strated in Figure 12( The interactive version of this 3D visualization is available at https html
itrlubb..////e oo/iclllreeoooooooooooooooooooe r6688 sensi i www.ssiiiitttiii....eiii. www.iiitttaaayyiiiiiaaaa highlighting the critic's s sensitivity outside the natural motion distribution. We also provide 2D scatter plot focusing on perturbed-score vs noise-scale, as demonstrated in Figure 12(A)
Non-smooth energy landscape. In summary, we discover that the critic model exhibits non- smooth and "bumpy" energy landscape, which as double-edged sword in our framework. On one hand, this non-smoothness is desirable property, making the critic highly discriminative and sensitive subtle motion perturbations, aligning well with human perceptual abilities. On the other hand, it introduces challenges to motion optimization. Future work could investigate better optimization methods, such leveraging reinforcement learning strategies (Liu et al. 2024). to mitigate challenges associated with this non-smoothness.
Non-distance measure. Based on the training objective, the Bradley-Terry model, the critic scores are intended represent relative preferences rather than absolute distances. Specifically. the scores indicate whether one motion judged as superior another, rather than their distance. Therefore. it's worth noticing that the critic score should not be interpreted as a distance function. This design aligns with the goal of producing a ranking- evaluation metric rather than continuous distance measure.
22

---

Published as a conference paper at ICLR 2025
senO (org
Noised Critic Score
-5
90
Accurasy
80
-10
60
-20
0.5
0.6
0.0
0.1
0.2
0.3
0.5
0.6
Nolse Scale
(A)
(B)
Figure 11: Sensitivity analysis results. (A) Accuracy vs noise-scale curve. (B) Average and standard deviation of critic scores vs noise-scale.
Makes Scale
(B): 3D seatter plot of noise scale, critic scores, and perturbed critic scores.
(A): 2D scatter plot of perturbed-score vs noise-scale.
Figure 12: Sensitivity analysis upon AMASS GT motions. Critic score differences (original per- turbed) demonstrated by colors.
B.4 TEST ON OUT-OF-DISTRIBUTION MOTIONS
We test method
motion dataset apart from HumanACT12 and UESTC. Specifically, we conduct additional analyses on AMASS using MotionCritic and observed some interesting findings:
Dataset
Mean
Std
Max
Min
AMASS
-1.10
4.14
20.77
-18,99
HumanAct12
-2.22
4.68
12.14
-21.96
MDM
-2.08
4.65
12.83
-30.65
Table Critic score statistics of the two datasets.
The critic scores AMASS are generally higher than those on HumanAct12, which aligns with our expectations, while the critic score statisties of the two datasets are shown in Table
The actions with the lowest scores on AMASS were indeed outliers, often violating physical constraints such gravity (mostly due object interactions that appear unreasonable when
23

---

Published as a conference paper at ICLR 2025
considering only the motion). At the same time, it was also able assign relatively high scores to some previously unseen complex motions.
Current results show that AMASS has higher average scores, maximum scores, and minimum scores compared to MDM-generated motions, even though some outliers in AMASS sigmificantly lowered the scores, as discussed above. This suggests that normal motions in AMASS generally outperform those generated by MDM. Additionally, we conduct pairwise comparison between MDM-generated motions and HumanAct12 GT motions, testing both the critic scores and user study results, which are presented in the table below. HumanAcr12 indeed has quality issues and both the critic score and users often favored the MDM-generated motions. Furthermore, the critic score continues to align well with human users' judgments when comparing these two types of motions, demonstrating its effectiveness.
Table 10: User study on HumanAct12 GT and MDM-generated motions
Motion Category (12
HumaAct12 MDM Pre-
User Simi- lar
Humact12 Critic
MDM Critic
User
Dis-
categories, 10 mo-
Preferred
ferred
agreement
tion pairs each)
0
10%
70%
20%
20%
80%
20%
1
30%
40%
30%
30%
70%
20%
2
30%
50%
20%
20%
80%
20%
3
20%
50%
30%
20%
80%
10%
4
0%
80%
20%
10%
90%
0%
5
0%
90%
10%
0%
100%
0%
6
40%
10%
50%
70%
30%
0%
90%
10%
90%
8
10%
90%
0%
40%
60%
30%
9
10%
80%
10%
50%
50%
30%
10
0%
60%
40%
10%
90%
0%
11
40%
40%
20%
30%
70%
20%
Overall (120 pairs)
15.83%
62.50%
21.67%
25.83%
74.17%
12.50%
1. HumanAct12 User Preference:
This column shows the percentage of cases where users preferred the HumanAct12 motions over MDM.
2. MDM User Preference: This column shows the percentage of cases where users preferred the MDM motions over the HumanAct12 motions.
3. User Similar: This column indicates the percentage of cases where users found it difficult to distinguish between the HumanAct12 and MDM motions.
4. HumanAct12 Critic Score Win: This shows the percentage of cases where the Human- Act12 motion received a higher critic score.
5. MDM Critic Score Win: This shows the percentage of cases where the MDM motion received higher critic score.
6. User Disagreement with Critic Score: This column indicates the percentage of cases where the user's choice did not align with the critic score.
These examples demonstrate that MotionCritic is effective tool for automatically comparing and evaluating motion quality, whether individually or in bulk. It also helps uncover valuable insights, further highlighting its utility and potential.
B.5 IMPACT OF SEQUENCE LENGTH, FRAMERATE, AND INTERPOLATION
To further address concerns regarding the impact of sequence length, framerate, and interpolation the performance of the MotionCritic model, we conduct a detailed analysis using the AMASS GT dataset.
24

---

Published as a conference paper at ICLR 2025
Dataset and Pre-processing. We extracted total of 6,346 OOD (Out-of-Distriuuti) motion examples from the AMASS GT dataset. Among these, 547 examples (approximately 8.62%) have lengths shorter than 60 frames. We analyzed longer and shorter OOD motions separately to evaluate the model's s performance under varying conditions. To further assess the robustmess of the MotionCritic model, we generated pseudo-GTT dataset by adding Gaussian noise at different scales to the motions. We formed better-worse pairs by comparing unnoised and noised versions of the same motion.
For longer motions, we experimented with three different pre-processing approaches:
1. Unprocessed Motion: The full-length motion was fed directly into the model to obtain critic scores.
2. Cut-down to 60 Frames:
The motion was truncated to 60 frames before being input into the model, expanding the dataset to a total of 12,011 motions.
3. Uniform Frame Extraction: 60 frames were uniformly extracted from the complete motion, which was then fed into the model for critic scores.
The results, presented as shown in Figure 13 demonstrate that the Critic model performs optimally when the motion lengths are 60 frames, consistent with the original dataset.
(A) Short Motion Sequences
(B) Long Motion Sequences
100
90
Accuracy (%)
80
Accuracy
70
60
60
Pallergi
50
Lenghh
sa/pine Ou
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
Noise Scale
Noise Scale
Figure 13: Performance of the Critic model across different motion lengths and interpolations.
For shorter motions, two pre-processing approaches were evaluated:
1. Unprocessed Motion: The full-length, short motion input directly into the model for critic scores.
2. Interpolation 60 Frames: The motion was uniformly interpolated 60 frames before being input into the model.
The results, also presented as accuracy- noise line plots (on the left, Fig (A)), indicate that the Critic model can effectively handle shorter sequences with minimal performance loss after interpolation.
The results demonstrate that Moti in length, consistent with the training dataset, which could be achieved through interpolation or extraction. The model also exhibits strong performance across variable lengths and framerates, with only minor reductions in accuracy. This robustness highlights the advantages of the MotionCritic model.
works best when the motion sequences are 60 frames
B.6 ENSEMBLE LEARNING USING DIFFERENT METRICS
To further explore the interrelationship between the different motion quality metrics, treat the dif- ferent metrics as features and train a simple classifier to perform binary classification (ensemble learn- ing). To train the ensemble model, create separate train-test split from the Mot i iettioonereegg MDM test set. We explore three ensemble approaches to combine the metrics and evaluate the complementary effects between the heuristics and Mot
25

---

Published as a conference paper at ICLR 2025
Metric
Accuracy (%)
Method
Accuracy (%)
Mot MotionCritic
85.07
Logistic Regression
81.12
Acceleration
64.26
SVM
82.49
Jerk
65.48
MLP
85.17
GroundContact
71.78
Mot MotionCritic Only
84.96
PFC
64.79
Table 12: methods.
Accuracy of ensemble learning
Table 11: Initial accuracy of metrics on the MotionPercept MDM test set.
1. Logistic Regression. Binary classification using the five normalized features.
2. Support Vector Machine (SVM). Binary classification using the five features across two motions better-worse pair.
3. Multilayer Perceptron (MLP). A simple MLP with one hidden layer of 128 dimensions was trained using the five features as input, producing a single score to determine better or worse outcomes. The training objective was based on the Bradley- -erry model, similar to MotionCritic.
The accuracy of each combined method is presented in Table 12 the metries does not significantly improve accuracy beyond what MotionCritic alone can achieve, suggesting that MotionCritic already incorporates much of the heuristic knowledge.
The results indicate that combining
Visualization of Heuristics and MotionCritic Overlap We visualized the overlap between heuris- ties and MotionCritic on the entire MotionPercept MDM test set using Venn diagram Figure demonstrating a significant overlap in correctly classified pairs. This suggests that when MotionCritic makes a mistake, other metrics tend err in the same cases.
Ablation Study in SVM To further understand the contribution of each metric, we conducted an ablation study using SVM by incrementally adding features in the order of MMotinnrritc, Acceleration, Jerk, GroundContact, PFCJ. The results are shown in Table 13
Features Added
Accuracy (%)
Mot intiCritiic Mot MotionCritic Acceleration Mot notionCritic + Acceleration + Jerk
82,49
82.40
82,45
Mot MonoCiiii + Acceleration + Jerk + GroundContact
82.49
All Features
82.49
Table 13: Ablation study results in SVM.
Feature Removal in SVM We also conducted experiments by sequentially removing features observe the impact on accuracy, as presented in Table [4] The results indicate that removing MotionCritic has substantial negative impact on model performance, while removing other features has minimal effect.
The coefficients from the logistic regression model presented in Table 15 The positive coefficient for MotionCritic indicates that higher values are associated with better outcomes, while the negative coefficients for other metrics suggest that lower values preferable.
Logistic Regression Coefficients
SHAP Values Visualization We visualized the SHAP (SHapley Additive exPlanations) values, which provide insights into the contribution of each feature to the model's 's predictions (see Figure [4]. The figures show the SHAP values for the five metrics tested on a series of samples from our test set, evaluated with the MIP ensemble model. Across these samples, the SHAP values for the
26

---

Published as a conference paper at ICLR 2025
Feature Removed
Accuracy (%)
Feature
Coefficient
Mot LnCriiic
66.52
Mot onCritic
2.49
Acceleration
82.49
Acceleration
-0.61
Jerk
82,40
Jerk
-0.33
GroundContact
82.40
GroundContact
-0.39
PFC
82.49
PFC
-0.07
Table 14: Accuracy after feature removal in SVM.
Table 15: Logistic regression coefficients.
MotionCritic feature generally have the largest absolute values, indicating that the critic score is the most important feature. Additionally, when the critic score is high, its SHAP value is positive, suggesting positive correlation between the critic score and motion quality, which aligns with expectations.
(A) Metrics Prediction Venn Diagram
(B) Metrics SHAP Values
High
MetionCritic
Aocal
Jerk
Feature valuee
Acceleration
PFC
-0.8
-0.6
-0.4
-0.2
0.0
0.2
0.4
Grouhoco Ocontbot
SHAP value timpact model outputt)
Figure 14: Metrics' venn fiagram and SHAP values.
The experimental results demonstrate that Mot Loncritic effectively captures the essential knowledge provided by the heuristics. Attempts to further enhance accuracy by combining MotionCritic with other heuristics yield marginal improvements, underscoring the robustness of MotionCritic in this context.
27

---

Published as a conference paper at ICLR 2025
C DETAILS ON MCTIONCRITIC: AS TRAINING SUPERVISION
C.I FINE-TUNING
Critic Score Clipping. Generally, a higher Mot onCrit score indicates better motion quality. However, this relationship has an upper limit. During our fine-tuning process, we clip motions with reward scores exceeding threshold when computing gradients before back pack-propagation. This threshold, determined through a series of comparative experiments, set at 7 = 12.0, approximately the upper bound of ground-truth critic scores. We found that this setting yields the best results. Fine-tuned motion generation models without reward clipping tend to artificially inflate reward scores on few specific motions, which increases the average Mot iortonCriie tritic score but degrades overall performance. Thus, reward clipping is essential to maintain the integrity and quality of the fine-tuning process.
Finetuning Details. Inspired by ImageReward et al.) 2024), we observe how the critic score
changes denoising steps to identify the optimal time window for ReFL intercept. As shown in A),, A), we set the hyperparameter step sampling range to [T]
Figure 15
[700, 900), where the
critic score witnesses rapid increase. The full fine-tuning curve, provided in Figure 15 C), demon-
strates that the critic output improves rapidly during the early stages of fine-tuning and stabilizes after certain point.
In addition, another discussion by Clark et al. (0224) proposed DRaFT-LV, employing single-step gradient backpropagation, observed to be the most efficient. To evaluate the impact of alternative step sampling strategies, we conducted experiments comparing our current setup with DRaFI-LV's methodology. The results, detailed in Figure C), show that single-step refinement can indeed improve fine-tuning efficiency, consistent with existing findings.
10
-15
-20
-25
-35
100
800
1.000
200
800
500
1002
1500
2000
2502
(A)
(B)
Figure 15: Fine-tuning process. (A): Critic score 1000-step denoising process. (B): Critic output in 800-step fine-tuning process. (C): Full finetuning process of our strategy based on ReFl. (Xu et al. 2024] and 1-step back propppropngntionn based on DRaFI-LV (Clark et al. 2024).
C.2 FINETUNED RESULTS
Improved Critic Score. As shown in Figure [6] the critic score increases after Mot iootonCritic supervised fine-tuning. This scatter plot collects all data points from the test set, with the critic score of motions before fine-tuning on the x-axis and the critic score of the corresponding motions after fine-tuning the y-axis. As demonstrated in Figure T6(A)) we first compare results with and without critic model supervision. In the latter case, the original MDM loss used for continued training without our Mot ionCriti c-based plug-and-play module. The scatter plot clearly indicates that the results with critic model supervised fine-tuning achieve significantly higher scores. The second experiment in Figure [6[B] examines different fine-tuning steps using 800 steps from the first set baseline. The results demonstrate that critic model supervised fine-tuning consistently improves the critic score throughout the fine-tuning process.
28

---

Published as a conference paper at ICLR 2025
10
10
Critic Score after Fine- ree-tuning
Critic Score after Fine- tee-tunnngg
5
0
-5
-10
-10
Iransertaued odbuatencceteeee Pio laned (600 Stenol
Pis-landd 1602 Siezal Paselared 1800 Sienal
5 Critic Score before Fine-tuning
-5
Critic Score before Fine-tuning
(A)
(B)
Figure 16: Visualization of critic scores fine-tuning experiments. (A): Fine-tuning 400 steps with and without Mot oonriiii supervision compared. (B): Fine-tuning with 400 and 800 steps compared.
Improved Motion Quality. We conduct an independent user study to compare motion pairs generated at various fine-tuning stages and calculate the Elo Rating (Flo 1978 Tseng et al. (002)). Figure 17 demonstrates that the quality of motions consistently enhance as fine-tuning advances, as indicated by the user study. This improvement aligns with the training objective of elevating the critic score.
MationCnticc & Multimodality
16
3.0
2500
1.2
2000
1.0
0.8
0-
0.68
1000
-1.0
-1-
0.2
200
600
200
Fine-tuning Steps
Fine-tuning
(A)
Figure 17: Results from fine- tinettuning process. (A): Elo ratings and Critic scores. (B): FID, PFC Tseng al. 2023). Multimodality and Critic scores.
We further inspect the change of different metries during the fine-tuning process in Figure B) and Table PFC (Tseng et al.) 2023) and FID are expected to be negatively correlated with motion quality (the smaller, the better), correlated (the greater, the better). The results indicate that existing motion quality metrics (e.g. FID, PFC) do not adequately reflect human preference, as they poorly correlate with Elo ratings from user studies. Meanwhile, improving the critic score does not necessarily conflict with the multimodality metric, which models the diversity of generated motions.
Mot onCritii and multimodality are expected be positively
29

---

Published as a conference paper at ICLR 2025
D DETAILS ON USER STUDIES
D.1 ANNOTATION DETAILS
The annotators of Mot oonPercept are recruited by an annotation company and are ordinary adults (5 men and 5 women). Previous psychological research has shown that the ability to perceive and discern biological motion is universal and consistent, with a neural basis (Johansson 1973), and does not rely on specific training or professional background. This finding also consistent with what we observed in Figure 2 As for potential biases, while some degree annotator bias may exist (as with any human-annotea annotated dataset), we believe it is minor and manageable, as indicated by our Overall, we consider the opinions of the annotators to be representative.
analysis in Section 3
D.2 ALONE USER STUDIES
Details. We conduct studies on GT subsets grouped from HumanAct12 (Guo et 2020) and motions generated during finetune steps as discussed in the main text. Our user study platform is shown Fig In user study. one motion pair of two motions are played simultaneously, with their critic scores and text prompts being hidden. Annotators should choose the better motion or choose "Almost the Same" if they can't make a decision. We perform user study on 5 different finetune steps and 5 GT batches grouped from HumanAct12 (Guo et al., 2020
Different annotators. For the stand-alone user study, we did not use large-scale annotators from the annotation company. Instead, recruit new volunteers through campus announcement who performed pairwise comparisons of two motions, selecting the better one. The five volunteers are students from two different universities with backgrounds computer science, engineering, arts, and history. During this process, they do not see any additional information such as scores or text. We ensure that they receive reasonable compensation for their time and effort. We then calculate the Elo Rating based on their results. The annotation interface and rating formulas can be found in this section as well.
Choose This Video
Choose This Video
Next Pair
Almost the Same
Figure 18: The interface of our user study platform.
30

---

Published as a conference paper at ICLR 2025
D.3 RATING FORMULAS
Win-rates. After annotation, we calculate win-rates of subsets pairs. In user study, each subset has the amount of motions. Given subsets pair (A, B), win-rates shows the percentage of motion pairs where motion of subset A win over motion of subset B in naturalness. Then we paint heatmaps of all subsets with their win rates. Since the result of one match maybe tie, the sum of win-rates of two subsets in a pair and data in symmetric positions of heatmap might be less than 1.
Elo Rating
Elo 1978 Tseng et al. 2023) After annotation, we calculate elo rating of each subsets
follows:
Suggest RA, Rg are the initial ratings of two compared subsets A and B. The expectated win rate of subset9s A and B, denoting EA, En can be calculated as follows:
ΣΛ 1 + 10( RAA)/400
EB + 11(0a-RR5)/ Rs)/400
(8)
The new ratings of subsets A and B are:
R'A RA + K(SA EA)
(9)
RB RB + K(SB FB)
(10)
where K is rating coefficient, we choose 32; and S is real score, which is for winner, 0 for loser and 0.5 if the result tie. We set the initial rating of each subset as 1500.
31


Reviews:
[{"review_id": "9oj4lGo39F", "reviewer": "Reviewer_ud4c", "summary": "This paper proposes a data-driven approach to align human motion generation with human perceptions. It introduces a large-scale dataset MotionPercept and a human motion critic model MotionCritic. The main contributions include providing a more accurate metric for motion quality assessment and showing that the critic model can enhance motion generation quality.", "questions": "(1) Why not use the latest models, but MDM and FLAME released in 2022? \n\n(2)Since currently, another two paradigms Noise De-masking and GPT-like Auto-regressive model show their potentials for motion generation task, how could this method of motion generation with critic model supervision be general to these methods?\n\n(3) Why the step sampling range set as [700,900] in Sec 5.1? Is there any experiment support this setting, because I think from 700 - 900 step to get x0 directly results in motion with more artifacts, it might be help for those motion with obvious errors, but if you want to improve the motion quality use critic supervision as post-training, maybe it doesn't make much sense. If I am wrong, please explain your thoughts with some proofs. \n\nBTW, I intially wanted to give 7 out of 10, but there is not this option. If authors can solve my concerns, I will raise the score.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "（1） The use of a large-scale human-annotated dataset and a critic model for automatic evaluation to align motion generation quality with human perceptions is a significant step forward.\n（2） The experimental design is comprehensive. It evaluates the MotionCritic model on different data distributions and shows its generalization ability. Also, it tests the model as a training supervision signal and analyzes its impact on motion generation quality.", "weaknesses": "1. Table 2 shows, after 700 steps finetuning, the MotionCrotic decrease at 800 step. I am not sure about the trends, could you provide results with much much more steps to let us see the potential of designed Critic Supervision. \n\n2. I think motion model trained on larger datasets HumanML3D and Motion-X is should be used to see Critic Supervision's potential.\n\n3. Sec 4.3 could be with more detail and Algorithm 1 could be much simpler and clearer maintaining only necessary procedures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lejvl8RIEI", "reviewer": "Reviewer_fHeh", "summary": "This work introduces MotionPercept, a large-scale synthetic human motion dataset which contains human-ranked synthetic human motion generations, and MotionCritic, a model learned on MotionPercept. MotionCritic can be utilized to evaluate motion quality but also as an additional loss for motion quality. \nThe authors show that MotionCritic follows human perception closer than existing evaluation metrics and can even be used to find outliers in existing human motion datasets.", "questions": "-", "limitations": "", "rating": 8, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "This paper addresses a common problem in generative motion synthesis: how to evaluate the motion quality of a generated sequence? In contrast to the often utilized FID score MotionCrititic operates on a per-sequence base, which allows for a much more granular evaluation.", "weaknesses": "One major concern of the motion critic is its quality ceiling: MotionPercept is highly dependent on the generative models that produces the motion, so in turn MotionCritic is highly dependent on those generative models as well. That means that, for the model, the best possible motion is the best generative model of MotionPerecpt, potentially limiting the usefulness of the critic once generative models produce significantly better motion than has been produced for MotionPercept. Could the authors comment if there is a way to remedy this or if they have considered this a problem?\n\nAnother concern would be: how does MotionCritic behave when provided with unseen motion modalities? I.e. lets say MotionCritic was not trained with dancing motion but is now tasked to judge the motion quality of dancing: do the authors expect MotionCritic to continue being reliable or would retraining be required?\n\nWhy did the authors choose 2.3s sequence lengths (60 frames @24Hz)? This seems very short for complex motion, i.e. a person “sitting down”. Were complex motions discarded for the dataset generation? Otherwise, only the motion initiation would have been recorded.\nDid the authors observe that the model tends to start from the same pose or position, i.e. a T-Pose? This might limit the application of MotionCiritic as it could favor motions and poses that are more similar to the “initialization” pose/motion.\n\nHow can MotionCritic be applied to sequences of different lengths, i.e. [A] How can MotionCritic deal with sequences shorter than 60 frames and how does it perform ?\n\nHow is the input motion to MotionCritic normalized? Is the first frame translation at the origin? How is the global rotation set up? Are the first frame rotations set to the same rotation for each generation?\n\n[B] How can MotionCritic be applied to sequences far longer than 60 frames, i.e. 1000 frames?\n\nTable 1: NDMS and NPSS are flipped.\n\nThe evaluation methods against which motion critic is compared should be discussed in the related work, and not just in Table 1.\n\nThe authors should describe in how they apply the other metrics that they compare against, i.e. how they normalize the data, what kind of data representation/hyperparameters they are using.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mi0xWbsCX2", "reviewer": "Reviewer_zYMP", "summary": "The paper introduces a novel evaluation metric for 3D human motion that aligns more closely with high-level human perception. A large-scale data annotation effort is conducted by asking users to select the highest quality motion from a set of samples. To capture human preferences, the paper proposes training a critic model that learns to assign a quality score to each motion sample. A comprehensive analysis compares the proposed critic model with commonly used evaluation metrics from prior work, revealing deficiencies in these established metrics for aligning with human perception, while the proposed critic model shows much higher alignment. Additionally, the paper presents a fine-tuning approach for improving the motion quality of pre-trained motion diffusion models by leveraging the proposed critic model.", "questions": "1- This aligns with my comment on sensitivity analysis. The paper addresses only fine-tuning but does not consider an end-to-end training scenario. Theoretically, the critic model could also be used to train a motion model from scratch. I mention this because, even in the MDM fine-tuning experiment, the critic term is tightly controlled (with a low learning rate, low critic weight, and high KL-term weight). Is this due to difficulties in training with the critic term, possibly because of a non-smooth energy landscape?\n\n2- Locomotion sequences with significant foot sliding receive high critic scores. Could this be due to a bias in the annotated samples, given that MDM generates many sequences with foot sliding? Have the authors considered training the critic with ground-truth samples as well? This should be straightforward, as ground-truth samples could always be preferred over synthetic ones.\n\n3- Are the critic scores for two different samples comparable? I assume it isn’t a distance function. Could the authors comment on this?\n\n4- This is minor but it would be great to have a comparison against a log-likelihood metric such as a normalizing flow.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper addresses a fundamental problem in human motion generation and is both well-written and well-motivated. I appreciate the authors' meticulous effort in their work.\n\nAnnotating a large sample set and training a critic model on this data is a clever idea. Additionally, the thorough analysis and comparisons effectively demonstrate the proposed method's impact. While the human motion modeling field has converged on a set of complementary metrics for quality assessment, these metrics still fall short of aligning with human perception, as shown in this paper. The proposed MotionCritic takes a more direct approach to addressing this issue. I believe this work is highly valuable for the community, contributing both to the research on evaluation and analysis of future human motion synthesis research. I also hope it will serve as an alternative to the counterintuitive FID score.", "weaknesses": "A sensitivity analysis could be included to assess the robustness and smoothness (e.g., Lipschitz continuity) of the critic model. For instance, how would the critic model respond if a pose is slightly perturbed, either randomly or in a structured way by rotating a joint beyond its limits?\n\nBy design, certain artifacts—such as foot-floor penetration, person-ground contact, etc.—might be missed, as the critic model lacks information needed to detect these issues (e.g., it only uses rotations and root translation as inputs). What are the authors' thoughts on this limitation?\n\nAnother potential concern is the reliance on a single model (e.g., MDM) for generating data used in the annotations. Although the paper presents results with an additional model (e.g., FLAME) to demonstrate generalization, it may be beneficial to curate an annotation set from various models and architectures to mitigate possible biases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Y4WUUEykEi", "reviewer": "Reviewer_pzfu", "summary": "This paper addresses the challenges in evaluating and improving human motion generation by proposing a human-aligned evaluation approach. The authors introduce MotionPercept, a large-scale dataset containing over 52,000 human preference annotations for generated motions, which provides a foundation for understanding motion quality from a human perspective. They also develop MotionCritic, a model trained on MotionPercept, which accurately predicts human-perceived motion quality and surpasses traditional metrics in alignment with human preferences. MotionCritic can be easily integrated into existing generation pipelines, where it functions as a supervision signal to improve motion quality with minimal fine-tuning.", "questions": "1. There is inherent diversity in the motion generated from text-to-motion. Regarding the fine-tuning comparisons in the supplementary materials, I observe that the generated motions generally align with semantic accuracy, and the physical accuracy appears to improve over time. Could it be that the critic score primarily reflects physical correctness? What I’d like to see is a visualization that compares multiple generated motions for a single text prompt, showing the distribution of critic scores (with five or more samples), as well as how motion diversity changes before and after fine-tuning. This result would better demonstrate the effectiveness of your proposed method and its balance in generating diverse outputs. \n\n2. When the text is simple, such as \"run\" or \"punching,\" the corresponding motions should naturally be diverse. However, based on the annotation process provided in the HTML document, it seems that only the best out of four generated motions is selected, which shows no particular consideration for diversity. This raises the question of how Multimodality would improve after fine-tuning. [1] aligns motion generation with human preference. They only select the best result, which significantly reduces diversity (MModality, the diversity of motion generated for one text description). A discussion with [1] would also be beneficial.\n\n3. The annotation of human preference in this paper for motion follows the same process as for images. I see that many useful insights (as metrics and supervision) also resemble those used in image-based tasks. Since motion data is a unique form of 3D data, differing greatly from 2D image data, are there any distinct insights here?\n\n4. The annotation of human preference in this paper for motion follows the same process as for images. I also see that many insights (as metrics and supervision) also the same as image-based tasks [2,3]. Since motion data is a unique form of 3D data, differing greatly from 2D image data, are there any distinct insights here?\n\n5. In Table 1, where MDM’s generated results are evaluated with different metrics, why does the metric with higher Acc is better?\n\nReference:\n\n[1] HuTuMotion: Human-Tuned Latent Motion Diffusion Models with Minimal Feedback.   \n[2] ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation.  \n[3] Better Aligning Text-to-Image Models with Human Preference.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "A large amount of human preference annotation work has been done for text-to-motion, and a sizable dataset has been provided, which is beneficial to the community.", "weaknesses": "The human preference annotation lacks consideration for the diversity of motion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)