Exploring Text-to-Motion Generation with Human Preference

Authors: mohammad gheshlaghi azar
mark  rowland
bilal  piot
daniel  guo
daniele  calandriello
michal  valko
ralph allan bradley
milton e terry rank
zixiang  chen
yihe  deng
huizhuo  yuan
kaixuan  ji
f  christiano
jan  leike
tom  brown
miljan  martic
shane  legg
geoffrey  cideron
sertan  girgin
mauro  verzetti
matej  kastelic
zalán  borsos
brian  mcwilliams
liane  lovitt
jackson  kernion
yuntao  askell
saurav  bai
ben  kadavath
ethan  mann
nicholas  570 perez
kamal  schiefer
andy  ndousse
sam  jones
anna  571 bowman
tom  chen
nova  conerly
dawn  dassarma
tom  henighan
danny  hernandez
tris- 574  tan hume
josh  jacobson
scott  johnston
shauna  kravec
chris  olah
jared  kaplan
jack clark red
teaming  lan- 578
hang  jiang
xinxin  zuo
michael bi mi
xinchao  wang
karl  cobbe
tyna  eloundou
gretchen  krueger
kevin  but- ton
matthew  knight
benjamin  chess
josh  achiam
steven  adler
sandhini  agarwal
derek  chen
sully  chen
ruby  chen
jason  chen
mark  chen
chester  cho
casey  chu
won  chung
dave  cummings
jeremiah  currier
yunxing  dai
tian  gibson
vik  goel
tarun  gogineni
gabriel  goh
rapha  gontijo-lopes
jonathan  gordon
morgan  grafstein
scott  gray
ryan  greene
joshua  gross
shane  gu
yufei  guo
chris  hallacy
jesse  han
jeff  harris
yuchen  he
mike  heaton
johannes  heidecke
chris  hesse
alan  hickey
wade  hickey
peter  hoeschele
brandon  houghton
kenny  hsu
shengli  hu
xin  hu
joost  huizinga
shantanu  jain
shawn  jain
joanne  jang
angela  jiang
roger  jiang
haozhun  jin
denny  jin
shino  jomoto
billie  jonn
hee- woo  jun
tomer  kaftan
łukasz  kaiser
ali  kamali
ingmar  kanitscheider
shirish  nitish
tabarak  keskar
logan  khan
jong wook kilpatrick
christina  kim
yongjik  kim
hendrik  kim
jamie  kirchner
matt  kiros
daniel  knight
łukasz  koko- tajlo
andrew  kondraciuk
aris  kondrich
kyle  konstan- tinidis
gretchen  kosic
vishal  krueger
michael  kuo
ikai  lampe
teddy  lan
jan  lee
jade  leike
daniel  leung
ming  levy
rachel  li
molly  lim
stephanie  lin
mateusz  lin
theresa  litwin
ryan  lopez
patri- cia  lowe
anna  lue
kim  makanju
sam  malfacini
todor  manning
yaniv  markov
bianca  markovski
katie  martin
andrew  mayer
bob  mayne
scott  mcgrew
mck- inney  mayer
christine  mcleavey
paul  mcmillan
jake  mcneil
david  medina
aalok  mehta
jacob  menick
luke  metz
an- drey  mishchenko
pamela  mishkin
vinnie  monaco
evan  morikawa
daniel  mossing
tong  mu
mira  murati
oleg  murk
david  mély
ashvin  nair
reiichiro  nakano
ra- jeev  nayak
arvind  neelakantan
richard  ngo
hyeon- woo  noh
long  ouyang
cullen  o'keefe
jakub  pachocki
oliveira  pinto
michelle  pokrass
vitchyr  pong
tolly  powell
alethea  power
boris  power
eliza- beth  proehl
raul  puri
alec  radford
jack  rae
aditya  ramesh
cameron  raymond
francis  real
carl  ross
bob  rotsted
henri  roussez
nick  ryder
sarah  shoker
pranav  shyam
szymon  sidor
eric  sigler
maddie  simens
jordan  sitkin
katarina  slama
ian  sohl
jason  wei
c j weinmann
akila  welihinda
welin- 716  peter
jiayi  der
lilian  weng
matt  weng
dave  wiethoff
  willner
clemens  winter
samuel  wolrich
hannah  wong
sherwin  workman
jeff  wu
michael  wu
kai  wu
  xiao
tao  xu
sarah  yoo
kevin  yu
qiming  yuan
rowan  zaremba
chong  zellers
marvin  zhang
  zhang
peter  askell
paul f welinder
jan  christiano
ryan lowe leike
rafael  rafailov
archit  sharma
eric  mitchell
d  manning
stefano  ermon
chelsea  finn
  direct
sharan  narang
michael  matena
yanqi  zhou
wei  li
zhihong  shao
peiyi  wang
qihao  zhu
runxin  xu
mingchuan  song
y k zhang
y  li
daya  wu
  guo
ryan  lowe
chelsea  voss
dario  amodei
Venue: thecvf_com_CVPR_2024_Workshop_HuMoGen
Published: 2024-03-16 07:14:04

Abstract: This paper presents an exploration of preference learning in text-to-motion generation. We find that current improvements in text-to-motion generation still rely on datasets requiring expert labelers with motion capture systems. Instead, learning from human preference data does not require motion capture systems; a labeler with no expertise simply compares two generated motions. This is particularly efficient because evaluating the model's output is easier than gathering the motion that performs a desired task (e.g. backflip). To pioneer the exploration of this paradigm, we annotate 3,528 preference pairs generated by MotionGPT, marking the first effort to investigate various algorithms for learning from preference data. In particular, our exploration highlights important design choices when using preference data. Additionally, our experimental results show that preference learning has the potential to greatly improve current text-to-motion generative models. Our code and dataset will be publicly available to further facilitate research in this area.

Main paper: 
 preliminary: 
 This section reviews MotionGPT [19] , the supervised baseline upon which we use preference learning to finetune. Additionally, we present our data collection pipeline that uses sample pairs generated by MotionGPT.MotionGPT. Formulating text-to-motion generation as a sequence modeling problem allows building upon LLMs. This holds the premise of transferring language's compositional semantic structure to other modalities, thereby achieving off-the-shelf, out-of-distribution generalization.Casting motion generation as a sequence modeling problem requires discretizing the motion modality into tokens, as done by MotionGPT. The discretization process is akin to the tokenization of strings to tokens in language processing.In particular, they first map the motion dataset into a set of discrete tokens using a vector quantized variational autoencoder [37] . Then, a pretrained LLM is finetuned to generate corresponding motion tokens from the textual prompt. Thus, MotionGPT is an autoregressive model where the completions are motion tokens instead of word tokens.Collecting preference data. As shown in Fig. 2 , we build a labeling platform with Gradio [1], where labelers are presented with two different completions from a prompt.The labelers are tasked to read each prompt and choose the motion that corresponds best to the prompt. Additionally, the labelers provide a degree of preference for their choice, choosing from "Negligibly better/unsure," "Slightly better," "Better," and "Much better." We find that MotionGPT produces samples that are hard to distinguish when given a prompt from the training dataset, thus indicating signs of overfitting. Accordingly, we prompt gpt-3.5-turbo-0125[25] to generate a new set of prompts similar to those in the training set. For each prompt, we sample two completions from MotionGPT by using different seeds and a temperature of 1.2 to promote diversity. Our labelers are graduate student in computer science. We find that it is important to recruit labelers with prior exposure to generative models. Our initial exploration indicates that labelers with similar prior experience is crucial for achieving a high level of agreement. Quantitatively, we obtain an agreement of 84% on average (42 samples out of 50 samples). Note that in some cases, the model completely fails to generate perceptually realistic motion for both seeds. Accordingly, the labelers mark them as "Skipped." Upon inspection, we find that two cases often occur: (1) one generation is "Much better," with one seed failing to generate reasonable motion while the other seed partially achieving the motion, (2) the motion pair is "Skipped," with both seeds failing to generate reasonable motions. The resulting dataset contains 3, 528 annotated pairs, with 996 pairs labeled as "Much better," 607 pairs labeled as "Better," 497 pairs labeled as "Slightly better," and 116 pairs labeled as "Negligibly better/unsure."Additionally, there are 1312 examples labeled as "Skipped."We randomly select 10% of the total dataset to be the test dataset, with the remaining data designated for training. 
method: 
 We organize this section as follows. Sec. 4.1 presents the objective function in preference learning. Then, we present practical algorithms for optimizing this objective: RLHF based on reinforcement learning in Sec. 4.2 and DPO based on supervised learning in Sec. 4.3.x = (x 1 , x 2 , ...) is a textual prompt and y = (y 1 , y 2 , ...) is a completion, i.e. a generated motion sequence.We formulate the objective function for learning from human preference data as in Azar et al. KL(π θ ∥π ref ) = H[π θ , π ref ] cross entropy -H[π θ ] entropy .(2)The cross-entropy term acts as a regularizer that prevents . Accordingly, we have Ψ(q) = 283 log(q/(1 -q)). In practice, the Bradley-Terry model is im-284 plemented as a sigmoid function σ: During policy optimization, due to the overfitted reward, the policy tries to hack the reward function and selects tokens that are very improbable under the reference model.EQUATIONAs a result, the KL divergence explodes, and so does the reward. Accordingly, the value network is also affected by these sudden spikes. We experimentally find it hard to prevent these spikes. In particular, we found that removing dropout is essential to diminishing the spikes. Surprisingly, we observe that preventing these spikes is not related to better performance as evaluated by FID and R-precision. Overall, we find RLHF particularly difficult to tune in our setup, owing to the instabilities resulting from a reward model's inability to evaluate samples accurately. Instead, we recommend using DPO, which we present next.In DPO, we skip the step of learning a reward model and directly train our policy on the preference data. In particular, we rewrite Eq. 4 the reward function as a function of the optimal policy π θ ⋆ to Eq. 1:EQUATIONOriginally in RLHF, we had a loss function on the reward functions to turn our preference data into a reward function.We use Eq. 7 to turn the loss function over reward functions in Eq. 5 into a loss function on policies. In particular, we write Eq. 7 in terms of our current policy π θ instead of the optimal policy π θ ⋆ , which we denote as rθ (x,y):EQUATIONIntuitively, the logarithmic ratio yields a positive value when the policy assigns a higher probability to the response compared to the reference model, indicating a preference.Conversely, it results in a negative value when the policy deems the response less probable than what the reference model suggests, signifying a lesser preference. Then, the objective for DPO is: 1 . Preference data improves alignment. We find that DPO performs better than RLHF. It is important to note that the FID metric is an inaccurate measure of the quality of the motion. In particular, our labelers prefer outputs from DPO over MotionGPT.EQUATIONWe directly train our policy with Eq. 9 on the preference dataset. Its gradient formula yields a very intuitive understanding of the optimized objective: it increases the likelihood of the preferred sample and decreases the likelihood of dispreferred samples. In formulae, each gradient step is: EQUATION 
experiments: 
 We organize this section as follows. First, we present details of our implementation of both methods: RLHF and DPO. Second, we present the evaluation metrics that follow standard practice in text-to-motion generation. Then, our main results show an improvement in alignment with text, compared with MotionGPT. Finally, we present ablations to understand key design choices in DPO. In particular, we find that proper regularization is important in DPO.During all training runs, we train for 20 epochs in total and take the epoch with the best validation set performance on HumanML3D [14] . We evaluate quantitatively on the HumanML3D test set and qualitatively on our human preference test set.Implementation Details. Our implementations of RLHF and DPO build upon TRL [39] . We implemented RLHF with separate value and policy networks because empirically, we observed greater training stability. The value SFT model is stochastic. For better performance, we add 419 reward margin (3 for "Much better," 2 for "Better," 1 for 420 "Slightly better") [36] , reward whitening, and score scaling 421 [53] . We performed a hyperparameter sweep and found that Furthermore, when considering quality metrics, RLHF and 453 2 https://github.com/OpenMotionLab/MotionGPT/tree/main DPO demonstrate comparable performance to MotionGPT, suggesting their efficacy in producing high-quality outputs. Notably, our findings highlight DPO's superiority over RLHF in alignment metrics, underscoring its potential as a more effective approach for learning from preferences in motion generation tasks. Since the FID metric is not an accurate measure of the actual quality of the motion, we perform human evaluation of MotionGPT generation against DPO. We compare MotionGPT baseline generations and DPO generations at different temperatures. Given 50 random prompts taken from the test set of our preference dataset, we ask labelers to pick which generation is the best or mark a tie if they cannot make a choice. In Fig. 3 , we show the DPO win rate, MotionGPT win rate, and tie rate. On average, DPO generations perform better than MotionGPT generations at all temperature levels, indicating not only that humans prefer DPO outputs over MotionGPT outputs, but also that its good performance is sustained at different temperature levels. Additionally, in Fig. 4 , we train our models on the different 497 preference splits. We find that samples labeled as "Much 498 better" provide most of the performance gains. Our resultssuggest that labelers should focus on labeling samples with 500 a considerable visual difference. https://huggingface.co/OpenMotionLab/MotionGPT-base 
: 
 mains abundant with datasets, particularly in language tasks benefiting from ample and high-quality data, its application in fields constrained by limited, multi-modal data presents a unique challenge. The current landscape of learning from preference data is rife with intricate engineering details and subtle design choices, often concealed within implementations and validated solely through empirical experimentation [26, 36, 40, 53 ]. Yet, there are currently no existing motion datasets tailored for exploring preference learning techniques. As a result, initiatives to extend preference learning to these low-data, multi-modal setups remain absent, for they lack empirical evidence to substantiate the intricate design decisions pivotal for applying preference learning in motion generation. This absence underscores an intriguing gap in our understanding and presents an exciting opportunity to investigate how preference learning performs in motion generation tasks where data is scarce.Previous endeavors address data scarcity by aligning to large language models' (LLMs) rich representation [19, 51] .While this approach transfers some of the compositional structure of language, it nonetheless requires a large dataset of text and motion pairs, failing to circumvent the data issue.Other methods resort to pseudo-labeled data [21] to offset the dearth of large-scale datasets in motion generation. However, such approaches often introduce noisy learning signals that may amplify problems, such as perceptually unrealistic motions. Alternatives involve injecting noise into existing labels and learning from the resulting ranked generations [41] . Nonetheless, this approach neglects training on the actual policy distribution, leading to a distributional gap wherein the reward model is not trained to supervise the actual policy.We classify existing methodologies according to the approximations employed to represent the preference distribution. In particular, current methods make one or both of the following approximations. First, they assume that pairwise preferences can be substituted by a scalar reward. In particular, they employ the Bradley-Terry probabilistic model [5] to connect scalar rewards to preferences. Second, they assume that a reward model trained with the Bradley-Terry model generalizes so that it can accurately evaluate sam-Figure 1 . Text-to-Motion Generation with Human Preference. We gather preferences over generated completion (i.e., motion) pairs and use them to finetune MotionGPT. In preference learning, the likelihood of preferred completion is increased while that of dispreferred completion is decreased. We explore two types of practical algorithms for preference learning. First, RLHF trains in an online manner; it trains a reward model on the data and uses it to perform RL on MotionGPT. Second, DPO trains in an offline manner with supervised learning; it directly performs MLE on the data. The online/offline aspect is related to whether or not the policy performs exploration, i.e., training on completions outside of the preference dataset. We explore the aforementioned methods along their variants, and summarize our contribution as follows: a significant opportunity to advance our understanding of how these methods can be leveraged effectively in contexts where data availability is constrained, thereby opening new avenues for research and innovation in motion generation and related disciplines. Its application is particularly relevant to motion generation, where evaluating two motions is considerably easier than collecting motion data with costly motion capture systems. 
 
 Appendix:  


Decision: The paper received positive reviews. The reviewers appreciated the idea of introducing RLHF and DPO in the motion synthesis context. However, a recurring concern is regarding the quality of the results and whether the proposed modules actually help in the generation. Nonetheless, the paper is a step in an interesting direction and would be of interest to the motion generation community. The authors are encouraged to augment the supplementary material with more examples of the failure/success cases of the method for the camera ready.

Reviews: review: title: title: The authors point out the explorations using human preference data to improve t2m generation both in quality and text alignment manner. The authors argues that as currently the field of t2m relays of sכ mall datasets collected with expensive motion capture systems and annotators, it can be easier generating high quality training data by getting experienced people (as annotators) to compare two generate motions to decide which is better aligned to the text. They created a new dataset of human preferences for motion generation and show that using this data can improve the results of t2m quality and alignment. By that the authors are pioneering the RLHF and human preferences in the motion domain.
main_review: main_review: The authors point out the explorations using human preference data to improve t2m generation both in quality and text alignment manner. The authors argues that as currently the field of t2m relays of sכ mall datasets collected with expensive motion capture systems and annotators, it can be easier generating high quality training data by getting experienced people (as annotators) to compare two generate motions to decide which is better aligned to the text. They created a new dataset of human preferences for motion generation and show that using this data can improve the results of t2m quality and alignment. By that the authors are pioneering the RLHF and human preferences in the motion domain. 


Pros
- Pioneering in the field of T2M RLHF field - generated a dataset to tackle this issue
- Although not innovative as its self, the authors bring the field of RLHF to the human motion. 
- The authors collected and release an important dataset that can be later used for additional work.  


Cons
- From the way that the sup. As provided, it is extremely hard to understand the improvement claimed in the paper.
- As in contrast to what has been done in T2I domain, the data generated only by one specific model instead of generating a more diverse dataset.
- The authors did not provide a visual comparison figure to prove that the RLHF/DPO is better than the baseline model.
- The additional work doesn't show significant improvement in quality over the baseline (as can be understood from the user study).
 

Questions and suggestions for the next revision:

While the labelled data of comparison between two motions includes also the “how much better the motion is”, this signal was not used during the training. Can you explain why you are not using it, and suggest a way to use this signal as well?
Please add qualitative comparison between the methods and the baseline. 
It is not clear to me how the user study was conducted, how many pairs were generated, and how many users conducted the user study ?

Overall, the paper tackles an important and rising task that was already adopted by the t2i community, bringing it and emphasising the importance of collecting human preferences in the t2m field. Although I couldn’t find any significant innovation within the method, I do think that this is an important work and thus I would recommend accepting the paper.
title: title: A Valuable Work Requiring Clearer Exposition
main_review: main_review: While the overall idea of the article might not be groundbreaking, it indeed validates important issues within motion generation and offers valuable insights for future research. I think the main issue with the article is its somewhat confusing organizational structure, coupled with a lack of intuitive visualizations of the results.
Here are some minor issues:
- The authors frequently claim that their method surpasses labeling motions because it obviates the need for motion capture devices. However, such claims can be somewhat misleading, as their approach also necessitates a model that has been pre-trained on paired data. Moreover, the mocap and labeling processes are often not conducted simultaneously, and there is an abundance of mocap data that has not yet been labeled. Therefore, I believe it would be more accurate to frame the comparison as between collecting "more" labeled data and gathering some preferred data.
- 015-017: "greatly improve", however, the subsequent metrics did not show significant improvement, and the authors also observed that labeling more preference data did not yield more benefits.
- 024-028: "current motion generation research", I think authors mean text-to-motion generation research.
- LINE 292 (EQU4): Z(x) is not defined here, which should be some integration over x
- LINE 296-299: It feels somewhat odd to discuss this topic here. The author mentions the issue of overfitting and then moves on to a completely unrelated topic in the following paragraph. I initially expected the next section to offer a solution to overfitting. If the author does not intend to address the issue here, perhaps it would be more appropriate to bring it up before the IPO section?
- LINE 331-338: same as above 
- Tab 4 and LINE 490: In table, authors compare DPO with/without PEFT, while in LINE 490, they say LoRA is important. While there may be significant overlap in implementation or content, it's important to maintain consistency in how it's described.
- I think incorporating pseudocode would greatly enhance the clarity of the article's presentation.

Here are some major issues:
- The authors have shown that after RLHF or DPO, there is a decrease in the quality metrics. Have they considered proposing alternative metrics? It's noteworthy that the quality metrics mentioned in the article are mostly based on statistical indicators, fundamentally assessing whether the distribution of generated motions and the ground truth are statistically similar. However, the authors have fine-tuned using additional preference data, which could cause a certain shift in the distribution of generated motions. Is it possible to define metrics that are not based on overall statistical data but instead on the evaluation of individual motions?
- I believe the authors should showcase a few motions that are performed better after DPO in the article, providing an intuitive comparison with MotionGPT.