SoPo: Text-to-Motion Generation Using Semi-Online Preference Optimization
Authors: Xiaofeng Tan
Hongsong Wang
Xin Geng
Pan Zhou
Venue: NeurIPS 2025 poster ()
Keywords: Motion Generation
DPO
RLHF
Primary area: applications
TL;DR: We propose SoPo, a semi-online preference optimization method, combining the strengths of online and offline direct preference optimization to overcome their individual shortcomings.

Abstract: Text-to-motion generation is essential for advancing the creative industry but often presents challenges in producing consistent, realistic motions. To address this, we focus on fine-tuning text-to-motion models to consistently favor high-quality, human-preferred motions—a critical yet largely unexplored problem. In this work, we theoretically investigate the DPO under both online and offline settings, and reveal their respective limitation: overfitting in offline DPO, and biased sampling in online DPO. Building on our theoretical insights, we introduce Semi-online Preference Optimization (SoPo), a DPO-based method for training text-to-motion models using ``semi-online” data pair, consisting of unpreferred motion from online distribution and preferred motion in offline datasets. This method leverages both online and offline DPO, allowing each to compensate for the other’s limitations. Extensive experiments demonstrate that SoPo outperforms other preference alignment methods, with an MM-Dist of 3.25\% (vs e.g. 0.76\% of MoDiPO) on the MLD model, 2.91\% (vs e.g. 0.66\% of MoDiPO) on MDM model, respectively. Additionally, the MLD model fine-tuned by our SoPo surpasses the SoTA model in terms of R-precision and MM Dist. Visualization results also show the efficacy of our SoPo in preference alignment.  Project page: https://xiaofeng-tan.github.io/projects/SoPo/.

SoPo: Text-to-Motion Generation Using Semi-Online Preference Optimization
Xiaofeng Tan 1.2
Hongsong Wang 1,2+ Xin Gengle 1.2
Pan Zhou
DDpaarmeent of Computer Science and Engineering, Southeast University, Nanjing, China "Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University). Ministry of Education, Nanjing, China Singapore Management University (xiaofengtan, hongsongsang, xgeng) edu. en, panzhoudsmu. edu. edu.sg.
Abstract
Text-to-motion generation is essential for advancing the creative industry but often presents challenges in producing consistent, realistic motions. To address this, we focus on fine-tuning text-to-motion models to consistently favor high- quality, human-preferred motions critical yet largely unexplored problem. In this work, we theoretically investigate the DPO under both online and offline settings, and reveal their respective limitation: overfitting in offline DPO, and biased sampling in online DPO. Building on our theoretical insights, we introduce Semi-online Preference Optimization (SoPo), DPO-based method for training text-to-motion models using "semi-online" data pair, consisting of unpreferred motion from online distribution and preferred motion in offline datasets. This method leverages both online and offline DPO, allowing each to compensate for the other's limitations. Extensive experiments demonstrate that SoPo outperforms other preference alignment methods, with an MM-Dist of 3.25% (vs e.g. 0.76% of MoDiPO) on the MLD model, 2.91 % (vs e.g. 0.66% of MoDiPO) on MDM model, respectively. Additionally, the MLD model fine-tuned by our SoPo surpasses the SoTA model in terms R-precision and MM Dist. Visualization results also show the efficacy of our SoPo in preference alignment. Project page: https: eixiaofeng---o tan. winn.gittu....... io/projecta/Sooo/
Introduction
Text-to-motion generation aims to synthesize realistic 3D human motions based on textual descrip- tions, unlocking numerous applications in gaming, filmmaking, virtual and augmented reality, and robotics [1-4]. Recent advances in generative models [5-7], particularly diffusion models [1, 8- 14], have significantly improved text-to-video generation. However, text-to-motion models often encounter challenges generating consistent and realistic motions due to several key factors.
Firstly, models are often trained on diverse text-motion pairs where descriptions vary widely in style, detail, and purpose. This variance can cause inconsistencies, producing motions that do not always meet realism or accuracy standards [15-17]. Secondly, text-to- to-ttoooooon models are probabilistie, allowing diverse outputs for each description. While this promotes variety, it also increases the chances of generating undesirable variations [4]. Lastly, the complexity of coordinating multiple flexible human joints results in unpredictable outcomes, increasing the difficulty of achieving smooth and realistic motion [16]. Together, these factors limit the quality and reliability of current methods of moxt-to-motion generation.
"Corresponding Author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Frowt
right Es
w/o Soflo wi SaPo
wia SoPo w/ SoPo (b) kncels down then stands back up
wio SuPo
w/ SoPo
wio Solo
wi SoPo
(a) Somcone with difficulty
The person dribbling busketball backwards
(d) Person is sprinting to their left
right
SoPo
SoPo
(c) A persun, standing.
(1) A person and lost his
A person stands left log forward in karate pose, leans forward and makess high karate poso
hand
balanee, .a.cht himself
their right log
Figure I: Visual results on HumanML3D dataset. We integrate SoPo into MDM [13] and MLD [1]. respectively. Our SoPo improves the alignment between text and motion preferences.
In this work, we focus on refining cox-tto-motion models consistently generate high-quality and human-predt numan-preferred motions, largely unexplored but essential area given its wide applicability. To our knowledge, MoDiPO [9] is the only work directly addressing this. MoDiPO applies a preference alignment method, DPO [18], originally developed for language and text-to-image models, to the text-to-motion domain. This approach fine-tunes models on datasets where each description pairs with both preferred and unpreferred motions, guiding the model toward more desirable outputs. Despite MoDiPO's promising results, challenges remain, as undesired motions continue arise, as shown in Fig. 1. Unfortunately, this issue is still underexplored, with limited efforts directed at advancing preference alignment approaches to mitigate it effectively.
Contributions. Building upon MoDiPO, this work addresses the above problem, and derives some new results and alternatives for text-to-motion generation alignment. Particularly, we theoretically investigate the limitations of online and offline DPO, and then propose a Semi-Onlin Preference Optimization (SoPo) to solve the alignment issues in online and offline DPO for text-to-motion generation. Our contributions are highlighted below.
Our first contribution is the explicit revelation of the limitations of both online and offline DPO. Online DPO is constrained by biased sampling, resulting in high-preference scores that limit the preference gap between preferred and unpreferred motions. Mcanwhile, offline DPO suffers from overfitting due to limited labeled preference data, especially for unpreferred motions, leading to poor generalization. This leads inconsistent performance aligning preferences for existing methods.
Inspired by our theory, we propose a novel and effective SoPo method to address these limitations. SoPo trains models "semi-online" data pairs that incorporate high-quality preferred motions from offline datasets alongside diverse unpreferred motions generated dynamically. This blend leverages the offline dataset's human-labeled quality counter online DPO's preference gap issues, while the dynamically generated unpreferred motions mitigate offline DPO's overfitting.
Finally, extensive experimental results like Fig. show that our SoPo significantly outperforms the SoTA baselines. For example, on the HumanMLL.D dataset, integrating our SoPo into MLD brings 0.222 in Diversity and 3.25% in MM Dist improvement. By comparison, combining MLD with MoDiPO only bring 0.091 and -0.01% respectively. These results underscore SoPo's effectiveness in improving human-preference alignment in ext-to-motion generation.
Related Works
Text-to-Motion Generation. Text-to-motion generation [10, 19-21] is a key research area with broad applications in computer vision. Recently, diffusion-based models have shown remarkable progress by enhancing both the quality and diversity of generated motions with stable training [2, 11-13]. MotionDiffuse [14] is pioneering text-driven diffusion model that enables fine-grained body control

---

and flexible, arbitrary-lenget motion synthesis. Tevet et al. [13] propose a transformer-bases diffusion model using geometric losses for better training and performance. Chen al. [1] improve efficiency by combining latent space and conditional diffusion. Kong et al. [8] enhance diversity with a discrete representation and adaptive noise schedule. Dai et al. [2] present a real-time controllable model using latent consisteney distillation for efficient and high-quality generation. Despite these advances, generating realistic motions that align closely with text remains challenging. Despite significant progress in skeleton-based motion understanding achieved by unified foundational models [22, 23), these generative models still exhibit limitations in the semantic and spatial complexities understanding. Thus, how to enhance the generative ability by discriminative model remain necessary to explore.
Direct Preference Optimization. RLHF [24] aims to align model distributions over pre-defined preference distributions under the same conditions. As representative RL method, Direct Preference Optimization (DPO) has shown great success in large language models (LL.Ms) [18, 25J, text-to- 3D [26], and image generation [27-31], offering a promising solution to the aforementioned issue. Existing methods are broadly categorized into offline [27, 32] and online DPO [28-31] Offline DPO trains on fixed datasets with preference labels from humans [27] or AI feedback [9]. In contrast, online DPO generates data online using a policy [31] or a reference model [29], and forms preference pairs via human [28] or AI feedback [32]. While effective in text-to-image generation, DPO methods for text-to-motion e.g. MoDiPO [9]-remane remain underexplored and face challenges such as overfitting and insufficient preference gaps. More discussion about recent RL research are shown in App. D.
3 Motivation: Rethink Offline & Online DPO
Preliminaries. Here we analyze DPO MoDiPO explain its inferior alignment performance for text-to-motion generation. To this end, we first briefly introduce DPO [18]. Let 'D be a preference dataset which comprises numerous triples, each containing a text condition e and a motion pair x. > x' where xh and xf respectively denote the preferred motion and unpreferred one. With this dataset, Reinforcement Learning from Human Feedback (RI.F)) [33] first trains a reward model r(z, c) to access the quality of z under the condition c. Then RLHF maximizes cumulative rewards while maintaining a KL constraint between the policy model Te and reference model Tref:
max (-1e
I
(r(x,c) DKi. (No(a)c) || [ea(()))))]
(1)
Here one often uses the frozen pretrained model as the reference model mer and current trainable text-to-motion model as the policy model me-
Building upon RLHF, DPO [18] analyzes the close solution of problem in Eq. (I) to simplify its loss:
Lovo(0) (0) E( ,0 log (3Ho(a" (
x², ,e)
(2)
where Ha< ,x,,c) = h= c) , c), (2, c) = log xu(z]c) )t((2])) and a is the logistic function.
When there are multiple preferred motions (responses) under a condition c, i.c., x'>x²>... x.K (K 2 2), by using Plackett-Luce model [34], DPO can be extended as:
exp( (xx .c) Loa -Eegli 10)- log II
(3)
Ej=t exp( (2))
When K = 2, Coff degenerates to Lpoo- Since MoDiPO uses multiple preferred motions for alignment, we will focus analyze the general formulation in Eq. (3).
3.1 Offline DPO
Analysis. In Eq. (3), its training data are sampled from an offline dataset So DPO in Eq. (3) is also called "offline DPO". Here we analyze its preference optimization with its proof in App. C.I
Theorem 1. Given a preference motion dataset D, reference model Tyef and ground-truth prefer- ence distribution Pet, the gradient of V of can be written as:
VoLost (0) -E-DD... D&DKu (pp)).)).
(4)
Here po pa(2::K10 = IIK -- (x) represents the likelihood that policy model generates motions (exp hata" ( ee1²
x! matching their rankings, where po (xx (c) Ex sexp hota) .013
3

---

Theorem 1 shows that the gradient of offline DPO aligns with the gradient of the forward KL divergence, Drr. (Paullo)) This suggests that the policy model Po (i.e., the trainable text-to-motion model) is optimized to match its distribution with the ground-truth motion preference distribution Pzt-
Discussion. However, since training data is drawn from fixed dataset D, the model risks overfitting, particularly on unpreferred samples. Due to limited annotations, text-to-motion datasets typically contain only one preferred motion group =1:K per condition c, making Pga(-lc resemble a one-point distribution, i.e., Pgt (x! (c) 1. In this case, minimizing Prc.(Pet Po) reduces maximizing likelihood: min DKL (Pgt llpe)
min - lom--logoo Po (2r1. (c)- As result, offline DPO progressively increases po lc), widening the preference gap between preferred and unpre- ferred motions. As illustrated in Fig. 2, the model primarily learns from the fixed motion group xe for each e, caus- ing the internal gap within x!.K expand. This overfitting effect, also noted in [35], suggests that with limited unpre- ferred data, the model learns to avoid only specific patterns (e.g., red regions in Fig. 2) while ignoring many common un- preferred motions. Despite this limitation, the offline dataset is manually labeled and provides valuable preference infor- mation, where the gap between preferred and unpreferred motions is large, benefiting learning preferred motions.
Overlinin
Prokebill salak)
Aligael Probeliiiy
Oypegiona reginnu
Figure Overfitting in offline DPO: green/red points are preferred/unpre- ferred motions: blue shows bias from fixed unpreferred data, red indicates uncovered unpreferred regions.
3.2 Online DPO
Analysis. In each online DPO training iteration, the current policy model ITe generates K samples for given text c. A pretrained reward model r ranks them by preference as xhe > 2 > ... > zg. where x is sampled from n0 without gradient backpropagation. Using the Plackett-Luce model [34], the probability of a being ranked k-th is given by:
exp rxpr
Pr ( To lc)
(5)
Σ expr
Then we can analyze online DPO below. Theorem 2. Given a reward model and reference model Teet, for the online DPO loss Con. its gradient is:
oo (0) = BewD. Vo Pry (=,(x (x.: (x) (c)
(6)
where Pa, (c):K(c) T PR, (x*1c) with pao (x* (c) being the generative probability of policy model to (exp (ag (0))
generate x² conditioned on c, and ps p=(x²) = Σx "i (exp ()( denotes the likehood that policy model
generates motion I& with k-th largest probubility.
See the proof in App. C.2. Theorem 2 indicates that online DPO minimizes the forward KL divergence DKL (p-llpe). Thus, online DPO trains the policy model ne, i..., the text-to-motion model, to align its moxt-to-motion distribution with the online preference distribution pr (z|c).
Discussion. We diseuss the training bias and limitations of online DPO. Specifically, motions with high generative probability Pa, (I=, (c) are frequently synthesized and thus dominate the training of mo- In contrast, motions with low generative mobabiiiiy-dessppp rarely generated and scarcely contribute to training. Notably, when Pro (Ia, (e) + 0 but the reward C( + 1, the gradient still vanishes: limp,, (=+, (F)e,,, VeLon
potentially high human preference-
(see derivation in App. C.2). This highlights a key limitation: online DPO tends to ignore valuable but infrequent preferred motions, focusing instead on commonly generated ones regardless of their actual preference.
Additionally, online DPO aligns the generative probability Pa, (Iag le) with the preference distribution (I (c). leading to a positive correlation. Thus, motions with higher generative probabilities often exhibit higher preferences. However, since preference rankings are determined by reward model, roughly half of these high-preference motions -hhioe with lower rankings i despite high scores still treated unpreferred. As result, many unpreferred training motions retain
r(*,, R.. c) considerable preference, reducing the preference gap compared to manually labeled offline datasets.
On the other hand, online DPO dynamically generates diverse motions, particularly unpreferred motions, in each iteration. This dynamic process enriches preference information and mitigates the overfitting observed in offline DPO, enabling the model to avoid the undesired patterns.

---

Unpreferred motions do focus the weakness model
Preferred motions sampled from probb-iroaability regions
Combine their benefits and overcome their disadvantages
Rcward Darikase
Olline prrkmali miliets
Oitiine cepestence masson Deet
Sampling distribution
doaal sppestennd merion (Lerect)
Roverl Diinceses Oelire potered mutsen Coliee aqpeefared zudieaa
-... Kower Dhititakes
Offier peefered tutkea
Onina sxaefore natiees
Berrtaak
Office peckearreuttn
Optimized distribution
Office capudfered castion (See)
Lag roteree masiion (Unset)
Reward Daniiiatio
Rowar Dieribution offiie preflrred makkes Dalies expetinnt invises
Ordine peefirnd medees Ord ceteeeoppeeenaaanoooo raudimm
Offline DPO
Online DPO
SoPo (Ours)
Figure 3: Comparison of offline, online DPO, and our SoPo synthetic data. Offline DPO suffers from mining unpreferred motions with high probability, and online DPO is limited by biased sampling. Our SoPo utilizes the dynamic unpreferred motions and preferred motions from unbiased offline dataset, overcoming their advantage. Here, the blue region the distribution of generative model.
3.3 DPO-based methods for Text-to-Motion
Analysis. DPO in MoDiPO [9] uses offiine dataset 'D that is indeed generated by pre-trained model xp. denoted as:
x 22 argmax 1:K "p Enp expr xxy argmin ER, expr(xt,,,))
(0",, of c)(c offline textural sets).
(7)
For discussion, we formulate its sampled distribution as:
Mo Pgt (Fw, = I(( 21, D),
(8)
where the indication function I(E) 1 event & happens; otherwise, I(E) = =
From Eq. (7), we observe that, like online DPO, MoDiPO samples preference motions from the distribution Pr, (xree induced by the pre-trained model Tp- This leads two main issues like online DPO. 1) Samples with low generative probability Pm, (z|c) but high preferences r(z, c) are rarely generated by up and thus seldom contribute to training, even though they are highly desirable motions. 2) As discussed in Sec. 3.2, the motions T,, generated by πp typically exhibit both high generative probability and preference scores, which causes half the preferred samples to be selected as unpreferred, skewing the model's learning process. See the detailed discussion in 3.2.
Additionally, from Eq. (8), we see that for a given condition c, MoDiPO trains on fixed preference data, similar to offline DPO. Consequently, MoDiPO is limited to avoiding only the unpreferred motions valued by the pre-trained model up rather than those relevant to the policy model nø. Thus, it inherits the limitations of both online and offline DPO, constraining the alignment performance.
5

---

Semi-Online Preference Optimization
4.1 Overview of SoPo
We introduce our Semi-Online Preference Optimization (SoPo) to address the limitations in both online and offiin DPO for text-to-motion generation. Its idea is to train the text-to-motion model on semi-online data pairs, where high-preference motions are from offline datasets, while low-preference and high-diversity unpreferred motions are generated online.
As discussed See. 3, offline DPO provides high-preference motions with a clear preference gap from unpreferred ones but tends to overfit due to reliance fixed, single-source unpreferred motions. In contrast, online DPO benefits from diverse, dynamically generated data but often lacks a sufficient preference gap and overlooks low-probabiligg preferred motions. To leverage the strengths of both, SoPo samples diverse unpreferred motions =²,, from online generation and high-preference motions T. from offline datasets, ensuring broad gap between them. Thus, SoPo mitigates the overfitting of offline DPO and the insufficient preference gaps in online DPO. Accordingly, we arrive at our SoPo:
LDSoPo (0) = Es (0)= (et...e)eBBe....e... (elnagtll)) log o (BHo(x") .c)),
(9)
where Ha(x", x², c) is defined below Eq. (2), x is preferred motion from the offline dataset, and x unpreferred motion sampled from online DPO. To demonstrate the advantages of SoPo, we conduct experiments synthetic data, as shown in Fig. 3 (Detailed experimental settings in App. A.2).
However, direct online generation of unpreferred motions from the policy model presents challenges, given the positive correlation between the generative distribution ps, and preference distribution pr- Additionally, large gap between preferred and unpreferred motions remains essential for effective SoPo. In Sec. 4.2 and 4.3, we receptively elaborate on SoPo's 's designs to address these challenges.
4.2 Online Generation for Unpreferred Motions
Here introduce our generation pipeline for diverse unpreferred motions. Specifically, given condition c, we first generate K motions {a 1K the lowest preference value:
=== from the policy model тT. and select the one with
R9 arrminn (r4,)) r( (
(10)
However, xik, could still exhibit a relatively high preference r(ak,; c) due to the positive correlation between the generative probability Px, and preference distribution Pr (see Sec. 3.2 or 3.3). To identify genuinely unpreferred motions, we apply a threshold T to the set {a [==1 and check if any preference score below it. This leads to two training strategies based on the result.
The group (=+, H contains a low-preference unpreferred motion ot,- Then we select these unpreferred motions iteratively which ensure diversity due to randomness of online generations and address the diversity lacking issue in offline DPO.
Case 1:
The group contains no low-preference unpreferred motion 2 meaning all sampled motions are of high preference and should be treated as unpreferred. This suggests the model performs well under condition c, so training should focus on high-quality preferred motions from offline data further enhance generation quality.
Case 2:
To operationalize this, we apply: (1) distribution separation and (2) training loss amendment.
(1) Distribution separation: With a threshold separate the distribution Pr, (#1 (c) into two sub-distributions:
1:K (e) Ps. (I
Pao (a 1:K (c)p, (r(e'ng 2)2T)
+ (1 1:K (e)p, (r(a't c)<7).
(11)
relanively high-ppefereecce unpreferred motions s/i
valuable unpreferred mnttions
where P=, 1: K. (c) =
= Pao (x*1 c). Pao (x* is the generative probability of policy model mo to generate I conditioned on c, pr (r(rrts c)2T) is the probability of the event r(x1,) 2 and p- Dr(r(1t ()) has similar meaning.
Eq. (11) indicates that the online generative distribution (- 1:K (c) can be separated according to whether the sampled motion zi:K group contains valuable unpreferred motions. Accordingly, our

---

objective loss in Eq. (9) can also be divided into two ones: LDSOPO (0) - (0) + Luu (f), where (0) targets valuable unpreferred motions and (0) targets tigh-preference unpreferred motions:
Lva = ((.m.c)) (c) 1 a . (10 log o (3Ho(x" x));
(12)
Dbu = Re ku (c) R 1:K ~ ( (Hcc
log a (5Ho (a ;)),
where Ha(a") 2 is defined in Eq. (2), Pag.. ())
Z.u(c) and Fio ())
Pey
(-)
Znule) respectively
Pty()
denote the distributions of valuable unpreferred and high-preference unpreferred motions. Here
Zva (c) Pa: (a)) )x and Zu (c) = f Pag- (ar)dx are the partition functions, and are unnecessary to be computed in implementation (More discussion provided in App. C.3).
Pag
(2) Training loss amendment: As discussed above, unpreferred motions in case have relatively Segh-preference (score 2 T), and thus should not be classified into unpreferred motions for training. Accordingly, we rewrite the loss Chu (6) into LusoPo-hu(B for filtering them:
LUSoPo-bee hu (0) Ba . Zau log (ofo(zz..)) LuSoPo(0) LusoPo- Lv) (13)
See more discussion on LUsopo/LDSoPo in App. C.4.
4.3 Offline Sampling for Preferred Motions
As discussed, online DPO suffers from a limited preference gap between preferred and unpreferred motions. While high-quality motions from offline datasets can help mitigate this issue, they may not always differ significantly from generated motions especially when the model is well-aligned with the dataset. Thus, motions with larger preference gaps (Sec. 4.2) crucial and should be prioritized.
To utilize the generated unpreferred motion set Dr conditioned on from Sec. 4.2, we calculate its proximity with the unpreferred motions in De using cosine similarity:
S(x min cos(x"" """)""")))))))) x rt ~D.
Then we reweight the loss using Bu (Iw) = 3(C - S(x² with a constant C' 2 1: (c) log a (ww("") "oo("" - Sho ct)]
Lsoro = Fe .) a ~ (.lc)
(14)
.)) Zho (c) log (L. Mole"" (c)(e"")))..
As similar samples have similar preferences, this reweighting strategy guides the model to priori- tize preferred motions with significant preference gap from unpreferred ones. Accordingly, this reweighting strategy relieves and even addresses the small preference gap issue in online DPO.
4.4 SoPo for Diffusion-Basee Text-to-Motion
Recently, diffusion coxt-to-motion models have achieved remarkable success [2, 6, 11, 12], enabling the generation of diverse and realistic motion sequences. Inspired by [27], we derive the objective function of SoPo for diffusion-based text-to-image generation (See proof in App. C.5):
SoPo
Adiff So +
Aiif SSPoobuu
(15)
CaPe -7, w (o)) (c) lo Twi
B200 ref, =b1)]]
diif BaPo-ba = -tw.a--o-o--odd............... (c)
B ref, xef,
(16)
where L(0, ref, xx) L(0, L(ref, 2z), and C(0/ref, xIt) loss of the policy or reference model. Equivalently, we optimize the following form
||E0/ret (Tz, - e112 denotes the
log a loga
Tw. ((0 xF) ref,
if e) otherwise.
Cai -B ().""" oo 1i (1())
(T.)
(17)
where xf = a M ~o r r c). Proof and more details are provided in App. B.

---

Table I: Quantitative results of preference alignment methods for text-to-motion generation
the HumanMLLD test set. Results are borrowed from those reported in [9]. The subscripts in each cell denotes the relative performance change. Superscript "t" marks the largest improvement across all models; gray background highlights the largest improvement for each model. "Time" denotes estimated online/offinee motion generation time, with "IX" the time for MIDD [1] to generate all HumanMI motions and "K" (unspecified in [9I. typically 2~6) as the number of motion pairs.
Methods
Time
R-Precision
MM Dist 2
Diverait +
FID
Top
Top
Top
Real
0 4.511-42003
0,703-220M
0.7024-NG
2.7740
9.50g-M006
0,002
MLD [1]
-0.X
0,453
0.67961
0.755
3.2924000
9,79
0.15920m
McDiPO-T [9]
-121K'X X
0.455-8/012
0.683-22/03
0 0.758 178g
3.2671.00 TuTE
9.74744075
0.303430633
McDiPO-G
-121K
0.678-2-0052 1015
0.75paruus aaaus
1.20 ond
7024 tm
10.re1
SoPo (Ouss
McDIPO-O [9]
+
0.682571000
0.6099 1813
D1 01765en0
3. 1155
3.701 tn tn
4N
9.525 +0.z04
om
0.374
MDM [13]
-0.X
0,418422055
0.604-22005
0. 0.503-400
9.546 10.
0.501
MoDIPO T 19]
11A355
117706
3.634
9.531
0-45)
MDM (fas)) [15]
McDIPO
-121K
0.42940.006
40.00
0.632422.005
0.645-22007
0.74 to
10 14.1555
3,641 4n
9.504 oea
10 4655
9.455 10.07
949
in
+00005
0.486 +n.00
SoPo (Durs)
-60 X
0,478
0.674
surs
0.770
1/2 sis
9.6
0.480
10
Experiment
Datasets & Evaluation Metrics. For text-to-motion generation, we evaluate SoPo on two widely used datasets, HumanML3D [3] and KIT-ML [36], focusing on two key aspects: alignment and generation quality. Alignment is assessed using R- E-Precision and MM Dist, while generation quality is measured by Diversity and FID. For text-to-image generation, we utilize Flux-Dev [37] as the foundational model and employ HPSv2 [38] as the reward model. Further results and details are in App. I.
Implementation Details. Due to limited preference-labeled motion data, we use existing datasets (e.g., HumanML3D, KIT-ML) as offline preferred motions. For online generation of unpreferred motions, we use TMR, a texl-to-motion retrieval model [39]. as the reward model. Hyperparameters K and 7 are tuned through preliminary experiments to balance performance and efficiency, with T = 0.45, C = 2, and 8 = 1 in Eq. (14). We set K = 4 for MDM [40] and K = 2 for MLD [1]. All models are trained in 100 minutes on single NVIDIA GeForce RTX 4090D GPU. Since MLD" [2] is tailored for HumamML3D, we use MLD [1] for KIT-ML. More details are in App. A.4.
5.1 Main Results on Text-to-Motion Generation
Settings. We evaluate SoPo for preference alignment and motion generation, comparing it with state-of-the-art preference alignment [9] and texl-to-motion methods [1, 7J. For fairness, we fine-tune MLD [1] and MDM [13] with SoPo, using a fast diffusion variant [13] with 50 sampling steps. We also fine-tune MLD* [2] as stronger baseline. Since MLD" is not adapted to KIT-ML, we use MLD [1] and MoMask [44] for diffusion- and autoregressive methods, respectively.
Table 2: Quantitative comparison of ttate-of-the-art text-to-motion generation on the Hu- Man test set. "MLD* refers to the enhanced reproduction of MLD [1] from [2]. For fair comparison, we selected the "LMM-T" [41] with a similar size to ours.
Methods
Year
R-Precision 1
MM Dist
Diversity + Multimodal
FID 1
Top
Top
Top 3
Avg.
Real
0
0.703960005
0,79750.002
0.670
2.79480000
9.503=5.063
0.002
TEMOS [400
2022
0,424=
0.61250.0zz
0.72240.002
0.586
3.703
8.973=5.071
0.368
3,734
T2M [3]
2022
0.457
0.639 60.000
0.740 40.000
0.612
3.340
9.188 0.000
2.090 10.03
1.067 40.07
MDM [13)
2022
0.418
0.604555000
4.705400006
0.575
3.658
9.546=6.00
2.799:00077
0.501
2023
0.481
10.00
0.673
0.772 5..020 1000
1146
9.02
0.473-10-00
MationGPT [42]
0.402
0.681 20000s
0.650
9.528 mm
2000
0.232:20000
Mitionilluse [14]
2024
0.191 20004
0.681
0.782
0.651
3. 3.113 11300.
9.410=5.00
1 15555400042
0.630
GMG [43]
2024
0.784
9.657=9.0es
0.38 110.000
Wang ct.
0.033
n0.co
k0.000
0.598
3.4 0GL
2,835
0.352
McDiPO.T [9]
2024
0.455:0000
0.68255-000
0.75
3.267=40
9.247
2.603-00111
0.303
PriceDM [12]
2024
0.48]
5.610
9.620=6.00
Ii 1.60p
LMM-T [41]
2024
0.4% 2000
0.685 200000
0.78559.00
0.685
9.176=5.0nz
0:415:20.00
CrossDin [11]
2024
3.358-1.011
9.577
0.28 230.00
Motion Mambu [7]
2024
0
0.69
t.792
0.662
9.871 9.871
2 2.
0,28 20001
MLD' [I.
2021
0,50
0.69g
0,795
50.002
0.666
3.052
9.634
2 22675
0.100
MLDD (2).s. sec.
0.528 44 mu56
0.722 3.46%
0,827
0.692
2,939
4% uot
9.584
15
0.174 an 181
8

---

Comparison with Preference Alignment Methods. Table I compares preference alignment methods. MoDiPO, DPO-based method for motion generation, faces overfitting and biased sampling issues [18]. Conversely, our SoPo method uses diverse high-probabilityy unpreferred and high-quality preferred motions, improving generation quality and reducing unpreferred motions. SoPo excels in most metrics except FID, with R-Precision gains of 5.27%, 4.50%, and 2.80% (vs. baseline 0.42%) and a 3.25% MM Dist, improvement (vs. MoDiPO's -12.4% to +0. 76%). SoPo boosts Diversity by 0.268 (vs. MoDiPO's 000018 to 0.091). Despite MoDiPO's slight FID edge, SoPo's results are comparable, owing to conservative training on pwwporbabilityy, high-preference samples. SoPo also eliminates pairwise labels and cuts preference data generation time to ~ 1/10 of that MoDiPO. Comparison with Motion Generation Methods. We evaluate SoPo on Hu- manML33D [3], with results in Table 2. Us- ing preference alignment, SoPo surpasses state-of-the-art methods in R-Precision, MM Dist, and FID, achieving the best per- formance. Although Mo [42] has slightly higher Diversity (9.584 vs. 9.528), SoPo improves R- R-Precision by 6.46%, FID by 33.5%, and MM Dist by 5.34%. Com- pared to Motion Mamba and CrossDiff, SoPo increases Diversity by 0.287 and re- duces MM Dist by 12.5%. It also enhances 'LD" FIID by 61.3% On KIT-MI. (Table 3). SoPo with MoMask [44] achieves the best results across all metrics: Top-k R-Precision (0.446, 0.673, 0.797), MM Dist (2.783), and FID (0.176). MLD w/ SoPo ootoooouppoomnooo outperforms its original version, confirming its effectiveness across model architectures.
Table 3: Comparison of text-to-motion generation performance on the KIT-ML dataset.
Methods
Precision
FID I
MM Dist
Divenity
Top
Top 2
Top 3
0.649
0.779
0.031
2.788
11.08
TEMOS [10]
0.370
0.569
0.693
2.770
3.401
10.91
T2M
0.361
0.559
0.681
3.022
2.052
10.72
MLD [1]
0.734
0.404
10.50
[2M-GPT[4S]
0.416
0.627
0.745
0.514
3.007
10.86
MotionGPPT [42]
0.366
0,558
0,680
0,510
3.527
10:35
Motion Diffose 14]
0.621
0.739
11.10
Mamba [7]
0.419
0.645
0.765
0.307
3.021
11.02
McMask [44]
0.435
0.656
0.781
0.204
2.779
10.71
MLD [T]. sate
0.646
0.384
10.93
MoMask (44).. Sate
0.446
0.673
0.797
0.176
2.783
10,96
Quantitative Evaluation of Ppaatil-PPrception Motion Generation via SoPo. We quantitatively analyze the efficacy of our SoPo in resolving issues related Spatial-Perception Motion Generation shown in Fig. Experimental setting detailed in App. A.3. As exhibited in Fig. 4(a), these results confirm SoPo's effectiveness in enhancing pertal-peroettion capabilities.
5.2 Ablation Studies
Impact of Sample Size K. Due to computational and memory constraints, we recommend keeping
K < 8. As shown in Table 4, increasing K significantly improves generation quality. A larger sample pool allows the reward model to better evaluate and filter unpreferred motions, leading to more accurate guidance and higher-quality results.
Impact of Objective Functions. We fine-tune MDM [13] using four ob- jectives: DSoPo (Eq. (12)), USoPo (Eq. (13)). SoPo without value- unpreferred (VU), and full SoPo (Eq. (14)). As shown in Table 4. DSoPo alleviates limitations of of- fline/onliee DPO (Sec. 4.1) and im- proves FID by 7.30%. Removing VU further boosts FID to 8.98% by em-
Table 4: Ablation study on alignment methods, thresholds T, and sampled number K.
Methods
Precision
FID
MM Dist
Diversity
Top
Top
Top
MDM dastt [13]
.655
9.948
.534
DSaPP
.600,1 oxx 655 .1.51.55
55%
30297.000x
9.925-p.00m
.455 17.444
SoPP VU
460, A60.
3.205
20.0x
9.915
486
USaPo
475
668
367
3.226
9.901
556
SoPo
.799 2577
.770
3,206, 32208,
9.906
.450
ir 0.40)
661.
J68.2.51
3.272..077
10.04_p.cos
6600_
Co 0.45)
.779
.770
.270
3.208, :2005
9.906.p.002
.880_.0.155
(o 0.50)
468.2.99
46542299
764
3.256,11405%
4856
9.900-.0.00
401
phasizing preferred motions that differ
(r 0.55)
A66.2.415
660.
7633
3.263,
9.896
.130..10.55 10.9%
(o 0.60)
461.1.1
3.288.
9,803
3399
from unpreferred ones. USoPo, using a threshold r to filter unpreferred mo- tions, enhances R- R-Prrcision (+3.96%).
.71 .200s
3.212.27705
9.907
-0.0c)
550._..009
+SPPo 4)
6644
3770,2002
3,208,2m0g
...06.....0.
1800.1011 15
MM Dist (+2.36%), and Diversity (+0.047), though FID slightly drops (-4. (.4.2%)) Combining all advantages, SoPo achieves the best results: +5.27% R-Precision and +10.1 FID.
Impact of Cut-Off Thresholds T. Table reports results with T ranging from 0.40 to 0.60. A lower T leads to stricter filtering, yielding more reliable unpreferred motions. As decreases, R-Precision and MM Dist improve, indicating better alignment. contrast, higher 7 values improve FID and Diversity, suggesting enhanced generative quality due to exposure to more diverse samples. More
9

---

0.402
MLD (wie SaPo Preferred
Same
MLDD (w/ SoPo) Preferred
0.75
A24
Modlan Text Aliammnt
15%
38%
47%
Accuracy
0.70
0.65
Physical Fideliry
22%
18%
65%
0.60
Colerence and Flanacy
27%
0.50
60
100
TMR (Rewant Model)
MLD (Blaed Model)
MLD w/ SoPo (Owns)
Preference Rate (%)
(a)
(b)
Figure 4: Quantitative results on (a)
motion generation, and (b) user study.
experimental results, including ablation of training strategy and DPO hyper-parameter are shown in
App. A.5.
Impact of Training Strategy. To compare different training strategy, we conducted new experiments com- paring online DPO (ON. DPO), of- fline DPO (Off. DPO), their naive combination (Com. DPO), and com- bination with our proposed strategies (SoPo), as shown in Table 5. These results highlight that SoPo's hybrid semi-online design provides more effective and kata-efficient alignment, avoiding the limitations of both pure online and offline DPO.
Table 5: Ablation study on training strategy.
Precision
Methods
FID +
MM Dist J Diversity
Top 1
Top 2 Top 3
MLD [2] 0.504
0.698 0.796 3.052
9.634
0.450
Off.DPO
0.498
0.692 0.791
3.080
9.620
0.470
On.DPO
0.514
0.709
0.808
3.010
9.610
0.410
Com.DPO
0.517
0.712
0.811
2,985
9.605
0.340
SoPo 0.528
0.722
0.827
2.939
9.584
0.174
5.3 Discussion on Reward Hacking.
User Study & Visualization. To assess whether our fine-tuned model exhibits reward hacking, we conducted a user study and visualized the corresponding motions, as shown in Fig. 4(b). Additionlly, we visualize results of our SoPo and existing methods, provided in App. A.6. These results confirm that our SoPo can avoid reward hacking by KL.-Divergence in (q)
6 Conclusion
In this study, we introduce a semi-online preference optimization method: a DPO-based fine-tune method for the text-to-motion model to directly align preference on "Semi-online data" consisting of high-quality preferred and diverse unpreferred motions. Our SoPo leverages the advantages both of online DPO and offline DPO, to overcome their own limitations. Furthermore, to ensure the validity of SoPo, we present simple yet effective online generation method along with an offline reweighing strategy. Extensive experimental results show the effectiveness of our SoPo.
Limitation discussion. SoPo relies on reward model to motion quality evaluation and identify usable unpreferred samples. However, research on reward models in the motion domain remains scarce, and current models, trained on specific datasets, exhibit limited generalization. Consequently, SoPo inherits these limitations, facing challenges in seamlessly fine-tuning diffusion models with reward models across diverse, open-domain scenarios.
Acknowledgements
This work was supported by the Jiangsu Science Foundation (BK20230833, BG2024036, BK20243012), the National Science Foundation of China (62302093, 52441503, 62125602. U24A20324, 92464301). the Fundamental Research Funds for the Central Universities (2242025K30024) the Open Research Fund of the State Key Laboratory of Multimodal Artifi- cial Intelligence Systems (E5SP060116) the Big Data Computing Center of Southeast University, and the Singapore Ministry of Education (MOE) Academic Research Fund (AcRF) Tier grant (Proposal ID: 23-SIS-SMU-0700 Any opinions, findings and conclusions or recommendations expressed this material are those of the author(s) and not reflect the views of the Ministry of Education, Singapore..
10

---

References
[1] Xin Chen, Biao Jiang, Wen Liu, Zilong Huang, Bin Fu, Tao Chen, and Gang Yu. Executing your commands via motion diffusion in latent space, In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 18000-18010, 2023. 1, 2. 3, 8,8.9
[2] Wenxun Dai, Ling-Hao Chen, Jingbo Wang, Jinpeng Liu, Bo Dai, and Yansong Tang. Motionlem: Real- time controllable motion generation via latent consistency model. In Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol, editors, European Conference on Computer Vision, pages 390 408, Cham, 2024. Springer Nature Switzerland. ISBN 100-3-031-726400
2, 7, 8, 10
[3] Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang. Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5142-5155 2022. 8, 9
[4] Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. Motiongpt: Human motion as foreign language. In A. Oh, T. Naumann, A. Globerson. K. Saenko, M. Hardt, and S. Levine. edi- tors, Advances in Neural Information Processing Systems, volume 36, pages 20067-20079. Curran As-
neurips. cts.p..c..pp..........eeeeooc.e. cc/ paper_______ ppaprrfflesse wttp:/eeeeeeeeeeeeeeeeeeereeeeee /2023/ file// wwtppper.pppeeppeeeeeeeee sociates, Inc., 2023. URL actps://proceediiee Conference erence. pdf. Paper-Conf
[5] Yin Wang. Zhiying Leng, Frederick W. B. Li. Shun- Chun-Cheng Wu, and Xiaohui Liang. Fg-12m: Fine-grained text-driven human motion generation via diffusion model. In IEEE/CVF International Conference on Computer Vision, pages 21978-21987, 2023.
[6] Zan Wang, Yixin Chen, Baoxiong Jia, Puhao Li, Jinlu Zhang, Jingze Zhang, Tengyu Liu, Yixin Zhu, Wei Liang, and Siyuan Huang. Move as you say, interact as you can: Language-guide guided human motion generation with scene affordance. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 433 2024. 7,8
[7] Zeyu Zhang. Akide Liu, Ian Reid, Richard Hartley, Bohan Zhuang, and Hao Tang. Motion mamba: Efficient and long sequence motion generation. In Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol, editors, European Conference Computer Vision, pages 265-282, Springer Nature Switzerland, 2024. 1, 9,3
[8] Hanyang Kong, Kehong Gong, Dongze Lian, Michael Bi Mi, and Xinchao Wang. Priority-Centric Human Motion Generation in Discrete Latent Space. . In IEEE/CVF International Conference on Computer Vision, pages 14760-14770, Los Alamitos, CA, USA, October 2023, IEBE. 1,3
Massimiliano Pappa, Luca Collorone, Giovanni Ficarra, Indro Spinelli, and Fabio Galasso. Modipo: text-lo-motion alignment via ai-feedback-driver direct preference optimization, 2024. URL https: / //axxiv. orgg aa/////a///5555 03803. 2, 3, 8
[10] Ekkasit Pinyoanuntapong. Pu Wang. Minwoo Lee, and Chen Chen. MMM: Generative Masked Motion Model. In JEEE/CVF Conference Computer Vision and Patiern Recognition, pages 1546-1555 IEEE, 2024. 2
[11] Zeping Ren, Shaoli Huang, and Xiu Li. Realistic human motion generation with cross-diflusion models. European Conference on Compuder Vision, 2024. 2, 7,3
[12] Yoni Shafir, Guy Tevet, Roy Kapon, and Amit Haim Bermano. Human motion diffusion generative prior. The Twelfth International Conference on Learning Representations, 2024. 7,8
[13] Guy Tevet, Sigal Raab, Brian Gordon, Yoni Shafir, Daniel Cohen-or, and Amit Haim Bermano. Human motion diffusion model. In International Conference on Learning Representations, 2023. URL https:
// /oppereeeeww..o foott//fommmtt wowoooouuissss Siskiiis 2, 9.
[14] Mingyuan Zhang, Zhongang Cai, Liang Pan. Fangzhou Hong. Xinying Guo, Lei Yang, and Ziwei Liu. Motiondifluse: Text-driven human motion generation with difflusion model. JEEE Transactions on Puttern Analysis and Machine Intelligence, 46(6):4115 4128, 2024. 1, 8.9
[15] Qiaosong Qi, Le Zhuo, Aixi Zhang, Yue Liao, Fei Fang. Si Liu, and Shuicheng Yan. Diffdance:: Cascaded human motion diffusion model for dance generation. In Proceedings of the 31st ACM International Conference on Multimedia, MM '23, page 3374-188. New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400701085. doi: 10. 11144//5881783331223777 URL https://doi. org/10 10. 1114//33518833 3612307.
11

---

[16] Wentao Zhu, Xiaoxuan Ma. Dongwoo Ro, Hai Ci, Jinlu Zhang, Jiaxin Shi. Feng Gao, Qi Tian, and Yizhou Wang. Human motion generation: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[17] Bizhu Wu, Jinheng Xie, Keming Shen. Zhe Kong, Jianfeng Ren, Ruibin Bai, Rong Qu, and Linlin Shen. Mg- motionllm: A unified framework for motion comprehension and generation across multiple granularities. In Proceedings of the Computer Vision and Puttern Recognition Conference, pages 27849-27855 2025. 1
[18] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: your language model is secretly a reward model. In Advances in Neural Information Processing Systems, Red Hook, NY, USA, 2024, Curran Associates Inc. 2, 3,
[19] Junfan Lin, Jianlong Chang, Lingbo Liu, Guanbin Li, Liang Lin, Qi Tian, and Chang-wer wen Chen. Being comes from not-being: Open-vocabulary texi-to-motion generation with wordless training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23222- 23222-33231, 2023. 2
[20] Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Yong Zhang, Hongwei Zhao, Hongtao Lu, Xi Shen, and Ying Shan. Generating human motion from textual descriptions with discrete representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14730- 14740, 2023.
[21] Matthias Plappert, Christian Mandery, and Tamim Asfour. Learning a bidirectional mapping between human whole-body motion and natural language using deep recurrent neural networks. Robotics and Autonomous Systems, 109:13-26, 2018. ISSN 0921-8890. doi:
201... 07.006.
URI. https: wwwp://wceneeewe..............ee
2
[22] Hongsong Wang, Wanjiang Weng, Junbo Wang, Fang Zhao, Guo-Sen Xie, Xin Geng, and Liang Wang. Foundation model for skeleton-baseh human action understanding. IEEE Transactions on Pottern Analysis and Machine Intelligence, 2025. 3
[23] Hongsong Wang. Xiaoyan Ma, Jidong Kuang, and Jie Gui. Heterogeneous akeleton-basee action representa- tion learning. In IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 19154-19164, 2025. 3
[24] Buhua Liu, Shitong Shao, Bao Li, Lichen Bai, Zhiqiang Xu, Haoyi Xiong, James Kwok, Sumi Helal, and Zeke Xie. Alignment of diffusion models: Fundamentals, challenges, and future. arXiv preprint axXiv:449.002553, 2024. 3
[25] Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. Direct language model alignment from online ai feedback. arXiv preprint arXiv: 047922 2024. 3, 13
[26] Junliang Ye, Fangfu Liu, Qixiu Li, Zhengyi Wang, Yikai Wang, Xinzhou Wang, Yueqi Duan, and Jun Zhu. Dreamreward: Text-to-3d generation with human preference. arXiv preprint arXiv:2403. 14613, 2024. 3
[27] Bram Wallace, Meihua Dang, Rafael Rafailov, I Lingi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion Model Alignment Using Direct Preference Optimization. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8228-8238, Los Alamitos, CA, USA, June 2024. IEEE Computer Society. doi: 10. 109 Tww.5227.3..22200000 2024.00786. URL /tttps://oii. ieeecomputersooiety org/10, /0. 1199/CVPR52733. 2024. 00786. 3, 7, 11
[28] Kai Yang. Jian Tao, Jiafei Lyu, Chunjiang Ge, Jiaxin Chen, Weihan Shen, Xiaolong Zhu, and Xiu Li. Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8941 8941-8951, Los Alamitos, CA, USA, June 2024. IEEE Computer Society. doi: 10.1 119/CVPR52733.22222222 2004000440.24. URL https: //doi leeeccomutersoociett org/10 10. 1w11/11/////////
2024. 00854. 3
[29] Daoan Zhang, Guangehen Lan, Dong-Jun Han, Wenlin Yao, Xiaoman Pan, Hongming Zhang, Mingxiao Li, Pengcheng Chen, Yu Dong, Christopher Brinton, and Jiebo Luo. Seppo: Semi-policy preference optimization for diffusion alignment, 2024. URL https: /tttpp:://rxxt
argg/bss/2411 2ttp///22...0.5.... 3
[30] Zichen Miao, Zhengyuan Yang, Kevin Lin. Ze Wang, Zicheng Liu. Lijuan Wang, and Qiang Qiu. Tuning dimeste--distilled diffusion model using pairwise sample optimization, 2024. URL https:/ h/tps:://///xvi .rrg/ abs/2410. 00111/90....99.....
[31] Zhanhao Liang. Yuhui Yuan, Shuyang Gu, Bohan Chen, Tiankai Hang. Ji Li, and Liang Zheng. Step-aware preference optimization: Aligning preference with denoising performance at each step. arXiv preprint arXiv:2406.de 04314, 2024. 3
12

---

[32] Sanghyeon Na. Yonggyu Kim, and Hyunjoon Lee. Boost your own human image generation model via direct preference optimization with ai feedback. ArXiv, abs/2405. 20211, 2024. URL https://api. semanticscholar Per//orruusDD :22rpuus::27012365.. 270123366. 3
[33] Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep rein- forcement learning from human preferences. NIPS' 17, page 4302-4310. Advances in Neural Information Processing Systems. 3
[34] R. L. Plackett. The analysis of permutations. Journal of the Royal Statistical Society. Seriey C (Applied Statistics), (4(2):193-200) 1975. ISSN 00359254, 14679876. URL http://wuw, jstor org/ stable/ 2346567. 4, 6,7
[35] Banghua Zhu, Michael Jordan, and Jiantao Jiao. Iterative data smoothing: Mitigating reward overfitting and overoptimization in RLHF. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41 International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 62405-6222 662488. PMIRR, 21-27 Jul 2024. URL http::ttp/////tttm //tpp//e////ccenng...mm. mlr pww.s/pessaeee htteess//35//eee 2224/// html.
[36] Matthias Plappert, Christian Mandery, and Tamim Asfour. The kit motion-language dataset. Big Data, (4):236-252, 2016, doi: 10.1089/big.2116. URL wwwphhttp/////oo wttp////....1////i//// /tt.////b...11... 0028.
PMID: PMID:22999220 27992262. 8
[37] Black Forest Labs. Flux.
www.roorrettttaaelsss-eeee 2024. 8,
[38] Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preferencc score v2: A solid benehmark for evaluating human preferences of text-to-image synthesis. arXiv preprint 2rX1v:2366.0934/ 2023. 8,2
[39] Mathis Petrovich, Michael J. Black, and Gül Varol. Tmr: Text-to-motion retrieval using contrastive 3d human motion synthesis. 2023 IEFF/CVF International Conference on Computer Vision (ICCV), pages 9454-9463, 2023. doi: 10.1 109/ICCV5100..222..0000
8
[40] Mathis Petrovich, Michael J. Black, and Gül Varol. TEMOS: Generating diverse human motions from textual descriptions. In Luropean Conference Computer Vision, 2022. 8,
[41] Mingyuan Zhang, Daisheng Jin, Chenyang Gu, Fangzhou Hong, Zhongang Cai, Jingfang Huang, Chongzhi Zhang, Xinying Guo, Lei Yang, Ying He, and Ziwei Liu. Large motion model for unified multi-modal motion generation. In European Conference on Computer Vision, page 397-421. Springer, 2024. 8
[42] Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. Motiongpt: Human motion as a foreign language. Advances in Neural Information Processing Systems, 36:20067-20079, 2023. 8, 9
[43] Han Liang, Jiacheng Bao, Ruichi Zhang, Sihan Ren, Yuecheng Xu, Sihei Yang, Xin Chen, Jingyi Yu, and Lan Xu. Omg: Towards apen-vocabulary motion generation via mixture of controllers. In Proceedings of the JEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 482-493, 2024. 8
[44] Chuan Guo, Yuxuan Mu, Muhammad Gohar Javed, Sen Wang, and Li Cheng. Momask: Generative masked modeling of 3d human motions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1900-1910, 2024. 8, 8
[45] Jianrong Zhang, Yangsong Zhang, Xiandong Cun, Yong Zhang, Hongwei Zhao, Hongtao Lu, Xi Shen, and Ying Shan. Generating human motion from textual descriptions with discrete representations. In Proceedings of the IEEE/CVF Conference Computer Vision and Patlern Recognition (CVPR), pages 14730-14740 14740, June 2023.
[46] Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a- pic: An open dataset of user preferences for text-to-image generation. Advances in Neural Information P Processing Systems, 36:36652-3666 2023. 2, 13
[47] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PmLR. 2021. 2
[48] Jiazheng Xu, Xiao Liu, Yuchen Wu. Yuxuan Tong. Qinkai Li, Ming Ding. Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-iman to-image generation. Advances in Neural Information Processing Systems, 36:15903 -5515590331155,...11111119993 2023. 2, 13
13

---

[49] Yibin Wang. Yuhang Zang. Hao Li, Cheng Jin, and Jiagi Wang. Unified reward model for multimodal understanding and generation. arXiv preprint arXiv:2503.05233 055336, 2025. 2
[50] Naureen Mahmood, Nima Ghorbann, Nikolaus F. Troje, Gerard Pons-Moll, and Michael Black. AMASS: Archive of Motion Capture As Surface Shapes In /EEE/CVF International Conference on Computer Vision, pages 5441-5450, Los Alamitos, CA. USA. 2019. IEEE Computer Society. doi: 10.1 N0.110//CCV22019.000 URL httpp://ab...eeccccpteeeeeeeeee ieeecomputersociety wtt/:o..1////....ooooo 1109/ICCV 2019. 00554.
[51] Chuan Guo, Xinxin Zuo, Sen Wang, Shihao Zou, Qingyao Sun, Annan Deng, Minglun Gong, and Li Cheng. Action2motion: Conditioned generation of 3d human motions. In Proceedings of the ACM International Conference on Multimedia, page
New York, NY. USA, 2020. Association
for Computing Machinery. ISBN 9781450379885.
doi: 10.1145/399417.33113333 URL https:
//doi.org/1e 11..0..111///3/944 3413635.
[52] Loshchilov. Decoupled weight deeay regularization. arXiv preprint arXiv: 1ex.iv11111.05110 2017. 3
[53] Haozhe Ji, Cheng Lu, Yilin Niu. Pei Ke, Hongning Wang, Jun Zhu, Jie Tang, and Minlie Huang. Towards efficient exact optimization of language model alignment. In Proceedings of the 41st International Conference on Machine Learning, ICML.24. JoLR. 2024. 7
[54] Kevin Clark, Paul Vicol, Kevin Swersky, and David J. Fleet. Directly fine tine-tuning diffusion models on differentiable rewards. In The Twelfth International Conference on Learning Representations, 2024. URL https://openrevieer
11=11dd111 1iddi:iddd11iiiii VmSEVL1... 13
[55] Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. Diffusionelip: Text-guided diffusion models for robust image manipulation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2426-2435, 2022.
[56] Mihir Prabhudesai, Anirudh Goyal, Deepak Pathak, and Katerina Fragkiadaki. Aligning text-to-image diffusion models with reward backpropagation, 2023.
[57] Xiaoshi Wu, Yiming Hao, Manyuan Zhang. Keqiang Sun, Zhaoyang Huang, Guanglu Song, Yu Liu, and Hongsheng Li. Deep reward supervisions for tuming text-to-image diffusion models. In Computer Vision and Pattern Recognition (ECCV), pages 108-124, Cham, 2025. Springer Nature Switzerland. URL 13
https:/ //link////nn//... springer com/ chapter /00. 1007 3-18------------00
[58] Kevin Black, Michacl Janner, Yilun Du, Ilya Kostrikow, and Sergey Levine. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305. 13301, 2023. 13
[59] Ying Fan, Olivia Warkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Picter Abheel, Mo- hammad Ghavamzadch, Kangwook Lee, and Kimin Lee. Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 79858-79885. Curran As- sociates, Inc., 2023. URL https: / /tstts::/proceeedeie http///ppopcee....e/e//o/o nwwrips........ ctppssccccppopceee cc/ccc/c/ppco cc/paper__ http:pppppfaaeppppeeeaeeeee fc65fab891 hww.aa399...8844333333333663eeee -1p-ee1er-----P------- Paper- Conference pdf. 13
file//
[60] Xiangwei Shen. Zhimin Li, Zhantao Yang. Shiyi Zhang. Yingfang Zhang, Donghao Li, Chunyu Wang. Qinglin Lu, and Yansong Tang. Directly aligning the full diffusion trajectory with fine-grained human preference. arXiv preprint arXiv:2509.0694, 2025. 13
[61] Zijing Hu, Fengda Zhang, and Kun Kuang. D-fusion: Direct preference optimization for aligning diffusion models with visually consistent samples. arXiv preprint arXiv:2505. 220022 2025. 13
[62] Meihua Dang, Anikait Singh. Linqi Zhou, Stefano Ermon, and Jiaming Song. Personalized preference fine- tuning of diffusion models. In Proceedings of the Computer Vision and Pattern Recognition Conference. pages 8020 8030, 2025. 13
[63] Runtao Liu, Haoyu Wu. Ziqiang Zheng, Chen Wei, Yingqing He, Renjie Pi, and Qifeng Chen. Videodpo: Omni preferirefeeence alignment for video diffuusin generation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 8009-8019, 2025. 13
[64] Zhenglin Zhou, Xiaobo Xia, Fan Ma, Hche Fan, Yi Yang, and Tat- Tat-Seng Chua. Dreamdpo: Aligning text-to- 3d generation with human preferences via direct preference optimization. arXiv preprint arXiv:2502.044370 2025. 13
[65] Navonil Majumder, Chia-Ya Yu Hung, Deepanway Ghosal, Wei-Ning Hsu, Rada Mihalcca, and Soujanya Poria. Tango 2: Aligning diffusion-based text-to-audio generations through direct preference optimization. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 564-572, 2024, 13
14

---

SoPo: Text-to-Motion Generation Using Semi-Online Preference Optimization Supplementary Material
Incline
SoPo (a) person running then takes big leap.
SoPo
SoPo (b) man jumps from tight to left.
SoPo
SoPo
w/ SoPo
(c) Person walks quickly down short incline
w/ SoPo (d) The person slides to right times, slides their left times, and slides their left times.
wla SaPo
w/o SaPa
w/ SuPo
wro SaPo
w/ SoPo
(e) man throws object his right hand while lifting his right leg off the ground.
running with side.
SoPo
w/ SoPo
w/o SaPo
SoPo
wio SoPe
w/ SoPo
(g) A person jumps in the sir, then abruptly stumbles to his left as he had been pushed, finally regnins balance.
(h) A person walks forward, briefly sits down, and then stands and walk the opposite direction.
(i) A person knoels down onto four, crawls towards the left, and then stands back up.
w/o SoPo
w SoPo
w/o SoPo
SoPo
person walks forward in xig zag pattemn stepping something along the way.
(k) person both
head while bending their elbows, they then bend their knees squat, and then it,
Figure S1: Visual results HumanML3D dataset. We integrate our SoPo into MDM [13] and MLD [1], respectively. Our SoPo improves the alignment between text and motion preferences. Here, the red text denotes descriptions inconsistent with the generated motion.
This supplementary document contains the technical proofs of results and some additional experi- mental results. It is structured as follows. Sec. A presents the additional experiment information, including additional experimental details (Sec. A.2 and A.4) and results (Sec. A.6). Sec. provides the implementation and theoretical analysis of our SoPo. Sec. C gives the proofs of the main results, including Theorem 1, Theorem 2, the objective function of DSoPo, the objective function of USoPo, and theorem of SoPo for text-to-motion generation. Sec. D provides more related works.

---

From Sheep" Animation.
Prompt: A history Pepe Frog.
Pepe Frog
PEPE T FROG
Flux
nPO
(Durs)
Flux
DPO
Solo (Ours)
Figure S2: Visualization of text-to-image generation on the HPD dataset.
A Experiment
A.1 Details of Experiments on Text-to-Motion Generation
For text-to-image generation, we utilize Flux-Dev [37] as the foundational generation model and employ HPSv2 [38] as the reward model. To construct the offline training pairs, we first sample data from the HPDv2 dataset. However. due the inferior image quality HPDv2, compared to that produced by Flux-Dev, we generated 20,000 high-fidelity image pairs using Flux-Der to create the final offline dataset. We evaluate text-to-image the performance on Pick Score [46], CLIP [47], Image Reward [48] and Unified Reward [49]. The text-to-image model was trained for 330 GPU hours across 8 NVIDIA GPUs using LoRA, configured with a rank of 128 and scaling factor 256.
Results on Text-to-Image Generation. As shown in Table S1, the proposed SoPo con- sistently achieves superior performance across all evaluated text-to-ing -iatge metrics, including (O.(0.321) and IR IR(1. (1.194), outperforming the base FLUX model and standard DPO variants. Visualization is shown Fig.S2.
Table S1: Comparison of text-fo-image genera- tion on HPD dataset.
Method
HPS [38]
CLI [47]
[46)
[R[48]
UR[49]
GPU Hours
FLUX
0.313
0.388
0.27
1,088
3.370
OnDPO
0.317
0.30
0.22
1.154
3.421
316
DIDeO
0318
0.92
0330
1.177
1.402
SoPo
0.321
0.3%6
0.332
1.194
3.499
A.2 Details of Experiments on Synthetic Data
To simulate preference optimization framework, we design 2D synthetic setup with predefined generation and reward distributions. The generator distribution no modeled as a Gaussian with mean [-2, 1] and covariance matrix diag(2.0, 2.0). The reward model is defined mixture of two
=0.5
Gaussians with means [-3, 2] and [2, 22, covariances
and equal weights of 0.5.
+0.5
For the offline dataset, preferred samples are randomly drawn from the reward distribution, while unpreferred samples are sampled from manually specified distribution dissimilar to the reward model. These are used fine-tune the generator via offline preference optimization. For the online setting, we draw samples from the reference model and assign preference labels using the reward model to distinguish preferred and unpreferred motions. This process is repeated iteratively to optimize the model online.In SoPo, we combine offline preferred samples with unpreferred ones to perform semi-online preference optimization, thereby leveraging the strengths of both offline and online data.
A.3 Details of Spatial-Percoption Motion Generation via SoPo
The core insight to solve this issue is that reward models (discriminators) are better at judging spatial semantics than generative models (generator), and SoPo leverages reward feedback to improve alignment.
We divide this issue into three sub-issues:
1. Can the reward model distinguish left/right correctly?
2. Can the diffusion model generate motions consistent with left/right prompts?

---

3. Can SoPo improve generation via preference optimization?
Reward Model Discrimination Ability Spatial Misalignment. From the HumanML.3D test set (2,192 prompts), 783 prompts (35.72%) contain spatial information (e.g., "left" or "right"), highlighting the prevalence of this issue. For text-motion pair (xr, t), we computed the reward score r(I, t). We then created a misaligned text E' by swapping "left" with "right" and computed r(x, L'). The reward model is considered successful r(a, 1) r(x,t) l').
Diffusion Model Generative Ability of Spatial Alignment Generation. We randomly selected 100 spatial prompts from the 783 and generated motions per prompt (500 total). Human annotators judged whether motions matched the spatial constraints.
These results in Fig. 4 (a) demonstrate that:
1. The reward model is capable of detecting spatial misalignments;
2. The original diffusion model struggles with spatial understanding:
3. SoPo effectively enhances spatial alignment in generated motions.
Thus, SoPo offers a practical solution to address spatial misalignment by integrating spatial semantic information from the text-motion-alige aaligned reward model.
A.4 Additional Experimental Datails
Datasets & Evaluation. HumanML3D is derived from the AMASS [50] and HumanAct12 [51] datasets and contains 14,616 motions, each described by three textual annotations. All motion is split into train, test, and evaluate sets, composed 23384, 1460, and 4380 motions, respectively. For both HumanMI.31 and KIT-MI. datasets, we follow the official split and report the evaluated performance on the test set.
We evaluate our experimental results on two main aspects: alignment quality and generation quality, Following prior research [2, 7, 11], we motion retrieval precision (R-Precision) and multi-modal distance (MM Dist) to evaluate alignment quality, while diversity and Fréchet Inception Distance (FID) are employed assess generation quality. (1) R- R-Precisson evaluates the similarity between generated motion and their corresponding text descriptions. Higher values indicate better alignment quality. (2) MM Dist represents the average distance between the generated motion features and their corresponding text embedding. (3) Diversity calculates the variation in generated samples. A diversity close real motions ensures that the model produces rich patterns rather than repetitive motions. (4) FID measures the distribution proximity between the generated and real samples in latent space. Lower FID scores indicate higher generation quality.
Implementation Details. For the preference alignment of MDM [13], we largely adopt the original implementation's settings. The model is trained using the AdamW optimizer [52] with a cosine decay learning rate scheduler and linear warm-up over the initial steps. We use batch size of 64, with guidance parameter of 2.5 during testing. Diffusion employs cosine noise schedule with 50 steps. and an evaluation batch size of 32 ensures consistent metric computation. For fine-tuning MLD [1]. we similarly follow its original parameter settings.
Table S2: Hyperparameters analysis of our SoPo.
Precision
Methods
FID L
MM Dist
Top
Top
Top
SoPo0C=1,8=00.55)
0.523
0.717
0.823
2.941
0.176
SaPo(C11,840.5)
0.524
0.718
2.940
SuPw(C=1,8=1)
0.525
0.719
0.825
2.939
0.174
SoPPo(C=2,8-0.25)
SuPw(C=2,3F005)
0.528
0.722
0.827
2.937
0.172
SoPorC=2,,8=1)
SoPo(C=3.9=00.5)
0.532
0.726
0.831
2.935
0.170
SoPo(C=3,0d11
SoPo(C=3.(=2))
0.529
0.723
0.828
2,936
0.171
3

---

down
down
up
w/o SaPo
w/ SoPo
wlo SoPo
w/ SaPo
(4) A their right and then curves the Jeft and continues to run then slups.
(b) man jumps and brings both arms above his head and then moves themn back into the original position.
Figure S3: Visual results on Human dataset.
A.5 Additional Ablation Results
Impact of Hyperparameters Setting. The hyperparameters of our SoPo can be divided into two
types: (1). From SoPo: filtering threshold т, candidate number K, weight C; (2). From DPO:
temperature 3. For SoPo-specific hyperparameters, Table. S2 shows they have minor influence. Below, we report results on MLD* to analyze the sensitivity to 3 and C:
A.6 Additional Experimental Results
We visualize the generated motion for our SoPo. As shown in Fig. S3, our proposed approach helps text-to-motion models avoid frequent mistakes, such as incorrect movement direction and specific semantics. Additionally, also present additional results generated by text-to-motion models with SoPo, as illustrated in Fig, S1. Our proposed SoPo significantly enhances the ability of text-to-motic models comprehend text semantics. For instance, Fig. S1 (i), model integrated with SoPo can successfully interpret the semantics of "zig-zag pattern", whereas a model without SoPo struggles to do so.
B Details of SoPo for Text-to-Motion Generation
In this section, we first examine the objective function of SoPo and argue that it presents significant challenges for optimization. Fortunately, then discover and derive an equivalent form that is easier to optimize (Sec. B. Finally, we design an algorithm to optimize it and finish discussing their correspondence (Sec. B.2).
B.1 Equivalent form of SoPo
In Eq. (15) and (16), the objective function of SoPo is defined as:
Ciif SoPo So +
Aditf
cdiff SoPo-hu
(SI)
diif SSPPo-vu
-B -........)......)) -aj Zon
logor( Tws (Bw (C ref, re) BL ref, 2f),
(S2)
dii LooPo-hu hu
loggo Tww, L ref, x:

---

However, these objectives can not be directly optimized, since the distribution Th" and Tg are not defined explicitly. To this end, we begin by inducing its equivalent form:
C (S) E a2 (.())))
if if log Twn (f(0, ref, (ef, (())),
<T, otherwise.
(S3)
log - - Tw. (0,,
where of argmin Vi Noo c). c).
Proof.
Recall definition of Solo (0) in Eq. (15) and (16). Through algebraic maneuvers,
have:
Cdite
Cdiff -o +
Cdif -SPPo-bu
ref, log o( Twi (Bm.(()))) (C(0, - - ref, ref())))l,,)))) =
(E .) Znn (c) log (am [ ref,
-B .) D ~ (11) Lcm(c)
log of (- Twi (z (C ref, x²") 3L ref, x()1)
Be ()) DD - (110) Zo (c) logo
log Twi (3w ref, re!, - 3C ref,
Be Bl 1:k (c)
logo (T))
(r c) <)
log a( Twy (B (Cw) ref, x/")
3C (0,, ref,,))))))
EL- .)- mont (r( 2 27) log ((0,
BB. (
logo
Twi (Bw (C ref, x²))
BC =i)))) ref,))),,
if ifr(z',)) <T, otherwise.
logor (ww (B, x.f.
where holds since Pagu. (.) = Peg and p (2 1: (c) = Pa. (c1 pr (r(xtt,) 2)) The x
Zon (c)
proof is completed.
B.2 The process of SoPo text-to-motion generation
Based on the equivalent form of SoPo in Eq. (S3), we can design an algorithm to directly optimize it, as shown in Algorithm
The SoPo optimizes policy model Ty for text-to-motion generation through an iterative process guided by a reward model. In each iteration, given preferred motion xt and conditional code c, a random diffusion step is selected, and K candidate motions are generated by πæ. The motion with the lowest preference score is then treated as the unpreferred motion. To determine the weight of the preferred motion x", the similarities between all generated motions are computed, and the lowest cosine similarity value used calculate its weight. Finally, the loss is calculated in two ways, determined based on the preference scores of the unpreferred motion. If the preference score of the selected unpreferred motion falls below threshold v, it identified as valuable unpreferred motion and used for training. Otherwise, it indicates that the motions generated by the policy model ny are satisfactory. In such cases, the policy model trained exclusively on high-quality preferred motions, rather than on both preferred motions and relatively nigh-preference unpreferred motions.
5

---

Algorithm SoPo for text-to-motion generation
Input: Preference dataset D; diffusion steps T; iterations samples K; ref model Rref' policy mos threshold
Output: Aligned model
1: for = to do
2:
for each (xx c) D do
3: Sample ~ (0.
4: Sample 28 11:K
(lc)
5: Compute S(x ming cos( .x
6:
x' = =arg min; cc
7:
if ifr(=',c) < then
8:
= log o - TT Br (x. (2(0, ref, x(() 32(0,, ref,
9:
else
10:
L(0, end L= log ( (x. L ref, x²²))
12:
Accumulate loss: S + = C
end for
14: Update using Vo Vo
end for
16:
return T
To further understand the objective function, we analyze the correspondence between the objective function in Eq. (S3) and Algorithm 1:
(S)
AiiF
(x c) ~ D,t (( 1:K #o(-Ic)
line
log
Twi (L ref, BL(O, ref, x())
If If <T.
(S4)
Lime
log or (-
( Bwr ref,
Otherwise,
Line 10
C Theories
C.1
Proof of Theorem
Proof. The offline DPO based Plackett-Luce model [34] can be denoted as:
E .0w log II exp(3ho
K
Cof (8)
(S5)
E exp()ho(ol,c))

---

where hz(2,c) = log mo(xle) (2)()
Then we have:
exp(Sho(x* .c)
Coft (0) (a ,0> log
2j= exp(Sho(z), c))
K
exp(3ho( c))
- D Patlall 1:
lc) log
j exp(
K
To no exp(6 log
--EodD Pat 1:
1c) log
j=k exp( @ log
explog( xe(x"le) Azat (x
=-Eo-D Pet (2
(c) 1:
log
23 k j/ exp log( No c)) None (a0
K
Toor(a*1e)) )
(S6)
-.EDD wi (at(oli(()) log
molan (c)
j= SToor
po(2
E-Ea-D Pet (
(c) log II po(x*1c)
pa(alia(())
-EEWDD Pat (a (c)
log pa (x 1: (c)
log Pat (x 1:K (c) + log Pet (xr 1:K (c)
(2 (c)
LENDD Pgt (a 1: (c)
log po 1: c))
log (z 1c)
Eew (PI Pat (x (c) Pa (x1:K(c)
Therefore, have:
V VoLott (0) = -BewDD V DI (Poc)
(S7)
The proof is completed.
C.2 Proof of Theorem
Proof. Inspired by [53], we replace the one-hot vector in DPO with Plackett-Luce model [34], and then the online DPO can be expressed as
(ark [c)
(DPO-on -EaDD Bl
(x (c) log
(S8)
Tel (1r7 (c) 8
1=

---

expr(1 :9)
where pr (, (c)
Then we have:
Di exp 2
K pr (r log
(ar(a )
To(a (c)
Coa(O) D. (co(
Tol le) j
Tarl
K Pr (r(x log
Ne (et (c)
= --EDD Pao (x (c)
Feed (x0
Ni (o (e((z)l))) lo! j= Ture
k=1
xo(x") Keat (20 )iP
EeWD Pao (x (c)
Σ pr (r log
(2x3 le) j=bb / Toar (202
(S9)
po(at(e)
K
= --EWD Pao (21: (c)
Σ
pr (og log po(2)10))
K
BEODD Pno (x (c)
Σ
loger(**)e) pe (og (xg
k=1
EEWD Pno (x lc)
D DKL (c logp
Therefore, we have:
V vaLon BEED Vo Pro (x De)
(S10)
The proof is completed.
Given a sample æ with a tiny generative probability P=olo(0) - O, and large reward value r(x, c) + 1, we have limp,, (=.()))) VeCon
O.
Proof. Since is contained in the sampled motion group 2 we have:
lim
voLon
Pro (2,
lim
Va Pao (x I:K (ez(kl(p))))))
Pog (0)
(S11)
lim
Ve Pso (z Dc)
Poo (0(0))))))
where ① holds since Pro (x (x = Pro (x S Pmo (x/c), and M denotes a motion group obtained by removing the given motion from the group 1:K i.e., satisfying that a 11:k {x) The proof is completed.
C.3 Proof of DSoPo
Proof. Eq. (10) suggests that DSoPo samples multiple unpreferred motion candidates instead of single unpreferred motion. Thus, should first extend Eq. (9)
LDSOPO Bew .c) i (ale) log a (8Hl(0" (x))))
(S12)
8

---

respectively denote the distributions of valuable where x' = (* J
~ ~a r c). Then, we have:
(DSoPo(O) = .)) (m(tee log [HHa(=",zt xd(c)) E peo (x (c
lc)
log o (8Ho(e" a ,e,,))
Substirating
B(e Baii Pro (T (clp c)27) + pro (1# (c)p- ())e))))))))))))) log ( ,0) (r)p, c)2r) log of [HHo("", x.,c))
- () B Pa, (o 1i
-E ( Buix Paa (x ()p- c)<r log o (Holo" xx,
Pgg (a
-B Znd (c)p bu (x 1:k
lo) o [Hol(" xx'
(x (N Zem (c)
1:K logo [Hol"" (2'
E(e" --E Da (c) hur 1:K (c) logo
--
Moll'" (2,c))
E( .) Zou C) PTo (a 1:K (c) log (Mola"" (z'io))
-E E(u" LA. (c)
log a (BHa(e", "o""","",,, .e,
Beg Zo (c)
logo (3" ,x,,,))
(0) + (0),
(S13)
where Paga-() (-) unpreferred and high-preference unpreferred motions. The proof completed.
P (
toj and hax (-)
Pe
tej
Accordingly, we rewrite Chu(() and obtain the objective function of USoPo:
LUSoPo (0) = -R Ra .D Znn (c) log [3ha(x" .e)).
(S14)
(UsoPo(O) = LUSoPo-bul (0) + Cvu(0). (0).
Now, we discuss how deal with the computation of Zon and Zku (c) in our implementation. As discussed in Sec. B, directly optimizing the objective function Cdiff (0) is challenging. and we used Algorithm optimized its equivalent form: ddiff
Implementation
SoPo(()
SaPo (0) - -E (t)_e/e__)......e (e.k)) D.
log
Twu (Bw (z (C(0, xx) ref, if (f <T, (S15)
log - Twr (w [(0, ref, xx(")
otherwise.
Similarly, we can optimize the equivalent form of UDoPo to avoid the computation of Zva(c) and Znu(c): (USoPo(O) -E -Bm D. K. woo
log o (Ho .x). (fr((',c) <T.
(S16)
log a (Bho(x" c)) Otherwise.
The proof of Eq. (S16) follows the same steps as the proof of Eq. (S15) in Sec. B.
C.4 Discussion of USoPo and DSoPo
In this section, discuss the relationship between USoPo and DSoPo and the difference between their optimization. Here, USoPo and DSoPo are defined as:
LUSoPO (0) = LE( Znu ( (c) log [She ,c)) + LV)
(S17)

---

LDSOPa(0) = Lvu( + Liv
(S18)
Relationship between USoPo and DSoPo We begin by analyzing the size relationship between USoPo and DSoPo:
(DSoPo(O) (f)
(m) + [0 e)> (un log (Sho(x" .c))
,))/ (c", Nu (c) way logo (3H ,000)"",,,c)))))))))))) xx,,c) + ,))),,)od2 ( log o (ho(a" .c) log σ (HHl(x" , ,2',c) .))) - log o (9ho(z",c))]
(c) (a"")))-D Zuu
(S19)
Considering that Hz(x", x ,²,c)) = ho(x", - ho ho(a', c) and ho ho c) = log xo(2]c) )ar(sr/c)
we have:
(f) (Os
.) () Zou (t)
log a (BHol100 at,, log (Bho(=";)))
exp (2"
exp (2 .c)
(( .j nr,, (c) ahu.
log
llog
exp Bho (x c) + exp 3hg (zt, (z),c)
exp Sho , +
exp (ho(x", c) +
B(z" .
(c)~ nu (c) ugu
log
exp Bhg (x c) + exp / (xt
[a(x= ya
Tar (e)
- Biz" .))=D Zu ~xpu.
log
[a Marl="
(S20)
In general, DPO focuses on reducing the generative probability of loss samples (unpreferred motions).
Consequently, the generative probability of the policy model To(x'(c) will be lower than that of
the reference model [ra((")(c). i.e., To(x'(e) le) S Tre((="(e), resulting in (e
S 1. Hence, the
nat [c]
following relationship holds:
To(2 (c)
<
(Te((x)1c)) (c)
(x=1 (c)
To (2("1c))
To(x le) )a Teer (or (zr1 (c)
+ 2
+
neet ( c))
net (x:0 (c)
Na(2" [c)
8
20
21 1
To (m
3
Ta(
)
Mael (z (c)
Nest (ar(( cj ).
Tc(
(S21)
= log
20 0
(o(w )3 + [o(2 Faelar (6)
)
me far le)
+
+ - (um, Zhu (c)
log
Meat
00
( [c)
Nola )8
(1n!!
(DSara(() Cu)na(
LDsoro < (O).
Eq. (S21) indicates that LusoPo one of upper bounds of LDSoPO-
Difference between the optimization of USoPo and DSoPo The difference between the opti- mization of USoPo and DSoPo can be measured by that between their objective function. Let La( Lusara (f) LDSoPo (0), the difference between their objective function can be denoted as:
La (0) -LSa.r LDSaro(0)
Tru (ar (c)
9
(S22)
Em .)~-D (c) Karho
log
0
Ma (2e Feed
Ic)
To
c
10

---

where ① holds due to Eq. (S21). As discussed above, the generative probability of the policy model To(2'(e) will be lower than that of the reference model Ter((" (c), and thus To (2 (c) falls in the range between 0 and Mrr(<" i... 0 I (e(x"(c) I (T)((*)))
Assuming that the value of No (x**|c) (c) is fixed, the value of (0) is negatively correlated with no (2" (c), since we have:
(ar (c) Keat (ee (2e (c)
.)> (nn VoLa(0) =Vo ())
logg
ccl
)
mata" icl
).8
(c)
To (2=2
To (z" lc)
Le .) Zira (c) (c) whyu Vo log
Tere (2:
Trerl (zo (c)
(S23)
(20 (2²1lo) ya Ter( (c)
( .) Zha (c) (c) wi
To [c) )
Ro(ar (c) Tarlar
Teat xx c)
To (x(11cc) )
Teet (z' c)
where ! holds since
0.
Tata 1c) 1oy
Tp(ofle) [er
)a
Hence, when the generative probability of unpreferred motions (1(1'cc) is lower, the difference between the optimization of USoPo and DSoPo is larger. However, the unpreferred motions are sampled from the relatively high-preference distribution whv+ and thus should be treated as unpreferred motions. Using CusoPo optimize policy model गe instead of CDSOPo (0) can avoid unnecessary optimization of these relatively high-preference unpreferred motion (0)..
C.5 Proof of Eq. (16)
Before proving Eq. (16), we first present some useful lemmas from [27].
Lemma 1. [27] Given winning sample Tho and losing sample the DPO denoted as
To(x (c) 3 log To(x'|c) (c) CDro = = (e) loog (slog
(S24)
Tee((>00 (c)
Tree (x' (c)
Then the objective function for diffusion models can be denoted
Lnpo-Difusion (0) = (ax MDDD log o BE By nono lat
Te (2())) [lo
(e( 1);
(S25)
log Trot (2
Tref (x
where x denoted the noised sample x" for the t-th step.
Lemma 2. [27] Given the objective function of diffusion-based DPO denoted Eq. (S25), has an upper bound LUB(O);
Lpro-Dijusion (0) S E(g
logo
To [x (xt²) 3T
log Tel 1a(2t_11))
-T
(S26)
Tret (x) (xte))
Teet ( (or)) )
Lue(P)
where T denotes the number of diffusion steps.
Lemma 3. [27] Given the objective function for diffusion model denoted as Eq. (S26), it can be rewrillen as:
LuB(0) B
-ell log logor-3TTa
(S27)
(Ile - {-)"" 1 - ||e (xe"" 1) (Ile (1l--co(tee c0(2,, t) - l|e foet +)112))
where axz + 0,6, (0, is a draw from the distribution of forward process ((0]126)
11

---

Now, we proof Eq. (16) based on these lemmas.
Proof. This proof has three steps. In each step, we apply the three lemmas introduced above in succession. We begin with the loss function of SoPo for probability models:
(So (o D.shs w (-1e) (c) log (Sw(z") ,c) Bho .0)(=e,c)) .c)]
ConPn-vn(0)
(S28)
=) log (e Mesl"" ,c))
CeaPa-ba(P)
Based on Lemma we can rewrite the objective function for diffusion models:
pdiff ori DSsPl-Diilusioo (0) = /diff- -va (0) + SSPPo-hu (6) ediff-ori (0) = (uy (..
(c)(c)
log To(x))r)
To (1'arr)
log o g mo ) gf +t)) =1)(
3log
Tree Teet(
SoPo-bo (0) [oy ,) nuu (c) log o (Be P bog [Sw Bull (x log no (x0TT)
Dr.
Tixed (2
(S29)
where 27 denoted the noised sample x for the t-th step. According to Lemma 2, the upper bound of -drf "SoPo-vu (f) and SoPo-hu (0) can be denoted as:
pdift- C (0) S (x of
(1))) "1 log (w (w T log Ta (2t lx(t)
128)) ).))) 1.1-
Te (2 |x|) neat 24 (x4))
BT log
Teet (Ir lx:"))
odies
(0)
To (x l,x²)
diif La
(0) < Eeyy (. mo la log
Bw (T log Teee (a |2:
SoPo hu
CSaPn SSiffuDiiftoiin (0)
-Cdift- - (0) Lo (0) & L80Pn (0)
adit
SSPPn hn (0)
ddiff
edil (0).
(S30)
Cdifl Applying Lemma to -SoPo-vu vu (0) and Cdiff SoPa hu (0), we have
diff Pu (0) = B La
logor
Twi (x ) (Ile c0 1 Ezet ()))))
8(1le - (oh, t)ll ||e God co (o)))t)1))))
(S31)
Adif SoPo (0) - BBygg, sD
1 1x8)))
log o Twn ( (1 co(=" ,))=),)))3 ||e erof (0) ,))
C (0)
Cditt LSoPo- (0)
ditf La
To simplify the symbolism, the objective functions can be rewritten as:
SoPo-vu
dift
= -.
Zou(c) [logo Tw: (Fe ref, - 3C ref, a:))))
(S32)
Po bu
[ [og o Twr, ref, )
where L(0, xx) = L(0,xz) - L(ref, ,x), and L(0/ref, xx) |1<0/set 1) - =el13 denotes the loss of the policy reference model. The proof is completed.
D
12

---

D More Related Works
Fine-tuning pre-trained diffusion models [54-57] using task-specific reward functions [46, 48] is widely adopted approach for adapting models to specific downstream tasks. Current approaches are broadly classified into three mechanisms: those relying on differentiable rewards [54, 57], conventional reinforcement learning algorithms [58, 59], and Direct Preference Optimization (DPO) [25]. Our work is most closely related to methods based on DPO, which provides a remarkably straightforward path to align the model with specific downstream objectives by directly utilizing pairs of motions reflecting human judgments. Recently, some research focus on the issues of fine- grained human preference [60]. visually consistence [61]. and personalized preference [62] for image generation. As powerful alignment method, DPO is also extended to video generation [63], 3D generation [64], and audio generation [65].
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims in the abstract accurately reflect our contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of this work Sec. 6.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

Justification:
The proof are provided in App. C.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide complete experimental details in Sec. A.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [NA]
Justification: We plan to release the code and detailed documentation after the acceptance of the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
Sablli//guddes/ooeSutbbaae sooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https: /nips. ctips.cc/public//uu guwdee/dss///// CCodeSubeesiaceecc simiisssinn ssssonoliccyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We describe the complete experimental details and hyperparameter choices in Sec 5.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The confidence intervals based on 20 independent repetitions are reported in Table 1.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
16

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We report the computational resource requirements of our proposed method in
Sec 5.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https /ttr/eeurpp..e/ cc cccppblicc/eieecccceccceeeeeeeee htw.liii//hiissiiiiiiiee cetthicsGuieelneeee
Answer: [Yes]
Justification:
Our research aligns with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our paper focuses on advancing the field of machine learning. Although our work may have various societal implications, we consider none are significant enough to warrant specific mention here.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
17

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We plan to release the code and datasets after the acceptance of the paper.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi wwwode.hcoe...c. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
18

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We plan release the code and detailed documentation after the acceptance of the paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
19

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: This paper does LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non non-standard components.
Please refer to our LLM policy (https / /at.:///////oooeeeecn///////222 for what should or should not be described.
20


Reviews:
[{"review_id": "0LLowLV4Kh", "reviewer": "Reviewer_E3Yi", "summary": "The paper introduces SoPo, a semi-online DPO alignment strategy, designed to improve text-to-motion alignment.\nThe authors identify limitations in both offline and online DPO, a framework commonly used for preference alignment.\nTo overcome these limits, SoPo uses motions from an offline dataset as preferred examples, while unpreferred motions are generated in an online fashion. The unpreferred motions are categorized into two types: high-preference unpreferred and valuable unpreferred motions. Both types are incorporated into the learning process using a weighted DPO loss.\nAccording to the authors, high-preference unpreferred motions are considered to have quality and should be encouraged. To this end, an additional loss term is introduced during optimization.\nThe authors evaluate SoPo against other text-to-motion methods. Compared to the base model (MLD*), SoPo shows notable improvements in RPrecision, MMDist, and FID.", "questions": "- Q1. Could the authors clarify why the Multi-Modality metric was omitted from the evaluation? Do they plan to include it in a revised version?\n\n- Q2. Are the generated motions produced using a frozen reference model or one that is being actively optimized during generation? This distinction would help clarify the experimental setup.\n\n- Q3. The paper claims faster inference compared to [9], but it appears the speed-up may stem primarily from using a faster base model (MLD* vs. MLD) rather than the proposed alignment technique. Could the authors clarify this point and possibly disentangle the contributions of each component?\n\n- I would be open to revising my evaluation if these issues are adequately addressed in the rebuttal.", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n \n- SoPo offers a significant improvement over previous state-of-the-art methods, which generally depend solely on either online or offline strategies. By combining the strengths of both, SoPo provides a more balanced and effective alignment technique.\n\n- The method is simple yet effective, well-grounded in existing literature, and shows potential for broader applications, such as text-image alignment.\n\n- Building upon existing works, the authors report improvements in R-Precision Top-3, MM Distance, and Diversity metrics.\n\n**Major Weaknesses**\n\n- Multi-Modality metric, which is a key measure in text-to-motion generation, is not used in the paper. This metric is commonly used in prior studies to assess whether the model suffers from mode collapse effect.\n\n- It is unclear whether the generated motions are produced by the reference model or the model currently being optimized.\n\n- The paper also lacks clarification on how generation time is reduced compared to [9]. It appears that the improvement in inference speed stems primarily from using MLD* instead of MLD, which is unrelated to the core contribution of their proposed method. Despite this, the authors suggest an overall improvement in inference times, which may be misleading as it is not a direct result of their approach.\n\n\n**Minor Weaknesses**\n\n- L216 There is a typo: Nore instead of More\n- L218 the authors do not explain the meaning of r in \"score >=r\". This may be a typo, and it seems more appropriate to use \\tau as in eq 11.", "was_revised": true, "final_justification": "Regarding Multimodal Evaluation: The authors have incorporated multimodal evaluation, albeit with limited scope restricted to the HumanML3D dataset and presented through partial tabular results. I recommend that comprehensive results for both the target model and competing approaches be included in the main paper, covering both HumanML3D and KIT-ML datasets to provide complete comparative analysis.\n\nRegarding Motion Sequence Generation: The authors have appropriately clarified that motion sequences are generated using the target model under optimization rather than the reference model, which addresses the previously raised concerns about methodology.\n\nRegarding Data Generation Cost Claims: During the discussion phase, the authors asserted that their data generation cost extends beyond mere inference time to encompass the method’s capacity for producing feedback-rich samples. However, this claim regarding superior sample quality remains unsupported by empirical evidence or quantitative metrics. The assertion that their approach generates “feedback-rich samples” of allegedly superior quality requires substantiation through rigorous experimental validation to avoid misleading characterizations of the method’s computational efficiency advantages.\n\nI encourage the authors to provide empirical validation for their claims regarding sample quality to strengthen the contribution’s credibility and ensure accurate representation of the method’s capabilities.", "rebuttal": "We sincerely thank the reviewer for raising this important point. Thank you for recognizing the strengths of our method and experimental results. You raised several important concerns about the evaluation metrics, implementation details, and algorithmic efficiency. **We believe your concerns mainly come from a misunderstanding of our experimental setup.**\n\n---\n\n## [Q1/W1]  Could the authors clarify why the Multi-Modality metric was omitted from the evaluation? Do they plan to include it in a revised version?\n\nBelow, we explain the reasons for omitting the Multi-Modality metric. We promise that we will include it in future versions.\n\n### [Q1-1] Why was the Multi-Modality metric omitted?\n\nThe reasons for omitting the Multi-Modality metric are as follows:\n1. **Some of recent works also omit it**:  We followed recent and concurrent works like LaMP [1] (ICLR’25), MotionStreamer [2] (ICCV’25), and Go to Zero [3] (ICCV’25), none of which report the Multi-Modality metric.  We believe this may be because the metric overlaps with other diversity indicators and can introduce certain inconsistencies, as discussed in Points 4 and 5.\n\n2. **Some of our baselines omits it**: Key baselines like PriorMDM and OGM do not provide Multi-Modality, and their code is unavailable. Including the metric would make our table incomplete and less comparable.\n\n3. **Not the primary metric of our work**: Our goal is to improve human preference, especially **motion-text alignment and quality**. As Reviewer 7xd7 pointed out, metrics reflecting human judgment are more relevant.\n\n4. **Diversity and Multi-Modality are similar**: Both metrics measure diversity using similar formulas. The main difference is that Diversity samples across all prompts, while Multi-Modality samples multiple motions per prompt. Therefore, Diversity also can be seem as a indication of mode collapse.\n   - **Diversity**:  For overall variation, we randomly sample two sets of motion features $\\{v_i\\}$ and $\\{v'_i\\}$ of size $S_d=300$ from all generated motions. The metric is:\n\n     $$\\text{Diversity} = \\frac{1}{S_d} \\sum_{i=1}^{S_d} \\|v_i - v'_i\\|$$  \n   - **Multi-Modality**:  For each of $C$ prompts, we generate two sets of $S_m = 10$ motions and extract feature vectors $\\{v_{c,i}\\}$ and $\\{v'_{c,i}\\}$. The metric is:\n\n     $$\\text{Multimodality} = \\frac{1}{C \\cdot S_m} \\sum_{c=1}^{C} \\sum_{i=1}^{S_m} \\|v_{c,i} - v'_{c,i}\\|$$  \n\n5. **This metric is sensitive to architecture**: We observe that diffusion-based methods consistently get higher Multi-Modality scores than autoregressive (AR) methods. This makes the metric unreliable for fair comparisons across model architecture.\n\n|Method|Architecture|Multi-Modality|\n|:-:|:-:|:-:|\n|MMM|AR|1.232|\n|MoMask|AR|1.241|\n|HumanTOMATO|AR|1.732|\n|MotionGPT|AR|2.008|\n|Motion Diffuse|DM|1.553|\n|Motion Mamba|DM|2.294|\n|MDM|DM|2.799|\n|MLD|DM|2.413|\n|MLD*|DM|2.267|\n\n### [Q1-2]  Will it be included in the revised version?\n\n**Yes**, we agree that Multi-Modality is a classic metric used in many previous studies to test mode collapse. We will include it in our revised version.\n\nFor methods without reported values, we are reproducing them to fill in the table. Below is a preliminary result on the HumanML3D dataset:\n\n| MMM | Motion Diffuse | Motion Mamba | MoMask | MLD* | MLD*+SoPo |\n|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|\n| 1.232 | 1.553 | 2.294 | 1.241 | 2.267 | 2.301 |\n\n\n[1] Zhe, Li, et al. \"*Lamp: Language-motion pretraining for motion generation, retrieval, and captioning.*\" arXiv preprint arXiv:2410.07093 (2024).\n\n[2] Lixing, Xiao, et al. \"*MotionStreamer: Streaming Motion Generation via Diffusion-based Autoregressive Model in Causal Latent Space.*\" arXiv preprint arXiv:2503.15451 (2025).\n\n[3] Ke, Fan, et al. \"*Go to Zero: Towards Zero-shot Motion Generation with Million-scale Data.*\" arXiv preprint arXiv:2507.07095 (2025).\n\n\n---\n\n## [Q2/W2] Are the generated motions produced using a frozen reference model or one that is being actively optimized during generation? This distinction would help clarify the experimental setup.\n\nThank you for your insightful question. The generated motions used in our preference optimization pipeline come from a **policy model $\\bar{\\pi}_\\theta$ that is actively optimized during training**, not from a frozen reference model.\n\nThis is clearly stated in method section in our paper:\n- In Line 189–191 states: *\"we first generate $K$ motions $\\\\{x^k\\_{\\bar{\\pi}\\_\\theta}\\\\}\\_{k=1}^K$ from the policy model $\\bar{\\pi}\\_\\theta$.\"*\n- Eq. (17) also makes this clear by denoting sampling as $x^{1:K}\\_{\\bar{\\pi}\\_\\theta} \\sim \\bar{\\pi}\\_\\theta(\\cdot \\mid c)$, rather than from the frozen reference model $\\pi_{\\text{ref}}$.\n- In Algorithm 1 (Appendix A), Line 4 explicitly performs:  *$x^{1:K}\\_{\\bar{\\pi}\\_\\theta} \\sim \\bar{\\pi}\\_\\theta(\\cdot \\mid c)$*, indicating that the samples are generated from the current policy $\\bar{\\pi}_\\theta$ being optimized.\n\n**We agree that this implementation detail could be more clearly emphasized in the experiment section, and we will add clarification in the next revision.**\n\n---\n\n## [Q3/W3] The paper claims faster inference compared to [9], but it appears the speed-up may stem primarily from using a faster base model (MLD* vs. MLD) rather than the proposed alignment technique. Could the authors clarify this point and possibly disentangle the contributions of each component?\n\nThank you for raising this important point. We believe there may have been some misunderstanding regarding our claims. Before addressing the component-wise contributions, we would like to clarify two key points.\n\n### [Q3-1] The paper claims faster inference compared to [9]\n\nOur claim concerns data generation cost, not inference speed.\n\nIn **Line 265**, we state that our method reduces *preference data generation time*. The caption of **Table 1** also clarifies that *\"Time\"* refers to the **online/offline motion generation time** used in preference optimization — not the inference speed of the generative model itself.\n\nExisting DPO frameworks typically consists of two stages data generation and preference optimization:\n- For **offline DPO**, a large-scale samples must be pre-generated and labeled by human or reward models, and then are used for preference optimization.\n- For **online DPO**, data generation and optimization are interleaved per iteration, requiring continual sampling and feedback.\n\nIn fact, data generation is a critical bottleneck for fine-tune cost. **The number of generated samples, the efficiency of the alignment paradigm, and the inference time all affect the overall cost of data generation.**\n\n### [Q3-2] The speed-up may stem primarily from using a faster base model (MLD* vs. MLD) rather than the proposed alignment technique\n\nWe apologize for not explicitly explaining the inference time of MLD\\* and MLD in the experiment section. **In fact, they share the similar inference speed.**\n\nMLD\\* refers to a stronger version of MLD reproduced by [1], which achieves better generation quality **with nearly the same inference speed.** According to [1], MLD* has an inference time of 0.225s, compared to 0.217s for MLD, indicating that MLD is slightly faster. **Thus, the observed speedup in preference data generation does not stem from faster inference of the base model.**\n\nMoreover, in the speed comparison with [9], **we used the original MLD model for a fair comparison.**\n\n[1] Wenxun, Dai, et al. \"*Motionlcm: Real-time controllable motion generation via latent consistency model.*\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.\n\n### [Q3-3] Why is our data generation more efficient?\n\n**The efficiency stems from our Semi-online alignment paradigm**.\n\nPreferences are often derived from **objective criteria** like image-text alignment, realism, or safety. In motion generation, preferences are defined by motion-text alignment and motion quality.DPO-based methods aim to improve policy models using preference feedback from generated samples. The key question is:  \n**What kind of preference samples are most effective for generative models to learn preferences?**\n\nTo ensure the model learns **consistent** preferences while addressing **diverse** failure cases, ideal preference pairs should consist of:\n1. **Preferred samples** that strongly align with the desired preferences.\n2. **Unpreferred samples** that target the generative model's specific weaknesses\n\nInspired by this concept, we employ:\n- **Offline high-quality preferred samples**, which are challenging to generate but essential for alignment.\n- **Online unpreferred samples**, generated from the current policy model $\\bar{\\pi}_\\theta$, to reflect the model's current shortcomings and facilitate effective reward learning.\n\nThis Semi-online setup allows us to generate **the most informative training pairs** using fewer samples — leading to **data-efficient alignment**.\n\nIn contrast, MoDiPO generates large-scale candidate samples from multiple generative models:\n- These samples have low diversity in quality, making **preference gaps hard to identify**.\n- Generating strong negatives solely from pre-trained models risks **misalignment with the current policy**.\n\nAs discussed in [Q3-1], preference data generation cost is not solely determined by inference time, but also by how effectively the method produces **feedback-rich** samples. **We believe our Semi-online framework is the key to this improvement.**\n\nThank you for your question! We hope the respond addresses your concerns well."}, {"review_id": "9k6qbwbMnG", "reviewer": "Reviewer_tJvD", "summary": "This work introduces Semi-online Preference Optimization to train text-to-motion models that better align with textual inputs. The key idea is to use unpreferred motion from online distribution and preferred motion in offline datasets for DPO training. The authors provide theoretical analysis for this choice and show in experiments that it boosts performance on motion generation models like MLD and MDM.", "questions": "Following weaknesses part, I have several questions.\n\n1. Could the authors provide results of applying the proposed method to other domains and include them in the appendix?\n\n2. Could the authors provide clearer comparisons with online DPO, offline DPO and their naive combination?\n\n3. How well does the preference optimization solve spatial constraint misalignment for given texts? Could the authors design some experiments and show quantitative results?", "limitations": "Limitations are discussed in the paper.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The research scope of this work i.e. preference alignment for text-to-motion models is important yet under-explored in the community. \n\n1. Figuring out the respective limitation of online and offline DPO and combining them is technically sound and valuable. The core insight of the proposed method is concise yet interesting enough. \n\n2. The paper is well-written and of high quality. There are nice figures (especially Fig. 2 and Fig. 3) that make this work intuitive to understand. \n\nWeaknesses:\n\nMy concerns are mainly about the evaluation.\n\n1. Although applied to the text-to-motion task, the proposed method is essentially a general loss that is not dependent on specific datasets, diffusion models and retrieval models. This work lacks evaluation on generation tasks in other domains to prove its generalizability, such as for image generation.\n\n2. In order to better evaluate the merits of such combination, the paper should more clearly present the baseline results of only using online DPO, offline DPO and their naive combination. Currently, the ablation study in Table 4 is a bit vague and needs better clarity. Also from Table 4, it seems that the improvement of each design is not very significant.\n\n3. It is not clear what type of problem in motion generation is addressed by preference optimization. From Fig. 1, I assume that spatial constraints in text inputs are better aligned, such as left/right, back, etc. Consequently, besides qualitative examples, some quantitative statistics reflecting ratio of successful/failure samples can be added to show how well the preference optimization solves these issues.\n\n4. As for the related work, more discussion is required for the recent advances in preference alignment for generation tasks.", "was_revised": true, "final_justification": "The rebuttal addresses my concerns. I appreciate the authors' effort to validate the generalizability beyond text-to-motion tasks and the capability of solving spatial misalignment. It would help if the authors include these points in the final version. I keep my score as 4: borderline accept based on the overall contribution of this paper.", "rebuttal": "Thank you very much for your recognition of the novelty, writing quality, and significance of our work. You also raised several insightful points — including the **generality of SoPo to other domains**, **more complete ablations**, and **how SoPo addresses spatial perception through semantic feedback**. These are all valuable suggestions. In response, we have conducted additional in-depth experiments, including a new analysis inspired by your comments to **illustrate how preference alignment addresses spatial consistency in motion generation**. We sincerely appreciate your feedback — these experiments significantly improve the completeness of our work.\n\n---\n\n## [Q1/W1] Could the authors provide results of applying the proposed method to other domains and include them in the appendix?\n\nThank you for the suggestion — it helps clarify SoPo’s generalizability.\n\nWe demonstrate SoPo's transferability to the **text-to-image generation domain**, following the setup in [1]. We use Pick-a-Pic’s preferred image as the offline dataset, PickScore as the reward model, and Stable Diffusion 1.5 as the policy model. Evaluation is conducted on the Parti-Prompts (1632 prompts) using PickScore, HPSv2.1, ImReward, Aesthetic, and CLIP.\n\n|Model|PickScore↑|HPSv2.1↑|ImReward↑|Aesthetic↑|CLIP↑|\n|-|-|-|-|-|-|\n|SD1.5|21.40|24.97|0.121|5.355|33.11|\n|SFT|21.77|28.17|0.702|5.592|34.07|\n|DPO|21.62|25.79|0.386|5.442|33.54|\n|KTO|21.77|28.05|0.677|5.547|34.11|\n|SmPO (ICML'25)|22.00|**28.81**|0.911|5.636|34.72|\n|**SoPo (Ours)**|**22.23**|28.70|**0.958**|**5.749**|**34.80**|\n\nThese results show that **SoPo consistently improves multiple preference metrics across domains**. We will include the full results in the appendix.\n\n[1] Yunhong, Lu, et al. \"*Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferences.*\" arXiv preprint arXiv:2506.02698 (2025).\n\n---\n\n## [Q2/W2] Could the authors provide clearer comparisons with online DPO, offline DPO and their naive combination?\n\nWe appreciate your valuable suggestion to provide clearer comparisons.\n\nIn Tab.1 (in paper), we treat MoDiPO-G and MoDiPO-T as approximate online/offline DPO baselines, since MoDiPO only introduces minor technical changes to the standard DPO framework.\n\n**For a clearer comparisons, we conducted new experiments comparing online DPO, offline DPO, their naive combination, and combination with our proposed strategies** (Offline Sampling for Preferred Motions, Sec. 4.3, and Online Generation for Unpreferred Motions, Sec. 4.4). Specifically:\n- **Offline DPO**: We use MLD\\* as the base model, and follow MoDiPO to create an offline preference dataset.\n- **Online DPO**: We use TMR as the reward model, consistent with SoPo.\n- **Combination DPO**: We use offline dataset for preferred motion, and online generate unpreferred motion.\n\nResults are shown below:\n\n|Method|Top1↑|Top2↑|Top3↑|MM.Dist.↓|Div→|FID↓|\n|-|-|-|-|-|-|-|\n|Baseline (MLD*)|0.504|0.698|0.796|3.052|9.634|0.450|\n|OfflineDPO|0.498|0.691|0.790|3.081|9.618|0.469|\n|OnlineDPO|0.514|0.708|0.809|3.012|9.611|0.414|\n|CombinationDPO|0.517|0.712|0.811|2.988|9.606|0.339|\n|+Sec.4.3|0.520|0.715|0.814|2.973|9.602|0.318|\n|+Sec.4.4|0.525|0.719|0.820|2.951|9.591|0.243|\n|**SoPo (Ours)**|**0.528**|**0.722**|**0.827**|**2.939**|**9.584**|**0.174**|\n\nThese results highlight that **SoPo's hybrid semi-online design provides more effective and data-efficient alignment**, avoiding the limitations of both pure online and offline DPO.\n\n---\n\n## [Q3/W3] How well does the preference optimization solve spatial constraint misalignment for given texts?\n\nThis is a critical and valuable question — spatial misalignment is a common failure case in text-to-motion generation.\n\nThe core insight to solve this issue is that **reward models (discriminators)** are better at judging spatial semantics than **generative models** (generator), and SoPo leverages reward feedback to improve alignment.\n\nWe divide this issue into three sub-issues:\n1. Can the reward model distinguish left/right correctly?\n2. Can the diffusion model generate motions consistent with left/right prompts?\n3. Can SoPo improve generation via preference optimization?\n\n### [Q3-1] Reward Model Discrimination Ability of Spatial Misalignment\n\nFrom the HumanML3D test set (2,192 prompts), 783 prompts (35.72%) contain spatial information (e.g., “left” or “right”), highlighting the prevalence of this issue. For a text-motion pair $(x, t)$, we computed the reward score $r(x, t)$. We then created a misaligned text $t'$ by swapping “left” with “right” and computed $r(x, t')$. The reward model is considered successful if $r(x, t) > r(x, t')$. Results are shown below:\n\n|Model|Samples|Correct|Wrong|Accuracy|\n|-|-|-|-|-|\n|Random|-|-|-|0.500|\n|Reward Model|783|628|155|0.802|\n\n### [Q3-2] Diffusion Model Generative Ability of Spatial Alignment Generation\nWe randomly selected 100 spatial prompts from the 783 and generated 5 motions per prompt (500 total). Human annotators judged whether motions matched the spatial constraints:\n\n|Model|Samples|Correct|Wrong|Accuracy|\n|-|-|-|-|-|\n|Random|-|-|-|0.500|\n|Reward Model|783|628|155|0.802|\n|Diffusion Model|500|316|184|0.632|\n\n### [Q3-3] The Alignment Quality of SoPo\n\nAfter fine-tuning the diffusion model using SoPo, we evaluated the same 500 prompts:\n\n|Model|Samples|Correct|Wrong|Accuracy|\n|-|-|-|-|-|\n|Random|-|-|-|0.500|\n|Reward Model|783|628|155|0.802|\n|Diffusion Model|500|316|184|0.632|\n|**+ SoPo**|500|382|118|0.764|\n\nThese results demonstrate that:\n1. The reward model is capable of detecting spatial misalignments;\n2. The original diffusion model struggles with spatial understanding;\n3. **SoPo effectively enhances spatial alignment in generated motions.**\n\nHence, SoPo offers a practical solution to address spatial misalignment by integrating spatial semantic information from the text-motion-aligned reward model.\n\nSincerely, thank your insightful suggestion!\n\n---\n\n## [W4] As for the related work, more discussion is required for the recent advances in preference alignment for generation tasks.\n\nWe sincerely thank the reviewer for this suggestion. We will enhance the related work section to include recent advances in preference alignment, including **RL-based**, **differentiable-based**, **text-time alignment**, and **recent DPO-based** advancements, as well as applications in domains like **LLMs**, **video**, **3D**, and **audio**."}, {"review_id": "egfpYcDOKm", "reviewer": "Reviewer_GBbd", "summary": "This paper proposes a preference-learning-based generation method for text-to-motion generation. This paper attempt to utilize DPO in motion synthesis via text-to-motion model fine-tuning, and propose a Semi-online Preference Optimization. The training sample consisting of unpreferred motion from online distribution and preferred motion in offline datasets. The experimental results indicate better R-precision for motion retrieval.", "questions": "1. In the ablation studies, comparisons are made for different K and \\tau, it will be more important to see different sampling or weighting strategies.\n2. SoPo relies on a reward model to assess motion preferences. However, the current simple reward model struggles to evaluate human motion in a fine-grained manner. A more effective reward model should take into account richer physical and semantic information.", "limitations": "There is no obvious negative societal impact of their work.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. By combining high-quality offline \"preferred samples\" with dynamically generated online \"non-preferred samples\", the method improves the preference gap and prevents overfitting. \n2. The use of a threshold strategy can make the truly \"non-preferred\" samples be selected. \n3. The method is evaluated on different datasets and models, consistently outperforming baseline methods.\nWeakness:\n1 Preference Optimization for Motion Synthesis in MoDiPO [1] and combination of on-line and off-line preference optimization has been proposed in [2]. So, the contribution should be more clearly stated.\n2. According to Table 2, the R-precision increases, but the MM Dist and Diversity degrade, the proposed method may scarify some metrics for others.\n3. Lots of results are missing in Table 1 which make it seems to be unfinished.\n\n[1]Pappa, M., Collorone, L., Ficarra, G., Spinelli, I. and Galasso, F., 2024. MoDiPO: text-to-motion alignment via AI-feedback-driven Direct Preference Optimization. CoRR. 2024\n[2] Cen, S., Mei, J., Goshvadi, K., Dai, H., Yang, T., Yang, S., Schuurmans, D., Chi, Y. and Dai, B., Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF. ICLR 2025.", "was_revised": true, "final_justification": "The rebuttal provides the missing results. However, there are still concerns on the contribution of integration of off-line and on-line preference optimization built upon existing Preference Optimization. It is not enough. Meanwhile, the accuracy of the reward model is not convincing. So, I will keep my original rating.", "rebuttal": "Thank you for your valuable feedback and recognition of our method. Your comments on related work, ablations, and physical-semantic settings were very helpful. We reviewed the suggested references, clarified key differences, extended ablation studies, and added a new experiment to evaluate our method in physically and semantically grounded scenarios.\n\n---\n\n## [Q1/W1] Ablation studies about different sampling or weighting strategies.\n\nThank you for suggesting more comprehensive ablation studies on sampling and weighting strategies.\n\nSoPo builds upon on-/offline DPO, combining their strengths into a semi-online approach, termed Combination DPO, as noted by [GBbd]. However, this naive combination has limitations (in Sec. 4). To address these, we introduced Offline Sampling for Preferred Motions (Sec. 4.3) and Online Generation for Unpreferred Motions (Sec. 4.4). For offline dataset, we generate the preference dataset following MoDiPO. Results are shown below:\n\n|Method|Top1↑|Top2↑|Top3↑|MM.Dist.↓|Div→|FID↓|\n|-|-|-|-|-|-|-|\n|Baseline (MLD*)|0.504|0.698|0.796|3.052|9.634|0.450|\n|OfflineDPO|0.498|0.691|0.790|3.081|9.618|0.469|\n|OnlineDPO|0.514|0.708|0.809|3.012|9.611|0.414|\n|CombinationDPO|0.517|0.712|0.811|2.988|9.606|0.339|\n|+Sec.4.3|0.520|0.715|0.814|2.973|9.602|0.318|\n|+Sec.4.4|0.525|0.719|0.820|2.951|9.591|0.243|\n|**SoPo (Ours)**|**0.528**|**0.722**|**0.827**|**2.939**|**9.584**|**0.174**|\n\nThe results highlights the effectiveness of each SoPo component.\n\n---\n\n## [Q2/W2] Limitation of the Current Reward Model in Capturing Fine-Grained Human Motion Preferences and the Need for Richer Physical and Semantic Awareness\n\nThank you for raising this important point. Recent research in motion generation has indeed highlighted the need for **physical plausibility and semantic awareness**. We believe your concern touches on two aspects:\n1. Whether our simple reward model (text-motion alignment) can effectively **capture semantic information** in generated motions.\n2. Whether SoPo is extensible to **physically simulated reward environments**.\n\nOur work primarily focuses on how to leverage the reward model to improve motion generation quality, such as realism and text-motion alignment. Designing a more fine-grained reward model remains an important and open issue for future work. For these question, we think:\n1. Our reward model can capture fine-grained semantic imformation and improving the generative model’s spatial and physical awareness.\n2. Compared to online/offline DPO, SoPo has more potential to integrate with physically grounded environments.\n\n### [Q2-1] Experiment: The spatial semantic perception ability of Reward Model/SoPo\n\nOur key idea for improving text-motion alignment is to use feedback from a discriminator (reward model) to guide the diffusion model. Since discriminators better capture spatial semantics than generators, this enhances the generator’s spatial perception.\n\nWe focus on three questions:\n1. Can the reward model distinguish spatial semantic misalignment?\n2. Can the diffusion model generate spatially semantic aligned motions?\n3. Can SoPo improve the generative model’s spatial semantic perception?\n\n\n**[Q2-1.1] Can the reward model detect spatial semantic misalignment?**\n\nWe analyzed the HumanML3D test set with 2192 prompts and found that 783 (35.72%) contain spatial expressions such as “left” or “right”, showing this issue is common.\n\nFor each motion-text pair $(x, t)$, we calculate a reward score $r(x, t)$. Then, we flip the spatial keyword in the text prompt (e.g., “left” to “right”) to get $t’$, and compute $r(x, t’)$. If $r(x, t) > r(x, t’)$, we count this as a successful detection. The results are as the 2-nd row of the table.\n\n**[Q2-1.2] Can the diffusion model generate spatially semantic aligned motions?**\n\nWe randomly sampled 100 of the 783 spatial prompts and generated five motions per prompt (500 total) using the pre-trained diffusion model. Human evaluators then judged whether each motion aligned with its prompt. The results are shown in the 3-rd row of the table.\n\n**[Q2-1.3] Can SoPo improve the generative model’s spatial semantic understanding?**\n\nAs shown in the 4-th row of table, fine-tuning the diffusion model using SoPo significantly improved its spatial perception. \n\n|Model|Samples|Correct|Wrong|Accuracy|\n|-|-|-|-|-|\n|Random|-|-|-|0.500|\n|Reward Model|783|628|155|0.802|\n|Diffusion Model|500|316|184|0.632|\n|**+ SoPo**|500|382|118|0.764|\n\nThis highlights that  **SoPo effectively aligns the spatial semantics of generative model with the discriminator.**\n\n### [Q2-2] Discussion: SoPo for Richer Physically Simulated Optimization\n\nExisting works in physical simulation — such as TokenHSI [1], PhysHOI [2], and SkillMimic [3] — typically rely on imitation learning from physically simulated motion data. \n\nFor preference optimization setting, the motions used for imitation learning can be viewed as positive preference data, while failures during environment interaction serve as negative preference data. **This naturally aligns with SoPo’s semi-online optimization: combining offline motions with online failures for joint preference learning. It enables the model to both imitate high-quality motions and avoid undesirable ones during interaction.**\n\nCompared to pure online or offline DPO, SoPo is better suited for integration with simulated environments. Recent Embodied AI work applying Online DPO to tasks like grasping [4] further supports the extensibility of DPO-based frameworks.\n\nThank you for the insightful suggestion. We agree that stronger reward models with physical/semantic priors are valuable.  **However, our main contribution lies in proposing a novel preference alignment framework (SoPo),** rather than in designing new reward models per se.\n\n[1] *Tokenhsi: Unified synthesis of physical human-scene interactions through task tokenization.*\n\n[2] *Physhoi: Physics-based imitation of dynamic human-object interaction.*\n\n[3] *Skillmimic: Learning basketball interaction skills from demonstrations.*\n\n[4] *Evolvinggrasp: Evolutionary grasp generation via efficient preference alignment.*\n\n---\n\n## [W1] Related work Discussion\n\n### [W1-1] Preference Optimization for Motion Synthesis\n\nThank you very much for your valuable comment—this is indeed a critical and insightful point that we are eager to clarify. \n\nIn our paper (Lines 41–60), **we explicitly acknowledge and build upon the existing work of MoDiPO** [1], which first applies preference optimization to motion synthesis. **Rather than claiming to be the first to explore this direction, we carefully analyze why MoDiPO fails to deliver promising results** (Lines 41–50) and **propose SoPo as a new method to better unlock the potential of preference optimization** for this task (Lines 51–60).\n\n### [W1-2] Differences from VPO [2]\n\nThank you again for pointing out [2]—it is an excellent work. Below, we elaborate on the key differences between [2] and our method to more clearly state our contributions.\n\nIn [2], the authors formulate a unified preference optimization objective that incorporates both online and offline signals through reward regularization. **However, they design separate algorithms for the online and offline settings (see Algorithms 1 & 2) and report results for each setting independently, rather than integrating both within a unified algorithm.** In contrast, our work analyzes the theoretical strengths and limitations of online and offline preference optimization and proposes SoPo to combine their respective advantages. For experiment, **we jointly use both online and offline preference signals in a unified training pipeline, rather than treating them separately.** \n\nAdditionally, concurrent work in the LLM domain [3] (arXiv in 2025.06) tries to explore semi-online preference optimization, supporting the novelty and relevance of our approach.\n\nThank you again for your thoughtful feedback. We will revise our related work section. **Therefore, we believe our proposal of SoPo, which integrates online-DPO and offline-DPO into a unified semi-online optimization framework, represents a justified and novel contribution, as stated in Lines 41–60.**\n\n[1] *MoDiPO: text-to-motion alignment via AI-feedback-driven Direct Preference Optimization.*\n\n[2] *Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF.*\n\n[3] *Bridging Offline and Online Reinforcement Learning for LLMs.*\n\n---\n\n## [W2 (4-th point in \"Strengths And Weaknesses\")] According to Table 2, the R-precision increases, but the MM Dist and Diversity degrade, the proposed method may scarify some metrics for others.\n\nWe sincerely thank you for highlighting the potential trade-offs in Tab. 2. **To clarify, MMDist is better when lower, and Diversity is better when closer to ground truth.** \n\nIn Tab. 2, SoPo consistently improves R-precision. For MM Dist and Diversity: MLD benefits in both metrics, and MoMask improves in Diversity while maintaining comparable MM Dist (with only a slight drop). Only in one case does MM Dist degrade slightly, but it remains competitive.\n\n**Hence, while SoPo improves R-precision, it does not necessarily sacrifice MM Dist or Diversity.** We will make this clearer in the revised version to avoid misunderstanding.\n\n\n## [W3 (5-th point in \"Strengths And Weaknesses\")]  Missing Results in Table 1\n\nThank you for pointing out the missing entries in Table 1. These omissions were due to MoDiPO not reporting certain metrics. (Note: the MLD and MDM results are from MoDiPO, which differ from those in the original papers.) We have reproduced the missing values to the best of our ability, as shown below.\n\n||Top 1|Top 2|\n|-|:-:|:-:|\n|MLD|0.469|0.660|\n|+MoDiPO-T|0.471|0.662|\n|+MoDiPO-G|0.468|0.659|\n|+MoDiPO-O|0.416|0.581|\n|+SoPo|0.476|0.671|\n|MDM|0.418|0.604|\n|+MoDiPO-T|0.421|0.607|\n|+MoDiPO-G|0.416|0.605|\n\nWe will update the manuscript to include these values."}, {"review_id": "59kvTGgyAZ", "reviewer": "Reviewer_7xd7", "summary": "In SoPo, the authors introduce a novel preference optimization approach. \nOnline-generated samples are generated by the model, filtered by a reward function to serve as unpreferred data.\nOffline samples are designated as preferred data. \nThis combination addresses the issues with existing pure offline DPO (overfitting) and pure online DPO methods (biased sampling).", "questions": "1. Appendix (s6): The 5th line of the equation in line 44 has unmatched “)]”.\n2. line 102: Here pθ … with represents -> which represents\n3. Is the reward function just the TMR similarity score? What happens if TMR is overfitting?\nI think it will be necessary to test novel or out of distribution text and motion samples requested by the users.\n\n4. There’re a lot of engineering hyperparameters, which are “tuned through preliminary experiments”. I wonder how sensitive they are?", "limitations": "The authors address some limitations in the reward model, but as mentioned above, there are other limitations unaddressed in the paper.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 1, "originality": 2, "strengths_and_weaknesses": "Pros:\n1. Clarity and writing quality: The paper is well written with clear explanations and a logical flow of information.\n2. Theoretical rigor and practicality: The algorithm has a comprehensive theoretical analysis, but is also easy to implement engineeringly. The paper also provides an intuitive toy case to explain the motivation behind the algorithm.\n3. The paper contains a wide spectrum of motion experiments, and contains major variants of text to motion algorithms such as motion diffusion models and MMM.\n\nCons:\n1. The proposed algorithm is a general DPO improvement, but the experiments are only conducted in text to motion generation. It raises concerns on how generalized the solution is and how well the claim is supported. That is, if the claim is true, why don’t we apply it to other domains?\n\n2. The visual quality of the presented motion results is not entirely convincing. Specifically, the baseline performance without SoPo appears significantly inferior to what would be expected from a well-trained Motion Diffusion Model (as shown in the supplementary).\nThis observation suggests that the baseline model may be either undertrained or not optimally configured, potentially impacting the comparative efficacy demonstrated by SoPo.\n\n3. While the paper presents improved numerical results across metrics such as R-precision and FID, it is not clear that these quantitative gains consistently translate into a better alignment with human preferences.\nThis is particularly questionable since the “preferred samples” are directly chosen from the target dataset. With additional training it’s expected that those metrics will improve (unless I misunderstand how training works and the proposed algorithm actually trains from scratch?)\n\nTo address this critical ambiguity and to validate the practical efficacy of the approach, we need a comprehensive human evaluation study to support the claim of the paper.\n\n4. A more fundamental concern lies in the core assumption that 'preferred data' is exclusively offline and directly equates to the target dataset.\nThe application of text to motion authoring is an interactive process where preferred motions are chosen among generations or even directly authored by artists. By presuming preferred data is always pre-generated, the model's ultimate capacity to adapt to dynamic user preferences and its applicability in interactive creative workflows may be severely limited.\nThe fact that the model is always trying to give high scores to existing samples in the dataset also raises very big concerns on how well or if the model can generalize to novel text prompts from the users.", "was_revised": true, "final_justification": "Please refer to my below comments.\n\nDespite some concerns remain, the overall quality and novelty of the submission out shine the drawbacks.\n\nTherefore for the final rating, I recommend acceptance for this paper.", "rebuttal": "Thank you for your encouraging feedback. **We’ve addressed your concerns on overfitting, generalization, and preference diversity with added experiments and clarifications.** We also included an illustrative example to highlight our method’s effectiveness. We appreciate your insights and look forward to further discussion.\n\n---\n\n## [Q1/Q2] Typo\n\nThank you for pointing out the typo. We will correct it in the next version.\n\n---\n\n## [Q3/W3] Is the reward function just the TMR similarity score? What happens if TMR is overfitting? I think it will be necessary to test novel or out of distribution text and motion samples requested by the users.\n\nThank you for your insightful question about **the reward function and the potential overfitting of the TMR model**, particularly its impact on **out-of-distribution text and motion generation**. \n\nTo address your concerns, we conducted an OOD generation experiment.\n\n### [Q3-1] Is the reward function just the TMR similarity score?\n\nYes, our reward function currently relies on the TMR similarity score. However, designing a reward function may be not the core focus of our work. **Our method is fully extensible to multi-reward scenarios, allowing integration of additional reward signals as needed.**\n\n### [Q3-2] What happens if TMR is overfitting?\n\nThis is an important concern. To clarify, we use a **pretrained TMR model** rather than training one ourselves. As is standard in prior work [2,3,4,5], reward model training and policy fine-tuning are treated as independent stages. **It is thus critical that the reward model is trained properly to avoid overfitting [2,3].**\n\nIn fact, our method is **explicitly designed to mitigate this risk**. **If TMR exhibits some degree of overfitting, our generative model is not necessarily affected.**  As shown in Lines 88–89, we introduce a KL divergence constraint to regularize training:\n$$\n\\max\\_{\\pi\\_\\theta} \\mathbb{E}\\_{c\\sim\\mathcal{D}, x \\sim \\pi_\\theta(\\cdot|c)} \\left[ r(x,c) - \\beta D_{\\text{KL}} \\left( \\pi_\\theta(x|c)\\,\\|\\,\\pi_{\\text{ref}}(x|c) \\right) \\right].\n$$\nDPO-based methods inherently prevent overfitting by encouraging proximity to the reference model, acting as implicit regularization.\n\n### [Q3-3] Out-of-distribution Evaluation\n\nThanks for the suggestion. We clarify that for each benchmark, the preference data, pretrained generative model, and reward model are all trained on the training set of benchmark. **All evaluations are done on a held-out test set, which is not seen during training and thus nearly out-of-distribution.**\n\n**To further assess generalization, we conduct an OOD test by sampling 100 text prompts from the KIT-ML dataset and generating motions using a model fine-tuned on HumanML3D.** As a baseline, we also evaluate the original HumanML3D pretrained model. The results are:\n\n|Model|MotionCritic↑|\n|-|-|\n|Pretrained model|0.68|\n|+SoPo|1.36|\n\nNote: Since KIT and HumanML3D differ in their motion representation formats, ground-truth motions are incompatibility, making metrics like FID infeasible. Instead, we report MotionCritic scores for evaluation.\n\n---\n\n## [Q4] Sensitivity to engineering hyperparameters\n\nThank you for the question. Our method uses a **unified setting** across experiments: $C = 2$, $\\beta = 1$. The hyperparameters fall into two groups:\n1. **From SoPo**: filtering threshold $\\tau$, candidate number $K$, weight $C$  \n2. **From DPO**: temperature $\\beta$\n   \nFor SoPo-specific hyperparameters, **Tab.4**(in paper) shows they have minor influence. Below, we report results on MLD* to analyze the sensitivity to $\\beta$ and $C$:\n\n|Method|R@1↑|R@2↑|R@3↑|MMDist↓|FID↓|\n|-|-|-|-|-|-|\n|SoPo(C=1,$\\beta$=0.25)|0.523|0.717|0.823|2.941|0.176|\n|SoPo(C=1,$\\beta$=0.5)|0.524|0.718|0.824|2.940|0.175|\n|SoPo(C=1,$\\beta$=1)|0.525|0.719|0.825|2.939|0.174|\n|SoPo(C=2,$\\beta$=0.25)|0.527|0.721|0.826|2.938|0.173|\n|SoPo(C=2,$\\beta$=0.5)|0.528|0.722|0.827|2.937|0.172|\n|SoPo(C=2,$\\beta$=1)|0.528|0.722|0.827|2.939|0.174|\n|SoPo(C=3,$\\beta$=0.5)|**0.532**|**0.726**|**0.831**|2.935|0.170|\n|SoPo(C=3,$\\beta$=1)|0.530|0.724|0.829|**2.934**|**0.169**|\n|SoPo(C=3,$\\beta$=2)|0.529|0.723|0.828|2.936|0.171|\n\nThe results show our method is robust to changes in $\\beta$ and $C$.\n\n---\n\n## [W1] Concern on Generalizability\n\nThank you for raising this point — it clarifies SoPo’s generalizability.\n\nTo test beyond text-to-motion, we follow [1] and apply SoPo to image generation. We use Pick-a-Pic preferences as offline data, PickScore as reward, and SD 1.5 as the policy. Evaluation on PartiPrompts uses PickScore, HPSv2.1, ImReward, Aesthetic, and CLIP. Results:\n\n|Model|PickScore↑|HPSv2.1↑|ImReward↑|Aesthetic↑|CLIP↑|\n|-|-|-|-|-|-|\n|SD1.5|21.40|24.97|0.121|5.355|33.11|\n|SFT|21.77|28.17|0.702|5.592|34.07|\n|DPO|21.62|25.79|0.386|5.442|33.54|\n|KTO|21.77|28.05|0.677|5.547|34.11|\n|SmPO [1] (ICML25)|22.00|**28.81**|0.911|5.636|34.72|\n|SoPo(Ours)|**22.23**|28.70|**0.958**|**5.749**|**34.80**|\n\nWe will include these results in the appendix of the final version.\n\n[1] Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferences\n\n---\n\n## [W2] Concern on baseline quality (whether either undertrained or not optimally configured) and the reason of SoPo’s improvement\n\n### [W2-1] May the selected diffusion model be either undertrained or not optimally configured\n\nThank you for the question. We acknowledge that our qualitative results use classic MDM and MLD models (ICLR 23 & CVPR 23). **However, all pre-trained models are official open-source implementations with pretrained weights.** We selected these two classic because prior work (MoDiPO) only reported results on these models without releasing code. **Importantly, Tab. 2 and 3 use SoTA MLD\\* and MoMask as base models, showing consistent SoPo improvements on both classic and SoTA models.**\n\n### [W2-2] Visual improvement and quantitative analysis\n\nThis is a valuable question. **To further evaluate visual improvement, we now quantify the representative case in our paper, spatial alignment.**\n\nAmong all test prompts, 783 (35.7%) contain spatial cues (e.g., “left”, “right”). This raises an important question: **Why does SoPo help the motion diffusion model better understand left-right spatial information?**\n\n#### [W2-2.1] Reward model’s spatial understanding\n\nGiven a prompt $t$ and motion $x$, we compute reward $r(x,t)$. We then swap “left” and “right” to get $t'$ and compare $r(x,t)$ vs. $r(x,t')$. If $r(x,t) > r(x,t')$, it’s considered a success.\n\n|Model|Count|Success|Failure|Accuracy|\n|-|-|-|-|-|\n|Random|-|-|-|0.500|\n|Reward model|783|628|155|0.802|\n\n#### [W2-2.2] Diffusion model generation\n\nWe sampled 100 prompts with spatial terms. For each, 5 motions were generated (500 total). Human evaluation counted how many reflected correct spatial meaning.\n\n|Model|Count|Success|Failure|Accuracy|\n|-|-|-|-|-|\n|Diffusion model|500|316|184|0.632|\n|+ SoPo|500|382|118|0.764|\n\nResults suggest that the reward model captures spatial distinctions more effectively than the diffusion model. **SoPo leverages this supervision signal from the reward model to improve spatial understanding of generative model.**\n\n---\n\n## [W3] Concern: It's unclear whether improvements in metrics like R-precision and FID truly reflect better human preference alignment, especially since the \"preferred samples\" are drawn from the training set. Reviewer also questions whether improvements come simply from more training.\n\n### [W3-1] Relationship Between Metrics and Human Preference \n\nWe appreciate the reviewer’s concern. In our setting, **human preference is defined as a combination of text-motion alignment and motion quality**. R-precision and FID respectively quantify these aspects (in Line 242-243), and are therefore aligned with this definition.\n\n### [W3-2] Clarification on Training Data\n\nWe appreciate the concern regarding potential data leakage. **To clarify, the “preferred samples” used as training data correspond strictly to the benchmark’s original training set.** Both the pretrained generative model and the reward model are trained solely on this set **without access to any additional data**. \n\nThus, the reported improvements reflect better alignment within the benchmark distribution rather than from exposure to extra or out-of-distribution data. **Additionally, the reason for performance improvement is clarified in [W2].**\n\n\n## [W4] Concerns on Preference Diversity and Generalization\n\nThank you for raising these points. The concerns are:\n1. **Real-world preferences are diverse and dynamic**, but we use fixed preferred data and static reward models, limiting adaptability.\n2. **Offline-learned preferences may not generalize to out-of-distribution prompts.**\n\nThese are common challenges in offline post-training/RLHF and DPO approaches, which, while not the main focus of our work, are important for the broader community:\n- We focus on preferences like **motion realism and text-motion alignment**, which are relatively objective. In the image domain, common criteria include alignment and safety. **Handling subjective, diverse preferences with limited data remains a key open problem.**\n- Learning from fixed datasets is standard in offline post-training. **Prior work (e.g., SD3 [5]) shows strong generalization despite being trained on static data**.  In my opinion, this may be since post-training reshapes the pretrained distribution without compromising its continuity or semantic structure.\n\nWe hope this respond addresses your concerns well.\n\n[1] TMR: Text-to-Motion Retrieval Using Contrastive 3D Human Motion Synthesis\n\n[2] ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation\n\n[3] Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation\n\n[4] Diffusion Model Alignment Using Direct Preference Optimization\n\n[5] Scaling Rectified Flow Transformers for High-Resolution Image Synthesis"}]

Author rebuttal:

Meta-review: The AC and the reviewers thank the authors for their response.
This paper combines the advantages of offline and online DPO, and for the first time effectively addresses the problems of overfitting and sampling bias in text-to-motion preference alignment, surpassing the existing SOTA across multiple models.

After the rebuttal, three reviewers gave a borderline acceptance recommendation. However, one reviewer questioned the contributions and raised concerns about the accuracy of the reward model, and therefore gave a borderline reject recommendation.

After reading the paper, the reviewers’ comments, and the authors’ feedback, the AC considers the contributions of this work to be clear. The paper is the first to apply the combined strengths of offline and online DPO to text-to-motion, which is valuable for advancing motion generation and provides a useful reference for future work in this area. In addition, given the authors’ thorough response and discussion, the AC believes that most of the concerns have been addressed.

The AC concludes that the paper can be accepted after further refinement based on the reviewers’ suggestions, and therefore recommends acceptance.
Decision: Accept (poster)