LAPP: Large Language Model Feedback for Preference-Driven Reinforcement Learning
Authors: Pingcheng Jian
Xiao Wei
Yanbaihui Liu
Samuel A. Moore
Michael M. Zavlanos
Boyuan Chen
Venue: Accepted by TMLR ()

Abstract: We introduce Large Language Model-Assisted Preference Prediction (LAPP), a novel framework for robot learning that enables efficient, customizable, and expressive behavior acquisition with minimum human effort. Unlike prior approaches that rely heavily on reward engineering, human demonstrations, motion capture, or expensive pairwise preference labels, LAPP leverages large language models (LLMs) to automatically generate preference labels from raw state-action trajectories collected during reinforcement learning (RL). These labels are used to train an online preference predictor, which in turn guides the policy optimization process toward satisfying high-level behavioral specifications provided by humans. Our key technical contribution is the integration of LLMs into the RL feedback loop through trajectory-level preference prediction, enabling robots to acquire complex skills including subtle control over gait patterns and rhythmic timing. We evaluate LAPP on a diverse set of quadruped locomotion and dexterous manipulation tasks and show that it achieves efficient learning, higher final performance, faster adaptation, and precise control of high-level behaviors. Notably, LAPP enables robots to master highly dynamic and expressive tasks such as quadruped backflips, which remain out of reach for standard LLM-generated or handcrafted rewards. Our results highlight LAPP as a promising direction for scalable preference-driven robot learning.

Published in Transactions on Machine Learning Research (09/2025)
LAPP: Large Language Model Feedback for Preference- Driven Reinforcement Learning
Pingcheng Jian Duke University
pingcheng.Ban@dduee.ee
Xiao Wei Duke Universily
xãao. Stao.weistduuke.oo edu
Yanbaihui Liu Dake University
yonbaihui.liufeee liw@duke.Be.... edu
Samuel A. Moore
yonbaihui.luu@duuke liw@duke. edu
Dake University
Michael M. Zavlanos
michael.aoolnnos@dbbeecc.c edu
Duke University
Boyuan Chen Duke University
boguan. cetiwonncceendduue.co edu
Project Website:
www.gpeeealrooooiclaa.ooom//Z/
Reviewed on OpenReview:
Abstract
We introduce Large Language Model-Assisted Preference Prediction (LAPP), a novel frame- work for robot learning that enables efficient, customizable, and expressive behavior acqui- sition with minimum human effort. Unlike prior approaches that rely heavily on reward en- gineering, human demonstrations, motion capture, expensive pairwise preference labels, LAPP leverages large language models (LLMs) to automatically generate preference labels from raw statc-action trajectories collected during reinforcement learning (RL). These labels are used to train an online preference predictor, which in turn guides the policy optimiza- tion process toward satisfying high-level behavioral specifications provided by humans. Our key technical contribution is the integration of L.I.Ss into the RL foedback loop through trajectory-level preference prediction, enabling robots to acquire complex skills including subtle control over gait. patterns and rhythmic timing. We evaluate LAPP set of quadruped locomotion and dexterous manipulation tasks and show that it achieves efficient lcarning, higher final performance, faster adaptation, and precisc control of high- level behaviors. Notably, LAPP enables robots to master highly dynamic and expressive tasks such as quadruped backflips, which remain out reach for standard LLM-generated or handcrafted rewards. Our results highlight LAPP as a promising direction for scalable -efference-driven robot learning.
diverse
1 Introduction
Designing effective reward functions remains a fundamental challenge in training robots with reinforcement learning (RL) (Ratner al. 2018 Dang et al. 2023 Eschmann 2021 Sorg et al., 2010 Evans et al.) 2021 Ng et al. 1999 Grzes & Kudenko 2008; Devlin & Kudenko 2012). Reward functions define the objectives
and constraints of the learning process, but often hand-crafted through trial and error, a process that is

---

Published in Transactions on Machine Learning Research (09/2025)
Dofs
LAPP
*... Your preference feedback will be used reward signal to train robot to ..."
Large Action Space
Behavior Specification
Bestrtip
Adopt in Real World
Challenging Tasks
Performance Gain
Figure 1: Large Language Modcl-Assister Preference Prediction (LAPP) takes in language behavior instructions and generates preference feedback to guide reinforcement learning training from raw stale-action robot trajectories.
sabor-intensive, error- error-prone, and difficult to scale. Poorly designed rewards can lead suboptimal or unsafe behaviors, making robust and expressive policy learning especially difficult in real-world robotic systems.
Several alternatives have been proposed to reduce this burden. Inverse RL infers reward functions from
expert demonstrations (Arora & Doshi 2021 Ng et al. 2000; Abbeel & Ng 2004 Zhou & Small 2021
but requires significant human effort and data collection. More recently, large language models (LLMs) and
language models (VLMs) have been used to automate aspects of reward design (Ma et al., 2023 Yu
ett al. 2023 Xie al. 2024 Yu et al.] 2024 Wang et al., 2024). These methods typically refine reward functions by analyzing task descriptions or environment code. While promising, they often fail to capture nuanced and high-level behavioral qualitics, such rhythmic locomotion expressive timing, which are hard to specily with explicit reward terms.
Another line of work bypasses explicit reward engineering by learning from human or Al preferences over trajectory pairs (Christiano et al. 2017 Early et al. 2022 Kim et al. 2023) Wang et al. 2024 Venkataraman et al.) 2024]. By querying which of two behaviors is preferable, these methods convert preferences into supervision for training reward models. However, preference queries can be costly and cognitively demanding for humans, and recent VLM-based methods remain limited to relatively simple tasks with low-dimensional action spaces. These approaches also typically assume Markovian decision process, which may not hold for long-horizon and high-dimensional control tasks.
In this work, we propose LLM-Assisted Preference Prediction (LAPP) (Fig. & novel framework that enables robots to learn efficient, customizable, and expressive behaviors from human language specifications. The core idea of LAPP to leverage LLMs to generate preference labels on full state-action trajectories, which are then used to train an online transformer based reward predictor. This predictor produces dense, trajeclory-iformeed reward estimates that guide policy optimization and continually updated with new policy training and rollouts. Unlike prior approaches, LAPP integrates the LLM-generated feedback directly into the RL loop, enabling closed-loop refinement of learned behaviors.
We evaluate LAPP across a suite of challenging control tasks, including quadruped locomotion and dexterous manipulation with up to 52-dimensional action spaces. LAPP not only accelerates training and improves final performance compared to state-of-the-art baselines, but also enables nuanced control over high-level behavior attributes, such as gait symmetry, timing, and cadence, through simple languag inputs. LAPP also enables faster adaptation to unseen environmental conditions. Notably, LAPP successfully solves exploration-heavy tasks such as quadruped backflips, which were previously infeasible with human-designed, LLM-generated, or VLM-derived reward functions.
We summarize our key contributions as follows:
novel learning framework (LAPP) that uses LLM -ene-aeee preference feedback over state-action trajectories to guide reinforcement learning with language behavior instructions.
2

---

Published in Transactions on Machine Learning Research (09/2025)
2. A transformer-basee online preference predictor that models trajectory-level feedback and integrates it as dense supervision into the RT. policy learning loop through iterative policy & reward model improvements.
3. Empirical results showing the LAPP outperforms baselines in training speed, final performance, adaptation efficiency, and controllability of high-level behaviors across complex robotic tasks.
4. Ablation studies dissocting architectural and algorithm design choices to reveal how trajectory-level modeling and online reward model updates contribute to LAPP's success.
2
Related Work
Foundation Models for Robotics. Recent advances in foundation models have spurred applications in
robotic action generation (Octo Model Team et al.
2024: Szot et al.
2023 Zitkovich et al. 2023 Tang et al.
2023
simulation [Authors] 2024).
task planning [Lin et al.
2023
Ahn et al.
2022
Singh et al.
2023 Huang
ct al.
2024
Zhang et al.
2023
Hao et al. 2023 Liu et al.
2023
Ha et al. 2023 Wang et al. 2023c Ding
jet al.
2023
Silver et al.) 2024
Xie 2020
Huang et al., 2023
Liang et al.
2023), and sim-to-real transfer
(Ma et al.
2024). A growing line of work
focuses on using language or vision-language models to automate
aspects of reward engineering (Ma et al.
2023 Yu et al.) 2023
Xie et al.
2024 Yu et al. 2024
Wang et al.
2024) or generate training environments [Lingg et al.. 2024 Wang et al.
2023b d Faldor et al.
2024 Wang
et al. 2023c). However, these approaches remain limited in specifying high-level behaviors (Oclo Model
Team et al. 2024h, handling hard exploration challenges (Ma et al.) 2023), and scaling to high-dimensional
action spaces (Wang et al. 2004). In contrast, LAPP fills in this gap and achieves superior performance in
hard exploration tasks with high-dimensional action spaces given only high-level behavior specification.
Reward Signal Design for Challenging Robotic Tasks. Reward design is a crucial component of RL
[Ratner et al. 2018 Dang et al. 2023 Eschmann 2021 Sorg et al. 2010 Jian et al.
2021 Evans et al. 2021
Xia et al. 2024).
To address sparse reward issues, prior works explore reward shaping
Grzes & Kudenko
2008
Devle in & Kudenko 2012 Devidze et al. 2022 Marom & Rosman 2018
Hu et al.]
2020 Gupta et al.
2022
Grzes 2017 Zou et al. 2019 Goyal et al. 2019 Memarian et al. 2021
Hussein et al. 2017 Ng et al.
1999
However, complex agile motions, such as rapid locomotion (Margolis al.) 2024) and backflips (Tang
et al.
2021 Kim et al.
2024a remain difficult to learn with single reward function.
Reward design remains bottleneck for complex RL tasks, especially when rewards are sparse, brittle,
difficult to engineer. Prior elforts explore reward shaping (Grzes & Kudenko 2008 Devlin & Kudenko 2012
Devidze et al.) 2022) Marom & Rosman
2018
Hu et al.] 2020
Gupta et al., 2022 Grzes 2017 Zou et al.
2019 Goyal et al.
2019: Mcmarian ct al.
2021 Hussein ct al.
2017 Ng ct al. 1999.. curriculum learning
(Tang el al.] 2021 Margolis et al.j 2024
Ryu el al. 2024), and mulli-objective oplimization (Kim el al.
2024a: K Kyriakis & Deshmukh] 2022 Van Moffaert et al. 2013 Basaklar al.
2022; Xu et al.
2020 Cai
et al. 2024| Abdolmaleki al. 2020
Yang et al. 2019 Hayes et al.) 2022 Huang et al.)
| [0022]
Inverse RL
methods aim to infer reward signals from demonstrations (Arora & Doshi
2021 Ng et al. 2000
Hadfield-
Menell et al. 2016 Zakka et al. 2022 Brown et al.) 2018 Kumar et al.]
2023
Das et al.
2021
Abbeel &
Ng 2004 Zhou & Small 2021), but require curated expert data.
While recent works attempt to automate reward or curriculum generation using LLMs (Ma et al. 2023 Liang
et al. 2024). they still depend on explicit reward decompositions or low-level state supervision which can be
difficult to obtain for complex tasks and high-level behavior specifications. Our method complements these advances by using LLMs to generate implicit preference feedback. As our experiments show, our method achieves the best performance when combined with previous shate-of-the-art reward designs.
Human-Guide Machine Learning. Integrating human guidance into machine learning has been widely explored to improve training efficiency and model performance (Amershi el al. 2014 Gil et al.) 2019 Wu
et al.) 2022 Zhang et al.) 2019). Various methods incorporate human demonstrations (Pomerlean
1988
Schaal 1996), instructions (Zhou Small, 2021 Saran et al. 2021), ) and corrections (Chai & Li 2020 Ji
et al. 2024) to enhance imitation learning (Pomerleau] 1988 Schaal 1996 Saran et al.) 2021] or inverse RL
(Abbeel & Ng 2004 Zhou & Small 2021). Other works model human feedback reward functions (Knox &
3

---

Published in Transactions on Machine Learning Research (09/2025)
Stone 2008
Warnell et al.
2018 or advantage functions (MacGlashan et al.
2016 Arumugam et al. 2019)
to guide RI. Recent advancements extend these algorithms to continuous action spaces (Sheidlower et al. 2022), multi-agent scenarios (Ji et al. 2024), multi-agent scenarios (Ji et al. 2024 and real-time human foedback (Zhang ct al. 2024a b).
The most relevant works to ours are those that learn from human preferences WWitth et al.
2017 Akrour
et al.
2011 Daniel et al.
2015: Fürnkranz al.)
2012
Ibarz al.
2018 Wilson et al.)
2012
Wirth et al.
2016
Kim et al.) 2024b Dong et al. 2023 Liu et al.) 2024 Aroca-Ouelletee Ouellette et al.) 2024
Akrour et al.) 2012
Liu et al.
2020 Lee et al.)
2021b Knox et al., 2022
Ouyang et al.
2022 Park et al. 2022 Verma & Metcalf
2022
Christiano et al. 2017). with applications in LLM fine-tuning BBrwnn ct al. 2020), summarization (Wu
et al.
2021), robotic manipulation (Hejna III & Sadigh 2023), and locomotion (Yuan et al.) 2024 Yu et al. (2024) uses human preference to select the
2021), browser-assisted question answering (Nakano et al.)
reward functions generated by LLM, while the human preference is not directly predicted as a preference reward to guide the policy optimization.
Our work builds on reinforcement learning from human feedback (RLHF) CCristiano et al. 2017 where human preferences used to train MLP-based preference predictors for Markovian rewards. Later works ex- rewards using Preference Transformers (Kim et al., 2023). However, RL.FF methods require extensive human annotation, with human annotators evaluating thousands of trajectory pairs. Recent research proposes Reinforcement Learning from AI Feedback (RLAIF) (Bai et al.) 2022 Lee et al.) 2024 Wang et al.) 2024 Venkatara- man et al. 2024), replacing human annotators with AI models. However, these approaches are limited Markovian rewards and have only been tested in low-dimensional robotic tasks. Our work not only reduces annotation cost but allows for preference-driven RL in more complex task domains than those explored in existing RLHF or RLAIF frameworks.
tend this to non- MorkMiakovi settings with LSTMs (Early et al. 2022) and importance-weigee
Preference Feedback for Robot Learning. Learning from human or AT preferences has emerged as an alternative to explicit reward design (Christiano et al. 2017 Early et al. 2022 Kim et al. 2023 Yuan et al. 2024). These methods traīn reward models using preference labels over trajectory pairs, typically annolated by humans. While effective, annotation costs remain high.
More recent works adopt Al-generated feedback in place of human raters, such as RL-VLM-F (Wang et al, 2024 Venkataraman et al. 2024), which uses vision-language models to rank state images. However, such models operate under a fixed preference criterion throughout the entire policy learning process and assume Markovian rewards, limiting them to relatively simple low-DoF tasks like CartPole or tablctop manipulation. Specifically, the manipulation tasks in RI-VLM-F have 4 to degrees of freedom, while the dexterous manipulation tasks in LAPP have 52 degrees of freedom, and our locomotion tasks have 12 degrees of freedom.
LAPP offers advancements in this direction in several key aspects. First, LAPP is the first work to operate on raw state-action trajectories to provide effective preference foodback from LLMs. This method avoids reliance on vision-based snapshots to query VLMs, which currently come with much higher costs than LLMs and still do not yet demonstrate strong reasoning capabilities, hence limiting the task complexity they can solve. Second, LAPP models both Markovian and non- Mar-Maaoov preference rewards using transformer architectures to allow reasoning over long temporal sequences.
Moreover, LAPP enables dynamic preference shaping by prompting LLMs to evolve their evaluation criteria training progresses, while prior works (Wang et al., 2024 Venkataraman et al. 2024) rank the states with a static standard. To our knowledge, LAPP is the first method to fully automate preference alignment via LLMs for training policies in complex and high-dimensional tasks, including quadruped backflips and dexterous hand manipulation.
3 Preliminaries
We consider an agent interacting with an environment over a sequence of discrete timesteps a RL. framework SSttonn & Barto 2018). At each timestep t, the agent receives an observation or of the current state s and selects action a, based on its policy n. The environment. then provides reward r, based the
4

---

Published in Transactions on Machine Learning Research (09/2025)
Transformer Reward Predictor
Preference
LLM
(s Sf. s.)
Preference 1p Reward
LAPP
Reinforcement Learning
Trajectory Pair: (s 1p s()
Tø Environment Reward
Si...,
Rollout Trajectories
Env
(S ..., s(), (s ..., st),
Behavior Instruction:
"Jump
rorate
back to do backtlip"
(s SN s..,
Policy
Figure 2: LAPP generates preference feedback from an LLM based on rollout trajectories pairs of raw state and actions as well as a high-level behavior instruction. A transformer-basee reward predictor is trained using these preferences while simultaneously optimizing robot policy to maximize a weighted sum euvironment rewards and predicted preference rewards.
pre-designed reward functions and transitions the agent the next state 8141- The goal of RL is optimize to maximize the expected return R, 7 = where is the discount factor.
However, designing reward function that precisely captures high-level behavioral objectives human preferences remains challenging. To address this, preference- based RL learns reward model that predicts human preferences instead of relying on manually defined rewards. In this setting, we consider a pair of trajectory segments (o", o!) with length H: o = {($1, a1), (SH, aH)}. preference relation > o² indicates that segment o¹ preferable over segment σ². Given a pair (o", o!), human or AI provides a preference label y {0, 1.053:
o² 1 ol σ
0.5
and a¹ are equally preferable 0
The preference judgments are recorded in datasel D labeled preference triples (o",a1_ .y)
To obtain preference-basel reward model f, prior works (Christiano et al. 2017 Ibarz et al.
2018 Lee
2 Hejna III Sadigh 2023: Park et al. 2022] adopt the Bradley- Terry model (Badley 1952h, assuming Markovian rewards (i.e. the reward depends only on the current state and action) The probability of preferring one segment over another modeled as:
et al.
Terry
exp Σ E (s),a)) exp (st, exp Dxpp i as)
P o7l
However, Markovian rewards struggle with long-horizon tasks where preferences depend on past trajectoric rather than only the current state-action pair. To address this, recent works (Kim al. 2023 Early et al. 2022) propose non-Markovian rewards, where á considers the full preceding sub-trajectory segment
{(s,, a )
expr(i(s) Hi))) P o===
(2)
E Eje(aa exp (s (ss a
5

---

Published in Transactions on Machine Learning Research (09/2025)
The reward predictor & is then trained via supervised learning to fit the dataset D by minimizing the cross-entropy loss:
C CB (f)=
Σ (1- log P P[oO o o'] + ylog P [ oo's x0]
(o),02,weD
To mitigate the noisy LLM outputs, We assumee that the LLM has 15% of chance to provide preference feedback uniformly at random. Thercfore, the adjusted preference probability is:
P [o" - o'' -1 - c) [o" - +.-0.5
(4)
where 0.15 is the error rate. We also have:
P o' =1 p' [o0> o
(5)
Therefore, the adjusted cross-entropy loss becomes:
L =
Σ
[(1 - 4) log frr [oº x o'] y ylog PV [o' o']]
Σ
[(1 - v) log ((1 (1- c) [o" - or] + 0.5)
(6)
+ yog (1 ((1 ((1-e) c) [d o'] +e +.5)).
Once trained, the reward predictor f can be used to guide policy optimization, where an RL algorithm maximizes the expected return from the learned preference rewards. We leave the exploration of tuning future work. All our experiments have used the same value, hence have observed that our method is not sensitive to the selection of this value.
4 LLMs-Assisted Preference Prediction
LAPP is a novel framework that enables preference-driven RL by integrating LLM-generated leedback into the policy training loop. It consists of three main components: 1) Behavior Instruction: a prompting strategy clicit trajectory preferences from LLMs given language description of task objectives and preferred behaviors; 2) Preference Predictor Training: an ensemble of transformer-basee models that learn to predict preference rewards; and 3) Preference-Driven Reinforcement Learning: robot policy is optimized using both environment rewards and predicted preference rewards. An overview of LAPP is shown in Fig.
4.1 Behavior Instruction: Generating Preference Labels from State-Action Trajectories
Conventional RLHF frameworks rely on human annotators to label trajectory preferences. However, this process is labor intensive. To reduce this burden, LAPP replaces human annotators with LLMs by prompting them to generate preference labels for pairs trajectory segments od and o!.
Fig. illustrates an example of a behavior instruction prompt. The first part defines the LLM's role the robot's goal and desired behavior (e.g., "walk forward with bounding gait"). The second part provides numerical values and their descriptions of each trajectory (e.g., base velocity, orientation, foot contacts). The third part defines how preferences should be evaluated and formatted.
Unlike prior work that uses video clips (Christiano et al. 2017 Kim et al. 2023 for human annotation, we feed LLMs structured numerical state-action logs, since the current multimodal foundation models such as GPT-4o lacks fine-grained video understanding with high costs and slow responses.
Notably, enhance learning efficiency, encourage the LLM to refer to our provided evaluation criteria and generate adaptive evaluation criteria, allowing the LLM to dynamically adjust its preferences training progresses. For instance, in quadruped locomotion, early-stage training should prioritize learning to stand, followed by developing stable movement, and ultimately refining gait patterns and command adherence. Instead of providing these stages explicitly by humans, our prompts ask the LLM to actively decide the important factors for different training stages by itself.

---

Published in Transactions on Machine Learning Research (09/2025)
LAPP supports batched labeling of five trajectory pairs per prompt which can significantly reduce API latency and token costs, the output consists of pref- crence labels in 0, 2,3, indicating whether trajec- tory o² is better, worse, equally preferable, or in- comparable. To promote clear supervision, we en- courage the LLM to avoid ambiguous labels. All labels are stored as triples (o", o ,y) in a growing preference dataset DD Details of all state-action vari- ables used for all tasks and full prompts can be found in Appendix A.2
You are robotics engineer trying quadruped robot locomotion trajectories decide which one is better in each pair. Your feedback of the comparisons will be used as a reward signal to train a quadruped robot to
compare pairs
Each trajectory will contain 24 time steps of states of the robot trying to... The state includes:
decide which trajectory better pair, here are some
criteria:
1)
2)
4.2 Preference
The user will provide pairs trajectories in batch, and you should provide preference value each pair.
Predictor Training: Modeling LLM Feedback
1) If the trajectory better, the preference value should be O.
LAPP models LLM-generated preferences using ei- ther Markovian or non-Markovian reward functions, depending on task complexity. For tasks like flat- ground locomotion, a Markovian reward model fol- lowing the Bradley-Terry formulation (Eq. [ suf- fices. However, for more challenging tasks such as quadruped backflips or gait cadence control, non-Markovian reward function (Eq. 2 is neces- sary to caplure long-term dependencies in behavior. Training the predictor for non nor-Moakovinn rewards roquires additional computational resources, as it must process historical states to infer the reward at given timestep. TAPP adopts the appropriate reward model based on task requirements to balance the preference prediction accuracy and the predictor training elficiency.
2) If the trajectory 1 is better, the preference value should be 1.
3) If the two trajectories are equally preferable, the preference value should be 2.
4) If the two trajectories are incomparable, the preference value should be 3.
Please give response with only one list of preference values... If each trajectory has its pros and cons, instead saying they are equally preferable, you can decide which criteria are more important at this stage of training, and then decide which trajectory is preferable.
Figure 3: Behavior Instruction Prompt Exam- ple. The LLM prompt consists of three sections: (I) defining the LLM's role and the robotic task (blue box), (2) specifying the state variables and some evalu- ation criteria of preference (green box), and (3) estab- lishing rules and semantics for generating preference labels (purple box).
The preference dataset D, = {(a", ,),,)} is split into training (Dirain) and validation (Dpal) sets at a 9: ratio. We maintain an ensemble of M preference predictor networks, each trained to minimize the cross- entropy loss (Eq. 3) To prevent overfitting, training stops early if the validation loss exceeds times the training loss and the training has gone through minimum number of iterations Kmin- If carly stopping is triggered, the training will finish after Kmax iterations. After training all P predictors, we select the top C' models with the lowest validation losses and compute the final preference reward as their ensemble average. In practice, we set P p=0 == 9, C = 3, Kinn = 30, Kmax = 90, and a = 1.3. The full training procedure is detailed in Algorithm This ensemble approach can help increase robustness to LLM label noise.
4.3 Preference- Driven Reinforcement Learning
LAPP continuously aligns robot behaviors with high-level task specifications throughout RL training by iteratively updating both the preference predictor and the policy network. Unlike previous RLHF approaches (Christiano et al.) 2017 Kim et al. 2023) that train static preference models, LAPP dynamically refines preferences during training.
Initially, the policy generates rollout trajectory pairs (of ot), which are evaluated by the LLM to generate preference labels { To mitigate noisy outputs which could pose potential risks to training stability, we sample 15 preference labels for each trajectory pair and calculate the mode of them as the final selected preference labels {u:). The labeled data is stored in an initial preference dataset D, [[o). af Pp which is used to train the initial preference predictor via Algorithm [
Ppl.
During RL training, the reward at each timestep is computed as a weighted sum of the predicted preference reward Tp and the environment reward rg defined by built-in explicit reward functions in our evaluation
7

---

Published in Transactions on Machine Learning Research (09/2025)
Algorithm 1: LAPP Preference Predictor Training
1 Require:
Ensemble of preference predictors {ft), preference predictor training dataset Dp
2 Hyperparameters:: Minimum iteration mins Maximum
iteration Kmar; pool of predictors number P, selected predictors number C, overfitting scale a, LLM feedback error rate E.
//Split into training and validation sets
4 Derain Doal < pp)
5 val_loss_ _is__oss__sstt__________________ loss list + []
6 for P predictors do
7 Randomly initialize P;
for K + to Kman epochs do
H1 Sample from Derein (strain, at agrain) D
11
// Predict preference reward rtrain = * at afras))
// Train the preference predictor Calculate loss Lirain (rtrain) with Equation
14
A - -Adam (f,, Wtami (rtrain)
15
// Sample from Deal
16
(spal, aaaa)) ~ Dial D
// Predict preference reward
19
r/ =f(ss ,apal)
20
Calculate loss
with Equation G
if (rtal) > CCE, (rtrain) and k Kmin then val_ loss list.append (Ceal (reat))
23
break;
if k = Kmaz- 1 then
val loss list.append (C (rtaa))
25
26
// Select the predictors with smallest validation losses
27
Pr. Ply. fio + arg min ( val _dls_los_____ [oss_[ist(i] // Use the mean value of the selected predictors as the final
predictor
29 f = mean (fi, fin....,ffo) fto)
30 return
suites:
r Brp PE
(7)
where 3 balances their contributions. We set the B to be 1.0 in all the tasks except the Backflip. The Backflip task has some reward items with large scales, so the B is set to 50.0 ensure the effective influence of the preference rewards. We encode the key objectives from the environment rewards into LLM prompts human languages to have LLM understand the key objectives of the task, such as following the speed commands for locomotion. Additionally, LAPP prompts LLMs to provide effective feedback high-level behavior specifications that are difficult or impossible to ground into environment reward functions, such havinng natural trotting gail". The policy is optimized using PPO (Schulman el. al. 2017), while new trajectory pairs are continuously collected. Every M epochs, newly collected trajectories are evaluated by the LLM, added to D,, and used to retrain the preference predictor. LAPP only uses the latest labeled trajectories to retrain the preference predictor. As shown in our ablation studies, this design provides higher performance than including all past trajectories. TAPP's online preference learning allows the policy to

---

Published in Transactions on Machine Learning Research (09/2025)
Algorithm 2: LAPP
1 Require: Robot behavior prompt prompt, preference generator LLM LLM, environment E, policy x, preference predictor ř, preference predictor training dataset Dp, preference data buffer Bp
2 Hyperparaaeters:: Policy optimization epoch number N, preference predictors update interval epoch number M, per epoch trajectories pains collection number K, per epoch rollouts number 5, steps in cach epoch T, preference reward scale 3
3 Initialization: Randomly initialize æ, f.
Dp t {zeros triple; 1/rr Bp - fzoros
M.
4 // Collect initial preference dataset.
5 Rollout æ and sample [Dpl trajectories pairs {(o% ot) o
6 fuile11 LLM (fof,
1pl prompt)
7 Dp - ((of. (lof.o/..:)1 31))
8 Update with Algorithm obs F.reset() // reset E get initial observation for i to N - epochs do
11
//rollout ı in E for T sleps do
12
13
a ~ (obs) // sammle action from policy
ræ ~ Elobs) // get environment reward r' T₁ F(obs) // predict preference reward Ty r= B.rp rp re // calculate weighted sum
14
17 Update with PPO algorithm (Schulman et al. // Sample K pairs from S rollouts
2017
Hof. kb) ~ {a,)s=
19
20
// Push into the preference data buffer
Push {(a), ot, Nome) into Bp
21
22
// Update the preference dataset
if (i+1) % M == 0 then
// Generate preference labels
(yk MoR LLM (flof ob)) prompt)
// Place preference labels into the buffer
By + {lab: ot MaR
1/ Update preference dataset
Dp + {triple, /triple,/ripple, (tiple,
E D2J Mo U B.
// Update preference predictor Update f with Algorithm
B, 0
33 return
progressively align with LLM preferences based on dynamic evaluation criteria according different
learning stages. The full RL procedure is detailed in Algorithm 2 and the environment rewards rg of all the tasks can be found in Appendix A.3

---

Published in Transactions on Machine Learning Research (09/2025)
4.4 Network Architectures
The preference predictor is transformer network (Waswani et al. 2017) based on the GPT archilecture [Radford] 2018) with 6 masked self-attention layers. Inputs are embedded into a 128-dimessional space with sinusoidal positional encodings and processed by 8-headed attention layers. Each block includes 2-layer MLP with GELU activations (Hendrycks Gimpel 2016) and layer normalization Ba 2016) to the output tensor from the last self-attention block. A final decoder outputs a scalar reward.
For Markovian rewards, the imput sequence length is 1 and the casual mask in the self-attention layer is removed. For non- Markovinn rewards, the input sequence length is 8, with zero-padding applied for shorter trajectories.
For quadruped tasks, the policy is an MLP with layers [512, 256, 128] and ELU activations Clevert 2015 outputting 12 target joint angles. PD controller computes the torque commands. For dexterous manipu- lation, we use the same MLP architecture. The output 52-dimensional joint displacement vector for two 26-DoF Shadow Hands (ShadowRobot 2005
5
Experiments
We evaluate LAPP diverse set of quadruped locomotion and dexterous manipulation tasks to assess its ability to:
improve both training efficiency and task performance,
enable high-level behavior control via language instructions, and
solve highly challenging tasks that are very difficult or even infeasible with traditional reward engi- neering.
Additionally, conduct ablation studies to analyze key design choices in LAPP, identifying the factors contributing to its performance gains. Finally, deploy the trained policies on physical quadruped robot across various terrains and tasks to demonstrate LAPP's real-world applicability.
(a)
Plane
Stairs
Obstacles
Slope
Wave
(b)
(c)
Kettle
Hand Over
Swing Cup
Backflip
Figure 4: Simulation Tasks, (a) Quadruped locomotion. The robot learns to walk forward across various terrains following given velocity commands. The terrains include the flat plane, stairs pyramids, discrete obstacles, slope pyramids, and wave-pattern hills. (b) Dexterous manipulation. Fach dexterous hand has 26 degrees of freedoms. Kettle requires the robot to pick up the kettle with one hand, and the cup with another hand, and then pour water into the kettle. Hand Over requires one hand to pass a ball to another hand. Swing Cup requires two hands to hold the cup and rotate it for 180" (c) Quadruped backflip. The robot jumps in the air and rotate backwards for 360 and then land on the ground.
10

---

Published in Transactions on Machine Learning Research (09/2025)
We use GPT-4o mini (Achiam et al. 2023 (gpo-4o-minn-2222--0- variant) as the LLM backbone for LAPP. A full training run for each policy (5000 epochs) costs approximately $2.5 to $3, which is significantly lower than the $40 to $50 required for the larger GPT-4o variant, while still achieving satisfactory results. Since LAPP involves frequent online LLM queries, its ability to succoed with a smaller and cheaper LLM is crucial for broader practical adoption. For the Eureka baseline, we use GPT-4o (xpt-40-2024-08-00 variant) ensure a faithful reproduction of its full capabilities from the original work. Each evolutionary reward search with Eureka costs approximately $3. The videos of all the experiment results can be found our project website.
5.1
Baselines
This baseline uses a well-tuned Proximal Policy Optimization (PPO) implementation (Rudin et al, Schulman et al.) p In each task, PPO is trained with the same environment reward functions LAPP. These reward functions are directly adopted from state-of-the-art policies designed by expert robot learning researchers, representing the current best outcomes from human reward engineering, For a fair comparison, PPO shares all hyperparameters with LAPP. The only difference is that PPO does not incorporate preference rewards, allowing us to isolate and analyze the effect of LAPP's preforonce-guided learning design.
PPO.
2022
Eureka. Evolution-driven Universal Reward Kit for Agents (Eureka) (Ma et al. 2023) is a recent LLM- based approach for automated reward function design. It prompts LLM with reward design guidelines and environment source code to generate executable Python reward functions. Eureka then performs an evolutionary search to refine the reward function over multiple iterations based on observed training per- formance. By following the original implementation, we conduct 5 evolutionary search iterations with 16 reward samples per iteration. Out of the 80 = 16 x reward functions, the best-discovered reward function is then used to train the policy with PPO using the same hyperparameters as the PPO baseline.
5.2 Simulation Experiments
We evaluate LAPP on five quadruped locomotion tasks, three dexterous manipulation tasks, and one quadruped backflip task, as shown in Fig. 4 The Unitree Go2 robot [Robotics) 2023 is used for quadruped experiments, while the Shadow Dexterous Hand (ShadowRobot 2005) Andrychowicz et al.) 2020) is used for dexterous manipulation. The locomotion and manipulation tasks are established RT. benchmarks from prior
Tasks:
works (Ma et al., 2023 2024 Rudin et al. 2022). The quadruped backflip task is extremely challenging control problem, previously studied in multi-objective RL (Kim et al., 2024a). While some RLHF studies have explored backflips, they have primarily used the Hopper model in Gym-Mujoco (Christiano et al. 2017] Kim et al. 2023), which is significantly casicr due to its lower degroes of freodom (3 DoFs) with no real-world physical counterpart.
The quadruped locomotion tasks are derived from massively parallel RL experiments in Rudin et al. (0022) prior work. As shown in Fig. 4 [aa,, we evaluate LAPP five terrain types including flat plane, stairs pyramids, discrete obstacles, slope pyramids, and periodic wave terrain (with periodic wave-pattern hills).
The dexterous manipulation tasks are from the Bidexterous Manipulation (Dexterity) benchmark (Chen et al. 2022) and are also evaluated in Eureka (Ma et al., 2023). As shown in Fig. (b), we evaluate LAPP on the Kettle, Hand Over, and Swing Cup tasks. Kettle requires one hand to hold a kettle and pour water into a cup held the other hand. Hand Over requires to hand over a ball from one hand to another hand. Swing Cup requires the two hands to collaborate to turn cup for 180%
Finally, the quadruped backflip task (Fig. 4 (c)) requires the Unitree Go2 robot to perform 360 backward rotation mid-air and land successfully. Unlike Hopper-based backflip tasks in prior RLHF studies (Christiano) et al. 2017 Kim et al.) 2023 : which focus on low DoF's and lightweight dynamics, our set up utilizes the Go2's official simulator [Robotics] 2023), incorporating realistic physical parameters, which makes the task significantly more challenging for RL.
LAPP improves training efficiency. Fig. shows the learning curves of LAPP, Eureka, and PPO across five locomotion tasks and three dexterous manipulation tasks. Locomotion tasks evaluated with fixed
11

---

Published in Transactions on Machine Learning Research (09/2025)
Plane
Stairs
Obstacles
Slope
1.3
3:5-
150
1.0
1.50
Spaed (mos)
1.5
Speed (mos)
Spxeed (mob))
1.23
0.6
1,00
0.75
0.75
0.50
6.50
6.3
0.00
0.0
-0.55
50C
1000
1540 Epochs
2000
2500
1000
2000
2000
4003
5003
1000
2020 Epochs
3000
4000
100
250
303
400
500
Epochs
Epachs
Wave
Kettle
Hand Over
Swing Cup
1.75
1.0
1.50
1.25
500d (m2g)
0.6
Success Rate
13.6
Success Rate
0.75
0.61
9.4
0.50
LAPP
0.
0.25
Eureka
8.00
0.0
0.0
PPO
200
300 Epochs
400
500
250
532
750 1000: Epochs
1250
1502
6
500
5000 1290 Epochs
150
1500
250
500
753 Epochs
1002
1250
1500
Training with LAPP converges faster in the Plane, Stairs, Obstacles, Hand Over, Swing Cup and Kettle tasks, while also exhibiting more stable performance post-convergence in Swing Cup. In the Slope and Wave tasks, LAPP performs similarly to baselines as these tasks are relatively easier for exploration, converging quickly for all algorithms.
Figure 5: Training Efficiency.
velocity command of 1.0 m/s (Stairs) 1.5 m/s (other terrains). For manipulation tasks, report success rate progression throughout training.
LAPP
Eureka
PPO
LAPP demonstrates faster convergence in flat-plane, stairs, and discrete obstacle locomotion, as well as all manipulation tasks, achieving higher final success rates. These tasks pose non non-trivial exploration chal- lenges, where LAPP accelerates learning by dynam- ically adjusting preference rewards. This flexibil- ity prioritizes different behaviors at different train- ing stages, so that policy exploration can be guided more effectively. In contrast, Eureka struggles with reward balancing, as it relies on a static reward func- tion throughout the training process. While this ensures well-calibrated reward function, often results in inferior performance compared to LAPP.
1.000-
0.975
0.950
Success Rate
0.925
0.900
0.875
0.850
0.825
0.800
Swing Cup Figure 6: Convergence Success Rate. LAPP achieves higher success rates in Kettle, Hand Over, and Swing Cup after the training converges. shows that the preference rewards can continuously refine the robot motions to improve the performance beyond the reach of explicit reward shaping.
Kettle
Hand Over
Interestingly, in the Hand Over task, Eureka con- verges slower than PPO with human-designed re- wards. This occurs because Eureka's evolutionary search optimizes for final performance as the fitness score rather than training efficiency. Therefore, this method can help improve the policy performance, but it may not help with training efficiency.
For relatively easier tasks like Slope and Wave, LAPP exhibits similar performance to baselines. This is because randomized robot initialization on smooth slopes can naturally lead to sliding motions to facilitate early exploration of velocity tracking rewards. As a result, all methods converge within 300 epochs in these tasks, suggesting no significant advantage for LAPP in environments where task exploration is inherently easier.
LAPP achieves higher convergence performance. Although well-designed reward functions from human experts LLMs effectively train quadruped robots to follow velocity commands across various
12

---

Published in Transactions on Machine Learning Research (09/2025)
terrains, they often fail to reach optimal performance in more complex dexterous manipulation tasks such as Kettle, Hand Over, and Swing Cup. As shown in Fig. 6 LAPP significantly improves success rates over PPO desipte sharing the same environment reward functions, increasing from 92% to 99% in Kettle, 91% to 97% in Hand Over, and 96% to 99% in Swing Cup. These gains stem from the continnous motion refinement enabled by LAPP's dynamically updated preference predictor.
Compared to Eureka, LAPP achieves 6% higher success rate in Kettle and 5% improvement in Hand Over. In the Swing Cup task, both LAPP and Eurcka reach near-optimal 99% success rates, but LAPP converges faster and maintains more stable performance over extended training epochs as in Fig. 5
(a)
(b)
(c)
front left
rear left
near 1ghht
front right
Baunding Gait Synchronization Error
Step Trajectory Bounding Galt Preference
Synctonnaaton Error
0.577
LAPP-
0.
08
0,387
Eureka
0.334
02- LAPP
Eureka
PPO
0.00
0.25
0.50
0.75
1.00
125
1.50
1.5
2.00
Time 1s)
Figure 7: Bounding Gait Pattern Control. (a) Feot synchronization error calculated with Eq. 8 LAPP achieves the lowest syuchronization error, indicating its closest adherence lo a bounding gait. (b) Step trajectories of the robots. LAPP synchronizes both front and rear feet, while Eureka aligns only the rear feet, and PPO fails to produce a bounding gait. (c) Motion frames of the robots.
LAPP enables behavior control via instruction. Traditional RL can train robots complete tasks but cannot typically control how they perform them in way that aligns with high-level human preferences. Can LAPP guide robot behaviors using high-level specifications in the behavior instruction prompt? To investigate this question, we design two experiments: 1) enforcing a bounding gait in quadruped forward locomotion, and 2) controlling gait cadence to be either higher or lower in quadruped forward locomotion.
A bounding gait requires the quadruped's front and rear feet make simultancous ground contact in pairs. To quantify how closely robot's gait adheres to this pattern, we adopt synchronization error definition in Eq.
1 N N
sync error =
Σ ([FL, - FR + |RL, - RR.)),
(8)
where FL,, FR,, RLe, and RR, represent front left feet, front right feet, rear left feet, rear right feet contacts at time These are binary values: if the foot in contact with the ground and if it is in the air. A lower synchronization error indicates gait pattern closer to bounding.
As shown in Fig. 7 (a), LAPP trains the robot to achieve bounding gait with the lowest synchronization error. While Eureka also encourages bounding gait through reward shaping, its synchronization error remains higher than LAPP. PPO fails to enforce a bounding gait effectively.
To further illustrate gait patterns, Fig. 7 (b) presents the step trajectories of all methods, where each dot represents a foot contacting the ground. LAPP successfully trains the robot to synchronize its front and rear feet, ensuring that both front feet land simultaneously, followed by both rear feet. In contrast, Eureka achieves partial synchronization, aligning only the rear feet. PPO fails learn bounding gait with unsynchronized foot contacts.
Fig. 7 (c) provides motion frames of the robots trained with LAPP and the baselines. These visualizations complement the trajectory plots, clearly ratinng that LAPP exhibits stronger behavior control through effective implicit reward shaping to follow the high-level gait patterns.
We also evaluate LAPP's ability to control step cadence through high-level instructions. In this experiment, the behavior instruction prompt specifies a preference for either faster or slower stepping frequency. We
13

---

Published in Transactions on Machine Learning Research (09/2025)
(a)
(b)
front of
rear lit
rear night
foott ngg
Cadence Preference (steps/sec)
Trajectory Galt Cadence Preference
Eurcko
1Ta0
50ap Codence
Pr
3.0
2.70
Eureta
(slo)
28
(sloe)
(slow)
Eureaa (S0w))
PRO
Eurice (WR]
LAPP
0.5
1.5
2.0
2.5
3.0
40
Tiire (5)
Figure 8: Gait Cadence Control. (a) Step trajectories under different cadence instructions. LAPP effectively modulates step frequency based on the high-level prompts by following faster and slower gaits. Eureka can adjust the cadence slightly, but it is less effective.(b) Step cadence comparison. LAPP provides precise control over cadence, while Eureka has limited effect. (c) Motion frames illustrating cadence variation. Robots with high cadence take quick and shallow steps, while those with low cadence take larger and higher steps.
Figure 9: Quadruped Robot Backflip. LAPP successfully trains the Unitrec Go2 robot to acomplish the backflip lask. No baseline including PPO, curriculum learning, or Eureka able to solve this lask.
compare the effectiveness of LAPP and Eureka in enforcing these behaviors. Eureka is also prompted to generate reward functions that encourage the desired cadence. During testing, all robots follow a velocity command of 1.5 m/s. As shown in Fig. S (a), with a high-cadence instruction, LAPP achieves 4.54 steps/sec, siguificantly exceeding Eureka (3.06 steps/sec) and PPO (2.78 steps/sec). Similarly, with a low-cadence instruction, LAPP produces 1.67 steps/sec, notably lower than Fureka (2.38 steps/sec) and PPO (2.78 steps/sec). While Eureka can influence cadence through reward shaping, its effect is far weaker than LAPP's.
The step trajectories in Fig. (b) further highlight LAPP's superior cadence control. Compared to all other methods, LAPP produces the densest step trajectory under a high-cadence instruction and the sparsest trajectory under low-cadence instruction. Qualitatively, Fig. (c) presents motion frames illustrating the impact of cadence control. With fast cadence, LAPP trains the robot to take small and rapid steps with minimal foot lift. Conversely, with slow cadence, the robot takes larger strides, keeping its feet in the air for extended periods.
Notably, this experiment. uses the non-Markovian reward model from Eq. We the transformer preference predictor's input sequence length to 8. To control the step cadence, the preference predictor needs to consider the history states of the foet contacts to determine the latent reward value of the current state.
LAPP solves challenging tasks. Quadruped backflips have long been considered a challenging RI. prob- lem due to the need for precise whole- whole-body coordination, complex dynamics, and controlled landing. Some previous RLHF works have trained Hopper model to perform a backflip the Gym-Muuoco simulator (Arumugam et al. 2019), but the Hopper has only three joints, is lightweight, and lacks real-world coun- terpart.
A recent multi-objective RL (MORL) approach solves the quadruped backflip by dividing the motion into five stages, designing separate handcrafted reward function for each stage (Kim et al. 2024a). However, this method requires significant expert knowledge, as practitioners must manually define stage transitions and tnn-tune rewards for each specific robot.
14

---

Published in Transactions on Machine Learning Research (09/2025)
In contrast, LAPP solves the backflip without intensive human labor for preference feedback manual reward tuning. We train a Unitree Go2 robot using a weighted combination of human-designed environment reward and a predicted prefer- ence reward. Our process still requires an initial warm-up to encourage the exploration, but we limit our process with simple reward designs for each step. Specifically, we first pre-train the robot to jump vertically. We then randomly initialize the robot in the air. We also reduce the robot's weight during training but restore its real-world weight for testing.
Backflip Rotate Back Angle (degrees)
359.6
350
Rotate Back Angle
300
200
150
100
52.8
49.0
As shown in Fig. g and Fig. 10, LAPP successfully trains the robot jump, rotate backward 360", and land safely. We also evaluate PPO and Eurcka on the same task, using iden- tical exploration strategies (i.e., pre-training to jump, random air initialization, and reduced training weight). PPO follows the same human-designed backflip reward as LAPP but lacks preference reward, while Eureka generates its own reward function via GPT-4o. Neither PPO nor Eureka succeeds in training the robot for a full backflip. Instead, PPO learns to jump and oscillate the robot's torso up and down but strug- gles flip for over 180 with average maximum rotation of 49.0%. Despite the iterative reward scarch mochanism of Eurcka, it produces similar behavior to PPO with average maximum rotation of 52.8%. These results demonstrate that explicit reward engineering struggles to capture the complex dynamics of a backflip, while LAPP's weeference-driven learning and the adaptive online predictor updates enable successful learning of this highly dynamic capability. We believe that LAPP can shed light on automatically solving many difficult tasks in the near future that are previously unsolvable by conventional RL.
LAPP
Eureka
PPO
Figure 10: Backflip rotate angle. LAPP successfully enables the quadruped robot to complete backflips with full 360 rotation. In contrast, PPO and Eureka fail to gen- crate sufficient rotation, highlighting the advantage of preference-driven learning in solving highly complex and dynamic tasks.
LAPP enables faster transfer learning. We evaluate the transfer learning performance of LAPP For the Plane task with forward speed command of 1.5 m/s, the Go2 robot is first trained on flat ground with static friction of 1.0, dynamic friction of 1.0, and a restitution of 0.0. Then, it trans- ferred a different flat ground with static friction of 0.01, dynamic friction of 0.01, and restitution of 0.9. We fine-tune the policy from the source envi- ronment in the target environment with LAPP and other basclines. As shown in Fig. I1 the robot speed drops to about 0.9 m/s due to the more slip- pery ground surface, and LAPP trains the robot to adapt to this new environment faster than the other (wo baselines. The dashed curves show the learning processes of the robots trained in the target environ- ment from scratch. The results show that transfer learning with pre-trained policy in a different envi- ronment is much faster than training from scratch, and LAPP trains the robot to adapt faster than other baselines.
New Environment Adaptation
1.25
Spocdd (mess
0.75
1APP
Tuneka
0.25
ITO
L4P (hom scratch
Eureks (fom sonech
(fom scratch)
-0.25
203
400
1400
2000 2800
Epochs
Figure 11: Transfer Learning. For the Plane loco- motion task, the robot is transferred new environ- ment with different friction and restitution. The solid curves show that LAPP enables the robot. to adapt to the new environment faster. Compared with the dashed curves, transfer learning is generally faster than training from scratch.
5.3 Ablation Study
We carry out the ablation study analyze the contributions of two key design decisions of LAPP: 1) updating the preference predictor with the latest rollout trajectories instead of trajectories sampled from the
15

---

Published in Transactions on Machine Learning Research (09/2025)
(a)
Plane
Stairs
Obstacles
Slope
2.0
1.75
1.50
1.50
Speed (mi))
Spaeed (mis)
0.8
Speed (mis)
1.25
1.0
6.75
0.75
6:50
0.2
025
-0.25
-05
0.00
1900
2932
1000
2000
3000
6900
5000
1000
3000
4000
280
400
503
Epechs
Epochs
Epochs
Wave
Kettle
Hand Over
Swing Cup
1.0
1.0
Spodd (tr6)
Success Rate
Succesa Mate
6-6-
0.6
3.0
Success
0.2
0.0
100
309
400
590
250
500
r50 Epocht
1900
125
1590
250
z50 Epochs
1900
1250
1500
250
1600
3250 1590
Epochs
Epochs
(b)
(c)
Bounding Synchronization Error
Step Cadence with Preference (steps/sec)
Backflip Rotate Back Angle (degrees)
0.895
0.960
4.54
350
359.6
Synchronibato Erro
300
LAPP
3.0
4.0
LAPP (full process)
3.7
3.0
2.78
2.93
2.78
3,14
Rocate Sa0
200
LAPP (MLP)
130
95-
3/4
1.67
100
1.-
50
39.6
process) (MUP)
process)
(slow) (slow)
(Fast) (fest)
Figure 12: Ablation Study. We evaluate two key design choices of LAPP: (1) updating the preference predictor with the latest rollout trajectories (LAPP vs. LAPP (full process)), and (2) using a transformer- based reward predictor (LAPP vs. LAPP (MLP)). (a) Training Speed: LAPP generally converges faster, except in the Stairs task, where LAPP (full process) achieves similar speed, and in the Slope and Wave tasks, where both tasks simple enough for LAPP and its ablations to have similar performance. In the Kettle task, both ablations fail at different stages, suggesting that suboptinal training method for the preference predictor can disrupt policy learning, (b) Behavior Control: Only LAPP successfully controls gait pattern and cadence, while both ablations fail. (c) Challenging Task: LAPP enables successful backflips, achieving a full 360 rotation. In contrast, LAPP (MLP) and LAPP (full process) reach only 57.2 and 39.6", respectively, and fail complete the lask.
full RL process, and 2) adopting transformer architecture for the reward predictor network instead of a simple MLP.
To investigate the first design choice, we introduce LAPP (full process), which stores all rollout trajectorics pool and samples from the entire RL process rather than only the latest epochs (set 500 epochs in LAPP). The dataset size remains identical between LAPP and LAPP (full process), ensuring a controlled comparison. However, unlike LAPP, this variant may use trajectory data from early training, which potentially introduces biases to suboptimal policy rollouts. Note that we do not use the full trajectory pool for predictor training, as its continuous expansion would slow down online updates.
For the second ablation, LAPP (MLP) replaces the transformer-basee predictor with an MLP, limiting it Markovian preference rewards (Eq. D In contrast, LAPP supports both Markovian and non-Mvakovian rewards (Eq. 2 which are essential for modeling tasks like step cadence control and backflips.
16

---

Published in Transactions on Machine Learning Research (09/2025)
We compare the performance of LAPP and its two ablations in the simulation tasks in Sec. 5.2 (a) shows that replacing the transformer with an ML.P reduces training speed across most. locomotion tasks except for the Slope and Wave tasks, which are relatively simple and lead to similar performance of LAPP and its ablations. Morcover, sampling preference data from the full RL process docs not affect speed in Stairs, Slope, and Wave, but slows down the training in other locomotion tasks. In Kettle, both ablations initially improve performance but then drop to 20%, suggesting that biased preference data or a predictor network that does not capture long-term dependencies in past states can introduce misleading preference rewards, which ultimately destabilize the policy training.
Fig. 12
Fig. 12 (b) evaluates the ability to control gait pattern and step cadence in quadruped locomotion. Only LAPP succeeds, while both ablations fail. This supports the hypothesis that preference rewards must evolve dynamically with training. LAPP (full process) struggles because sampling from the full trajectory pool prevents the predictor from adapting to the current learning stage. LAPP (MIP) fails because MLPs lack the capacity to model non-Markovian rewards, which are essential for cadence control. Specifically, bounding gait requires analyzing foot synchronization from step history, and cadence control depends on tracking past states to predict step timing rewards. However, MLPs cannot effectively capture these temporal dependencies, leading to failure in high-lovel behavior control.
Fig. 12 (c) evaluates the quadruped backflip task. Neither ablation succeeds. LAPP (full process) reaches 39.6 degrees of backward rotation, while LAPP (MLP) improves slightly to 57.2". LAPP completes the full 360 backflip, demonstrating the importance of both preference data sampling and transformer-basee prediction for capturing complex dynamics.
5.4 Real World Experiments
To evaluate the feasibility of deploying LAPP-trained policies real robots, conduct experiments with the Unitree Go2 quadruped. The policy is first trained in the TsaacGym simulator (Makoviychuk et al. 2021) and then directly deployed in the real world. To mitigate the sim-to-real gap, we apply domain randomization, varying ground friction, robot mass, observation noise, and external disturbances during training.
We first test gait pattern control. By specifying preferred gait pattern in the behavior instruction prompt, LAPP successfully trains the robot to walk forward using either trotting or bounding gait as shown in Fig. 13
Trotting Gait
We then evaluate stair climbing. In Sec. 5.2 Fig. has shown that LAPP accelerates stair-climbing training in simulation. Here, deploy the trained policy at epoch 2200 onto real 17 cm-high stair- case. As shown in Fig. 14] LAPP-trained robot suc- cessfully climbs both up and down stairs. In con- trast, policies trained with PPO or Eureka under the same number of epochs often fail to maintain stability, stumble, and fall while navigating stairs. These results demonstrate LAPP's robustness in real-world deployment, effectively translating simulation- trained behaviors to real hardware while reducing manual reward engincering efforts.
Bounding
Figure 13: Gait Pattern Control. We can directly deploy the policy trained by LAPP trotting and bounding gait control, specified via high-level language instructions in the behavior prompts.
6 Conclusion
We introduced LAPP, novel framework that leverages LLM for preference feedback from raw state-action trajectories to guide reinforcement learning. Given only high-level behavior specifications in natural language, LAPP automatically learns a preference reward predictor using I.M--generated feedback and contimuously refines robot motions to align with the specified behavior through the adaptive reward predictor.
17

---

Published in Transactions on Machine Learning Research (09/2025)
Compared to conventional RL approaches that rely explicitly shaped reward functions, our exper- iments in both simulation and real-world deploy- ment demonstrate that LAPP achieves faster train- ing convergence while maintaining superior perfor- mance. Morcover, LAPP enables customizable high- level behavior controls. Finally, LAPP generalizes to complex and non-Markovian preference rewards, surpassing traditional reward engineering methods and LLM-generated reward functions. Notably, LAPP successfully solves the quadruped backflip task for the first time under a basic RL setting, aided only by simple exploration warm-up stops. In con- trast, all other baselines fail under the same conditions. boundaries of RL for robot learning, expanding robot capabilities through foundation model-driven behavior guidance.
Figure 14: Stairs Climbing. LAPP-trained policies enable the Unitree Go2 robot to climb up and down stairs in a real-world deployment. The stairs are ap- proximately 17 cm in height.
These results showcase LAPP's potential to push the
7 Limitations
We note several areas for future improvements of LAPP:
Frequent LLM queries. Despite our batch query process, LAPP still queries LLM very frequently due to its online adaptive training scheme for the preference predictor. Future work could explore more efficient data utilization to maintain or improve performance without biasing the predictor training while reducing LLM query frequency.
Exploration of Other LLMs We chose GPT-44-mini for LAPP it is among the state-of-the-art LLMs with reasonable cost. Our focus of this paper is to study how to leverage LLMs to provide effective feedback to train RL agents in challenging tasks with better sample efficiency and higher performance. Another interesting topic for future exploration is to evaluate the performance of LAPP with other LLMs.
Manual selection of state variables. Our trajectory state representation is curated for LLM queries to include appropriate amount of the information. Including too many variables results long prompts leads to higher costs, and potential confusion in LLM's responses, while too few variables may not provide sufficient context for accurate reasoning. Future research could develop automated methods for state variable selection, optimizing the balance between prompt length, cost, and label accuracy. One possible method is to warm up the training with different variable selections and choose the best set based on the warm-up performance.
A bsennee of visual inputs. LAPP currently does not consider the tasks that requires visual inputs to the policy networks such as reactive motion planning in locomotion tasks, and its feasibility to handle state trajectories with images remains unexplored. With the reasoning capability of VLMs continuously improving, future work can explore the similar idea. for tasks with visual trajectories. This could enhance preference prediction accuracy and enable richer robot capabilities.
Acknowledgments
This work is supported by ARL STRONG program under awards W911NF2320182, W91LNF2220113 and W911NF2420215, by DARPA FoundSci program under award HR00112490372, by DARPA TIAMAT HR001 12490114, and by AFOSR under award # #PA9550-19-1101699
Copyright 2025. Duke University. This paper is hereby licensed publicly under the CC BY-NC-ND 4.0 License. Duke University has filed patent rights for the technology associated with this article. For further license rights, including using the patent rights for commercial purposes, please contact Duke's Office for Translation and Commercialization (olcquestions@dduee..ll and reference OTC File 8724.
18

---

Published in Transactions on Machine Learning Research (09/2025)
References
Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning, pp. 1, 2004.
Abbas Abdolmalcki, Sandy Huang, Leonard Hasenclever, Michael Neunert, Francis Song, Martina Zambelli, Murilo Martins, Nicolas Heess, Raia Hadsell, and Martin Riedmiller. A distributional view on multi- objective policy optimization. In International conference on machine learning, pp. 11-22. PMLR, 2020.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lgge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXio:2303.088: 08774, 2023.
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do as can, not as say: Grounding language in robotic affordances. arXiv preprint wrXiv2220.006991 2022.
Riad Akrour, Mare Schoenauer, and Michele Sebag, Preference-based policy learning, In Machine Learn- ing and Knowledge Discovery in Databases: European Conference, ECML PKDD 2011, Athens, Greece, September 5-9, 2011. Proceedings, Part 11, pp- 12-27. Springer, 2011.
Riad Akrour, Mare Schoenauer, and Michèle Sebag. April: Active preference learning-based reinforcement learning. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2012, Bristol, UK, September 24-28, 2012. Proceedings, Part II 23, pp. 116-131. Springer, 2012.
Saleema Amershi, Maya Cakmak, William Bradley Knox, and Todd Kulesza. Power to the people: The role of humans in interactive machine learning. AT magazine, 35(4):105-12) 2014.
Ap Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowiez, Bob McGrew, Jakub Pa- chocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-21 20, 2020.
Stephane Aroca- Ouellet,,,,,,,, Natalie Mackraz, Barry-John Theobald, and Katherine Metcalf. Predict: Prefer- ence reasoning by evaluating decomposed preferences inferred from candidate trajectories. arXio preprint arXiv:2410.062. 2024.
Saurabh Arora and Prashant Doshi. A survey of inverse reinforcement learning: Challenges, methods and progress. Artificial Intelligence, 297:103500, 2021.
Dilip Arumugam, Jun Ki Lee, Sophie Saskin, and Michael L Littman. Deep reinforcement learning from policy-dependent human feedback. arXiv preprint Xrv: 04257, 2019.
Genesis Authors. Genesis: A universal and generative physics engine for robotics and beyond, December
2024. URL hattpp//gggtho...co////sssssttta
Jimmy Lei Ba. Layer normalization. ar preprint arXic:1607.06455 2016.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:221.008772 2022.
Toygun Basaklar, Suat Gummusso,, and Umit Y Ogras. Pd-morl: Preference-driven multi-objective reinforce- ment learning algorithm. arXiv preprint 2rXii:2208.0791. 2022.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: 1. the method of paired comparisons. Biometrika, 334343): 1952.
Daniel S Brown, Yuchen Cui, and Scott Nickum. Risk-awar awake active inverse reinforcement learning. Conference Robot Learning, pp. 062-772. PMI.R, 2018.
19

---

Published in Transactions on Machine Learning Research (09/2025)
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877 1901, 2020.
Xin-Qiang Cai, Pushi Zhang, Li Zhao, Jiang Bian, Masashi Sugiyama, and Ashley Llorens. Distributional pareto-optimal multi-objective reinforcement learning. Advances in Neural Information Processing Sys- tems, 36, 2024.
Chengliang Chai and Guoliang Li. Human-in-the-loop techniques in machine learning. IEEE Data Eng. Bull. 43(3):37-52, 2020.
Yuanpei Chen, Tianhao Wu, Shengjie Wang, Xidong Feng, Jiechuan Jiang, Zongqing Lu, Stephen McAleer, Hao Dong, Mong-Chun Zhu, and Yaodong Yang. Towards human-level bimanual dexterous manipulation with reinforcement learning. Advances in Neural Information Processing Systems, 35:5150-5163, 2022.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforce- ment learning from human preferences. Advances in neural information processing systems, 30, 2017.
Arnrk-Aréé Clevert. Fast and accurate deep network learning by exponential linear units (elus). arXii preprint arXiv:1511.07289, 2015.
Xuzhe Dang, Stefan Edelkamp, and Nicolas Ribault. Clip-motion: Learning reward functions for robotic actions using consecutive observations. arXiv preprint wrXio:2311.08448 2023.
Christian Daniel, Oliver Kroemer, Malte Viering, Jan Metz, and Jan Peters. Active reward learning with a novel acqusstiinn function. Autonomous Robots, 39:389-405 2015.
Neha Das, Sarah Bechtle, Todor Davchev, Dinesh Jayaraman, Akshara Rai, and Franziska Meier. Model- based inverse reinforcement learning from visual demonstrations. In Conference on Robot Learning, pp. 1940-1942. PMLR, 2021.
Rati Devidze, Parameswaran Kamalaruban, and Adish Singla. Exploration-guicn reward shaping for re- inforcement learning under sparse rewards. Advances in Neural Information Processing Systems, 35: 5829 5842, 2022.
Sam Michael Devlin and Daniel Kudenko. Dynamic potential-based reward shaping. In 11th International Conference on Aulonomous Agents and Mulliagent Systems (AAMAS 2012), pp. 433 440. IFAAMAS, 2012.
Yan Ding, Xiaohan Zhang, Chris Paxton, and Shiqi Zhang. Task and motion planning with large language models for object rearrangement. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2086 2092. IEEE, 2023.
Zibin Dong, Yifu Yuan, Jianye Hao, Fei Ni, Yao Mu, Yan Zheng, Yujing Hu, Tangjie Lv, Changjie Fan, and Zhipeng Hu. Aligndiff: Aligning diverse human preferences via behavior-customisable diffusion model. arXiv preprint arXiv:2310.022. 020042 2023.
Joseph Early, Tom Bewley, Christine Evers, and Sarvapali Ramchurn. Non-markovian reward modelling from trajectory labels via interpretable multiple instance learning. Advances in Newral Information Processing Systems, 55:27652-276633 2022.
Jonas Eschmann. Reward function design in reinforcement learning. Reinforcement Learning Algorithms: Analysis and Applications, pp. 25-33, 2021.
Benjamin Evans, Herman A Engelbrecht, and Hendrik W Jordaan. Reward signal design for autonomous racing. In 2021 20th International Conference on Advonced Robotics (ICAR), pp. 455-460. IEEE, 2021.
Maxence Faldor, Jenny Zhang, Antoine Cully, and Jeff Chunc. Omni-opic: Open-endedness via models of hu- man notions of interestingness with environments programmed in code. arXiv preprint arXiv:2405. 155668 2024.
20

---

Published in Transactions on Machine Learning Research (09/2025)
Johannes Fürnkranz, Eyke Htillermeier, Weiwei Cheng, and Sang-Hyeun Park. Preference-base reinforce- ment. learning: formal framework and a policy iteration algorithm. Machine learning, 89:123-11 2012.
Yolanda Gil, James Honaker, Shikhar Gupta, Yibo Ma, Vito D'Orazio, Daniel Garijo, Shruti Gadewar, Qifan Yang, and Neda Jahanshad. Towards human-guided machine learning. In Proceedings of the 24th international conference on intelligent user interfaces, pp. 614 624, 2019.
Prasoon Goyal, Scott Nickum, and Raymond J Mooney. Using natural language for reward shaping reinforcement learning. Xrviv preprint ar Xiv::99903 020.02000, 2019.
Marck Grzeš. Reward shaping in episodic reinforcement learning. In Proceedings of the 16th Conference on Aulonomous Agents and MulliAgent Systems, pp. 565 573, 2017.
Marck Grzes and Daniel Kudenko. Plan-based reward shaping for reinforcement learning. In 2008 4th International IEEE Conference Intelligent Systems, volume 2, pp. 10 22. IEEE, 2008.
Abhishek Gupta, Aldo Pacchiano, Yuexiang Zhai, Sham Kakade, and Sergey Levine. Unpacking reward shaping: Understanding the benelits of reward engineering on sample complexity. Advances in Newral Information Processing Systems, 35:15281 -12955, 2022.
Huy Ha, Pete Florence, and Shuran Song. Scaling up and distilling down: Tanguage-guided robot skill acquisition. In Conference on Robot Learning, pp. 3766 3777. PMLR, 2023.
Dylan Hadfield-Menell, Stuart Russell, Pieter Abbeel, and Anca Dragan. Cooperative inverse reinforcement learning. Advonces in neural information processing systems, 29, 2016.
Shibo Hao, Yi Gu, Haodi Ma, Joshua Hong, Zhen Wang, Daisy Wang, and Zhiting Hu. Reasoning with language model planning with world model. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 8154-8173, 2023.
Conor F Hayes, Roxana Rádulescu, Eugenio Bargiacchi, Johan Källström, Matthew Macfarlane, Mathieu Reymond, Timothy Verstraeten, Luisa M Zintgraf, Richard Dazeley, Fredrik Heintz, et al. A practical guide multi-objoctive reinforcement learning and planning. Autonomous Agents and Multi- Multi-Agena Agent Systems, 36 (1):26, 2022.
Donald Josoph Hejna III and Dorsa Sadigh. Few-shot preference learning for human-in-the-loop rl. In Conference on Robot Learning, pp. 2014 2025. PMLR, 2023.
Dan Hendryeks and Kevin Gimpel. Gaussian error lincar units (gelus). arXiv preprint arXiv: 08..000..81... 2016.
Yujing Hu, Weixun Wang, Hangtian Jia, Yixiang Wang, Yingfeng Chen, Jianye Hao, Feng Wu, and Changjic Fan. Learning to utilize shaping rewards: A new approach of reward shaping. Advances in Neural Information Processing Systems, 13:15931-15940 2020.
Sandy Huang, Abbas Abdolmaleki, Giulia Vezzani, Philemon Brakel, Daniel Mankowitz, Michael Ne- unert, Steven Bohez, Yuval Tassa, Nicolas Heess, Martin Riedmiller, et al. A constrained multi-objective reinforcement learning framework. In Conference on Robot Learning, pp. 883-898 PMLR, 2022.
Siyuan Huang, Zhengkai Jiang, Hao Dong, Yu Qiao, Peng Gao, and Hongsheng Li. Instruct2act: Mapping multi-modality instructions to robotic actions with large language model. arXiv preprint ar XXx:22305.11177 2023.
Wenlong Huang, Fei Xia, Dhruv Shah, Danny Driess, Andy Zeng, Yao Lu, Pete Florence, Igor Mordatch, Sergey Levine, Karol Hausman, et al. Grounded decoding: Guiding text generation with grounded models for embodied agents. Advances in Neural Information Processing Systems, 36, 2024.
Ahmed Husscin, Eyad Elyan, Mohamed Medhat Gaber, and Chrisina Jayne. Deep reward shaping from demonstrations. In 2017 International Joint Conference on Neurol Networks (IJCNN), pp. 510-517.
TEFE, 2017.
21

---

Published in Transactions on Machine Learning Research (09/2025)
Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. Advances in neural informalion processing systems. 31, 2018.
Zhengran Ji, Lingyu Zhang, Paul Sajda, and Boyuan Chen. Enabling multi-robot collaboration from single- human guidance. arXiv preprint arXiv:2409. 198831,, 2024.
Pingcheng Jian, Chao Yang, Di Guo, Huaping Liu, and Fuchun Sun. Adversarial skill learning for robust manipulation. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 2555 2561. IEEE, 2021.
Changyeon Kim, Jongjin Park, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. Preference transformer: Modeling human preferences using transformers for rl. arXiv preprint arXiv:2303.00957, 2023.
Dohyeong Kim, Hyeokjin Kwon, Junseok Kim, Gunmin Lee, and Songhwai Oh. Stage-wise reward shap- ing for acrobatic robots: A constrained multi-objective reinforcement learning approach. arXiv preprint arXiv:2409.15755, 2024a.
Minn Kim, Yongsik Lee, Schycok Kang, Jihwan Oh, Song Chong, and Sc- Young Yun. Preference alignment with flow matching. Xiviv preprint urXie:2405.1990 19806, 2024b.
W Bradley Knox and Peter Stone. Tamer: Training an agent manually via evaluative reinforcement. In 2008 7th IEEE international conference on development and learning, pp. 292 297. IEEE, 2008.
W Bradley Knox, Stephane Hatgis-Kessell, Serena Booth, Scott Niekum, Peter Stone, and Alessandro Allievi. Models of human preference for learning reward functions. arXiv preprint ar Xix::2200 02231, 2022.
Sateesh Kumar, Jonathan Zamora, Nicklas Hansen, Rishabh Jangir, and Xiaolong Wang. Graph inverse reinforcement learning from diverse videos. In Conference Robot Learning, pp. 55-66. PMLR, 2023.
Panagiotis Kyriakis and Jyotirmoy Deshmukh. Pareto policy adaptation. In International Conference on Learning Representations, volume 2022, 2022.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Ren Lu, Colton Bishop, Ethan Hall, Victor Carbunc, Abhinav Rastogi, et al. Rlaif vs. rlhf: Scaling reinforcement learning from human leedback with leedback. In Forby-firsl International Conference on Machine Learning, 2024.
Kimin Lee, Laura Smith, Anca Dragan, and Pieter Abbeel. B-pref: Benchmarking preference-based rein- forcement learning. arXiv preprint Xxx:22111.03020 2021a.
Kimin Lee, Laura M Smith, and Pieter Abbeel. Pebble: Feedback-efficient interactive reinforcement. learning via relabeling experience and unsupervised pre-training In International Conference on Machine Learning, pp. 6152 6163. PMLR, 2021b.
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 1493-9500. IEEE, 2023.
William Liang, Sam Wang, Hung-Ju Wang, Osbert Bastani, Dinesh Jayaraman, and Yecheng Jason Ma. Environment curriculum generation via large language models. In Conference Robot Learning (CoRL), 2024.
Kevin Lin, Christopher Agia, Toki Migimatsu, Marco Pavone, and Jeannette Bohg. Text2motion: From natural language instructions to feasible plans. Autonomous Robots, 47(8):1345-1365, 2023.
Bo Liu, Yuqian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone. Llm+ p: Empowering large language models with optimal planning proficiency. orXiv preprint (bb::2344.11477 2023.
22

---

Published in Transactions on Machine Learning Research (09/2025)
Fei Liu et al. Learning to summarize from human feedback. In Proceedings of the 58th Annual Meeting of the Associalion for Compulational Linguislics, 2020.
Runze Liu, Yali Du, Fengshuo Bai, Jiafei Lyu, and Xiu Li. Pearl: zero-shot cross-task preference alignment and robust reward learning for robotic manipulation. In International Conference on Machine Learning. 2024.
Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eureka: Human-level reward design via coding large language models. arXiv preprint arXiv: Arriv-2310.1 12931, 2023.
Yecheng Jason Ma, William Liang, Hungju Wang, Sam Wang, Yuke Zhu, Linxi Fan, Osbert Bastani, and Dinesh Jayaraman. Dreureka: Language model guided sim-to-real transfer. In Robotics: Science and Systems (RSS), 2024.
James MacGlashan, Michael L Littman, David Roberts, Robert Loftin, Bei Peng, and Matthew E Taylor. Convergent actor critic by humans. In International Conference on Intelligent Robots and Systems, 2016.
Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al. Isaac gym: High performance gpu-based physics simulation for robot learning, orXiv preprint ar Xiv:21008. 104 70, 2021.
Gabriel В Margolis, Ge Yang, Kartik Paigwar, Tao Chen, and Pulkit Agrawal. Rapid locomotion via reinforcement learning. The International Journal of Robolics Research, 43(4):572 587, 2024.
Ofir Marom and Benjamin Rosman. Belief reward shaping in reinforcement learning. In Proceedings of the AAAT conference on artificial intelligence, volume 32, 2018.
Farzan Memarian, Wonjoon Goo, Rudolf Lioutikov, Scott Niekum, and Ufuk Topeu. Self-supervised online reward shaping in sparsc-reward environments. In 2021 IEEE/RS, /RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2369 2375. IEEE, 2021.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXie preprint arXiv:2112 12.09332 2021.
Andrew Y Ng. Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Ieml, volume 99, pp. 278 287, 1999.
Andrew Y Ng, Stuart Russell, et al. Algorithms for inverse reinforcement lcarning. In Ieml, volume 1, pp. 2, 2000.
Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Charles Xu, Jianlan Luo, Tobias Kreiman, You Liang Tan, Lawrence Yunliang Chen, Pannag Sanketi, Quan Vuong, Ted Xiao, Dorsa Sadigh, Chelsea Finn, and Sergey Levine, Octo: An open-source generalist robot policy. In Proceedings of Robotics: Science and Systems, Delft, Netherlands, 2024.
Long Ouyang, Jeffrey WU, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
Jongjin Park, Younggyo Seo, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Loe. Surf: Semi-supervised reward learning with data augmentation for feedback-efficient preference-based reinforcement learning. arXiv preprint ar 2rXXt::203.1005/ 2022.
Dean A Pomerleau. Alvinn: An autonomous land vehicle in a neural network. Advances in neural information processing systems, 1, 1988.
Alec Radford. Improving language understanding by generative pre-training. 2018.
23

---

Published in Transactions on Machine Learning Research (09/2025)
Ellis Ratner, Dylan Hadfield- and Anca D Dragan. Simplifying reward design through divide-and- 2018.
conquer. Xixiv preprint
Unitree Roboties. Unitree rl gym. http:///ghhhbbooo///mrrrrrorroer rl_gym
2023. Ac-
cessed: 2025-01-22.
Nikita Rudin, David Hoeller, Philipp Reist, and Marco Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning. In Conference on Robot Learning, pp. 10-000 PMLR, 2022.
Kanghyun Ryu, Qiayuan Liao, Zhongyu Li, Koushil Sreenath, and Negar Mehr. Curricullm: Automatic task curricula design for learning complex robot skills using large language models. arXiv preprint arXiv:2409. 18382, 2024.
Akanksha Saran, Ruohan Zhang, E Short, and Scott Nickum. Efficiently guiding imitation learning algo- rithms with human gaze. In International Conference on Aulomomous Agents and Mulliagent Systems, 2021.
Stefan Schaal. Learning from demonstration. Advances in neural information processing systems, 9, 1996.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimiza- tion algorithms. arXiv preprint Xrxiv:1707. (al::707.06444 06747, 2017.
ShadowRobot. Shadowrobot dexterous hand, 2005. URL
Bexterous-hande Accessed: 2025-01-31.
Isaac Sheidlower, Elaine Schaertl Short, and Allison Moore. Environment guided interactive reinforcement learning: Learning from binary feedback in high-dimensional robot task environments. In Proceedings of the 21st International Conference Autonomous Agents and Multiagent Systems, pp. 1726-1728, 2022.
Tom Silver, Soham Dan, Kavitha Srinivas, Joshua B Tenenbaum, Leslie Kaelbling, and Michael Katz. Generalized planning in pddl domains with pretrained large language models. In Proceedings of the AAAT Conference on Artificial Intelligence, volume 38, pp. 20256 20264, 2024.
Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. Progprompt: Generating situated robot task plans using large language models. In 2023 IEEE International Conference on Robolies and Aulomation (ICRA), pp- 11523 11530. IEEE, 2023.
Jonathan Sorg, Richard L Lewis, and Satinder Singh. Reward design via online gradient ascent. Advances in Neural Information Processing Systems, 23, 2010.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Andrew Szot, Max Schwarzer, Harsh Agrawal, Bogdan Mazoure, Rin Metcalf, Walter Talbott, Natalie Mackraz, R Devon Hjelm, and Alexander T Toshev. Large language models as generalizable policies for embodied tasks. In The Twelfth International Conference on Learning Represenuations, 2023.
Yujin Tang, Wenhao Yu, Jie Tan, Hciga Zen, Aleksandra Fanst, and Tatsuya Harada. Saytap: Language to quadrupedal locomotion. In Conference on Robot Learning, pp. 3556-35700 PMLR, 2023.
Zuoxin Tang, Donghyun Kim, and Sehoon Ha. Learuing agile motor skills quadrupedal robots using curriculum learning. In International Conference on Robot Intelligence Technology and Applications, vol- ume 3, 2021.
Kristof Van Moffaert, Madalina M Drugan, and Ann Now6. Scalarized multi-objective reinforcement learning: Novel design techniques. In 2013 IEEE symposium on adaptive dynamic programmingg and reinforcement learning (ADPRL), pp. 191-199. IEEE, 2013.
Sreyas Venkataraman, Yufei Wang, Ziyu Wang, Zackory Erickson, and David Held. Real-world offline reinforcement learning from vision language model feedback. arXio preprint arx:2:241 1.05273, 2024.
24

---

Published in Transactions on Machine Learning Research (09/2025)
Mudit Verma and Katherine Metcalf. Symbol guided hindsight priors for reward learning from human preferences. arXiv preprint Xin:2200. 0.09151, 2022.
Guanzhi Wang, Yugi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv: Arziv-2305. 16291, 2023a.
Lirui Wang, Yiyang Ling, Zhecheng Yuan, Mohit Shridhar, Chen Bao, Yuche Qin, Bailin Wang, Huazhe Xu, and Xiaolong Wang. Gensim: Generating robotic simulation tasks via large language models. In Arriv, 2023b.
Lirui Wang, Yiyang Ling, Zhecheng Yuan, Mohit Shridhar, Chen Bao, Yuzhe Qin, Bailin Wang, Huazhe Xu, and Xiaolong Wang. Gensim: Generating robotic simulation tasks via large language models. arXiv preprint 2rXi:223100114 0116661, 2023c.
Yufei Wang, Zhou Xian, Feng Chen, Tsun-Hsuan Wang, Yian Wang, Katerina Fragkiadaki, Zackory Erickson, David Held, and Chuang Gan. Robogen: Towards unleashing infinite data for automated robot lcarning via generalive simulation. arXiv preprinl arXiv:2311.00 1..03111114455 0145, 2023d.
Yufei Wang, Zhanyi Sun, Jesse Zhang, Zhou Xian, Erdem Biyik, David Held, and Zackory Erickson. Rl-vlm-f: Reinforcement learning from vision language foundation model feedback. arxiv preprint (02 2024.
Zihao Wang, Shaofei Cai, Guanzhou Chen, Auji Liu, Xiaojian Ma, Yitao Liang, and Team CraftJarvis. Describe, explain, plan and select: interactive planning with large language models enables open-world multi-task agents. In Proceedings of the 37th International Conference Newral Information Processing Systems, pp. 34153-34189 2023c.
Garrett Warnell, Nicholas Waytowich, Vernon Lawhern, and Peter Stone. Deep tamer: Interactive agent
shaping in high-dimensional state spaces. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
A Waswani, N Shazeer, N Parmar, J Uszkoreit, I. Jones, A Gomez, I. Kaiser, and I Polosukhin. Attention is all you need. In NIPS, 2017.
Aaron Wilson, Alan Fern, and Prasad Tadepalli. A bayesian approach for policy learning from trajectory preference queries. Advances in nearal information processing systems, 25, 2012.
Christian Wirth, Johannes Fürukranz, and Gerhard Neumann. Model-free preference-based reinforcement
learning. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
Christian Wirth, Riad Akrour, Gerhard Neumann, and Johannes Fürnkranz. A survey of preferenco-based reinforcement learning methods. Journal of Machine Learning Research, 18(136):1 46, 2017.
Jeff Wu, Long Ouyang, Daniel M Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. Recursively summarizing books with human feedback. orXiv preprint arXiv:2109. 10862, 2021.
Xingjiao Wu, Luwei Xiao, Yixuan Sun, Junhang Zhang, Tianlong Ma, and Liang He. A survey of human- in-the-loop for machine learning. Future Generation Computer Systems, 135:364 381, 2022.
Boxi Xia, Bokuan Li, Jacob Lee, Michael Scutari, and Boyuan Chen. The duke humanoid: Design and control for energy efficient bipedal locomotion using passive dynamics. arXiv preprint orXin:2,09.19999, 19795, 2024.
Tianbao Xie, Siheng Zhao, Chen Henry Wu, Yitao Liu, Qian Luo, Victor Zhong, Yanchao Yang, and Tao Yu. Text2reward: Reward shaping with language models for reinforcement learning. In The Twelfth International Conference on Learning Representations, 2024. htttps:/oppaerriowwne///oooo EUM39YTR*HI
25

---

Published in Transactions on Machine Learning Research (09/2025)
Yaqi Xie. Translating natural language to planning goals with large-lngguag models. The International
Journal of Robolics Research, 2019:1, 2020.
Jie Xu, Yunsheng Tian, Pingchuan Ma, Daniela Rus, Shinjiro Sueda, and Wojeiech Matusik. Prediction- guided multi-objective reinforcement learning for continuous robot control. In International conference machine learning, pp. 10607-10616. PMLR, 2020.
Runzhe Yang, Xingyuan Sun, and Karthik Narasimhan. A generalized algorithm for multi-objective rein- forcement learning and policy adaptation. Advances in neural information processing systems, 32, 2019.
Chao Yu, Hong Lu, Jiaxuan Gao, Qixin Tan, Xinting Yang, Yu Wang, Yi Wu, and Eugene Vinitsky. Few-shot in-context preference learning using large language models. arxiv preprint arXiv:24 1000:2410.7720000 17233, 2024.
Wenhao Yu, Nimrod Gileadi, Chuyuan Fu, Sean Kirmani, Kuang- Huci Lee, Montserrat Gonzalez Arenas, Hao-Tien Lewis Chiang, Tom Erez, Leonard Hasenclever, Jan Humplik, et al. Language to rewards for robotic skill synthesis. In Conference on Robot Learning, pp. 374 404. PMLR, 2023.
Xinyi Yuan, Zhiwei Shang, Zifan Wang, Chenkai Wang, Zhao Shan, Zhenchao Qi, Meixin Zhu, Chenjia Bai, and Xuclong Li. Preference aligned diffusion planner for quadrupodal locomotion control. arXi preprint arXiv: 10. 13586, 2024.
Kevin Zakka, Andy Zeng, Pete Florence, Jonathan Tompson, Jeannette Bohg,, and Debidatta Dwibedi, Xirl: Cross emsoddment inverse reinforcement learning. In Conference on Robot Learning, pp. 537-546. PMLR, 2022.
Jesse Zhang, Jiahui Zhang, Karl Pertsch, Ziyi Liu, Xiang Ren, Minsuk Chang, Shao-Hua Sun, and Joseph J Lim. Bootstrap your own skills: Learning to solve new tasks with large language model guidance. Conference on Robot Learning, pp. 302-325. PMLR, 2023.
Lingyu Zhang, Zhengran Ji, and Boyuan Chen. CREW: Facilitating human-ai teaming research. Transactions on Machine Learning Research, 2024a.
Lingyu Zhang, Zhengran Ji, Nicholas R Waytowich, and Boyuan Chen. GUIDE: agents. In The Thirty-eighth Annual Conference on Newral Information Processing Systems, 2024b.
Real-time human-shaped
Ruohan Zhang, Faraz Torabi, Lin Guan, Dana Ballard, and Peter Stone. Leveraging human guidance for deep reinforcement learning tasks. arXiv preprint arXiv: arXix:1909.00 190900909,, 2019.
Li Zhou and Kevin Small. Inverse reinforcement learning with natural language goals. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 11116 11124, 2021.
Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, et al. Rt-2: Vision-languagc-action models transfer web knowledge to robotic control. In Conference on Robol Learning, pp. 2165 2183. PMLR, 2023.
Haosheng Zou, Tongzheng Ren, Dong Yan, Hang Su, and Jun Zhu. Reward shaping via meta-learning. arXiv preprint Xiv:: 1 1901. 093300, 2019.
26

---

Published in Transactions on Machine Learning Research (09/2025)
A
Appendix
A.1 Full Prompts
A.2
Full Prompts
In this section,we provide all the prompts for training with LAPP.
Prompt for Flat Plane
You are robotics engineer trying to compare pairs of quadruped robot locomotion trajectories and decide which one is better in each pair.
Your feedback of the comparisons will be used as reward signal (for reinforcement learning) to train a quadruped robot (Unitree Go2) to walk forward at some speed given by the commands, and the velocity range of the speed command is [0.0, 2.2] m/s.
The training method similar that in the paper "Deep Reinforcement Learning from Human Preferences", where humans provide preference of trajectories in different pairs of comparisous, but you will take the role of the humans to provide feedback which trajectory is better in pair of trajectories.
Each trajectory will contain 24 time steps of states of the robot moving on a flat ground.
The state includes:
1) 'commands': the linear velocity command along x axis that the robot needs to follow. its length is 24, standing for the 24 steps of a trajectory. value range each step is [0.0, 2.2] m/s. Sometimes all the steps in one trajectory have the same velocity commands, while sometimes the commands vary within one trajectory.
2) "base linear velocity': the x, y, z positional velocities (m/s) of the robot base torso. The data shape is (24, 3), standing for 24 steps, and x, y, dimensional velocities.
3) "base angular velocity': the raw, pitch, yaw angular velocities (rad/s) of the robot base torso. The data shape is (24, 3), standing for 24 steps, and raw, pitch, yaw angular velocities around the x, y, axes.
4) 'base height": the z position (height) of the robot basc torso. The data shape is (24,) standing for the 24 sleps of trajectory.
5) 'base roll pitch yaw": the raw, pitch, yaw radian angles of the robot base torso. The data shape is (24, 3), standing for 24 steps, and raw, pitch, yaw 3 rotation angles around the x, axes. 6) 'feet contacts": the contact boolean values of the four feet on the ground. means touching the ground while 0 means in the air. The data shape is (24, 4), standing for steps, and the 4 feet in the order of [front left, front right, rear left, rear right).
To docide which trajectory better in a pair, here are some criteria:
1) The robot. should follow the forward velocity command close as possible. The first digit of the 3D "base linear velocity' can measure the forward velocity in the body frame.
2) The robot should have 0 velocities in the and directions of the body frame. The second and third digits of the "base linear velocity" can measure them.
3) The robot should keep its body torso near the height of 0.34 meter. The "base height" value can measure the robot torso height.
4) The robot should not. have angular velocities all the 3 roll, pitch, yaw directions when walking forward. The values of the "base angular velocity' should be close to
5) The robot should not have roll pitch angles when walking forward. Since the linear and angular velocities of the robot are randomly initialized al each episode, the robot might has some yaw angle from start, but this yaw angle should not change when the robot is waling forward.
6) The robot is encouraged to take longer steps instead of small steps. In addition, periodic gait pattern better than random steps on the ground. The "fect contacts" can be used to analyze the gait pattern of the robot.
The user will provide pairs of trajectories (each pair has index 0 and 1) batch and you should provide 1 preference value for cach pair (5 values in total).
27

---

Published in Transactions on Machine Learning Research (09/2025)
1) If the trajectory better, the preference value should be O.
2) If the trajectory is better, the preference value should be 1.
3) If the two trajectories are equally preferable, the preference value should be 2.
4) If the two trajectories
incomparable, the preference value should be 3.
Please give response with only one list of preference values, c.g., [0, 0, 1, 2, 3).
Do not provide any other text such as your comments or thoughts. The preference value number can only be 0, 1, 2, or 3.
Please provide preference values and as many as possible, which clearly indication which one is better in a pair.
Please be careful about providing equally preferable value 2. If each trajectory has its pros and cons, instead of saying they equally preferable, you decide which criteria are more important at this stage of training, and then decide which trajectory is more preferable. Please be very careful about providing incomparable value 3! Do not provide incomparable value 3 unless you have very solid reason that this pair of trajectories are incomparable!
Prompt for Stairs
You robotics engineer trying to compare pairs of quadruped robot locomotion trajectories and decide which one is better in each pair.
Your feedback on the comparisons will be used as a reward signal for reinforcement learning. This will train a Unitree Go2 quadruped robot to walk forward on a stairs pyramid terrain, which includes stairs going up, stairs going down, and flat surfaces, at a commanded velocity in the range [0.0, 2.2] m/s.
The training method is similar to that in the paper "Deep Reinforcement Learning from Human Preferences", where humans provide preference of trajectories in different pairs of comparisons, but now you will take the role of the humans to provide feedback on which one trajectory is better in pair of trajectories.
Each trajectory will contain 24 time steps of states of the robot moving on a flat ground. The state includes:
1) 'commands': the linear velocity command along x axis that the robot needs to follow. its length is 24, standing for the 24 steps of a trajectory. its value range each step is [0.0, 2.2] m/s. Sometimes all the steps in trajectory have the same velocity commands, while sometimes the commands vary within one trajectory.
2) "base linear velocity': the x, z positional velocities (m/s) of the robot base torso. The data shape is (24, 3), standing for 24 steps, and x, y, 2 3 dimensional velocitics.
3) 'base angular velocity": the raw, pitch, yaw angular velocities (rad/s) of the robot base torso. The data shape (24, 3), standing for 24 steps, and raw, pitch, yaw angular velocities around the x, y, axes.
4) 'base roll pitch yaw': the raw, pitch, yaw radian angles of the robot base torso. The data shape is (24, 3), standing for 24 steps, and raw, pitch, yaw 3 rotation angles around the x, axes.
5) 'base height": the z position (height) of the robot base torso. The data shape is (24, ), standing for the 24 steps of a trajectory.
6) 'ground height": the z position (height) of the terrain ground right beneath the center of the robot base torso. The data shape (24, ), standing for the steps of a trajectory.
7) 'feet heights': the four height values of the four feet. The data shape is (24, 4), standing for 24 steps, and the feet in the order of [front left, front right, rear left, rear right].
8) 'feet contacts'; the contact boolean values of the four feet the ground. means touching the ground while 0 mcans in the air. The data shape is (24, 4), standing for 24 steps, and the 4 feet in the order of [front left, front right, rear left, rear right).
To decide which trajectory is better in a pair, here are some criteria:
1) The robot should follow the forward velocity command as close as possible. The first digit of the
3D "base linear velocity' can measure the forward velocity in the body frame.
28

---

Published in Transactions on Machine Learning Research (09/2025)
2) The robot should have 0 velocities in the y direction of the body frame. The second digit of the 'base linear velocity" can measure them.
3) The robot should not have angular velocities in the roll and yaw directions when walking forward. The first and third values of the "base angular velocity" should be close to O.
4) The robot should keep its base height about 0.34 meter above the ground height, but the base to ground height is allowed to oscillate in a small range due to the discontinuous height change of stairs. Compare the 'base height' and "ground height" values to measure this.
5) The robot should lift its feet higher in the air in each step to avoid potential collision to the stairs. Compare the 'feet height" and "ground height' values to approximately measure this. When the robot is climbing upstairs downstairs,
leet heights
be lower than the ground height beneath
the robot center due to the body pitch angle.
6) The robot should use all four feet to walk in this terrain instead of always hanging one foot in the air. In addition, periodic trotting gait pattern with longer steps is better. The 'feet contacts' can be used to analyze the gait pattern of the robot.
7) The robot should have 0 roll angle when walking forward. Pitch angle allowed for climbing upstairs and downstairs. Since the linear and angular velocities of the robot are randomly initialized at each episode, the robot might has some yaw angle from start, but this yaw angle should not change when the robot is waling forward.
The user will provide 5 pairs of trajectories (each pair has index 0 and 1) in batch and you should provide 1 preference value for each pair (5 values in total).
1) If the trajectory is better, the preference value should be 0.
2) If the trajectory is better, the preference value should be 1.
3) If the two trajectories are equally preferable, the preference value should be
4) If the two trajectories are incomparable, the preference value should be PLease note that the robot should should use all four feet to walk. It is highly preferable that all the four feet. have contacts to the ground when going downstairs. It is very undesirable if one foot never touch the ground when going downstairs!
Please give response with only one list of 5 preference values, e.g., [0, 0, 1, 2, 3]. Do not provide any other text such as your comments or thoughts. The preference value number can only be 0, 1, 2, or 3.
Please provide preference values and as many as possible, which clearly indication which one is better in a pair.
Please be careful about providing equally preferable value 2. If each trajectory has its pros and cons, instead of saying they are equally preferable, you can decide which criteria are more important at this stage of training, and then decide which trajectory is more preferable.
Please be very careful about providing incomparable value 3! Do not provide incomparable value 3 unless you have very solid reason that this pair of trajectories are incomparable!
Prompt for Obstacles
You are robotics engineer trying to compare pairs of quadruped robot locomotion trajectorios. Your lask is to provide feedback on which trajectory is better in given pair of trajectories.
Your feedback of the comparisons will be used as reward signal to train a quadruped robot to walk forward at some speed given by the commands, with speed range of [0.0, 2.2] m/s.
Each trajectory will contain 24 timesteps of states of the robot moving on discrete obstacles terrain.
To be specific, the lerrain features unevenly distributed rectangular platforms with varying heights and smooth edges, creating stepped, block-like like appearance. The state includos:
1) "commands": the linear velocity command along x axis that the robot needs to follow. its length is 24, standing for the 24 steps of a trajectory. its value range at each step is |0.0, 2.2] m/s. Sometimes the steps in one trajectory have the same velocity commands, while sometimes the commands vary within one trajectory.
29

---

Published in Transactions on Machine Learning Research (09/2025)
2) "base linear velocity": the x, y; z positional velocities (m/s) of the robot base torso. The data shape is (24, 3), standing for 24 steps, and x, y; dimensional velocities.
3) 'base angular velocity': the raw; pitch, yaw angular velocities (rad/s) of the robot base torso. The data shape is (24, 3), standing for 24 steps, and raw, pitch, yaw 3 angular velocities around the x, Yr axes.
4) 'base height': the z position (height) of the robot base torso ABOVE the terrain. The data shape is (24, ), standing for the 24 steps of a trajectory.
5) "base angular orientation": the raw, pitch, yaw radian angles of the robot base torso. The data shape is (24, 3), standing for 24 steps, and raw, pitch, yaw rotation angles around the x, y, z axes. 6) 'feet contacts': the contact boolean values of the four feet the ground. means touching the ground while 0 means in the air. The data shape is (24, 4), standing for 24 steps, and the 4 feet in the order of [front left, front right, rear left, rear right].
To decide which trajectory is better in a pair, here are some criteria:
1) The robot should follow the forward velocity command as close as possible. The first digit of the
3D "base linear velocity' can measure the forward velocity the body frame.
2) The robot should have no velocity in y axis of the base torso. The second digit of "base linear velocity' can measure.
3) The robot should keep its body torso near the height of 0.34 meter. 'base height' mcasure.
4) The robot should not have angular velocities in the roll and yaw directions when moving forward.
The first and third values of the "base angular velocity" should be close to O. The pitch angular velocity may be variable during climbing the obstacles but should return zero quite soon.
5) The robot should not have roll angle when moving forward. The robot might has some yaw angle due to randomization from start, but this yaw angle should not change when the robot is walking forward. Small pitch orientation acceptable so as to adapt the terrain.
6) The robot is encouraged to take **rotting** gait to move forward. The trotting gait features diagonal contact paltern where opposing diagonal legs (e.g., front left and rear right,) touch the ground simultancously, alternating in rhythm. The "feet contacts' can be used to analyze the gait pattern of the robot.
7) The robot is encouraged to take farther steps. 'feet contacts' can help measure.
The user will provide pairs of trajectories (each pair has index and 1) a batch and you should provide 1 preference value for each pair (5 values in total).
1) If the trajectory 0 is better, the preference value should be 0.
2) If the trajectory is better, the preference value should be 1.
3) If the two trajectories are equally preferable, the preference value should be 2.
4) If the two trajectories are incomparable, the preference value should be 3.
Examples for preference:
1) If both can move forward, the one with greater velocity x axis is better.
2) If both have close-to-commantt velocity x axis, the one with lower velocity y axis better.
3) If both cannot. move forward, the one that maintain body height close to 0.34 meter is better.
4) If both robots can walk forward, the one whose gait is more similar to trotting gait is better.
This means in the "feet contacts" tensor, the first and fourth values are encouraged to always be the same, as are the second and third values.
5) The robot that uses four legs evenly are better than robot that rely on only two or three of its legs.
This means period of non-zero values in all positions of 'feet contacts' tensor, and the periods should be similar.
6) The robot that takes longer steps better. This mcans longer period is preferable.
Please give response with only one list of 5 preference values, e.g., [0, 0, 1, 2, 3). Do not provide any other text such as your comments or thoughts. The preference value number can only be 0, 1, 2, or
3.
Please provide preference values and as many as possible, which clearly indication which one is
better a pair.
30

---

Published in Transactions on Machine Learning Research (09/2025)
Please be careful about providing equally preferable value 2. If each trajectory has its pros and cons, instead of saying they equally preferable, you decide which criteria are more important this stage of training, and then decide which trajectory is more preferable.
For example, if the two trajectories both show that the robots are moving forward at some given command speed, the robot whose gait pattern is more similar to trotting pattern more preferable. Please be very careful about providing incomparable value 3! Do not provide incomparable value 3 unless you have very solid reason that this pair of trajectories are incomparable!
Prompt for Obstacles
You are robotics engineer trying to compare pairs of quadruped robot locomotion trajectorios. Your lask is to provide leedback on which trajectory is better in given pair of trajectories.
Your feedback of the comparisons will be used as reward signal to train a quadruped robot to walk forward at some speed given by the commands, with speed range of [0.0, 2.2] m/s.
Each trajectory will contain 24 timesteps of states of the robot moving on pyramid slope terrain. To be specific, the lerrain features evenly spaced, volcano-like formations with smooth slope and platform top, and the height of each "volcano" is consistent. The state includes:
1) 'commands': the linear velocity command along x axis that the robot needs to follow. its length is 24, standing for the 24 steps of a trajectory. its value range at each step is [0.0, 2.2] m/s. Sometimes the steps in one trajectory have the same velocity commands, while sometimes the commands vary within one trajectory.
2) "base linear velocity': the x, y, z positional veloeities (m/s) of the robot base torso. The data shape is (24, 3), standing for 24 steps, and x, y, dimensional velocities.
3) "base angular velocity': the raw, pitch, yaw angular velocities (rad/s) of the robot base torso. The data shape is (24, 3), standing for 24 steps, and raw, pitch, yaw 3 angular velocities around the x, 2 axes.
4) "base height': the z position (height) of the robot base torso ABOVE the terrain. The data shape is (24, ), standing for the 24 steps of a trajectory.
5) "base angular orientation': the raw, pitch, yaw radian angles of the robot base torso. The data shape is (24, 3), standing for 24 steps, and raw, pitch, yaw 3 rotation angles around the x, y, 6) "feet contacts': the contact boolean values of the four feet the ground. means touching the ground while 0 means in the air. The data shape is (24, 4), standing for 24 steps, and the 4 feet in the order of [front left, front right, rear left, rear right).
To decide which trajectory better in a pair, here are some criteria:
1) The robot should follow the forward velocity command as close as possible. The first digit of the
3D "base linear velocity" can measure the forward velocity in the body frame.
2) The robot should have velocity in axis of the base torso. The second digit of "base linear velocity" can measure.
3) The robot should keep its body torso near the height of 0.34 meter. "base height" can mcasure.
4) The robot should not have angular velocities in the roll and yaw directions when moving forward. The first and third values of the "base angular velocity" should be close to O. The pitch angular velocity may be variable during adjustments from descending ascending (or vice versa), but should be zero when on platform.
5) The robot should not have roll angle when moving forward. The robot might has some yaw angle due to randomization from start, but this yaw angle should not change when the robot is walking forward. Small pitch orientation is acceptable so as to adapt to the terrain.
6) The robot is encouraged to take a **rooling*** gait to move forward. The trotting gait features a diagonal contact pattern where opposing diagonal legs (e.g., front left. and rear right) touch the ground simultaneously, alternating in rhythm. The "feet contacts' can be used to analyze the gait pattern of the robot.
31

---

Published in Transactions on Machine Learning Research (09/2025)
7) The robot encouraged take farther steps. "feet contacts' can help measure. The user will provide pairs of trajectories (each pair has index 0 and 1) in batch and you should provide 1 preference value for each pair (5 values in total).
1) If the trajectory 0 is better, the preference value should be 0.
2) If the trajectory is better, the preference value should be 1.
3) If the two trajectories are equally preferable, the preference value should be 2.
4) If the two trajectories are incomparable, the preference value should be Examples for preference:
1) If both can move forward, the one with greater velocity x axis is better.
2) If both have close-to-command velocity in axis, the one with lower velocity in y axis is better.
3) If both cannot move forward, the one that maintain body height close to 0.34 meter is better.
walk forward, the one whose gait is more similar to trotting gait is better. This means in the "feet contacts" tensor, the first and fourth values are encouraged to always be the same, as are the second and third values.
4) If both robots
5) The robot that uses four legs evenly are better than robot that rely on only two or three of its legs.
This means period of non-zero values in all positions of "feet contacts' tensor, and the periods should be similar.
6) The robot that takes longer steps are better. This means longer period is preferable.
Please give response with only one list of preference values, e.g., [0, 0, 1, 2, 3J. Do not provide any other text such as your comments or thoughts. The preference value number can only be 0, 1, 2, or 3.
Please provide preference values and as many as possible, which clearly indication which one is better in a pair.
Please be careful about providing equally preferable value 2. If each trajectory has its pros and cons, instead of saying they are equally preferable, you can decide which criteria are more important al this stage of training, and then decide which trajectory is more preferable.
For example, if the two trajectories both show that the robots are moving forward at some given command speed, the robot whose gait pattern is more similar to trotting pattern is more preferable. Please be very careful about providing incomparable value Do not provide incomparable value 3 unless you have very solid reason that this pair of trajectories are incomparable!
Prompt for Slope
You robotics engineer trying to compare pairs of quadruped robot locomotion trajectories. Your task is to provide feedback which trajectory is better in given pair of trajectories.
Your feedback of the comparisons will be used as reward signal to train quadruped robot to walk forward at some speed given by the commands, with speed range of [0.0, 2.2] m/s.
Each trajectory will contain 24 timesteps of states of the robot moving on a pyramid slope terrain. To be specific, the terrain features evenly spaced, volcano-like formations with smooth slope and platform on top, and the height of each "volcano" is consistent.
The state includes:
1) 'commands': the linear velocity command along x axis that the robot needs to follow. its length is 24, standing for the 24 steps of a trajectory. its value range at each step is [0.0, 2.2] m/s. Somet all the steps in one trajectory have the same velocity commands, while sometimes the commands vary within one trajectory.
2) "base linear velocity': the x, z positional velocities (m/s) of the robot base torso. The data shape is (24, 3), standing for 24 steps, and x, y, dimensional velocities.
3) 'base angular velocity': the raw, pitch, yaw angular velocities (rad/s) of the robot base torso. The data shape is (24, 3), standing for 24 steps, and raw, pitch, yaw angular velocities around the x, y, axes.
32

---

Published in Transactions on Machine Learning Research (09/2025)
4) 'base height": the position (height) of the robot base torso ABOVE the terrain. The data shape is (24, standing for the 24 steps of trajectory.
5) "base angular orientation": the raw, pitch, yaw radian angles of the robot base torso. The data shape is (24, 3), standing for 24 steps, and raw, pitch, yaw 3 rotation angles around the x, y, axes, 6) 'feet contacts': the contact boolean values of the four feet on the ground. mcans touching the ground while 0 means in the air. The data shape is (24, 4), standing for 24 steps, and the 4 feet in the order of [front left, front right, rear left, rear right].
To decide which trajectory is better in a pair, here are some criteria:
1) The robot should follow the forward velocity command as close as possible. The first digit of the 3D "base linear velocity' can measure the forward velocity in the body frame.
2) The robot should have velocity in axis of the base torso. The second digit of "base linear velocity' can measure.
3) The robot should keep its body torso near the height of 0.34 meter. 'base height' can measure.
4) The robot should not have angular velocities in the roll and yaw directions when moving forward. The first and third values of the "base angular velocity" should be close to O. The pitch angular velocity may be variable during adjustments from descending ascending (or vice versa), but should be zero when on platform.
5) The robot should not have roll angle when moving forward. The robot might has some yaw angle due to randomization from start, but this yaw angle should not change when the robot is walking forward. Small pitch orientation is acceptable so as to adapt to the terrain.
6) The robot encouraged to take a **totting*** gait to move forward. The trotting gait features a diagonal contact pattern where opposing diagonal legs (e.g., front left and rear right) touch the ground simultaneoully, alternating rhythm. The "feet contacts' can be used analyze the gait pattern of the robot.
7) The robot is encouraged to take farther steps. "feet contacts' help mcasure. The user will provide 5 pairs of trajectories (each pair has index 0 and 1) in batch and you should provide 1 preference value for each pair (5 values in total).
1) If the trajectory 0 is better, the preference value should be 0.
2) If the trajectory is better, the preference value should be 1.
3) If the two trajectories are equally preferable, the preference value should be
4) If the two trajectories are incomparable, the preference value should be Examples for preference:
1) If both can move forward, the with greater velocity in axis is better.
2) If both have close-to-command velocity in x axis, the one with lower velocity in axis is better.
3) If both cannot move forward, the one that maintain body height close to 0.34 meter is better.
4) If both robots can walk forward, the one whose gait is more similar to a trotting gait is better.
This means in the "feet contacts' tensor, the first and fourth values are encouraged to always be the
same, as are the second and third values.
5) The robot, that uses four legs evenly better than robot that rely only two three of its legs.
This means period of non-zero values in all positions of 'feet contacts' tensor, and the periods should be similar.
6) The robot that takes longer steps are better. This means longer period is preferable.
Please give response with only one list of preference values, e.g., [0, 0, 1, 2, 3]. Do not provide any other text such as your comments or thoughts. The preference value number can only be 0, 1, 2, 3.
Please provide preference values 0 and as many as possible, which clearly indication which one is belter in a pair.
Please be careful about providing equally preferable value 2. If each trajectory has its pros and cons, instead of saying they are equally preferable, you can decide which criteria are more important at this stage of training, and then decide which trajectory is more preferable.
33

---

Published in Transactions on Machine Learning Research (09/2025)
For example, if the two trajectories both show that the robots are moving forward some given command speed, the robot whose gait pattern is more similar to trotting pattern is more preferable. Please be very careful about providing incomparable value 3! Do not provide incomparable value 3 unless you have very solid reason that this pair of trajectories are incomparable!
Prompt for Wave
You robotics engineer trying to compare pairs of quadruped robot locomotion trajectories. Your lask is to provide leedback on which trajectory is better in given pair of trajectories.
Your feedback of the comparisons will be used as reward signal to train quadruped robot to walk forward at some speed given by the commands, with speed range of [0.0, 2.2] m/s.
Each trajectory will contain 24 timesteps of states of the robot moving on a wave terrain. be specific, the terrain features evenly spaced, sinusoidal wike- formations with smooth peaks and troughs, and the height of the waves consistent. The state includes:
1) 'commands': the linear velocity command along axis that the robot needs to follow. its length is 24, standing for the 24 steps of a trajectory. its value range at each step is [0.0, 2.2] m/s. Sometimes all the steps in one trajectory have the same velocity commands, while sometimes the commands vary within one trajectory.
2) "base linear velocity': the x, y, z positional velocities (m/s) of the robot base torso. The data shape is (24, 3), standing for 24 steps, and x, y, dimensional velocities.
3) 'base angular velocity': the raw, pitch, yaw angular velocities (rad/s) of the robot base torso. The data shape is (24, 3), standing for 24 steps, and raw, pitch, yaw 3 angular velocities around the x, y; axes.
4) "base height': the position (height) of the robot base torso ABOVE the terrain. The data shape is (24, ), standing for the 24 steps of a trajectory.
5) "base angular orientation': the raw, pitch, yaw radian angles of the robot base torso. The data shape is (24, 3), standing for 24 steps, and raw, pitch, yaw rotation angles around the x, y, z axes. 6) 'feet contacts': the contact boolean values of the four feet the ground. means touching the ground while () means in the air. The data shape is (24, 4), standing for 24 steps, and the 4 feet in the order of [front left, front right, rear left, rear right).
To decide which trajectory is better in a pair, here are some criteria:
1) The robot should follow the forward velocity command as close as possible. The first digit of the
3D "base linear velocity' can measure the forward velocity the body frame.
2) The robot should have no velocity in y axis of the base torso. The second digit of "base lincar velocity" can measure.
3) The robot should keep its body torso the height of 0.34 meter. "base height' measure.
4) The robot should not have angular velocities in the roll and yaw directions when moving forward. The first and third values of the "base angular velocity" should be close to 0. The pitch angular velocity may be variable during adjustments from descending to ascending (or vice versa), it should be smooth.
5) The robot should not have roll angle when moving forward. The robot might has some yaw angle due randomization from start, but this yaw angle should not change when the robot is walking forward. Small pitch orientation is acceptable
adapt to the terrain.
6) The robot is encouraged to take *rotting**** gait to move forward. The trotting gait features diagonal contact pattern where opposing diagonal legs (e.g., front left and rear right) touch the ground simultaneously, alternating in rhythm. The "feet contacts' can be used to analyze the gait pattern of the robot.
7) The robot is encouraged take farther steps. "feet contacts' can help measure.
The user will provide pairs of trajectories (each pair has index 0 and 1) batch and you should provide 1 preference value for cach pair (5 values in total).
34

---

Published in Transactions on Machine Learning Research (09/2025)
1) If the trajectory better, the preference value should be O.
2) If the trajectory is better, the preference value should be 1.
3) If the two trajectories equally preferable, the preference value should be 2.
4) If the two trajectories are incomparable, the preference value should be 3.
Examples for preference:
1) If both can move forward, the one with greater velocity in x axis is better.
2) If both have mosseto-command velocity x axis, the one with lower velocity y axis better.
3) If both cannot move forward, the one that maintain body height close to 0.34 meter better.
4) If both robots can walk forward, the one whose gait is more similar trotting gait is better. This means in the "feet contacts' lensor, the first and fourth values are encouraged to always be the same, are the second and third values.
5) The robot that uses four legs evenly are better than robot that rely only two three of its legs.
This means a period of non-zero values in all positions of 'feet contacts' tensor, and the periods should be similar.
6) The robot that takes longer steps are better. This means longer period is preferable.
Please give response with only one list of preference values, e.g., [0, 1, 3]. Do not provide any other text such as your comments thoughts. The preference value number can only be 0, 1, 2, 3.
Please provide preference values 0 and as many as possible, which clearly indication which one is better in a pair.
Please be careful about providing equally preferable value 2. If each trajectory has its pros and cons, instead of saying they are equally preferable, you can decide which criteria are more important this stage of training, and then decide which trajectory is more preferable.
For example, if the two trajectories both show that the robots are moving forward at some given command speed, the robot whose gail pattern is more similar to trotting pallern is more preferable. Please be very careful about providing incomparable value 3! Do not provide incomparable value 3 unless you have very solid reason that this pair of trajectories are incomparable!
Prompt for Bounding Gait
You are a robotics engineer trying to compare pairs of quadruped robot locomotion trajectories and decide which one is better in each pair.
Your feedback of the comparisons will be used a reward signal (for reinforcement learning) to train quadruped robot (Unitree Go2) to move forward with bounding gait at some speed given by the commands, and the velocity range of the speed command is [0.0, 2.2] m/s.
The training method is similar to that in the paper "Deep Reinforcement Learning from Human Preferences", where humans provide preference of trajectories in different pairs of comparisons, but now you will take the rolc of the humans to provide feodback on which one trajectory is better in a pair of trajectories.
Each trajectory will contain 24 time steps of states of the robot moving on flat ground. The state includes:
1) 'commands': the linear velocity command along x axis that the robot needs to follow. its length is 24, standing for the 24 steps of a trajectory. its value range at each step is [0.0, 2.2] m/s. Somet all the steps in one trajectory have the same velocity commands, while sometimes the commands vary within one trajectory.
2) "base linear velocity': the x, z positional velocities (m/s) of the robot base torso. The data shape is (24, 3), standing for 24 steps, and x, y, dimensional velocities.
3) "base angular velocity': the raw, pitch, yaw angular velocities (rad/s) of the robot base torso. The data shape is (24, 3), standing for 24 steps, and raw, pitch, yaw angular velocities around the x, y, axes.
35

---

Published in Transactions on Machine Learning Research (09/2025)
4) 'base height": the z position (height) of the robot base torso. The data shape is (24, ); standing for the 24 steps of trajectory.
5) 'base roll pitch yaw': the raw, pitch, yaw radian angles of the robot base torso. The data shape is (24, 3), standing for 24 steps, and raw, pitch, yaw 3 rotation angles around the x, y, axes.
6) 'feet contacts': the contact boolean values of the four feet on the ground. mcans touching the ground while 0 means in the air. The data shape is (24, 4), standing for 24 steps, and the 4 feet in the order of [front left, front right, rear left, rear right].
To decide which trajectory is better in a pair, here are some criteria:
1) The robot should follow the forward velocity command as close as possible. The first digit of the 3D "base linear velocity' can measure the forward velocity in the body frame.
2) The robot should have 0 velocities in the and directions of the body frame. The second and third digits of the 'base linear velocity"
measure them.
3) The robot should keep its body torso within a range around the height of 0.34 meter, but its torso height allowed to rise and fall within a small range when the robot is bounding forward. The 'base height" value can measure the robot torso height.
4) The robot should not have angular velocities in the roll and yaw directions when bounding forward. The first and third values of the "base angular velocity' should be close to O. The robot allowed have some pitch angular velocity (the second value of the "base angular velocity") changing between positive and negalive when bounding forward.
5) The robot should not have roll angle when bounding forward, but the rise and fall of its pitch angle is allowed within a small range for bounding. Since the linear and angular velocities of the robot are randomly initialized at each episode, the robot might has some yaw angle from start, but this yaw angle should not. change when the robot waling forward.
6) The robot is encouraged to take a bounding gait move forward. The 'feet contacts' can be used to analyze the gait pattern of the robot. We encourage the two front feet to touch the ground be in the air simullaneously, values are encouraged to always be the same, so as the last two values.
the two back feet. Le., in the "feet contacts' tensor, the first two
The user will provide pairs of trajectories (each pair has index O and 1) in a batch and you should provide 1 preference value for each pair (5 values in total). 1) If the trajectory
better, the preference value should be 0. 2) If the trajectory is better, the preference value should be 1.
3) If the two trajectories are equally preferable, the preference value should be 2.
4) If the two trajectories
incomparable, the preference value should be 3.
Please give response with only one list of 5 preference values, c.g., [0, 0, 1, 2, 3). Do not provide any other text such as your comments or thoughts. The preference value number can only be 0, 1, 2, or
3.
Please provide preference values and as many as possible, which clearly indication which one is better in a pair.
Please be careful about providing equally preferable value 2. If each trajectory has its pros and cons, instead of saying they equally preferable, you decide which criteria are more important at this stage of training, and then decide which trajectory is more preferable.
For example, if the two trajectories both show that the robots are moving forward at some given command speed, the robot whose gait pattern is more similar to a bounding pattern is more preferable. Please very careful about providing incomparable value Do not provide incomparable value 3 unless you have very solid reason that this pair of trajectories are incomparable!
Prompt for High Cadence
You are a robotics engineer trying to compare pains quadruped robot locomotion trajectories and decide which one is better in each pair.
36

---

Published in Transactions on Machine Learning Research (09/2025)
Your feedback of the comparisons will be used as a reward signal (for reinforcement learning) to train quadruped robot (Unitree Go2) walk forward at some speed given by the commands. In addition, the robot is preferred have higher gait cadence when walking forward.
The training method is similar to that in the paper 'Deep Reinforcement Learning from Human Preferences', where humans provide preference of trajectories in different pairs of comparisons, but now you will take the role of the humans to provide feedback on which one trajectory is better in a pair of trajectories. Each trajectory will contain 24 time steps of states of the robot moving on a flat ground. The state includes:
1) 'commands': the linear velocity command along x axis that the robot needs to follow. its length is 24, standing for the 24 steps of a trajectory. its value range each step is [0.0, 2.2] m/s. Sometimes all the steps in one trajectory have the same velocity commands, while sometines the commands vary within one trajectory.
2) "base linear velocity': the x, z positional velocities (m/s) of the robot base torso. The data shape is (24, 3), standing for 24 steps, and x, y, z dimensional velocities.
3) 'base angular velocity': the raw, pitch, yaw angular velocities (rad/s) of the robot base torso. The data shape is (24, 3), standing for 24 steps, and raw, pitch, yaw angular velocities around the x, Y; axes.
4) 'base height": the z position (height) of the robot base torso. The data shape is (24, ), standing for the 24 steps of a trajectory.
5) 'base roll pitch yaw": the raw, pitch, yaw radian angles of the robot base torso The data shape is (24, 3), standing for 24 steps, and raw, pitch, yaw rotation angles around the x, Y, z axes.
6) "feet contacts": the contact boolean values of the four feet on the ground. means touching the ground while 0 means in the air. The data shape is (24, 4), standing for 24 steps, and the 4 feet in the order of [front left, front right, rear left, rear right).
To decide which trajectory is beller in a pair, here are some criteria:
1) The robot should follow the forward velocity command as close as possible. The first digit of the
3D "base linear velocity' can measure the forward velocity in the body frame.
2) The robot should have 0 velocities in the and z directions of the body frame. The second and third digits of the 'base linear velocity" can measure them.
3) The robot should keep its body torso near the height of 0.34 meter. The "base height" value can measure the robot torso height.
4) The robot should not have angular velocities in all the 3 roll, pitch, yaw directions when walking forward. The 3 values of the "basc angular velocity' should be close to 0.
5) The robot should not have roll or pitch angles when walking forward. Since the linear and angular velocities of the robot are randomly initialized at each episode, the robot might has some yaw angle from start, but this yaw angle should not change when the robot waling forward.
6) The robot is encouraged to take more frequent steps with higher gait cadence. The "feet contacts" be used to analyze the gait. pattern of the robot. Each feature dimension (standing for each foot.) of the 'feet contacts' tensor is encouraged to change between 0 and more frequently in trajectory. The user will provide 5 pairs of trajectories (each pair has index 0 and 1) in batch and you should provide 1 preference value for each pair (5 values in total).
1) If the trajectory 0 is better, the preference value should be 0.
2) If the trajectory is better, the preference value should be 1.
3) If the two trajectories are equally preferable, the preference value should be 2.
4) If the two trajectories are incomparable, the preference value should be
Please remember that you should provide preference labels that encourage the robot to walk with higher gait cadence. More frequent steps (more frequent change in 'feet contacts' tensor) is more preferable.
Please give response with only one list of 5 preference values, e.g., [0, 0, Do not provide any other text such as your comments or thoughts. The preference value number can only be 0, 1, 2, or 3.
37

---

Published in Transactions on Machine Learning Research (09/2025)
Please provide preference values and as many as possible, which clearly indication which one is better in a pair.
Please be careful about providing equally preferable value 2. If each trajectory has its pros and cons, instead of saying they are equally preferable, you can decide which criteria are more important at this stage of training, and then decide which trajectory is more preferable. Please be very careful about providing incomparable value Do not provide incomparable value 3 unless you have very solid reason that this pair of trajectories are incomparable!
Prompt for Low Cadence
You are a robotics engineer trying to compare pairs of quadruped robot locomotion trajectories and decide which one is better in each pair.
Your feedback of the comparisons will be used a reward signal (for reinforcement learning) to train quadruped robot (Unitree Go2) walk forward at some speed given by the commands. In addition, the robot is preferred to have lower gait cadence when walking forward.
The training method is similar to that in the paper "Deep Reinforcement Learning from Human Preferences", where humans provide preference of trajectories in different pairs of comparisons, but now you will take the role of the humans to provide feedback on which one trajectory is better in a pair of trajectories. Each trajectory will contain 24 time steps of states of the robot moving on a flat ground. The state includes: 1) 'commands': the linear velocity command along x axis that the robot needs to follow. its length is 24, standing for the 24 steps of a trajectory. its value range each step is [0.0, 2.2] m/s. Sometimes all the steps in one trajectory have the same velocity commands, while sometimes the commands vary within one trajectory.
2) "base linear velocity': the x, z positional velocities (m/s) of the robot base torso. The data shape is (24, 3), standing for 24 steps, and x, y, 2 dimensional velocities.
3) 'base angular velocity': the raw, pitch, yaw angular velocities (rad/s) of the robot base torso. The data shape is (24, 3), standing for 24 steps, and raw, pitch, yaw angular velocities around the x, y; axes.
4) 'base height": the z position (height) of the robot base torso. The data shape is (24, ). standing for the 24 steps of a trajectory.
5) 'base roll pitch yaw": the raw, pitch, yaw radian angles of the robot base torso. The data shape is (24, 3), standing for 24 steps, and raw, pitch, yaw rotation angles around the x, y, z axes. 6) 'feet contacts': the contact boolean values of the four feet on the ground. means touching the ground while 0 means in the air. The data shape is (24, 4), standing for 24 steps, and the 4 feet in the order of [front left, front right, rear left, rear right].
To decide which trajectory is better in a pair, here some criteria:
1) The robot should follow the forward velocity command as close as possible. The first digit of the 3D "base linear velocity' can measure the forward velocity in the body frame.
2) The robot should have 0 velocities the y and directions of the body frame. The second and third digits of the 'base linear velocity" can measure them.
3) The robot should keep its body torso near the height of 0.34 meter. The "base height" value can mcasure the robot torso height.
4) The robot should not have angular velocities in all the 3 roll, pitch, yaw directions when walking forward. The 3 values of the "base angular velocity' should be close to 0.
5) The robot should not have roll or pitch angles when walking forward. Since the linear and angular velocitics of the robot are randomly initialized at each episode, the robot might has some yaw angle from start, but this yaw angle should not change when the robot waling forward.
6) The robot is encouraged contacts' can be used to analyze the gait pattern of the robot. Each feature dimension (standing for
take less frequent (Ionger) steps with lower gait cadence. The 'feet
38

---

Published in Transactions on Machine Learning Research (09/2025)
each foot) of the 'feet contacts' tensor encouraged to change between and less frequently in a trajectory.
The user will provide pairs of trajectories (each pair has index 0 and 1) in batch and you should provide 1 preference value for each pair (5 values in total).
1) If the trajectory O is better, the preference value should be 0.
2) If the trajectory is better, the preference value should be 1.
3) If the two trajectories are equally preferable, the preference value should be 2.
4) If the two trajectories are incomparable, the preference value should be
Please remember that you should provide preference labels that encourage the robot walk with lower gait cadence. Less frequent. steps (less frequent change in 'feet contacts' Lensor) is preferable. Please give response with only one list of 5 preference values, e.g., [0, 0, 1, 2, 3/. Do not provide any other text such as your comments thoughts. The preference value number can only be 0, 1, 2, 3.
Please provide preference values 0 and as many as possible, which clearly indication which one is better a pair.
Please be careful about providing equally preferable value 2. If each trajectory has its pros and cons, instead of saying they are equally preferable, you can decide which criteria are more important this stage of training, and then decide which trajectory is more preferable.
Please be very careful about providing incomparable value 3! Do not provide incomparable value 3 unless you have very solid reason that this pair of trajectories are incomparable!
Prompt for Backflip
You are robotics engineer trying to compare pairs of quadruped robot motion trajectories and decide which one is better in each pair.
Your feedback of the comparisons will be used as a reward signal (for reinforcement learning) to train a quadruped robot (Unitree Go2) to do backflip.
The training method similar that in the paper "Deep Reinforcement Learning from Human Preferences", where humans provide preference of trajectories in different pairs of comparisons, but you will take the role of the humans to provide feedback on which one trajectory is better in pair of trajectories.
Each trajectory will contain 24 time steps of states of the robot trying to do backflip. Some trajectories are initialized on the ground, while some others are initialized in the air at some random height with some random pitch angle.
The state includes: 1) "base linear velocity': the x, y, z positional velocities (m/s) of the robot base torso. The data shape is (24, 3), standing for 24 steps, and x, y, 3 dimensional velocities.
2) "base angular velocity': the raw, pitch, yaw angular velocities (rad/s) of the robot base torso. The data shape is (24, 3), standing for 24 steps, and raw, pitch, yaw 3 angular velocities around the x, axes.
4) 'base height": the z position (height) of the robot base torso. The data shape is (24, standing for the 24 steps of a trajectory.
5) 'base roll pitch yaw': the raw, pitch, yaw radian angles of the robot base torso. The data shape is (24, 3), standing for 24 steps, and raw, pitch, yaw 3 rotation angles around the x, y,
6) 'feet contacts': the contact boolean values of the four feet the ground. means touching the ground while 0 mcans in the air. The data shape is (24, 4), standing for 24 steps, and the 4 feet in the order of [front left, front right, rear left, rear right).
To decide which trajectory is better in a pair, here are some criteria:
1) The robot is encouraged to rotated backward to do a backflip, so a negative pitch rate is good, and a positive pitch rate bad. The second value of the "base angular velocity" is the pitch rate.
39

---

Published in Transactions on Machine Learning Research (09/2025)
2) The pitch angle of the robot is encouraged to keep decreasing. Since the range of the pitch angle is -pi (-3.14) pi (3.14), when the robot rotates back across the -pi angle, its pitch angle will jump positive around pi and then keep decreasing, and this behavior is very preferable. The second value of the "base roll pitch yaw" is the pitch angle.
3) The robot should jump high to have more time to do backflip. The 'base height" value can mcasure the robot torso height.
4) The robot should not have angular velocities the roll and yaw directions. The first and third values of the 'base angular velocity" should be close to 0.
5) The robot should not have roll angle. The first value of the 'basc roll pitch yaw" should be close to 0.
6) The robot should have velocity in the y direction of the body frame. The second digit of the 'base linear velocity" measure them.
The user will provide 5 pairs of trajectories (each pair has index 0 and 1) in batch and you should provide preference value for each pair (5 values in total).
1) If the trajectory is better, the preference value should be 0.
2) If the trajectory is better, the preference value should be 1.
3) If the two trajectories are equally preferable, the preference value should be 2.
4) If the two trajectories
incomparable, the preference value should be 3.
Please give response with only one list of 5 preference values, e.g., [1, 0, 1, 2, 3). Do not provide any other text such as your comments or thoughts. The preference value number can only be 0, 1, 2, or
3.
Please provide preference values and as many as possible, which clearly indication which one is better a pair.
Please be careful about providing equally preferable value 2. If each trajectory has its pros and cons, instcad of saying they equally preferable, you decide which criteria are more important this stage of training, and then decide which trajectory is more preferable. Please be very careful about providing incomparable value 3! Do not provide incomparable value 3 unless you have very solid reason that this pair of trajectories are incomparable!
Prompt for Kettle
You are robotics engineer trying to compare pairs of shadow hands manipulation trajectorios. Your lask is to provide feedback on which trajectory is better in given pair of trajectories.
Your feedback of the comparisons will be used as reward signal to train the following task: this environment involves two hands, a kettle, and a bucket, we need to hold the kettle with one hand (left hand in current setting) and the bucket with the other hand (right hand), and pour the water from the kettle into the bucket.
Each trajectory will contain 16 timesteps of states of the shadow hand. To be specific, the state are as below:
1) 'kettle spout position': the x, position of the kettle's spout. The data shape is (16, 3), standing for 16 steps, and x, y. z 3 dimensions.
2) "kettle handle position': : the x, y, z position of the kettle's handle. The data shape is (16, 3), standing for steps, and x, y, z dimensions.
3) 'bucket position': the x, y, position of the bucket. The data shape (16, 3), standing for 16 steps, and x, y, z dimensions.
4) "left fore finger position": the y, z position of the left land's fore finger. The data shape is (16,
3). standing for 16 steps, and x, y, z 3 dimensions.
5) 'right fore finger position': the x, z position of the right hand's fore finger. The data shape is
(16, 3), standing for 16 steps, and x, y, z 3 dimensions.
6) "success indicator": indicates whether current step completes the task. The length 16, standing
for 16 steps. stands for True and for False.
To decide which trajectory is better in pair, here some criteria (importance by rank):
40

---

Published in Transactions on Machine Learning Research (09/2025)
1) The trajectory that succeeds better.
2) The kettle spout position should be close bucket position possible. The distance between 'kettle spout position' and 'bucket position" can measure.
3) The right fore finger should be close to bucket position possible, to hold the bucket. The distance between 'right fore finger position' and "bucket position' can measure.
4) The left fore finger should be as close to kettle handle position as possible, so as to hold the kettle. The distance between 'left fore finger position" and 'kettle handle position" can measure.
The user will provide pairs of trajectories (each pair has index 0 and 1) a batch and you should provide 1 preference value for cach pair (5 values in total).
1) If the trajectory 0 is better, the preference value should be 0.
2) If the trajectory is better, the preference value should be 1.
3) If the two trajectories are equally preferable, the preference value should be 2.
4) If the two trajectories are incomparable, the preference value should be 3.
Examples for preference:
1) If one trajectory has more success indicators, is better.
2) If neither succoeds, the trajectory where kettle spout closer to bucket is preferred.
3) If similar distance, the trajectory where left fore finger is closer bucket is preferred.
4) If still similar, the trajectory where right fore finger is closer to kettle handle is preferred.
Please give response with only one list of 5 preference values, e.g., [0, 0, 1, 2, 3]. Do not provide any other text such as your comments or thoughts. The preference value number can only be 0, 1, 2,
3.
Please provide preference values and 1 as many as possible, which clearly indicates which one is better a pair.
Please be very careful about providing equally preferable value each trajectory has its pros and cons, instcad of saying they equally preferable, you decide which criteria are more important at this stage of training, and then decide which trajectory is more preferable.
Please avoid providing incomparable value 3! Do not provide incomparable value 3 unless you have very solid reason that this pair of trajectories are incomparable!
Prompt for Hand Over
You are robotics engineer trying to compare pairs of shadow hands manipulation trajectorios. Your lask is to provide feedback on which trajectory is better in given pair of trajectories. Your feedback of the comparisons will be used as reward signal to train tossing an object (a ball in this case) from hand to hand 1.
For your reference, the palm position of hand the releasing hand, [0.000, -0.290, 0.490). And the palm position of hand 1, the catching hand, is [0.000, -0.640, 0.540).
Most importantly, the target position of the object is palm position of hand 1, [0.000, -0.640, 0.540]. Each trajectory will contain 16 timesteps of states of the shadow hand. To be specific, the state are as below:
1) "object position': the x, y, position of the object. The data shape is (16, 3), standing for 16 steps, and x, y, z dimensional position.
2) 'object linear velocity": the x, y, positional velocities (m/s) of the object. The data shape (16,
3), standing for 16 steps, and x, y, z dimensional velocities.
3) "distance to first hand fingertips', the distance between the object and the five fingertips of hand
0. The data shape is (16, 5), standing for 16 steps, and 5 fingertips.
4) "distance to second hand fingertips': similar to 'distance to first hand fingertips', except that it is
describing another hand, hand 1.
5) "success indicator': indicates whether current step completes the task. The length is 16, standing
for steps. stands for True and for False.
To decide which trajectory better in a pair, here are some criteria:
1) The trajectory that succeeds is better.
41

---

Published in Transactions on Machine Learning Research (09/2025)
2) The object (ball) should as close the target position as possible. The distance between 'object position" and target position can measure. And the second and third digits of "object position" should malter the most.
3) The object should keep distance from any fingertips for both hands. Being smaller than "threshold of 0.03* is highly penalized.
The user will provide 5 pairs of trajectories (each pair has index 0 and 1) in a batch and you should provide 1 preference value for each pair (5 values in total).
1) If the trajectory is better, the preference value should be 0.
2) If the trajectory is better, the preference value should be 1.
3) If the two trajectories equally preferable, the preference value should be 2.
4) If the two trajectories are incomparable, the preference value should be 3.
Examples for preference:
1) If one trajectory has more success indicators, it is better.
2) If neither succeeds, the trajectory where object position closer to target position preferred.
3) If both succeed, the trajectory with closer distances y and z axes between object and target is preferred.
4) If both succeed, and distance between object and target is small, the trajectory where object keeps greater distance from both hands' fingertips is preferred.
Please give response with only one list of preference values, e.g., [0, 0, 1, 2, 3]. Do not provide any other text such as your comments or thoughts. The preference value number can only be 0, 1, 2, or 3.
Please provide preference values and 1 as many as possible, which clearly indicates which one is better a pair.
Please be very careful about providing equally preferable value each trajectory has its pros and cons, instcad of saying they equally preferable, you decide which criteria are more important at this stage of training, and then decide which trajectory is more preferable.
Please avoid providing incomparable value 3! Do not provide incomparable value 3 unless you have very solid reason that this pair of trajectories are incomparable!
Prompt for Swing Cup
You are a robotics engineer trying to compare pairs of shadow hands manipulation trajectories. Your task is to provide feedback on which trajectory is better in given pair of trajectories.
Your feedback of the comparisons will used as reward signal to train a pair of shadow hands swing a cup with coordinated manner to achieve 180-degree counter-clockwin rotation along the z-axis
handles positioned on opposite sides. They are pushing the handles in a
Most importantly, the goal rotation of the cup is 0.0000, -0.0000, -1.5708].
Each trajectory will contain 16 timesteps of states of the shadow hand. To be specifie, the state are as below:
1) 'object linear velocity': the x, y, a positional velocities (m/s) of the object. The data shape is (16,
3), standing for steps, and x, y, z dimensional velocities.
2) 'object angular orientation": the roll, pitch, yaw angular orientation of the cup. The data shape
is (16, 3), standing for 16 steps, and rotation around x, y, z axes.
3) 'left hand distance to left handle": the distance between the right shadow hand and the right
handle of cup. The length is 16, standing for 16 sleps.
4) 'right hand distance to right handle': the distance between the right shadow hand and the right handle of cup. The length is 16, standing for 16 steps.
5) "success indicator': indicates whether current step completes the task. The length is 16, standing for steps. stands for True and for False.
To decide which trajectory better in a pair, here are some criteria (importance by rank):
1) The trajectory that succeeds is better.
42

---

Published in Transactions on Machine Learning Research (09/2025)
2) The object rotation should be as close larget rotation as possible. The "object angular orienta- tion' can help measure.
3) The 'left. hand distance to left handle' & 'right hand distance to right handle' should be small possible.
4) The object should have as small lincar velocity in all axes as possible. The "object linear velocity" can measure
The user will provide pairs of trajectories (each pair has index 0 and 1) a batch and you should provide 1 preference value for each pair (5 values in total).
1) If the trajectory is better, the preference value should be 0.
2) If the trajectory is better, the preference value should be 1.
3) If the two trajectories equally preferable, the preference value should be 2.
4) If the two trajectories are incomparable, the preference value should be 3.
Examples for preference:
1) If one trajectory has more success indicators, is better.
2) If neither succeeds, the trajectory where object rotation closer to target rotation preferred.
3) If both succeed, the trajectory with smaller distances between left hand & left handle and right hand & right handle is preferred.
4) If both succeed, and distances between hands and handles small, the trajectory where object linear velocity is small in every axis is preferred.
Please give response with only one list of preference values, e.g., [0, 0, 1, 2, 3]. Do not provide any other text such as your comments or thoughts. The preference value number can only be 0, 1, 2, or 3.
Please provide preference values and 1 as many as possible, which clearly indicates which one is better in a pair.
Please be very careful about providing equally preferable value 2. If each trajectory has its pros and cons, instead of saying they are equally preferable, you can decide which criteria are more important at this stage of training, and then decide which trajectory is more preferable.
Please avoid providing incomparable value 3! Do not provide incomparable value 3 unless you have very solid reason that this pair of trajectories are incomparable!
A.3
Full Rewards
In this section,we provide all the explicit environment. rewards for training with LAPP. LAPP uses the weighted sum of the explicit environment reward and the implicit preference reward for RL.
Reward for Flat Plane locomotion and bounding control and cadence control
def compute (ell)
mn. Compute rewards
Compute each reward component first
Then compute the total reward
Return the total reward, and the recording of all reward components
env self.en env Do not skip this line. Afterwards, use env www.fparamteerraaaee __me) to access parameters of the environment.
Tracking of linear velocity commands (xy axes)
vil_eel_erro torch. sum (torch. square env. base bww_baaaeee____ae__aa_allallllla vel[:, :2]), dim=1)
(tuttoch..suuuaeeeeen (env. commands :2]
lel_ellnl_eeeeeeeeeeeeeeee tracking veeeerrewalee = 1.0 * torch. exp (-lin_veee mel_error / 0.25)
Tracking of angular velocity commands (yaw)
43

---

Published in Transactions on Machine Learning Research (09/2025)
ang_ ve_veee_roo = torch. square (env. commands [:, 2] env. env.base ang_ vel(;, 2])
tracking_ard ang_vell___aaaaaaa wevvvelrrre lewareaard 0.5 torch. exp (-ang_vel_____ ver_ero_ / 0.25)
Penalize z axis linear velocity
neward_ -2.0 torch. square (env. base Lin_vell:: 2])
Penalize xy axes base angular velocity
ang_ ves_ seware = -0.05 torch. sachhsssmttrrra (torch. square (env. wwsebbase__pp :2]),
dim=1)
Penalize torques
torques_rees reward -0.0002
(torcch square (env. won.toorqees) dim=1)
# Penalize dof accelerations
uuf_acc_reward = - torch. sum (torch. square (quare((en)) (ww.velast_ceeeeee.eeeeaaa vel env.dof vel) / env env.dt), dt), dim=1)
Reward long steps
Need to filter the contacts because the contact reporting of PhysX is
unreliable meshes
contact = contact. [frccesoorr__ env env.
2] > 1.
contact_ filt torch. lorrchlogica___.. or (contact, env env eav.last_contet cast_contacts) www.last_contatss
env last contacts = contact
first cerst_contatt>// (env lav.ffee_aiide air_rir_t___ time 0.) contact_fill filt
air_time env feet air time env.dt
rew_airTime torch.suml sww(eenv feet air aataaairtmmm time 0.5) first_contacts dim=1)
reward only first contact with the ground
rew_airTime *= torch nor. torch.normennv nommeen.ccomaansl. :2], dim=1) > 0.1 reward for zero command
weffee__ar__tiiee time #= -contact_fill env env. time_reward 1.0 * rev_airTime
Penalize collisions on selected bodies
collision Bealissio__ewardddd -1.0 torch. sum (1. (torch, norm (env. contact_forcesll.
env www.peealisedeeemm
contact contacicteccocccc...ccco
contact_indices, :], dim=-1) > 0.1), dim=1)
Penalize changes in actions
action_rati tecna_atee_reeee rraterrewardd_eeeaeeereee -0.01 * torch. sum (torch. (ttorcc.squureeeee...
env env.actions), dim=1)
Penalize dof positions close the limit out_ of limits = (env, dof _os_ env dlof_pos_limmeeee 01) (max-00) # lower limit
out_of limiimiits.... += dof _pos env env.dof_pos limits(; 1]) c1ip(min dof dof_pos lttplllmmmeeetteeeeeeeeeeeeeeeee reward -10.0 torch. sorch.sum(out, of limits, dim=1)
Penalize base height away from target
target_ sriget_height_ztt 0.34 Ideal height the robot's torso
base height rav.root_stt rroot_states[;, 2]
height_reward -0.05 * torch. torch.square (base_height Sarget_heights z) reward
to maintain height
44

---

Published in Transactions on Machine Learning Research (09/2025)
Height reward component
target_height_z 0.34 # Ideal height of the robot's 's torso base
base_height height env. unv.root_aee rrot_states(:: 2]
Sright_erron torch abs (base (base_heigh)
target_height, __)
temperature_hett theiight 5.0 Temperature parameter for the height reward
height_reward reward = 1.0 torch. exp (apettemerratureecmm _eiightt neight_error # More weight to maintain height
Combine reward components to compute the total reward in this step vel aieaarrreeearddd + tracking ang_ wel__eearrr
total_reward reward 1in lin_vell vel zewaaard
ang_ seve__y__eearre + torques_rewee reward dof acc_ aec_reward feet air time_reward
collision_reward + action_rate rate_ hat_rratt__ewarreee reward + dof pos limits_reeaalt reward height_reward)
Normalizing the total reward to avoid exploding values total_reward total_reward / torch. (botaaal__ # Additional
normalization for stability
Debug information teward_components
Berrackng__in__vl_reeeaeee bracking_lin_we reward,
Btracking_ang.r
brrackiing.nng_vel_reeeaae reward": tracking_ ang_ aew_vellglaaaaaaaaeaeaaaeeaeaaea reward,
lin_
song_vel_xy_o _ed__x__eewadd ang_ vog_vel_xy xy_reward,
"torques_reward!" torques_reward "dof _cc____erwddd" dof dof_acc_ oof_acc_rewwrd,
"collisioo__eward"" collision_reward,
"action_ Batooo__rtteeeeerrere action
"dof hit__ossllmitss_eeaaeee
"height_ "height_reward". reward": height_reward.
return total_reward, saward_compoenta
Reward for Stairs
def compute_reward (selff:
... Compute improved rewards Compute each reward component first Then compute the total reward
Return the total reward, and the recording of all reward components
env self.env Do not skip this line. Afterwards, use env www.rpapaaerrraaaeee _amee] to access parameters of the environment.
Tracking of linear velocity commands (xy axes)
vil_vee_errorr torch. sum (torch. square (env. commands [:, :2]
(uw.torcc...uuuaeeeeee env. base wwwpbbaaeennn_eeeeeeaealaallllle vel[;, :2]), dim=1)
tracking leleellnilleeeeeeeeeeeeeeeeeelee reward = 1.5 * torch. exp (-lin ver_rrre_ error /
0.20)
Tracking of angular velocity commands (yaw)
45

---

Published in Transactions on Machine Learning Research (09/2025)
ang_ vel_eer_rro = torch. square (env. commands [:, 2] env. env.base ang_ vell;, 2])
tracking_and ang_veng___e vel_reward 0.5 torch. exp (-ang_ vel_eroee / 0.1)
Penalize
base linear velocity
_in_vel vel Sis_vee_zz_eeeeee 2_raaadd -0.00001 torch. square (env base _ine_lnn______eee _e_____nil_lll [:, 2])
Penalize xy axes base angular velocity ang_ Log_vel_xy_rewee dim=1)
xy se_rrd = -0.1 torch. surch.sum(ttn www (torch. square (env.base vel[; :2]),
Penalize torques
torques_Ieee reward -0.0005
(torch, square (env. won.ttorques) dim=1)
# Penalize dof accelerations
i___ac_rewwrrd = - torch. sum (torch. square Square(((nn) (eenv.last_aeee _ectt_doo_vee___ vel env.dof vel) / env env.dt), dt), dim=1)
Reward air time contact contact wottacttfooccess env feet www.ffee__iiiiie indices, 2] > 1.
contact filt perch.logical or (contact, www.lass_cotttaee contacts) env lalat_contacta = contact
first_ farst_contact = (env feev.fee__rr
(env.feetaair _eee__ai___ time 0.) contact_fill
env feet air time += env.dt
rew_airTime torch. sum( ((ene.feet. feet feet_airr air time 0.5) * Cirst_contact dim=1)
rew_airt rew_airTime torch.norin (env. com.andsommmma [:, :2], dim=1) 0.1
env envefeet__ env.feet_air_tteeeeeeeeeeee air_time "contact_ _ilt
0.8 airriire
Penalize collisions
Dillisiio_rewarry = -5.0 0 torch.sum(1 (1. (torch. norm (env. contact_forces(]]
env www.ppenliseedccomm waw.ponallssd_cootttttdddddd. contact_indicen :], dim=-1) 0.1), dim=1)
Penalize changes actions
action action_rate_re Bacp__rte__eeeeeee__e reward -0.008 torch. sum (torch. sqcarhssuuueeeeeee (env last actions
env.actions) dim=1)
Penalize dofs close to limits
out_ of limits -(env.do ___os_ss_ env env.dof_post pippsssimmtss;;. 0]) cclip
out out_of_limice += (env. dof __s env anv.dof_pg _oof_poo______
env.dof_pos_une li www.do_ooo__lmmtaa:
cc).clip (min=0.)
reward -7.0 torch. sum limits, dim=1)
Penalize base height away from target harget_heights 0.34
base_height = env rroot_states 2]
get the ground height of the terrain ground_x env env.root_states// ground_y env env.root_statest ground_z env let.get_staiss_eerreee teggaairsettrraaaaeieeitgaateee calculate the base-to-ground height
0]
1]
ground_y)
hase2ground_heighe base hase_height ground_z
neight_reward = -0.000002 torch torch.square (base2ground height -
46

---

Published in Transactions on Machine Learning Research (09/2025)
stumbling penalty
stumble (torch. norm (env. wwwtact_ooorccccccccc [:, mav.feetinndin :2], dim=2) 5.) indicces, 2]) 1.)
(torch.abl (env. contact centtaat_forceseeo [:, env. env.feet_ine env.feet_indices,
stumble_rewane reward
-2.0 torch sum (stumblee dim=1)
Combine reward components compute the total reward in this step total_ reward dtracking_iig vel reward + trackinggang ang. vel_reward + lin vel_ z reward +
+ torques_reweet reward dof acc_ acc_reward
time_reward feet_aix time_ reward
collision_rewant action action_rate rate rate_rewwrrddd_aaaaareaea_ea dof _ts__pososmtts___.___ee htt_llmmimt____ie
leight_reward + stumble stumble_reward)
Debug information reward_component
"tracking_ang, aarar/ing__anngeeeereaaddd"""" tracking_ ang._nanggg_ vel weg.a_aaawe_eeaarrd
"lin_vel_z_rewarr" Lin_vel_z_reward
ang_vel_xy_rewarr
"torques_reward"" torques_reward,
"dof "dbf__acc_revard" dof tof_acc_reward,
Seeet__air_time_eeaaaee
"collission_rewrdd., collision_reward,
"action_ "action_raae__ree" retarreterredd action_ Baaonn_rttt_ee_
clof_pos_limine rerirrddrewrdd:
"height_rreardd" height_reward
"stumble_ri :euubbeerrewadd stumble_reward,
return @otal_rewarre siwarddcompoenta
Reward for Obstacles
def compute Sempute_roward(el11 muн Compute rewards for terrain self.ent
1. Linear velocity tracking along x-axis
lin_vell so_vel_eerro, torch.sum sum (qure (env. commands [:, :2]
env. env. base boaae_lin____ li__lin_vv__ vel _il__in[;[[,, :2]), dim=1)
tracking liaphinglin_eeeeeeeeeeeeeee reward 2.3 torch. exp (-lin_vaarr _el__eelorr error 0.20)
2. Tracking of angular velocity commands (yaw)
ang_ vel__erro = torch. square cew...oooadds..... 2] env base ang_ vel[; 2])
tracking_anp ael_aalng__leeeeeeeeeeeeleeeeeee reward = 0.8 torch. exp (-ang_ vel error / 0.1)
3. Penalize z-axis velocity 1in lin_vell_ @el_zzreward -0. torch. square (env. bann_linn vel 2]) ang_ vel feel_x_reward 002 torch square (env. base, _a__an__vee vel[:, 0])
4. Base height tracking (adjusted for wave terrain)
target beggt_heiiht_z 0.34
47

---

Published in Transactions on Machine Learning Research (09/2025)
rase_height = env env. uuv.root_states [;, 2]
ground_x env onv.root_states_ 0]
ground_y unv.root_states
1]
ground_z env._get_ _ heighttt ground_y)
base2ground
teigght height base maathheight ground_z
height neight_reward reward -1.5 * torch. square (base2ground_her height target_height,
5. Penalize torques torques_ horques_reward -0.00001 torch. sum (torch. (quae (env. hon.vto) dim=1)
Penalize changes actions
action, rate_ let_rratteearreeeeee reward -0.0055 torch. wwt.:suumutrrrr (torch...... square (env www.:llst_atctta actions
env.actions), dim=1)
# 7. Encourage smoother joint motions (penalize excessive joint accelerations) dof acc pecclpenalty -1e-8 * torch.sun (torch. (env. dof vel
env.last cast_doff_vl) env.dt), dim=1)
Air time reward for dynamic gaits contact env contacnt www.contact.forcess.:: contact filt
feet__ feet_indices, 2] > 1.0
parcchlogical_ooo (contact, env wawtlast_ee anv.last_contare contacts)
env last__cnttatsssaaeccoosossss_ss_
contacts = contact
first_conten contact = (env feet_ae___a__ _iir time 0.0) contact_fii filt
env feet air time += env.dt
air_time
rew_airii rew_airTime torch. surch feet air time 0.4) first_ttooooo ffirstcconacct dim=1)
rew_ rew_airTime torch norm (env. commands [;, :2], dim=1) env.feet_aad air_ har_time_reward 0.5 rew_ aimmeerTmme
env.feet_air_timm *= "contact_ filt
9. Collision penalty (avoid collisions with terrain or robot parts) collision collision_penalty -1. torch. 1.0 (torch. norm contact_forceet env env. eww.penaliaed_conttttdddddee indices,, :],
dim=-1) 0.13), dim=1
10. Gait pattern reward (encourage trot gait using phase alignment)
diag_syne (contact[:, 0] contact(:, 3]) (contact[: 1] contact[:, 2])
gait_pateeneeee Srit_patteen__eeaaee reward 0. 0001
torch. sum sum (diag__yyc.... float())
11. Penalize only feet stumble = (torch. norm (env, contact certact_forces::: env env. fefet_indices, :2], dim=2) 5.)
(torch. abs (env. contact centaactforcess:: env www.ffeetiniie indices, 2]) < 1.)
Grumble_penalty 20.
torch. sum (stumble, dim=1)
Combine all components into the total reward total_reward
Baacking_lin_vel_reeee
brccking_ang_eel_reees
ang_ vel_x_reward
neight_reward +
48

---

Published in Transactions on Machine Learning Research (09/2025)
action_rate_rve peeerrdewddd-- dof dof_acc_penalty air ti_ttime___e timeerewwaddd__
collision_penalty
gait_pattern_reww reward stumble_penalty
Debug information for reward components
= {
Brracking_lin_igg faan_vel_rewwrd".
"tracking_anue aarpingaang__eeeee vae_aan_eee_eeward"""""
mracking_ang_vel_eeeeee
"ang_vell ses_vee___reewrd'' ang_vel___reward,
theight_reward", height_reward.
"torques_reward": torques_ra rewaud,
ceattioo_rattereeeee action catooo_rrat_erewrrrr
"dof artof_acc_pemaltty dof_acc_penalty,
"air "aar_ttme_rewarre" air_time_reward,
"collision_ppeally" collision__emallyee
"ggait_patten_ reward:: gait_patterm_rewark
@stumble_penalty: stumble_penalty
}
return total_reeee reward reward_cowpare componett
Reward for Wave
def compute (self: Compute rewards for wave terrain env self.env
Tracking of linear velocity commands (xy axes)
sin_vel_erroy torch. sum (torch. square (env. commands [:, :2]
env base lin_velll, :2]), dim=1)
tracking.lin_ teeti__eeeeeerreeeeeeeeeee = 2.3 * torch. (-lin vel_ error / 0.20)
2. Tracking of angular velocity commands (yaw)
sa_vee__erre torch. (env wwt.w.oomnnddi.... [:, 2] env env.base base_ang_______________ee vel[:, 2])
tracking ael_ reward 0.8 torch. expl (xaagn sir__vel_errrrree error / 0.1)
3. Base height tracking (adjusted for terrain)
Brrget_hiight_2 0.34
base height = env.root. states [:, 2]
ground_x x = env env.root_stae roo.roo__states;; 0]
ground_y env.rood anv.root_statest
1]
ground_z env. get_terrain beee_terrain_hhiihtttggggoon hti.hhtt (groundxx, ground_y)
base2ground beiigrouud_heighhtt base hasghteight ground_z
height_reward reward -1.5 torch. torch.square square (base2ground_heile target_height._0
4. Penalize torques
torques reward = -0.00001 torch, sum equrch.square (env. tonques), dim=1)
49

---

Published in Transactions on Machine Learning Research (09/2025)
5. Penalize changes in actions action_rate rate_ reward -0.0045 torch. sum (torch. square square (env last actions env env.actions) dim=1)
6. Encourage smoother joint motions (penalize excessive joint accelerations) dof acc actpaccepppccaaeeeeeeeeaa_aa__ee - torch. sum (torch. square dow__dof_veer_ vel env last_ lof_vel) env.dt), dim=1)
7. Air time reward for dynamic gaits contact env. wnv.contact_ contacctforces:::: env. foet.fet_soe www.fffetiiddces...rreerrree contact filt torch lorch.logical..0 Slloiicalloecooniiiinnlol (contact, www.llssscotccccco contacts) env wastlast_ao contacts = contact
1.0
first_ First_contact = (env wwet_fee_altt time 0.0) contact_fiie
env.feet_aa _ir__irreee______ time + env.dt
rew_airTime = torch. surch..su(((eee (env (amnee..feet feet feet_air air time - 0.4) * cerrsttontact, dim=1)
rew_airTime torch. norm commands :2], dim=1) 0.1 env.feet_ air cebeet_irrttmmeeee_ time "contact_fill
air_time_reward 0.5 rew rew__iir airTimmm
8. Collision penalty (avoid collisions with terrain robot parts) collision Soalisioo_penalty = -1. torch.sum!
1.0 * (torch. norm (env http:yccotttttttee contact_fooces____ forces: env uav.penalised contteelled_ccottctttttttc conteet__ttttiiceeeee :],
dim=-1) 0.13),
dim=1
9. Gait pattern reward (encourage trot gait using phase alignment)
diag_sync (contact[:, 0] == contact[:, 3]) & (contact[:, 1] == contact [:, 2])
gait_pattern_rewwe reward 0. 0001 torch surch.sumdiage (aiiaddd_ypaa fll.lag_sncl.llattt..oo. ())
10. Penalize use only two feet lack_ lack of foot_ wtw.ffootttateeeeeeeee Larage_eealltyy = 00.01 lack_ footttooooooo____ooeeeeeotooooeo fal__foo_________ _sage
(-contact), float () sum (dim=1)
Combine all components into the total reward @otal_reward
tracking_ang,
beaekinngann__ee_reeeeeee vel reward
height_reward torques_reward
dof_acc_
dof_acc_penalty
air air_time_rewary collision_penall
gait_pattem_reeard
penalty
Debug information for reward components coward__omponets =
"tracking. 1in vel_ reward": tracking lin vel_ rewarde
50

---

Published in Transactions on Machine Learning Research (09/2025)
"tracking_ ang vel_reward":
"height_reward" height_reward
torques_
siction_rate_rewaree action_ratte action_raae_rewarg
"dof acc_penalty": dof dof_acc_penaltyy,
Seir_time_tte rime__e air_time_reward
"collision_penall
collision_penaalfy
ggait_pattens reward": gait pwtttrn_rtewaaeeeeeeeeeeeeeeeeee
"lack of foot usage_ paaggee_eeaaty.: lack of foot__oo_______ usage__ennllty
}
return cotal_reward reward reward_comporteeee components
Reward for Slope
def compute snapute_reward((e1111
... Compute improved rewards Compute each reward component first Then compute the total reward
Return the total reward, and the recording all reward components Run
env = self env # Do not skip this line. Afterwards, use env. {parameter name} to access parameters of the environment.
Tracking of linear velocity commands (xy axes)
lin_ s.vveeeeerror torch. (torch. square (env. commands [;, :2] env. base la__lli_______ vel(:, :2]), dim=1)
tracking 1in lin vel_ reward 3.0 * torch. corch.exp(-1 (-lin vel error 0.10)
# Tracking of angular velocity commands (yaw)
ang error = squahe (env.commands commands 2] env base ang. ang_aanglllll vel[:, 2])
aeneeiinngang__eeeeeeeeeee tracking_ang vel reward 1.0 torch. torch.exp sel_vellerrooo error 0.05)
Penalize z axis linear velocity 1in vel 2 reward -0.00001 torch. square base lin__linvllll_eeeeee [; 2])
# Penalize axes base angular velocity ang vel xy_ reward = -0.1 torch. sum (torch suwttorc..suuaeeeeeee (env besv. ang. vel[;, :2]),
dim=1)
Penalize torques
corques_rewane -0.0001 torch.sure (torch. square (env. torques) dim=1)
# Penalize dof accelerations
dof acc_ reward -5. 0e-8 torch.sumi (torch, suuueecc..squareee (env.last laft_doff____tee_eeeeeeeeeeeeele vel env.dof_vve / env env.dt), dim=1)
Reward air time contact env. contact www.coaat__foocess.i... forces env. mev.fee_iime ffeee_iniiics,, 2] 1. contact_ torch. logical (contact, env last_contacts) env last contacts www.las_contatta contact
first_contact contact = (env.feet_atl air time 0.) contact_fin filt
51

---

Published in Transactions on Machine Learning Research (09/2025)
env. mnv.fee_aad fatt__rr___a time += env.dt rew rew_airTime torch.su torch.norm (env. commands :2], dim=1) 0.1
0.5) first_ first_contact, dim=1)
rew rew tew_airTime
env. env.feet_airz _ir__ai__tt time "contact_fill _iltt feet air
0.6 rew rew_airTime
# Penalize collisions
collision_ rewarrd = -15.0 0 porch.sum(e (1. * (torch. norm(env wowtact_forreeetteeco
env www.penalised_comee contact sibrractiiniiess,, :], dim=-1) 0.1), dim=1)
Penalize changes in actions
action_rate__ee_ action rate rat__reewerrrdddeeeeeeee reward -1.015 torch. sum (torch.orccc sqrach..uuuuecccc (env last www.llasstiioon.. actions
anv.actions) dim=1)
# Penalize dofs close to limits
out of limits (envv _po env dof _om___imstm_s ()).clip(maxeer out out_of_limge (env.dof_po) env. env.dof_pos [ipps_siittta,,. 1]). clip (min=0.) c11i(miin==..
-6.0 torch. sorch.sumoout_or. limits, dim=1)
# Penalize base height away from target Sarget_heights 0.34
ease_heigh env onv.root_statar srootsstates[; 2]
get the ground height of the terrain
ground_x x sorrooottttteei,..... 1] ground_y env.
0]
ground_z env. wwr._get_terrat
seciiaa__iighhttttteee (ground,, ground_y)
calculate the base-to-grous height
base2ground_heiigee base mase_height ground_z
heighttreward rewardrewardddd = 000001 torch. square (baseegruuud height Sarget_height,
stumbling penalty stumble (torch.norm(e, wow.oeeo.ooooottttttttooooo.a [frrecttfoorces:; [;, env. env.feet_ing -feet_indices, :2], dim=2)
5.) (torch.abe (env. contact_forcest env. env. anv.feet_indicee indices, 2]) 1.)
# stumble reward = -4.0 torch. sum (stumble, dim=1)
# Combine reward components to compute the total reward in this step total_reward
reaard total_ reward Baracking_lin, lin_velll__eeeeeeeeeeeeeeeeellnl vel_reward + trackinggang ang_ reward
lin vel z reward
ang_ vel_eey_leeeeeeeeeeee reward torques_rent _eward dof acc_ reward rof_ard_rewwrd_ collision reward + action ret__raat_eeerrrd_1
feet air timettimee_eeeee reward
dof ltt__mmmittsttss li__liismt__rrrddddaaaa + neight_rewards
stumble (embble_rewar)
Debug information
reward_ cowpon__ompsnetss
(""reciiggll__vvarrrvaddd tracking_lin_
Saracking_ang__ee_rree reward"'
boccking_ang_e___ee
"lin togg_vel_xy_reeeee ang_ vel_xy__rewrrd,
vel_eel_2___ vel_z_reewrrd
Etorques_rewarde parquesreeardd
"dof acc aeljacccrewaaee dof acc pacc_re
52

---

Published in Transactions on Machine Learning Research (09/2025)
"feet_air_ teerrddi__rewrrd. feet feet_air_tte simeetime___e reward,
"collision_revars collision_rewars reward,
"action "action_rate rraonnnrttt__e retrrion_rate_rrarrd.. action_rate retiion
"dof ciof_pos_llmiiee sewartdd__wwadd dof_pos _od___p___miis____________ _eenit,
Theight_rewards height_reward,)
return @etal_reward, reward_ compoeetnmm
Reward for Jump high
def compute compute_reward reelf:
Compute rewards for the backflip task env self.env Do not skip this line. Afterwards, env www.ametaeeerrccmmm name}
to access parameters of the environment.
Penalize angular velocity around x-axis (roll) and z-axis (yaw)
role_rate env www.root_statellie 10] Angular velocity around x-axis (roll)
yaw_ env. proot_states(; 12] Angular velocity around z-axis (yaw)
roll [ill_rate_rit Soll_rate_reward reward -0.05
torch. square (roll_rate) # Penalize deviation in
roll, org -5.0, 0.5
aew_rate_reward -0.2 square (yaw_rate)
# Penalize deviation in yaw,
org -5.0, -2.0
Penalize roll and yaw around x-axis (roll) and z-axis (yaw) roll env.rpy [ 0]
pitch env.rpy[:, 1]
yaw rny.rpyl: 2]
# Penalize deviation in roll pitch_reward = -0.2 torch. (pitch) # Penalize deviation in pitch
toll_reward = -0.2 torch. (ro1))
yaw yaw_reward -1.0 * torch. (yaw) Penalize deviation in yaw
Encourage axis base linear velocity 1in vel _a_l___ree reward 0.4 norch.square
conaareeenvrootsttttte. hwwp:/ooot_sseee stateetttsssssss.....t 9])
Penalize x axis base linear velocity forward
lin lin_vel vel x reward -0.4 torch. torch.square square
(torch. squuuretuuttttcch
relu (env root_ stoot_sstttes;. 7)))
Penalize torques perques_reward -0.0005 torch.suml (torchh square (env. wew.torques). dim=1)
Penalize dof accelerations
dof a___cc_rreeeee reward = -e-7-- torch. sumh (torch. square (onare((en)) ((env lass_lasdddoo dof vel
[nv.dof_vel) / env. adv, dim=1)
Penalize dofs close to limits
out limits - -envv pos env enf dof_pos_____________eep___e_ed__ pas_pps__mmmme limitsms 0]) clip(maxx:..)
out out_of limits out_of_limits (env. dof pos env dof faa__po__lmitte [:, 1)) clip(min=0.
-7.0 * torch. sum (out_ limits, dim=1)
Penalize base torso hitting the ground
53

---

Published in Transactions on Machine Learning Research (09/2025)
base_hit _iit_go torch. any norm (env. contact crotaactforces__s ;], dim=-1) > 0.1, dim=1) >1.0
env. www.errnaaatooootttttttiiiiitoo
- originally
base www.hiitggroud_eeeeee
-20.0 Dase_hit_eee
# 10
Penalize flat base orientation
gravity_rewa = -0.01 torch. sum (torch. square (env. projjected_gravityy:: :2]),
dim=1)
Reward for highest z position reached in an episode height_history history env env. height history buff buf height_ higgh_hhsttoryreeaaaaa
1.0 height_history
0.5
Combine reward components to compute the total reward in this step ___rtteeeewwrr roll_reward + pitch_ reward
total_reward [roll_rate pester
yay yaware + lin vel_ reward lin_vel_ x_rewardd
torques corques_reward + dof cef_acc_rewarre base_hi saae_hittgrruud_reaaaa +
gravity_reward height_ hitprpiitttttyraeeeeee reward)
Debug information
=
"roll_ ratee_ reward": : roll __atrrtte______ reward,
"yaw_rate reward": = yaw_rate _eate_r
roll_reward,
"pitch_reward"
"yaw_reward", yaw_reward,
"lin_vel_z_rewae
"lin_vel "eel_x_rearrd" lin_vel_x__ reward,
"torques_reward" jorques_reward,
"dof aac_reearrd" dof_acc_revard,
"base Saaaeehit___e ptp__hit__gooooe hait_ground_eeward" base hwtphhgoouddreeaaaeeee
"gravity_rewwee jewardd" gravity_rewwe bewardd
srightt_hitttry_eaaaae
Biight_histooy_rewaall
}
return rotal_reward,
Reward for Backflip
def compute_reee rself::
и** Compute rewards for backflip task .".
self.env Do not skip this line. Afterwards, www.rpppatrraaaaee name]
to access parameters of the environment.
Penalize angular velocity around x-axis (roll) and z-axis (yaw)
roll_rate env www.root_states.: 10] Angular velocity around x-axis (roll)
yaw_rate enw.root_states/e 12] Angular velocity around z-axis (yaw)
roll_rate, Boll_rate_reward -0.05 torch.
Penalize deviation
+ roll, org -5.0, 0.5
Baw_rate_rewant -0.2 torch. square (yan_rate) Penalize deviation in yaw, org -5.0, -2.0
54

---

Published in Transactions on Machine Learning Research (09/2025)
Penalize roll and yaw around x-axis (roll) and z-axis (yaw) roll env. env.rpyl:, 0]
yaw env. 2]
roll_reward -0.2 torch. (rulre
Penalize deviation roll
yaw raw_reward -1.0 torch.squee square (yaw) Penalize deviation in yaw
Reward for angular velocity around y-axis (pitch), encourage backflip rotation
pitch_rate env. base ang_vell: 1] # Angular velocity around y-axis (pitch)
Clamp the pitch rate to minimum of -7 rad/s (maximum negative rotation - speed)
clamped 1lamped_pitch pitch rate torch. clampppitchh rate, max=5.0, min=-7.0)
Compute the backflip reward pitch_rate_res
torch.where( where( = torch.
clamped_pitch_ rate 0,
rate), ; -0.1 clamped_pitch
1.0 * (-clamped_piteel pitcc_tttheeeee
# Positive reward for negative pitch rate # Negative penalty for positive pitch rate
pitch_raaee__ppppep rate_reward 1.0 [-clamped_pitch_raatee (aattchh_att))_ Reward negative pitch rates
Reward for pitch frontwards for the back flip pitch env. env.rpy[:, 1]
last_pitch env. wastlast_.. last__ry[:;, 1]
delta_pitch (pitch last aas_pitthi porch.pi) % (2 torch.pi) torch.pi Dolta_pitch_rewarre torch. where (delta_pitch delta_pitch,
torch.zeros hiiero__iiieddeetappttthhl
delta_pitch_ree reward 20.0 Baata_pitch_rerr reward
Encourage z axis base linear velocity lin_ _enl_eel____ z reward 0.2 sprch.ssuaeeenn..roo
9])
Penalize torques torques_rewase reward -0.0005
(torch. square (env. (env.torques), dim=1)
Penalize dof accelerations dof __f_acc__ree reward _of_acc_reward -1.0e-7 torch. sum (torch, square ((env. wwsttllatt_eoe caat__df__ree vel
env dof vel) / env env.dt) dt), dim=1)
Penalize dofs close to limits out_ož of limits - -env. ppos env dof cof__oos_iiie mos_limits[; 0]) clip(max=0.) of limits
_immits += (env dof _pos env dof dof_pos___aieeceooeeepeeeeesool_ lespsos_lmmle moo__imits:[; 1]).cllt clicimmiiiii
dof pof__po__lmiis____ lirartseetterdddddd = -7.0 * torch. sum (out_ of limits, dim=1)
Penalize base torso hitting the ground
base hase_hit_ground torch. torch.any (torch norm (env. contact Sentacttforces_::
env. www.termination_come www.coontttnniiiccc originally
:], dim=-1) 0.1, dim=1) >1.0
base hit_ hhi_ggound__aeeeeeeeoohnnngg wewgggrrrowarrrrre reward -20.0 * base hit hit_ground # 10
Penalize projected gravity along axis
55

---

Published in Transactions on Machine Learning Research (09/2025)
gravity_reward = -0.01 torch. Bercch.quaeeeeee (prr 1])
Encourage backwards rotation the projected gravity
delta_gravity_ env. euv.projecte__graittyl. 0] env www.:last_ppooo htt.:aaat_pojettee__aa.tty
0] condition projected_rravitty; 2] Corrected the z-axis per your description desired_directine direction = sarch.were( condition,
-1.0, # Encourage decrease in delta_ gravvity
1.0 Encourage increase in delta_gravity.x
Compute the alignment between desired and actual change al alignment desired desireeddirection * delta_gravity_x
Compute the reward with different scaling factors rotate weteoegaavvtttrrrrrrr torch. where
alignment 0,
20.0 torch. abrch.ab (deltaa _),, # Positive reward Negative penalty
-2.0 torch borchhabb
)
Reward for highest 2 position reached in an episode height giisttryy env height history buf
0.8 neight_histony
0.5
Combine reward components to compute the total reward in this step
retard (roll_ mate_reward + ratteeeewwwdd + roll_reward + yaw_reward + rertch_rate_rewardd
delta_pitch__ewark lin_vel__ 2e torques_revard
caf_acc_rewaare
reward gravity_reward rotate_ notate_gravity_reaar
Debug information
cowardd_omponentt =
"roll_rate "aa__rraterewardd roll_ Balllrrtt____
reward,
"yaw_rate_rr rewardde we__raterrewaar
"roll_rewarde roll_ roll_reward,
"yaw_remard": yaw_reward,
"pltch_ratee_evarre"
caapita_pitch__reard
caltapitcc_reeward
"lin "lin_vel_z_rewarre sin_vel_z_revae
"torques_revard": torques_revard,
"dof "dof_acccreward": dof_ dof_acc_reward,
"base_ "baae_iit_grrouddrrwwrr'""" base
"gravityy_reward": gravity_reward,
"rotate.ggrvvtyyreeard"" votate_gravity_reward,
"height_historry_ewwrdd" Noight_historyyrrete rewardd
56

---

Published in Transactions on Machine Learning Research (09/2025)
return Setal_reward, reward_ component
Reward for Kettle
def kettle_ heeeec/cnppptteeeeeeeeeeeeeeeeee
torch. Tensor, bucket saaitt_handle__p__ torch, Tensor, left hai_hand__ops _pss: torch. Tensor, right_hand__t _pss torch. Tensor, left Soft_hand_ig ff _ps: torch.Tnnson Light_hand__e right hand ff fos: torch, Tensor
Tuple [torch. Tensor, [ict[str, torch. torch.Tensor. Tensor]]
Redefine proximity reward for less dominance kettle_ _erile_tt_uucke_eee _ torch. norm norn(ketleeespouu dim=1)
bucket _aaett_hande__pss.__
temp_ crmppppoximity 1.0
proximity_reward torch. exp wercchxxppketteeep... htapclleettouuutte_eeeeeeeeeee / cemp_proximity)
Rescale grip rewards and refine detection logic left_ wtt.grrppdddsseeee torch. norm (1erm hand ff pos kettle lottlle_spuu____e _pss, dim=1)
= torch normmright, hand ff_ ffs_ pucket_handie __pss, dim=1)
temp_grip 0.5
seft_grip_rewane torch. exrch.exp(leett_eee wtt_ggggpppdiaaiaeee distancc temp temp_grip .ripp
reward torch, purch.ep((11) exp(-right_grin beiigghtgrip_dsseeeet distance temp_grip)
Improved task-specific reward detection logic
task las___uccesss_iiiccess torch.tenson tensor ([0], device=kattle_spot,, device) #
Replace with task success condition
task wtwpcppeeeionntreehheeeeeeeeeeee
0.1 Define threshold to reflect task
completion
task_comple complete kettle to bucket_distance
task
distance task_completion transformed, task_ score cask_complete.eo (
Total reward balancing individual components
total reward
0.2 proximity_reward +
0.3 left_grip_reward +
0.3 left_grip
1.0 * transformed, task score
Reward components provided for diagnosis
reward_ componettmmoonetss
"kettle_ene htt___ppoot_poiitttteoeo proximity_rewarde
"kettle Shettle_handle_rrie handle_grip": left_grip.reward,
"bucket ceaet__hand___p__ee right_grip_reward
transformed_task..ee
57

---

Published in Transactions on Machine Learning Research (09/2025)
return reward_ conpoeenmm
Reward for Hand Over
def hand over_ reeatdee_eewadddaboeaeeo torch. Tensor,
object_rots torch. Tensor, object_ linee_linvel: torch. Tensor, goal_pos torch.Tensor, goal_rot: torch.Tensor, fingertip_pos. Tensor, fingertip_anoon another_poo _poo: torch.Tensor Tensor -> Tuple [torch. Tensor, [ict torch. [ensor]]]
Constants
distance threshold: float 0.03
rotation thrrssolddd float 0.1
retch_revard_weighh float = 30.0
toss_rewarte re___reward_eeeee weight: float 10.0
toss tosp_teepe: float 1.5
catch_ tetchtteepp float 0.5
penalty_neight: float 10.0
Compute distance and rotation differences between object and goal obj abject_aoe distance = torch. corbjjjettt pos goal_pos dim=-1) coaet__gal_rotatiooo diff
(objecttbbtt_ro goal rot, dim=-1)
Reward for object being close the goal position and rotation toss reward = torch. erciheeepppptttt (-toss reward weight * (object_goal distance voss_temperature)
Calculate the catch reward based on the difference between the object' linear velocity and the goal position
catchhreward torch. noroooollogg object_linvel, dim=-1) catchhreward torch. exp herraacchreear_eeeeeeeee catch_ )asnch_tempeatuue))
(catch_rewary
Penalty for direct rolling or touching the target instead of tossing and catching
penalty = torch. (oweeeoeeeeeeooooee [oa_ljcct_gal_ddsttacee)) distance)
for (anggefineertipppe pttpeerrttppppssssapaeee shape
cest_to_fingerii] torch norm (object _poccttppp_______ fingertip_pos.e i, :], dim=-1)
tist_to_fingert another torch. norm (object_poo
http:rttippanooherrooooeoo i, :], dim=-1)
penalty = torch. torch.where(
(dist_to_f ses__toffingertae < thrassolltr (dist_to to celat_to_fingrreee Singertip_anotiin <
distance threshold),
penalty 1.0, penalty
penalty_ratia torch. (pealid menalty_weight * 0.5
penalty = catch_ reward penalty_ratee ratio # Adapt the penalty to the
catch_ reward's s dynamic range
58

---

Published in Transactions on Machine Learning Research (09/2025)
Total reward
total_ total_reward reward toss_reward reward + catch_ reward penalty
reward_ terms {"toss_ _eoa__de toss toss_reward, "catch_reward" catch_reward,
"penalty": penalty)
return total_ reward reward_ terms
Reward for Swing Cup
def wttp:cccccccppteeereeeeeee
torch. Tensor, object_ torch. Tensor, object :invel cup_ri uup_right_ wrigghthhnnnle_pppprrrre www_left_handle__pe torch. Tensor, left_hand Torsor
Norch.Tensor
right settt_hand_oss- torch. Tensor,
goal_pos: torch.Tensor,
goal_rot: torch.
dim=-1) ) -> Tuple[torch, Tensor, Dict torch. Tensor]]: object_goale distance distance distance_reward temperature
_ = torch. horrch..rmmobjet_pppp
temrrrreerreteeeruuuueeeeeeeee = 0.1
object poact_goal_distane
reward_ weight 0.1
object_goas diataaddittneeeemmm geaett_goalddistaceeeeeoooe distance) (oaet__gal_ddstanee)
@eward reward -torch. exp (-distance (-disstance_revardd rtwpaeeeteeeeereeeeeeeeeeeeeeeee temperature
object_goal.
object_goad _eward weight
right_cup_
ctt__cuu_hanndeeee hwwdle_nddlae..l. dist torch, norm
handle _pos Bight_hand_pon
dim=-1)
_ist = torch. normmmummmuueeettt http__lletthhhheeeeeeeeeeeaa left Saan___and___ pand_pos, dim=-1)
cup cup_orientation -inff torch. sum (torch, mul (objectt_rro, goal_rot) dim=-1)
cup cup.oorientaatooo.eeaarrrvvvepg 1.
cup cap_orrlentaiion_eewardd (ccup_crieeeniiipeeeeee www.onaatiinnnoooom. diff cup_orientation_rwaara rewarrrddiiirrrrrr_aaaa_ear weight))
grasp_tempant http__tteerrauure__p 0.25
grasp_ 2trp_pateerr 0.25
right_grasp_r torch. expp(rgraapptemperrtuueeee 1 right, cup_ hap_handle,__ _dstt
left_grasp_reete reward torch. worchhexp((eee) exp(-grasp___ae. wwsppggrapp_tempeeeeeeeeeee
temperature 2 left_ left_cup happ_aande___dee dist) ctp___up___aeeee
grasp_reward Fright_grasp_reewe
left_grasp_reet reward 1.0)
sap_linvel_none norm torch. normmojeet _invel, dim=-1)
hta_iinvel__enaltyy_veeeeee 0.2
cup_linvel_penalty (cup_ liup_linvel_nore norm cup_Iinvel_.enatyy.eeeee weightt
touch_ couch_rewwrd_eempprrtuuuee 0.25
Bauch_reward_weighhe 0. 125
touch_reward (torch. exp (-touch reward_waeeee temperatureer right_ cup handhhannleeeee (
exp (-touch reward temperature * left_ cup_cupppaae_e handle handle_dist) - 1.
Bauch_reward_weiie
59

---

Published in Transactions on Machine Learning Research (09/2025)
total_reward grasp_reward cup_ pp__orrenttiionn _ewarrdreeaarrrrrr + cup pelinvel_penallt. souch_rewand
reward_dict
(grasp_rewarde grasp_reward,
"object_goad aisstancerreaardi object_gee _aall_ggal__aaa _el__ddstance_eee reward,
"cup_ orientation " cup sup_orientation_eet reward,
Siup_linvel_ae painvell_enaly" cup_ hop_linvel_pene siavel_penatt,,
Stouch_rewarde souch_reward
return total_reward, reward_dict
A.4 Preference Reward vs. Environment Reward
Env vs Pref Reward
Reward
-2
Environment Reward
Preference Reward
200
400
600
800
1000
1200
1400
Iteration
Figure 15: Preference Reward vs. Environment Reward. Comparison of the preference reward and the environment reward of the Backflip task throughout the training process.
To understand the effect of the preference reward and the environment reward in the training process, we take the Backflip task as an example and visualize them Figure 15. The overall trends of both rewards are similar, growing from lower values and converging to higher values. The environment reward is relatively more continuous, showing its effectiveness of stabilizing the training and ensuring the basic performance. The preference reward has periodic jumps, indicating the variation of the priority of the preference criteria throughout the training process. This ficxible and dynamic reward signal enables the success in very challenging lasks. In sunmary, both types of rewards are essential for stable and effective learning.
A.5 Reward Predictor Training Cost
In order understand the extra cost training the reward predictor in LAPP, we take the Backflip task an example and analyze the time consumption of training the robot policy, reward predictor, and generating the preference labels with the LLM. The training process NVIDIA RTX A6000 GPU. As shown
60

---

Published in Transactions on Machine Learning Research (09/2025)
Cost
Policy Training
Predictor Training
LLM Labeling
Time (socond) Percentage
11237
1995
2333
72.2%
12.8%
15.0%
Table 1: Cost of training the reward predictor in the Backflip task.
in Table [ the policy Training takes 72.2% of the total training time, which is the majority. The reward
predictor training takes 12.8% the time and the LLM Labeling process takes 15.0%. This result shows that extra cost training the reward predictor in LAPP the minority of the entire training cost. With the future advancement of the LLM responding efficiency, we believe the LLM Labeling time will be further shorten, making the extra cost more neglectable.
61


Reviews:
[{"review_id": "PG1cT7ThnH", "reviewer": "Reviewer_7jqc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The proposed LAPP framework effectively integrates LLM-generated feedback into the RL loop, bypassing the need for costly human labeling or complex reward engineering. It makes the whole training process more scalable and effective.\n- The empirical experiments demonstrates faster convergence and higher final performance across diverse robotic tasks, including manipulation and locomotion.\n\n**Weaknesses**\n- The choice of the LLM seems to be essential. The paper only adopts GPT-4o and GPT-4o-mini as the backbone model. It is highly recommended to incorporate more popular open-sourced models, like the Llama series, Qwen series or Deepseek series, into the experiments to show the effectiveness of LAPP across models.\n- As indicated by the authors in the limitation section, the current framework still requires manual selection of state variables. Therefore, the human efforts are not completely avoided in the training loop.\n- The framework requires updating the preference model every $M$ epoch, it is appreciated if this additional computational overhead is included in the final result.\n- The ablation study can be further extended. For instance, \n\t- How the error probability $15\\%$ of LLM is selected? It is a model-specific parameter and requires further validations.\n\t- I am curious how the $\\beta=50$ is determined in the reward design. It seems the preference feedback is dominating the reward, then why not remove the environmental reward $r_E$? Is the choice of $\\beta$ task-dependent?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces LAPP (Large Language Model-Assisted Preference Prediction), a framework that uses the feedback of LLMs to train the preference models to generate the preference feedback over the trajectories of reinforcement learning. It aims to reduce the dependence of human annotations during the reinforcement learning process, so that the whole learning process can be more efficient and scalable. Both the policy and the preference model are updated iteratively during the training process. Empirical studies are carried out to demonstrate LAPP is superior compared to the baselines in terms of training efficiency, convergence speed, behavior control.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "None.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jFOBYeOsGV", "reviewer": "Reviewer_Rgnk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths \n1. The proposed idea is both interesting and novel. The online learning procedure effectively leverages the capabilities of LLMs, transferring their rich knowledge into robot learning.\n2. The writing is clear and comprehensive. The paper first introduces the fundamentals of reinforcement learning, and then clearly explains the motivation and details of each module.\n\nWeaknesses\n1. Figure 2 could be improved for better clarity and visual appeal. Additionally, the diagram of the Transformer Reward Predictor currently resembles that of an MLP network, which may cause confusion.\n2. Given the numerous hyperparameters involved in the proposed method, the authors should explain how the optimal values are selected and provide a sensitivity analysis.\n3. The statements regarding the advantages are not sufficiently precise. For example, the abstract mentions \"efficient, customizable behavior acquisition,\" but the experimental results primarily demonstrate faster convergence in terms of epochs. Compared to vanilla PPO, the proposed method leverages additional components such as the LLM and preference predictor, potentially leading to differences in computational time and resources. Therefore, the claim of efficiency may not be fully justified. Furthermore, the experiments do not clearly demonstrate the \"customizable\" aspect.\n4. The experimental evaluation should include more LLMs, such as Llama and DeepSeek, to better illustrate the effects of different language models.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces Large Language Model-Assisted Preference Prediction (LAPP), a novel robot learning framework that leverages feedback from large language models (LLMs) to guide the training of both the preference predictor and the policy network. At regular intervals, new trajectories are collected and evaluated by the LLM, which are then used to retrain the preference predictor. This forms an online learning strategy that progressively improves the policy over time. Experimental results demonstrate the effectiveness of the proposed method across a variety of tasks, achieving rapid convergence and superior performance compared to baseline approaches.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the weaknesses.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "cFp8uhGthM", "reviewer": "Reviewer_kmca", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Sufficient Experimentation: The paper evaluates LAPP on a diverse and challenging set of tasks, including quadruped locomotion over various terrains, high-DoF dexterous manipulation, and the challenging backflip task.\n2. Significant Results: The experimental results strongly support LAPP's effectiveness. It not only improves training efficiency and final performance on multiple tasks but also successfully achieves fine-grained control over robot behaviors via high-level language instructions (e.g., gait patterns and cadence). Furthermore, it solves the backflip task, which traditional methods struggle with.\n\nWeaknesses & Questions:\n1. Inconsistent Writing: The Authors use LAPP in most places but use Lapp occasionally and are advised to further polish the writing.\n2. Incomplete Related Work: In the first section of related work (i.e., Foundation Models for Robotics), the authors fail to give a brief introduction. I advise authors to reorganize the related work section as it is not clear enough and it shares overlap with the introduction section when it comes to the details of LAPP method.\n3. Why do authors use ensemble learning strategy when training the preference predictor? Authors mentioned that it could increase robustness but I wonder if training multiple independent predictors help. If there is any evidence back up?\n4. Can authors analyze more about the two rewards used? For example, their distinct effect on the results, their correlation and intuitive interpretation?\n5. Is the title very appropriate? It does not convey any robot learning related information.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "1. LAPP proposes a novel framework of using LLMs to generate feedback for preference signals in robot learning. This process mitigates issues introduced by human annotation.\n2. LAPP demonstrates through experiments that it outperforms state-of-the-art baselines in training speed, final performance, adaptation efficiency, and controllability of high-level behaviors across a range of complex robotic tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address the Weaknesses & Questions mentioned above.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZV402XQtsl", "reviewer": "Reviewer_VNHv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. The proposed method to use LLMs to augment RL training with generated preferences seems to be reasonable\n2. Experiments show the proposed method is effective compared with the PPO baseline\n\n**Weaknesses**\n1. The key differences of the proposed *framework* compared to RL-VLM-F was not that clear \n  - The reviewer believes that the key difference at the framework level is that this work also incorporate manually designed environment rewards, while RL-VLM-F does not.  Is that the case? \n2. Training efficiency is measured mainly by the number of epochs\n  - There is limited discussion of the overhead of the using the LLM to generate the preferences and training of the reward model.  The cost (e.g. time, compute, memory) should be included in comparisons and discussions.\n3. Despite the initial discussion of the challenges and limitations of reward function design, it seems that the proposed framework actually introduces an additional component that needs to be designed as it requires the following:\n  1) designing the prompt for input to the LLM\n  2) writing out the code for the explicit environment reward \n  It is not clear whether 2) is different from manual reward function design, but now there is an step of designing the prompt.\n4. Limited comparisons and ablations\n  - The reviewer would have liked to see a comparison with RL-VLM-F / other recent methods that uses LLMs for preference modeling \n  - The work uses an ensemble of M preference predictor networks, there should be an ablation on the this  \n5. It is difficult to determine the quality of the results - videos should be included\n6. Some claims are not fully substantiated / accurate \n  - \"Unlike prior approaches, LAPP integrates the LLM-generated feedback directly into the RL loop\" \n     - The reviewer believes that RL-VLM-F also integrates the LLM-generated feedback directly\n  - the framework \"enables robots to learn efficient, customizable, and expressive behaviors from human language specifications\" \n     - There is limited evidence of this - the efficiency, customizability, and expressiveness of behaviors are not evaluated", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents Large Language Model-Assisted Preference Prediction (LAPP), which 1) uses LLMs to generate preference labels for state-action trajectories, 2) use the generated data to train a reward predictor, 3) the reward predictor is then used to train a policy that maximizes both the original rewards from the environment and the predicted preference rewards.\n\nThis works aims to address the following limitations of prior work that uses LLMs for generating preference labels are that 1) those works are limited to simple tasks with low-dimensional action spaces, 2) they assume a Markovian decision process, and 3) do not integrate LLM-generated feedback directly into the RL loop.  To tackle these, the work is evaluated on complex control tasks (1), use transformers for non-Markovian reward preference prediction (2), and direct integration of the LLM-generated feedback into the RL loop (3). \n\nThe proposed framework is evaluated on control tasks with quadruped locomotion / backflip and dexterous manipulation, with comparisons to baselines (PPO, Eureka) and ablations (on sampled trajectories, use of MLP vs transformer-based network for the reward predictor network).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Improve presentation to clarify the key difference of this work wrt prior work such as RL-VLM-F.  \n   - Figure 1 can be replaced, or Figure 2 updated to show the key elements of this work that is different from prior work on using LLMs for generating preference data and modeling preferences\n2. Add a clear discussion of what parts would a person need to specify for this framework \n3. Add more discussion and comparison of the overhead of using LLMs for generating preference data and cost of training the ensemble of reward models\n4. Ablation of the number of ensembled models (including M=1)\n5. Add discussion of hyperparameter selection (e.g. error rate for Eq 4 set to $\\epsilon = 0.15$, how is that value determined? Selection of M for number of epochs for retraining the preference model in section 4.3, etc)\n6. Add videos to allow for better comparisons of the learned behaviors\n7. Other writing improvements\n   - Either tone down and adjust claims\n   - Add comparison of tasks handled in prior work using LLMs (e.g. why are those tasks less complex, what is the action space) vs this work.\n   - Avoid using of the same letter for different things (e.g. M is used both for the number of models that are ensembled in section 4.2, and the number of epochs in section 4.3)\n   - Shorten main paper (the text can be made more concise with some details, e.g. Algorithms and some experiments can potentially be moved to the appendix)\n   - \"Consequently, We have:\" => \"We also have:\"", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "nUq4ZRkKeB", "reviewer": "Reviewer_68np", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n* The presentation is clear, and the paper is well-written.\n* The experiment supports the effectiveness of the method.\n\nWeakness:\n* As the LLM could judge the reward, why additionally train a reward Transformer to generate reward?\n* For Figure 5,the LAPP convergence speed may be lower than PPO at the early stage. Any explanation?\n* The detailed prompt example should be provided.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work uses LLM to generate rewards and then uses Transformer for reward generator. The experiment prove the efffectiness of this work.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "N/A", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)