Language to Rewards for Robotic Skill Synthesis
Authors: Wenhao Yu
Nimrod Gileadi
Chuyuan Fu
Sean Kirmani
Kuang-Huei Lee
Montserrat Gonzalez Arenas
Hao-Tien Lewis Chiang
Tom Erez
Leonard Hasenclever
Jan Humplik
brian ichter
Ted Xiao
Peng Xu
Andy Zeng
Tingnan Zhang
Nicolas Heess
Dorsa Sadigh
Jie Tan
Yuval Tassa
Fei Xia
Venue: CoRL 2023 Oral (2023)
Keywords: Large language model (LLM)
Low-level skill learning
Legged locomotion
Dexterous manipulation

Abstract: Large language models (LLMs) have demonstrated exciting progress in acquiring diverse new capabilities through in-context learning, ranging from logical reasoning to code-writing. Robotics researchers have also explored using LLMs to advance the capabilities of robotic control. However, since low-level robot actions are hardware-dependent and underrepresented in LLM training corpora, existing efforts in applying LLMs to robotics have largely treated LLMs as semantic planners or relied on human-engineered control primitives to interface with the robot. On the other hand, reward functions are shown to be flexible representations that can be optimized for control policies to achieve diverse tasks, while their semantic richness makes them suitable to be specified by LLMs.
In this work, we introduce a new paradigm that harnesses this realization by utilizing LLMs to define reward parameters that can be optimized and accomplish variety of robotic tasks. Using reward as the intermediate interface generated by LLMs, we can effectively bridge the gap between high-level language instructions or corrections to low-level robot actions. Meanwhile, combining this with a real-time optimizer, MuJoCo MPC, empowers an interactive behavior creation experience where users can immediately observe the results and provide feedback to the system.
To systematically evaluate the performance of our proposed method, we designed a total of 17 tasks for a simulated quadruped robot and a dexterous manipulator robot. We demonstrate that our proposed method reliably tackles 90% of the designed tasks, while a baseline using primitive skills as the interface with Code-as-policies achieves 50% of the tasks.
We further validated our method on a real robot arm where complex manipulation skills such as non-prehensile pushing emerge through our interactive system.

Language to Rewards for Robotic Skill Synthesis
Wenhao Yu; Nimrod Gileadi', Chuyuan Fu, Sean Kirmani', Kuang- Sueng Leel, Montse Gonzalez Arenas, Hao-Tien Lewis Chiang, Tom Erez, Leonard Hasenclever, Jan Humplik, Brian lchter, Ted Xiao, Peng Xu, Andy Zeng, Tingnan Zhang, Nicolas Heess, Dorsa Sadigh, Jie Tan, Yuval Tassa, Fei Xia Google DeepMind
Large language models (LLMs) have demonstrated exciting progress in acquiring diverse new capabilities through in-context learning, ranging from logical rea- soning to code-writing. Robotics researchers have also explored using L.LMs to advance the capabilities of robotic control. However, since low-level robot actions are hardware- dependent and underrepresented LLM training corpora, existing efforts in applying LLMs to robotics have largely treated LLMs as semantic planners or relied on human- engineered primitives interface with the robot. On the other hand, reward function is a flexible representation that can be optimized for control policies to achieve diverse tasks, while their semantic richness makes them suitable to be specified by LLMs. this work, we introduce a new paradigm that hamnsssss this realization by utilizing LLMs to define reward parameters that can be optimized and accomplish variety of robotic tasks. Using reward as the intermediate interface, we can effectively bridge the gap between high- level language instructions to low-level robot actions. Meanwhile, combining this with a real-time optimizer, MuloCo MPC, empowers an interactive behavior creation experi- ence where users can immediately observe the results and provide feedback to the system. To systematically evaluate the performance of our proposed method, we designed a total of 17 tasks for simulated quadruped robot and a dexterous manipulator robot. We demonstrate that our proposed method reliably tackles 90% of the designed tasks, while a baseline using primitive skills as the interface with Code-as as policies achieves 50% of the tasks. We further validated our method on a real robot arm where complex ma- nipulation skills such as non- oon-prehensile pushing emerge through our interactive system.
Abstract:
Keywords: Large language model (LLM), Low-level skill learning, Legged locomotion, Dexterous manipulation
1
Introduction
The recent advancements in large language models (LLMs) pretrained on extensive internet data [1, 2] have revolutionized the ability to interpret and act user inputs in natural language. These LLMs exhibit remarkable adaptability to new contexts (such as APIs [3], task descriptions [4], or textual feedback [5]), empowering tasks from logical reasoning [6, 7] to code generation [8]. These diverse applications have extended to robotics as well, where substantial progress has been made in using LLMs to drive robot behaviors [5, 4, 9, 10, I]: from step-by-step planning [4, 9, 12], goal-oriented dialogue [10, 11], to robot- code-writing agents [3, 13]. While they impart new modes of generalization, they focus on using language concatenate together new behaviors from an existing library of control primitives that are either manually- engineered or leamedd priori. Despite having internal knowledge about robot motions, LLMs struggle with directly outputting low-level robot commands due to the limited availability of relevant training data (Fig. 1). As result, the expression of these methods are bottlenecked by the breadth of the available primitives, the design of which often requires extensive expert knowledge or massive data collection [14, 15, 16).
To tackle these challenges, we need to operate at a level of abstraction that allows harnessing the capabilities offered by LLMs. Our key insight is to use reward functions as an interface that bridges the gap between
"Co-first authors, equal contribution
1 'Core contributors
i Corresponding emails: {magicmelon, lon, wiiiiroddxiifeiiaggogogll. See Contributions in Appendix A.1
7th Conference on Robot Leaming (02R),2023), Atlanta, USA.

---

User
User
User
Make robot dog
Make dog stand up on two foot.
Make robot dog
Reward Translator (LLM)
stand up on two feet
stand up on two feet.
Ressee oode
LLM
LLM
Set torso rewards
set, torso__ hwt.rrooowarrsssiiiio....oo www.rdshhiiggh-00 picchnnp.eggrra0000)
Melion description
The robort ding's torap uprignt balenced over it' hind feed, which are shoulder width spert. The front lgga hang locsely, polsed mid-us mimicking a human's arma.
#t Sot foot rewards
Low-lvel
set, feet rewerdel Tront left, height-077)
set joins LergetfoO. 0.2 07. 00, -033 08, D.,, 02 0., -0.3, 0.6)
set_feet__ps. leeeeeesssreeereeeee rewwardddddssst back__eftt height=OO))
fett_fnet__te postt_oosrrwarrctee front, right, height-(17)
set, pes_t_pss_re_eeeee_ rewerdel back height=O.D)
Optimized lowe level actions
Motion Controller
Figure I: LLMss have some internal knowledge about robot motions, but cannot direetly translate them into actions (left). Low-level action code can be exccuted robots, but LLMs know little about them (mid). We bridge this gap by propos- ing a system (right) consisting of the Reward Translator that interprets the user input and provides reward specification, which is then consumed by a Motion Controller that interactively synthesizes a robot motion given the reward.
language and low-level robot actions. This is motivated by the fact that language instructions from humans often tend to describe behavioral outcomes instead of low-level behavioral details (e.g. "robot standing up" versus "applying 15 Nm to hip motor"), and therefore we posit that it would be easier to connect instructions to rewards than low-level actions given the richness of semantics in rewards. In addition, reward terms usually modular and compositional, which enables concise representations of complex behaviors, goals, and constraints. This modularity further creates an opportunity for the user to interactively steer the robot behavior. However, in many previous works in reinforcement learning (RL) or model predictive control (MPC), manual reward design requires extensive domain expertise [17, 18, 19, 20, 21J. This points to a missing link between the reward structures and task specification which is often in natural language. As such, we propose to utilize LLMs to automatically generate rewards, and leverage online optimization techniques to solve them. Concretely, we explore using LLMs to translate task semantics to reward functions, and use MuJoCo MPC, a real- time optimization tool to synthesize robot behavior in real-time| [22]. Thus reward functions generated by LLMS can enable non-technical users to generate and steer novel and intricate robot behaviors without the need for vast amounts of data nor the expertise to engineer low-level primitives.
Across a span of 17 control problems on simulated quadruped and a dexterous manipulator robot, we show that this formulation delivers diverse and challenging locomotion and manipulation skills. Examples include getting a quadruped robot to stand up, asking it to do a moonwalk, or tasking a manipulator with dexterous hand to open faucet. We perform a large-scale evaluation to measure the overall performance of our proposed method. We compare our method to a baseline that uses fixed set of primitive skills and an alter- native formulation of grounding language to reward. We show that our proposed formulation can solve 40% more skills than baselines and is more stable in solving individual skills. We further deploy our approach to real robot manipulator and demonstrate complex manipulation skills through language instructions.
Our work makes the following core contritutions: i) We explore novel interactive framework of using reward function as the interface to bridge language models and low-level robot actions. ii) We introduce a two-layer prompting scheme that effectively improve the reliability of the system in harnessing internal motion knowledge from LLMs. iii) We demonstrate superior performance of our proposed system to baseline methods on two robot embodiments with 17 challenging control tasks and show validation of the approach on robotic hardware.
2 Related Work
Language to Actions. Directly predicting low-level control actions based on a language instruction has been studied using various robot learning frameworks. Early work the language community studied mapping templated language to controllers with temporal logic [23] or learning parser to motion prim- itives [24], while more recent work utilize end-to-end models that produce actions conditioned on natural language descriptions. One example is instruction following methods in navigation [25]. However, they often assume low-dimensional actions navigating from one node of the graph to another [25, 26). To extend the end-to-end approaches to manipulation, common approach is to utilize latent embeddings of language

---

commands as multitask input context, and train with behavioral cloning [14, 27, 16], offline reinforcement leamning [28], coulicondiiioned reinforcement learning [29], or in shared autonomy paradigm [30]. While end-to-end trained policies can be performant, they require significant amount of data in the form of offline datasets or online environment interaction. In contrast, we study less data hungry approach where low-level actions are not directly produced by an end-to-end policy but instead by an optimal controller.
Language to Code. Code generation models have been widely studied both in and outside robotics context [31, 8, 32j. The capability of those models range from solving coding competition questions [33] and benchmarks [34], to drawing simple figures [35], generating policies that solve 2D tasks [36], and complex instruction following tasks [3]. In this work, we study LLMs for generating code for reward functions, and show that the expression of the rewards can lead to expressive low-level policies.
Language to Rewards. The idea of translating natural language instructions to rewards has been explored by several prior work [37,38, 39, 40, 41, 42, 43, 44]. A common strategy in this direction is to train domain- specific reward models that map language instructions to reward values [38, 42, 40] or constraints [39]. Although these methods can achieve challenging language conditioned robotic tasks such as object pushing [39], and drawer opening [40], they require considerable language-labelee data to train the reward model. Recent works investigated using LLMs directly as a reward function for inferring user intentions in negotia- tion games or collaborative human-Al interaction games [37, 44J. By leveraging LLMs to assign reward val- ues during RL training, they demonstrate training agents that are aligned with user intentions and preferences without explicit reward modeling, However, these works receive reward values of rollouts when training RL policies, which requires large number of queries LLMs during training. In contrast, we levrage LLMs to produce a parameterized reward function that can then be optimized. A similar direction to this work is automated parameterization of reward functions, which had been explored in AutoRI. [21], however, they don't provide a language interface. Finally, a concurrent work by Huang et al. explored extracting 3D value maps from large foundation models, which are used by motion planners to perform manipulation tasks [45].
Incorporating Iterative Human Feedback. Correcting plans with iterative language feedback has also been explored in the past. Broad al. enable efficient online corrections using distributed correspondence graphs to ground language [46]. However, this work relies on semantic parser with pre-defined mappings to ground language corrections. More end-to-end approaches have also demonstrated learning a language correction conditioned policy, but they are similarly data hungry and thus fall back to shared autonomy to reduce complexity [47]. Later work explore mapping language corrections to composable cost functions similar to our work by training prediction model from demonstration and apply trajectory optimization to perform control [39]. Followup works further simplifies the system by integrating language corrections to directly modify the waypoints of a trajectory using extensive datasets of paired corrections and demon- strations [48, 49J. In contrast to these prior work, we demonstrate a flexible and data-efficient approach that leverages LLMs to allow for multi-step correction of reward functions based on human feedback.
3 Grounding Language to Actions Using Rewards
3.1 Background and Reward Interface
Our system takes user instruction in natural language and synthesizes corresponding reward functions for the desired motion. We define the reward function in the context of Markov Decision Process (MDP): (S.A.R, P.po). where S is the state space, A is the action space, R: S x A+ IR is the reward function, P:Sx S is the dynamics, and Po is the initial state distribution. Given a reward function R, an optimal controller finds a sequence of actions a = (a) a.r..,r) } that maximizes the expected accumulated reward: J(a.:))= = (a0.
EH R a.), where H is the rollout horizon.
In this work, we assume the reward takes a particular form, suitable for use with MJPC. The reward is the sum of a set of individual terms:
M
R(s.a) = > we ni (Fi(s,,,)))))) 2=0
(1)
where w E R4 is non-negative weight, n(-): R. is a prric-difffeeentiabe norm that takes its minimum at 0, r R a residual term that achieves optimality when 0- and dh is the parameters of the im residual term. For example, if we want to have the robot raise its body height / to desired height, we may design a residual term rn (hh,))) =h- where the reward parameter 4 denotes the desired height, and use the 12 norm to construct the final reward function: RG =
= ww]lr/]l In principle, one may design
3

---

task-specific residual temms that can solve particular controller tasks. However, designing these residuals requires domain expertise and may not generalize novel tasks. In this work, we use set of generic and simple residual terms, and leverage the power of L.M to compose different terms to generate complex behaviors. The full set of residual terms used in this work can be found in the Appendix A.6.
Our proposed system consists of two key components (Fig. right): i) a Reward Translator, built upon pre-trained Large Language Models (LLMs) [10]. that interacts with and understands user intents and modulates all reward parameters db and weights w,, and ii) a Motion Controller, based on MuJoCo MPC [22]. that takes the generated reward and interactively optimize the optimal action sequence a1:H. Below we provide more details on the design of Reward Translator and Motion Controller
3.2
Reward Translator
Uper
Mattin Deazzipoor Prompt
Make robot dog stand on two foot.
Describa the motion of dog nanng form: height
robat should pitch upward INUSE 0.00 degrees.
Motion tamplate
(NUM 0.3]
Reward Translator
Motion Descriotor
Remember:
1. you see phroses INUM: default value]. replace endre phrase numer cal value.
(start of description)
The torso of the robot should pitch upward at 900 degrees
see phroses (CHOICE: choice choroe2, -.... means you should replace antira phrese with one the choioes listad.
The height of the robot's CaM or sorso center should be at 0.7 meters tront, left foot lited to 077 moters high.
tront,. ngght foot lited to 0.7 meters high.
(and descr [ption]
Rhoaaed Cooke Phuergt gt.
We have a doscription of robat' motion and we want you to turn that Info the corresponding program with following functions: sot, lorse rewardsineigh leght, phsh)
Roward Coder
height: heigtt target for the robot tonso
plic: pitsh angle of the torso
Set tonso
set, torso. rewerdsheiggt--07 pitch-np.oegrradleeee deg2rad(ec))
set._ feet poot_ rewarded front. Jeft, height:() /1
12 Sut feel rewards
Example answer code:
pest__pos______ rewardsf
Jeft. heig v.-0.)
import numpy
Example
ptwtt_ess__ooaaaaeaeaeeeeee fron,_
pot_torso_ 2argetst 1, np (p.ceg2roo(5))
feet post_pos_tte reworded back right,, height=0.0)
Remember:
1. Always format the code code blocks
Motion Controllor
Figure 2: Detailed dataflow of the Reward Translator. A Motion Descriptor LLM takes the user input and describe the user-specified motion in natural language, and Reward Coder translates the motion into the reward parameters.
Inspired by recent progress on Large Language Models (LLMs), we propose to build the Reward Translator based on LLMs to map user interactions reward functions corresponding to the desired robot motion. As reward tuning is highly domain- specific and requires expert knowledge, it is unsurprising that LLMs trained on generic language datasets (e.g. [1] cannot directly generate a reward for a specific hardware. Instead, we explore the in-context learning ability of LLMS to achieve this goal, inspired by prior work that demonstrated variety of in-context learning skills for LLMS [2, 50). Furthermore, we decompose the problem of language to reward into two stages: motion description and reward coding task, as illustrated in Fig. 2.
Motion Description In the first stage, design a Motion Descriptor LLM that interprets and expands the user input into a natural language description of the desired robot motion following a pre-defined template (see example in Fig. 2). Although it is possible for LLMs to directly generate reasonable reward functions for relatively simple task, it often fails for tasks that necessitates complex reasoning. On the other hand, as observed in Fig. 1 left, LLMs can describe complex motions in detailed natural language successfully.
Inspired by this observation, we design template that describes common movements of robot (see Fig. top right for an example) to effectively harness I.MMs internal knowledge about motions. Motion Descriptor produces more structured and predictable outputs and improves stability of the overall system. are describing the motion in natural language, do not need to provide any specific examples in the prompt and can rely entirely on LLMs to generate the result.
In addition,
Reward Coding In the second stage, we translate the generated motion description into the reward function using a second LLM. We formulate the problem of language to reward function as a wwritwriting task to benefit from the LLMs' knowledge of coding and code structure, thus we name the second LLM the Reward Coder. We design a prompt for instructing the LLM to generate reward specifying code (see example in Fig. 2 bottom right). The prompt consists of three parts: i) description of the reward APIs that the L.M can call to specify different parameters of the reward function, ii) an example response that we expect the Reward Coder to produce, and iii the constraints and rules that the Reward Coder needs

---

to follow. Note that the example is to demonstrate to the LLM how the response should look like, instead of teaching it how to perform a specific task. As such, the Reward Coder needs to specify the reward parameters based on its own knowledge about the motion from the natural language description.
3.3 Motion Controller
The Motion Controller needs to map the reward function generated by the Reward Translator to low-level robot actions a1: that maximize the accumulated reward J(a::)). There are a few possible ways to achieve this, including using reinforcement learning (RL), offline trajectory optimization, or, in this work, model predictive control (MPC). At each control step, MPC plans a sequence of optimized actions a1- and sends to the robot. The robot applies the action comesponding its current timestamp, advances to the next step, and sends the updated robot states to the MPC planner to initiate the next planning cycle. The frequent re-planning in MPC empowers its robustness to uncertainties the system and, importantly, enables interactive motion synthesis and correction. We use an open-source implementation based on the MuJoCo simulator [51], MJPC [22]. MIPC has demonstrated the interactive creation of diverse behaviors such as legged locomotion, grasping, and finger-gaiting while supporting multiple planning algorithms, such as iLQG and Predictive Sampling. Following the observation by Howell [22], recond-order planners such as iLQG produces smoother and more accurate actions while zeroth-order planners such as Predictive Sampling is better at exploring non-smooth optimization landscape, we use iL.QG for legged locomotion tasks, while use Predictive Sampling for manipulation tasks in this work.
4 Experiments
We design experiments to answer the following questions: 1) Is our proposed method, by combining L.Mss and MJPC, able to generate diverse and complex robot motions through natural language interface? 2) Does interfacing with the reward function result in a more expressive pipeline than interfacing directly with low-level or primitive actions and is Motion Descriptor necessary for achieving reliable performance? 3) Can our method be applied to real robot hardware?
4.1 Experiment Setup
We first evaluate our approach on two simulated robotic systems: a quadruped robot, and a dexterous robot manipulator (Fig. 3). Both robots are modeled in MuJoCo MPC [22]. We use GPT-4 as the underlying LLM module with temperature parameter set to 0.3 [52]. Here we describe the key setups of each robot. More details regarding the full prompts and reward function can be found in Appendix A.5 and A.6.
Quadruped Robot In this example, we command a four legged, 12 DoF robot (Fig. 3 (a)) to perform variety of motor skills. Quadruped robots have been demonstrated to perform large variety of skills including locomotion [35], hopping [18), biped standing [53, 54], parkour [55] etc. We apply our system to the quadruped robot to perform similar suite of skills while only using natural language as input.
Dexterous Manipulator In the second example, we demonstrate our system on a dexterous manipulator robot consisting of 7 DoF Franka Emika arm and 20 DoF' shadow hand as the end-effector (Fig. 3 (b)). This creates large action space, making it challenging to manually design a controller for this robot.
4.2 Baselines
We compare our proposed system to two baseline methods: i) an variant of our approach that only uses Reward Coder without having access to the Motion Descriptor, and ii) Code-as-Policies [3] where the LLM generates a plan using a set of pre-defined primitive skills. For the Code-as-Policies (CaP) baseline, we design the primitive skills based on common commands available to the robot. Due to limited space we put the full list of primitives in Appendix A.3.
4.3 Tasks
We design nine tasks for the quadruped robot and eight tasks for the dexterous manipulator to evaluate the performance of our system. Fig. 3 shows samples of the tasks. The full list of tasks and sampled videos can be found in Appendix A.2 and project website 4.
'I Language- to- reward, neward.github... github.io
5

---

down
part
Rol over
Turn on faucel
Open ower
Flp the bowl
Biped eland
Face eunset
Keep banane uprigh:
the apple
(a) Quadruped robot
(b) Dexterous manipulator rohot
(c) Example rollout for the robots.
Figure 3: The two rohots used in our experiments and sampled tasks. (a) Quadruped robot with 12 DoFs. (b) dexterous manipulator robot with 27 DoFs. (c) example rollouts produced by our algorithm.
4.4 Evaluation results
For each task and method considered, we generate responses from Reward Translator each evaluated in MJPC for 50 times. Fig. 4 shows the results for both robots. Our proposed approach achieves notably higher success rate for 11/17 task categories and comparable performance for the rest tasks, showing the effectiveness of the proposed method. When compared to the CaP baseline, our method achieves better success rate in almost all tasks. This due to that CaP can perform well on tasks that can be expressed by the given primitives (e.g. Touch object) or very close to the given examples in prompt (e.g. Sit down), but fails to generalize to novel skills. On other hand, using Reward Coder only can achieve success on some tasks but fails in ones that requires more reasoning. For example, when asked open drawer, the baseline often forget task the robot hand to get closer to drawer handle and only design the reward for encouraging the drawer to be open. Sampled responses from different method can be found in Appendix A.8.
To further understand the overall performance of different systems, we show the pass rate Fig. right, which is a standard metric for analyzing code generation performance [8]. Each point in the plot represents the percentage of tasks the system can solve, given that it can generate pieces code for each task and pick the best performing one. As such, the pass rate curve measures the stability of the system (the more flat is, the more stable the system is) as well as the task coverage of the system (the converged point represents how many tasks the system can solve given sufficient trials). It is clear from the result that for both embodiments, using reward as the interface empowers LLMs to solve more tasks more reliably, and the use of Structured Motion Description further boosts the system performance.
4.5 Ablation Study
In this section, we perform an ablation study to investigate how our proposed pipeline responds to varying amount of rules and reminders provided Reward Translator and Motion Controller Specifically, for Reward Translator, we removed all rules in the prompt except for rule 5 that specifies the output format. Similarly for Motion Controller, we removed all rules except for rule to make sure LLMs respond in the correct format. We compare our full prompt to three variants: 1) reduced rule for Reward Translator only, 2) reduced rule for Motion Controller: in Figure 5. We can see that the full prompts outperforms the ablated variants, showing the usefulness of the rules. Furthermore, we find reducing the nules the Motion Controller caused a larger performance degradation than reducing the rules in Reward Translato...
reduced rule for both modules. The results are shown

---

1.0
1.0
Success Rate
0.8
0.8
Poss rate
0.6
0.4
0.4
0.0
Facing sunrise
facing sunset
Sit down
Roll
Spin
Lift paw! paw higher Spin and
Giped standing
0.0
Number of generated code
2
4
6
8
0.8
0.8
Pess rate
Success
0.6
0.1
0.4
0.2
0.0
Touch object
object
Move object
Upright object
Flip object
two objects
Turn faucet Open drawer
generated
Reward Coder only
Code-as as-picicees
Figure 4: Comparison of our method and altemative methods in terms of rate: if generate N pieces of code for each task and pick the best performing one, what's the percentage of tasks that the system successfully tackle.
Success Rate
0.8
0.6
0.4
0.2
Facing sunrise
facing sunset
Sit down Roil over
Spin
Lift one paw
paw higher Spin and lift
Biped standing
Ours
Reduced Descriptor Rules
Recuced Coder Rules
Figure 5: Ablation study for different amount of rules in the designed prompts.
4.6
Interactive Motion Synthesis Results
One benefit of using real time optimization tool like MJPC is that humans can observe the motion being synthesized in real time and provide feedback. We showease two examples where we teach the robot to perform complex tasks through multiple rounds of interactions. In the first example, task the quadruped robot to stand up and perform a moon-walk walk skill (Fig. 6a). We give four instructions to achieve the task, shown in Fig. 6. Each instruction improves the behavior towards the desired behavior based on the interactively synthesized results. This showcase that users can interactively shape the behavior of the rohot in natural language. In the second example, showcase different way of leveraging the interactivity of our system by sequentially commanding the dexterous manipulator robot to place an apple a drawer, seen Fig. 6b. Results of the interactive results are best viewed the supplementary video and full code output from our method can be found in Appendix A.9.
Make robot stand
back humanr.
Open the
Good you wen unly don't reed
from pawe certenn ghrt.
now put the xpple inaiie deawwe wi keep open hand
Goodt now make the robir do mod neve
New close the drawer
Moooo wolk mears ratch ahound walk wit
they
(a) The quadruped perform moon-walk.
(b) The manipulator places apple in the drawer.
Figure The two interactive examples using our proposed system.

---

4.7 Real-robot experiments
User Lift the apple
User the cube
Real
Planned Trajectory
Sim
ser, disttteee reward axt_
set costanceeec.... resnet aeL.ttLaeesttoo poearst
State Estimation
Figure 7: Implementation and rollouts of the proposed system in the real world.
We implement a version of our method on a mobile manipulator, and tested it on nonprehensile manipulation tasks in the real world. To obtain object states in real-world, we first use an open-vocabulary detector: F-VLM [56] to segment the object and then extract the associated points from point cloud behind the mask and perform outlier rejection for points that might belong to the background. From birds-eye view, we fit a minimum volume rectangle and take the extremes to determine the extent in the z-axis. We use this 3D bounding box as state estimation for corresponding object in simulation. To detect the surface of the table with proper orientation, we use an AprilTag [57]. In addition, as seen in the supplementary video, MJPC sometimes discover highly dexterous and dyanmic maneuvers to accomplish the desired task that are beyond the capabilities of current real hardwares. To mitigate this issue, we design regularization residual term specific to encourage steady and stable robot movements when applying our system to the real robot (set_ www.siimaaaal aaraallrrrglllaa www.eeeggiiiiae perrriiiiion tatatioorrewwee sonnrreeer((() ) in Fig. 7, see Appendix A.6.3 for details for this term).
We demonstrate sim-to-real transfer three tasks: object pushing, grasping, and drawer opening. Our system is able to generate relevant reward code and synthesize the correct motion. We also measured success rate of our system on three scenarios: 1) picking up an apple, 2) picking up a rubic's cube, and 3) opening the middle drawer by repeating our approach for 10 time. Our method achieved 70%, 70%, 80% success rate respectively. For example rollouts please refer the supplementary video/website and Fig. 8.
5 Discussion and Conlusion
We propose new paradigm for interfacing an LLM with a robot through reward functions, powered by a low-level model predictive control tool, MuJoCo MPC. Using reward function as the interface enables LLMs to work in a semantic-rich space that play to the strengths of LLMs, while ensures the expressiveness of the resulting controller. To further improve the performance of the system, we propose to use motion description template to better extract internal knowledge about robot motions from L.Ms. We evaluate our proposed system on two simulated robotic platforms: quadruped robot and a dexterous manipulator robot. We apply our approach to both robots to acquire a wide variety of skills. Compared to alternative methods that do not use reward as the interface, or do not use the motion description template, achieves significantly better performance in terms of stability and the number of tasks can solve.
method
Limitations and Future Work. Though we show that our system can obtain diverse set of skills through natural language interactions, there are few limitations. First, we currently design templates of motion descriptions for each type of robot morphology, which requires manual work. An interesting future direction to unify or automate the template design to make the system easily extendable to novel robot morphologies. Second, our method currently relies on language as the interaction interface with human users. As such, can be challenging to design tasks that are not easily described in language (e.g. "walk gracefully"). One potential way to mitigate this issue is to extend the system to multi-modal inputs to allow richer forms of user interactions (c.g., by showing a video of the desirable behavior). Thirdly, we currently use pre-defined reward terms whose weights and parameters are modulated by the LLMs. Constraining the reward design space helps improve stability of the system while sacrifices some flexibility. For example, our current design does not support time-varying varying rewards and would require re- re-designing the prompt to support that. Enabling LLMs to reliably design reward functions from scratch is thus an important and fruitful research direction.
Acknowledgments
The authors would like to acknowledge Ken Caluwaerts, Kristian Hartikainen, Steven Bohez, Carolina Parada, Mare Toussaint, and the greater teams at Google DeepMind for their feedback and contributions.
8

---

References
[1] A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung.
C. Sutton, S. Gehrmann, al. PaLM: Scaling language modeling with pathways. arXiv preprint arXiv:220.023111, 2022.
[2] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, Dhariwal, A. Neelakantan, Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1900 2020.
[3] J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng. Code as policies: Language model programs for embodied control. arXiv preprint arXiv:2209.07733 2022.
[4] A. Zeng, A. Wong. S. Welker, K. Choromanski, F. Tombari, A. Purohit, M. Ryoo, V. Sindhwani, J. Lee, V. Vanhoucke, et al. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint 2004:2204400098, 2022.
[5] W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng, J. Tompson, I. Mordatch, Y. Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint 0Xiv:2207.05608 2022.
[6] J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, Xia, Chi, Q. Le, and D. Zhou. Chain of thought prompting clicits reasoning in large language models. arXiv preprint arXiv:2201. 11903, 2022.
[7] T. Kojima, S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa. Large language models are zero-shot reasoners. arXiv preprint arXiv:2205. /916, 2022.
[8] M. Chen, J. Tworek, H. Jun, Q Yuan, H. P. d. O. Pinto, Kaplan, Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
[9] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O Cortes, B. David, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, et al. Do as can, not as i say: Grounding language in robotic affordances. arXiv preprint 2rX1:2204.00161,, 2022.
[10] S. Vemprala, R. Bonatti, A. Bucker, and A. Kapoor. Chatgpt for robotics: Design principles and model abilities. Microsoft Auton. Syst. Robot. Res, 2:20, 2023.
[11] C. Snell, S. Yang, J. Fu, Y. Su, and S. Levine. Context-aware language modeling for goal-oriented dialogue systems. arXiv preprint arXiv:2204.1011111 10198, 2022.
[12] W. Huang, P. Abbeel, D. Pathak, and I. Mordatch. Language models zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference on Machine Learning, pages 9118-9147. PMLR, 2022.
[13] I. Singh, Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg. Progprompt: Generating situated robot task plans using large language models. arXiv preprint arXiv:2209.11302, 2022.
[14] E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn. BC-z: Zero-shot task generalization with robotic imitation learning. In 5th Annual Conference on Robot Learning, 2021. URL htttps://openreeeee
[15] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, Hsu, et al. Rt-1: Robotics transformer for real-world control scale. arXiv preprint arXiv:2212.06817. 2022.
[16] C. Lynch, A. Wahid, Tompson, T. Ding, J. Betker, R. Baruch, T. Arnstrong, and P. Florence. Interactive language: Talking to robots in real time. arXiv preprint
2022.
[17] J. Lee, J. Hwangbo, and M. Hutter. Robust recovery controller for a quadrupedal robot using deep reinforcement learning, arXiv preprint 1rXivv1901.075530 2019.

---

[18] J. Siekmann, Y. Godse, A. Femn, and J. Hurst. Sim-to-real learning of all common bipedal gaits via periodic reward composition. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 7309-7315. IEHE, 2021.
[19] F. Xia, C. Li, R. Martin-Martin, O. Litany, A. Toshev, and S. Savarese. Relmogen: Leveraging motion generation in reinforeement learning for mobile manipulation. arXiv preprint arXiv:2008.07792. 2020.
[20] M. Toussaint, Harris, J.-S. Ha, D. Driess, and W. Hönig. Sequence-of-constraiits mppe: Reactive timing-optimal control of sequential manipulation. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 13753-13760. IEEE, 2022.
[21] H.-T. L. Chiang, A. Faust, M. Fiser, and A. Francis. Learning navigation behaviors end-to-end with autorl. IEEE Robotics and Automation Letters, 4(2):2007 2019.
[22] T. Howell, N. Gileadi, S. Tunyasuvunakooi, K. Zakka, T. Erez, and Y. Tassa. Predictive Sampling: Real-time Behaviour Synthesis with MuJoCo. dec 2022. 1.011044850//axi12222.04444 URL
https://arxiv httg/ab//22111 00541.
[23] H. Kress-Gazit, G. E. Fainekos, and G. J. Pappas. Translating structured english robot controllers. Advanced Robotics, 13112):133331359 2008.
[24] C. Matuszek, E. Herbst, L. Zettlemoyer, and D. Fox. Learning to parse natural language commands to a robot control system. In Experimental robotics: the 13th international symposium on experimental robotics, pages 403-415. Springer, 2013.
[25] A. Ku, P. Anderson, R. Patel, E. le, and J. Baldridge. Room-across-room Multilingual vision-and- language navigation with dense spatiotemporal grounding. arXiv preprint arXiv:2010.07954, 2020.
[26] A. Kamath, P. Anderson, S. Wang, J. Y. Koh, A. Ku, A. Waters, Y. Yang. J. Baldridge, and Z. Parekh. A new path: Scaling esion-and-langusee navigation with synthetic instructions and imitation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10813-10823) 2023.
[27] O. Mees, Borja-Diaz, and W. Burgard. Grounding language with visual affordances over unstructured data. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), London, UK, 2023.
[28] F. Ebert, Yang, K. Schmeckpeper, B. Bucher, Georgakis, K. Daniilidis, C. Finn, and S. Levine. Bridge data: Boosting generalization of robotic skills with cross-domain datasets. arXiv preprint arXiv:2109.13300 2021.
[29] J. Fu, A. Korattikara, S. Levine, and S. Guadarrama. From language to goals: Inverse reinforcement leaming for vision-based instruction following. arXiv preprint
2019.
[30] S. Karamcheti, M. Srivastava, P. I Liang, and D. Sadigh. Lila: Langunge-informeed latent actions. In Proceedings of the 5th Conference on Robot Learning (CoRL), 2021.
[31] J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. Program synthesis with large language models. arXiv:2108.0772 077322 2021.
[32] K. Ellis, C. Wong, M. Nye, M. Sable-Meyer, L. Cary, L. Morales, L. Hewitt, A. Solar-Lezama, and
J. B. Tenenbaum. Dreamcoder: Growing generalizable, interpretable knowledge with wake-sleep bayesian program learning. arXiv:2006.08381, 2020.
[33] Y. Li, D. Choi, J. Chung, N. Kushman, J. Schritwieserr, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. Dal Lago, et al. Competition-level code generation with alphacode. Science, 378(6624): 1092-1097, 2022.
[34] F. Alet, J. Lopez-Contreras, J. Koppel, M. Nye, A. Solar-Lezama, T. Lozano-Perez, L. Kaelbling, and J. Tenenbaum. A large-scale benchmark for few-shot program induction and synthesis. In ICML. 2021.
[35] L. Tian, K. Ellis, M. Kryven, and J. Tenenbaum. Learning abstract structure for drawing by efficient motor program induction. NeurIPS, 2020.
10

---

[36] D. Trivedi, J. Zhang, S.-H. Sun, and J. J. Lim. Learning to synthesize programs as interpretable and generalizable policies. NeurIPS, 2021.
[37] M. Kwon, S. M. Xie, K. Bullard, and D. Sadigh. Reward design with language models. In International Conference on Learning Representations (ICLR), 2023.
[38] J. Lin, D. Fried, D. Klein, and A. Dragan. Inferring rewards from language context. arXiv preprint arXiv:2204.025115 2022.
[39] P. Sharma, B. Sundaralingam, V. Blukis, C. Paxton, T. Hermans, A. Torralba, J. Andreas, and D. Fox. Correcting robot plans with natural language feedback. Robotics: Science and Systems (RSS), 2022.
[40] S. Nair, E. Mitchell, K. Chen, b. ichter, S. Savarese, and C. Finn. Learning language-conditionee robot behavior from offline data and crowd-sourced annotation. In A. Faust, D. Hsu, and G. Neumann, editors, Proceedings of the 5th Conference on Robot Learning, volume 164 of Proceedings of Machine Learning Research, pages 1303-1315, PMLR, 08-11 Nov 2022. URL https://proceediing mur. pwtes//vv4/naii2222/ /erpoeeeiigg...t.p.eees//////aaa htmm.. html.
[41] L.. Fan, G. Wang, Y. Jiang, A. Mandlekar, Y. Yang. H. Zhu, A. Tang, D.-A. Huang, Y. Zhu, and Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. In Thirty-sixth Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2022. URL https::/t//ttt /owpr//vieeeeeeeeee/meomo. net net/forum? becerrro8__88PXX
[42] P. Goyal, S. Niekum, and J. Mooney. Using natural language for reward shaping in reinforcement learning arXiv preprint arXiv:
2019.
[43] D. Bahdanau, F. Hill, J. Leike, E. Hughes, A. Hosseini. Kohli, and E. Grefenstette. Learning to understand goal specifications by modelling reward. arXiv preprint arXiv:1806.01946 2018.
[44] H. Hu and D. Sadigh. Language instructed reinforcement learning for human-ai coordination. In 40th International Conference on Machine Learning (ICML), 2023.
[45] W. Huang, C. Wang, R. Zhang, Y. J. Wu, and L. Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. arXiv preprint arXiv:2307.05973, 2023.
[46] Broad, J. Arkin, N. D. Ratliff, M. Howard, and B. Argall. Real-time natural language corrections for assistive robotic manipulators. International Journal of Robotics Research (IJRR), 36:684 698, 2017.
[47] Y. Cui, Karamcheti, R. Palleti, N. Shivakumar, P. Liang, and D. Sadigh. "no, to the right" language corrections for robotic manipulation via shared autonomy. arXiv preprint arXiv:2301.02555 2023.
[48] A. F. C. Bucker, L. F. C. Figueredo, S. Haddadin, A. Kapoor, S. Ma, and R. Bonatti. Reshaping robot trajectories using natural language commands: A study of multi-modal data alignment using trans- formers. In International Conference on Intelligent Robots and Systems (IROS), pages 978 778-984, 2022,
[49] A. F. C. Bucker, L. F. C. Figueredo, S. Haddadin, A. Kapoor, S. Ma. S. Vemprala, and R. Bonatti. Latte: Language trajectory transformer. arXiv preprint arXiv:2208.0291 2022.
[50] N. Ziems, W. Yu, Z. Zhang, and M. Jiang. Large language models are built-in autoregressive search engines. arXiv preprint arXiv:2305.096) 2023.
[51] E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSI International Conference Intelligent Robots and Systems, pages 5026-5033. IEEE, 2012. doi: 10.10..11//RRSS.200 201/ 2.366616.
[52] OpenAI. Gpt-4 technical report. arXiv, 2023.
[53] L. Smith, J. C. Kew, T. Li, L. Luu, X. B. Peng, S. Ha. J. Tan, and S. Levine. Learning and adapting agile locomotion skills by transferring experience. arXiv preprint UrXiv:2304.09834 2023.
[54] Fuchioka, Xie, and M. van de Panne. Opt-mimie: Imitation of optimized trajectories for dynamic quadruped behaviors, arXiv preprint arXiv:2210.01247.0000 2022.
11

---

[55] K. Caluwaerts, A. Iscen, J. C. Kew, W. Yu, T. Zhang, D. Freeman, K.-H. Lee, L. Lee, S. Saliceti, V. Zhuang, et al. Barkour: Benchmarking animal-Jevel agility with quadruped robots. arXiv preprint arXiv:2305. 14654, 2023.
[56] W. Kuo, Y. Cui, X. Gu, A. Piergiovanni, and A. Angelova. F-vim: Open-wocabulary object detection upon frozen vision and language models. arXiv preprint arXiv:2209. 2023.
[57] E. Olson, Apriltag robust and flexible visual fiducial system. 2023.
12

---

A Appendix
A.1 Author Contributions
Author contributions by type, ordered alphahetically within each category:
Method (conception, implementation, iteration, evaluation): Nimrod Gileadi, Kuang-Huel Lee, Yuval Tassa, Fei Xia, Peng Xu, Wenhao Yu.
Infrastructure Development:
Tom Erez, Nimrod Gileadi, Yuval Tassa, Fei Xia, Wenhao Yu.
Hardware Deployment: Chuyuan Fu, Nimrod Gileadi, Leonard Hasenclever, Jan Humplik, Sean Kirmani, Yuval Tassa, Fei Xia, Ted Xiao, Wenhao Yu.
Project Advising: Tom Frez, Nicolas Heess, Brian Ichter, Dorsa Sadigh, Jie Tan, Yuval Tassa, Fei Xia, Andy Zeng, Tingnan Zhang.
Paper Writing/Revision Montse Gonzalez Arenas, Hao-Tien Lewis Chiang, Nimrod Gileadi, Brian Ichter, Dorsa Sadigh, Fei Xia, Andy Zeng, Wenhao Yu, Tingnan Zhang,
A.2
Full task list
Here we show the list of tasks used in our evaluation as well as the instructions used for each task.
Tusk
Instructions
Expected Behavior
Facing sunrise
It's early in the morning, make the robot head towards the sun.
Robot face towards East.
Facing sunset
It's late in the afternoon, make the robot head towards the sunset.
Robot face towards West.
down
down low to ground with flat.
Rahat's CoM drops lower and remain flat.
Roll
rull by degnees.
Rohot's belly up.
Spin
Spin fast.
Robot reach fast turning speed.
Lift one paw
I want the robot to lift its front right paw in the air.
The front right paw of the robot lifts up in the air.
I want the robot to lift its front right pawe in the air.
Lift paw higher
Lift even highor.
The
right paw higher before.
Lift front left paw.
Spin puws
Good, now lift diagonal paw well.
Robot front rear right paws while spin fist.
Good, in addittion want the robot to spin fast.
Stand up lieet
Make the robot stand upright two back feet like humun.
Robot stands back feet and keep balance.
Trotting
Deseribe wotting gair, thon make the robot walk trotting gait.
Robot walks with diagonal synchronous
Pacing
Describe pacing gait, then
robot in pacing gait.
Robot walks on same side in synchronous
Bounding
Describe bounding gait, then make the rohot walk bounding gait.
Rohot walks with front and back feet in synchromous respectively.
Table I: List of tasks used in evaluation for the quadruped robot.
Instructions
Expected Behavior
Touch objject
Touch the {objact)
Robat fingers in contact with the object.
Lift object
Lift the (ubject) to 0.5
The object needs to stay above 0.4 m for Is.
Move objoct
Move the (object_a) to (object_b)
The distance between object noeds to be smaller than 0.1
Upright object
Place the fobject) upright
The axis of the object needs be parallel x-y
Flip object
Plip the (object)
local up vector object shoukl pointing downward.
Lift two objects
Lift the {object a} and (object b) the same time.
Both objects need stay above 0.4 m for 1s
Turn on the faucet
Tur on the faucet.
The valve of the fiucct needs to be turned 90 degrees.
Open the drawe
Open the drwwer
The driwer needs be pulled fully open.
Table 2: List of tasks used in evaluation for the dexterous manipulation.
13

---

A.3 Baseline details
For the quadruped robot, we use the following three primitive skills:
(rrooaard(directtinn specifies a target heading direction direction for the robot to reach.
walk( fwwowoorddspppee sideway_speeds turning_speed) controls the robot to walk and turn in different directions. This a common interface used in quadruped robots to navigate in different environments.
cet_joint_ppone (leg_ Leg_nmm,, joint_angless directly sets the joint positions for each DoF on the robot. To help the LLMs understand the joint angles, we provide a of examples in the prompt.
For the dexterous manipulator robot, we use three primitive skills to control the robot motion and also function get access the position of object in the scene:
end effector_ttopooitionn moves the center of the robot hand's palm the given position.
end_ uad__fffctor_ppenf opens the hand of the robot by extending all fingers.
end_effector_cllse(( closes the hand to form a grasping pose.
set_chhect_poostiinnnob_aaaa gets the position of a certain object in the scene.
get_ jet_joint_positionooonntt, gets the position of certain joint the scene.
A.4 Additional illustrations for real-world results
apple
User Lift the cube
box neer the apple
Open the middle drawer
Figure More illustrations for the real world results for the proposed system.
A.5 Full Prompts
Here we list the full prompts used in Reward Translator for all experiments used in this work.
i) Motion Descriptor Prompt for Quadruped
14

---

Describe the motion of a dog robot using the following form:
[start of description]
The torso of the robot should roll by [NUM: 0.0] degrees towards right, the torso should pitch upward
at [NUM: 0.0] degrees.
The height of the robot's CoM or torso center should be at INUM: 0.3] meters.
The robot should (CHOICE: [face certain direction, turn at certain speed]). If facing certain direction, it should be facing {CHOICE: [east, south, north, west]]. If tuming, it should turn at [NUM: 0.0] degrees/s.
The robot should (CHOICE: [go to certain location, move certain speed]]. going to certain location, should go to (x=[NUMM 0.0], y=[NUM: O.0D). If moving at certain speed, it should move forward at [NUM: 0.0jm/ss and sideways at [NUM: 0.0Jmn's (positive means left).
[optional] front left foot lifted to INUM: 0.0] meters high.
[optional] back left foot lifted to INUM: 0.0] meters high.
[optional] front_right foot lifted INUM: 0.0] meters high.
[optional] back back_ foot lifted to INUM: 0.0] meters high.
[optional] front_left foot extend forward by INUM: 0.0] meters.
[optional] back left foot extend forward by [NUM: 0.0] meters.
[optional] front right foot extend forward by [NUM: 0.0] meters.
[optional] back right foot extend forward by [NUM: 0.0) meters.
[optional] front Jeft foot shifts inward laterally by INUM: 0.0] meters.
[optional] back__et left foot shifts inward laterally by [NUM: 0.0] meters.
[optional] front right foot shifts inward laterally by INUM: 0.0] meters.
[optional] hack Bach_ foot shifts inward laterally by [NUM: 0.0] meters.
[optional] front_left foot steps on the ground at a frequency of [NUM: 0.0] Hz, during the stepping
motion, the foot will move INUM: 0.0) meters up and down, and [NUM: 0.0) meters forward and back,
drawing a circle if it's walking (CHOICE: forward, back], spending [NUM: 0.0J portion of the time in the air gait cycle.
[optional] back left foot steps on the ground at a frequency of [NUM: 0.0] Hz, during the stepping
motion, the foot will move INUM: 0.0] meters up and down, and [NUM: 0.0] meters forward and back, drawing a circle as if it's walking (CHOICE: forward, back), spending [NUM: 0.0] portion of the time in the air gait cycle.
[optional] front_ right foot steps the ground at frequency of [NUM: 0.0] Hz, during the stepping
motion, the foot will move [NUM: 0.0] meters up and down, and [NUM: 0.0] meters forward and back,
drawing a circle as if it's walking (CHOICE: forward, back], spending [NUM: 0.0] portion of the time
in the air vs gait cycle.
[optional] back right foot steps on the ground at frequency of [NUM: 0.0] Hz, during the stepping
motion, the foot will move INUM: 0.0J meters up and down, and INUM: 0.0] meters forward and back,
drawing a circle as if it's walking (CHOICE: forward, back), spending [NUM: 0.0J portion of the time in the air vs gait cycle.
[optional] The phase offsets for the four legs should be front_left: INUM: 0.0], back_left: [NUM: 0.0), front_ right: INUM: 0.0), back_right: INUM: 0.0].
[end of description]
Rules:
1. If you see phrases like [NUM: default value], replace the entire phrase with a numerical value.
2. If you phrases like CHOICE: [choicel, choice2, ...]. it means you should replace the entire phrase with one the choices listed. Be sure to replace all of them. If you are not sure about the value, just use your best judgement.
3. Phase offset is between [0. 1]. So if two legs' phase offset differs by O or 1 they are moving in synchronous. If they have phase offset difference of 0.5, they are moving opposite in the gait cycle. 4. The portion of air vs the gait eycle is between [0, 1]. So if it's 0, it means the foot will always stay on ground, and it's 1 means the foot will always be in the air.
5. I will tell you a behavior/skilltast that I want the quadruped to perform and you will provide the full description of the quadruped motion, even if you may only need to change a few lines. Always start the description with |start of description) and end it with [end of description).
6. We can assume that the robot has a good lowelevel controller that maintains balance and stability as long as it's in a reasonable posc.
7. You can assume that the robot is capable of doing anything, even for the most challenging task.
8. The robot is about 0.3m high in CoM torso center when it's standing on all four feet with horizontal body. It's about 0.65m high when it stand upright two feet with vertical body. When the
robot's torso/body is flat and parallel to the ground, the pitch and roll angles are both
9. Holding a foot 0.0m in the air is the same as saying it should maintain contact with the ground
10. Do not add additional descriptions not shown above. Only use the bullet points given in the template.
15

---

11. a bullet point is marked loptional], do NOT add unless it's absolutely needed.
12. Use as few bullet points as possible. Be concise.
ii) Reward Coder Prompt for Quadruped
We have description of a robot's motion and want you to umm that into the corresponding program with following functions:
def set set_torso__m (aarrss_trret(((eee targets(target, (target_torso torso sooroo_hiight,
target_torso torso ppttch target_torae porso_roll, target_tore torso
ttt_tttooooottttto_ee lol_llocacton___. xy, target_torse parget_torso_heading, target_turning_sper
segee__torso_elocittyy
_eight: how high the torso wants to reach. When the robot is standing all four feet in normal standing pose, the torso about 0.3m high.
target_toroo_ppee pitch: How much the torso should tilt up from a horizontal pose in radians. A positive number means robot looking up, e.g. the angle is 0.5 the robot will looking upward, if angel is O, then robot will be looking forward.
target_ _orettttosoo___s velocity_ xy: target torso moving velocity in local space, is forward velocity, is sideway velocity (positive means left). target torso heading; target torso heading is in the range of 0 to 2*pi, where and 2 both mean East, pi being West, etc.
desired direction that robot should face towards. The value
surningsspeed:: the desired turming speed of the torso in radians per second. and target_torse tettttoroooooooetteeeee vclocity_ must be None. of
Remember: one of target_tore corso_location targot_torso corso_hcading and target_ srrget_tuninggspooe must None. other inputs can None.
def set set_feet feet ht..pooppprameters(ffeeeeeee Lift_height, extend_ forward nove_inward)
feet name one of ("front_lefh
"back_leftt" ftront_right" "back_right")
lift height: how high should the foot be lifted in the air. If is None, disable this term. If it's set to 0, the foot will touch the ground.
extend forward: how much should the foot extend forward. is None, disable this temm.
move inward: how much should the foot move inward. If is None, disable this term.
def het_feet___ Saarriii__prammteeee((ee
air_ratio, phase_offset, twing_up_down swing_ fowwfrdwwwddbbaaa should_activate.
Klepping_frequent
feet_nams of Jeft", leetr_lett., ffront_right,, "rear right".
ratio (value from to describes how much time foot spends in the versus the whole gait cycle. it's the font will always stay on ground, and it's it'll always stay in air. phase_offsect (value from 0 to 1) describes how the timing of the stepping motion differs between different feet. For example, if the phase_ _otteee_ooettttte between two legs differs by 0.5, it means one leg will start the stepping motion the middle of the stepping motion cycle of the other leg. swing up down is how much the foot swings vertical during motion cycle.
swing forward back how much the foot swings horizontally during the motion cycle. If swing_forwer forward bacckddbcckk back is positive, the foot would look like it's going forward, if it's negative, the foot will look like it's going backward.
If should_actee activate False, leg will follow the stepping motion.
def execute_plan( (arpaaanplaa_dureeeeee (ilaandduratio=22))
This function sends the parameters to the robot and exccute the plan for "plan_duration" soconds, default to be 2
Example answer code:
import numpy np import numpy because are using it below
reset_rewarde
# This is a new task so reset reward; otherwise we don't need it set_torso, porso_tarrgetsee1,, np.deg2rad(s), dp_degarad(15), (2, 3), None, None, np. ap.degarad(1o))
set Liret_pos_prrameeers""to
left", 0.1, 0.1, None)
set [___fett_pos_pammeetrrree ("back_ ("backlleft", None, None, 0.15) set s___feet_poo_paramettrsss (trrrrrttttersroot_rrrrrrrre "ffoot_right",, None, None, None) [t__feet_pospparaeeee ("back_right" e.e, 0.0, None)
16

---

[o, 0.75, 1.5l) set beeetsstepiin_parameeee "frontrright,, 2.0, 0.5, 0.2, 0.1, -0.55, True)
set feeee_sstepping_prraaaeee ("back_left', 3.0, 0.7,
0.05, True)
(ttaammteettsrotteleeeeeee Froot_leftt, 0.0, 0.0, 0.0, 0.0, 0.0, False)
set
"backrright", 0.0, 0.0, 0.0, 0.0, 0.0, False)
execute_plan(4)
Remember: Always fomnat the code in code blocks.
2. Do not invent functions classes. The only allowed functions you call the listed above. not leave unimplemented code blocks your response.
3. The only allowed library is numpy. Do not import or use any other library. If you use np, be sure
to import numpy.
4. you are
what value use, just your best judge. Do not None for anything. 5. not calculate the position or direction of any object (except
ones provided above). Just
use a number directly based on your best guess.
6. For set st_torso targets, only the last four arguments (target_tons terstttrs____ooeee lecttlooccttoon_ __nttoon_,,,,o target_ torso cerro__elocity.... target_ torso_ tottooooosssutooeeeeoooooooooeee headdigg target_ target_turming turning (uming_speed) can he None. Do not set None for any other arguments. 7. Don't forget to call execute plan the end.
iii) Baseline: Code-as-Polickes Prompt for Quadruped
We have quadruped robot. has joints in total, three each leg. can use the following functions to control its movements:
def ser__targe__oiit_anglleee
waaggneeeleggnnnnaaneeeee (leg_name, target_joint_ange
leg. name is of ("front_Jeif left", "back left", "front right", "back right").
target_ joint angles: 3D vector that describes the target angle for the abuluction/akkluction,, hip, and knee joint the each leg.
def wa lalk( forward_ speedd,_ sideway_speed. turning_speed)
forward speed: how fast the robot should walk forward
sideway_speen how fast the robot should walk sideways tuminngspeed:: how fast the robot should be tuming (positive means turning right)
def towards (headrds(eeadinng....
heading direction: larget heading for robot to reach, range to 2pi, where means East, 0.5pi means North, pi means West, and 1.5pi means South.
def execute_plan( (aannpppanndaaaaaa (aaa__duratioo1o))
This function sends the parameters to the robot and execute the plan for "plan duration" seconds, default be 2
Details about joint angles each leg: abduction/alluucion joint controls upper leg to swinging inward/outward. When it's positive, legs will swing outward (swing to the right for right legs and left for left legs). When it's negative, logs will swing inward. hip joint controls the upper the torso (hip same height as shoulder), pointing backward. When it's positive, upper leg rotates downward so the knee is below the shoulder. When it's 0.5pi, it's perpendicular to the torso, pointing downward. When it's negative, the upper leg rotates upward so the knee is higher than the shoulder. knee joint controls the lower leg rotate around the knee. When it's zero, the lower leg is folded closer the upper leg. knee joint angle dicular to upper leg. When it's pi. the lower leg fully streching out and parallel to the upper leg. Here are few examples for setting the joint angles to make the robot reach few key poses: standing on all four feet:
rotate around the shoulder. When it's zero, the upper leg is parallel
only be positive. When it's 0.5pi. the lower leg perpen-
set tattttarrrttnnttee (aneet__oonn__alle((sseee ("rloss"f _eftt, [e, 1.5])
set set
wew.annesssoootriiiiii [o, 1, 1.5l) set set_target_joime saagaaanngassaeee execute_plan()
[o, 0.75, 1.5))
sit down on the floor:
set set_target_joint_anglle (otiiiiiiinnnggeeefottteteeeee [o, o, o))

---

set "ainet__jont__anlees"baaa beabbaakkk"k,,, [e, a, o])
set set_target_jone aoont_angles"""root__eoo rightt", [e, o])
ret_target_joing Weoiinnnnnlssnnmee (angless"back_rieeee right", [e, 0, o]) execute_plan()
lift front left foot:
[e, 0.45, 0.35l)
set leaoooonnnnnnnnnaoaoo
[o, 1, 1.5l)
aange__jooittaneees"fforott set set_target_joint angles ("baee__ighhhht [o, 1, 1.5))
[e, 1.4, 1.5l)
execute_plan()
lift back left foot:
set_ tatttaaggeooinn ("rrott __nntt," [e, 0.5, 1.5])
[e, 0.45, 0.35l) set_target_joint_ann trrnaannness"frottrriiiee [e, 0.5, 1.5)) set srs_target_jointe joini_angnenn(((( angglesbback_rihht [o, 0.5, 1.5l) execute_plan()
Remember:
I. Always start your response with [start analysis]. Provide your analysis of the problem within 100 words, then end with [end analysis].
2. After analysis, start your code response, format the code in code blocks.
3. Do not invent new functions or classes. The only allowed functions you can call are the ones listed above. Do not leave unimplemented code blocks in your response.
4. The only allowed lihrary is numpy. Do import use any other library. you np, be
to import numpy.
5. If you are not sure what value to use, just use your best judge. Do not use None for anything.
6. Do not calculate the position or direction of any object (except for the ones provided above). Just
number directly based your best guess.
7. Write the code concisely possible and try
define additional variables. 8. you define a new function for the skill, be sure to call somewhere.
9. Be sure to call execute_ plan at the end.
iv) Motion Descriptor Prompt for Dexterous Manipulator
We have dexterous manipulator and we want you to help plan how it should move perform tasks using the following template: [start description]
To perform this task, the manipulator's palm should move close to (CHOICE: apple, banana, box, bowl, drawer handle, faucet handle, drawer center, rest position).
object)=(CHOIEE: apple. banana, box, bowl, drawer handle, faucet handle, drawer center] should be close (Cjoct2=(HOOIE: apple, banana, box, bowl, drawer_handyy faucet_handle, drawer_center nothing].
[optional] objectl needs to be rotated by INUM: 0.0] degrees along x axis. [optional] object2 needs to be rotated by [NUM: 0.0] degrees along axis. [optional] objectl needs to be lifted [optional] object2 needs to lifted a height [NUM: 0.0Jm at end. [optional] object3=[CHOICE drawer, faucet) needs to be (CHOICE: open, closed). [end of description] Rules:
height of [NUM: 0.0Jm at the end.
1. you see phrases like INUM: default_valuc],, replace the entire phrase with numerical value. (CHOICF: choicel, chnice2, ...), means you should replace entire
2. you see phrases
phrase with one of the choices listed.
3. If you see [optional], it means you only add that line if necessary for the task, otherwise remove that line.
4. The environment contains apple, banana, box, bowl, drawer handle, faucet handle. Do not invent new objects not listed here.
5. The bowl is large enough to have all other object put in there.
6. I will tell you behavior/skilltask that want the manipulator to perform and you will provide the
full plan, even you may only need change a few lines. Always start the description with [start plan] and with fend of plan].
18

---

7. You can assume that robot is capable of doing anything, even for the most challenging task.
8. Your plan should be as close to the provided template as possible. Do not include additional details.
Reward Coder Prompt for Dexterous Manipulator
We have plan of robot arm with palm to manipulate objects and we want you to turn that into the corresponding program with following functions:
def set set_la_distt Lla_disttnce_reeee reward _aamm______________ name_obj__0
I"palm", "apple", "banana", "box", "bowl", where name obj and name _aeee__o______ are selected "rest_poittoo") "] This term sets a reward for "drawer handle", "faucet_ "faucet_handle",, "drawer_center", "rest_poone
minimizing 12 distance between name hame_obj__ _bj__A and name_obj. so they get closer to each other. rest_position is the default position for the palm when it's holding in the air.
def set_obj cert_bjj_orienttte httpi_irrenttttreeaaaaaeee
_aaam__b__ x_ __axis__eea aaxisssotatioo_rateee radians)
this term encourages orientation name_ obj to close to target (specified by
x x_axis rotatisotttionnnnono r
def execute p duration=2)
This function sends the parameters to the robot and execute the plan for "duration" seconds, default to be 2.
def set_ set_joint_ ceant__frattioo_eweeeera
name_joint fraction)
function sets the joint to certain value between and means close and means open. name_jor joint needs to be select from ['drawer', 'faucet"]
def www.ooo__possiin_ooo
name_obj, z_height)
this term encourages the orientation of name obj to be close to the height (specified by height).
def reset_reward()
This function resets the reward default values.
Example plan: perform this task, the manipulator's 's palm should move close to object =ppple. objectl should be close to objec12=bowl. object2 needs to be rotated by 30 degrees along x axis. object2 needs to be lifted to height of 1.0.
This is the first plan for Example answer code:
task.
import numpy as np
reset_ (sset_rewar(()
This is task reset reward; otherwise don't 't need it "apple")
set set_l2_dee www.:tttnceeewarrd""" set_l2_ wwwttanceeewwrrreeeeoeoeooee
"bowl")
set_obj_orientatine wtt__oorennntio_rvveeeeee reword("bowl", np. op.degzra((30))
set_obj___position reward( "bowl", 1.0)
execute_plan(4)
Remember:
1. Always format the code in code blocks. In your response execute plan should be called exactly once
at the end.
2. Do not invent new functions classes. The only allowed functions you can call are the ones listed above. Do not leave unimplemented code blocks your response.
3. The only allowed library is numpy. Do not import or use any other library.
4. If you are not sure what value to use, just use your best judge. Do not use None for anything.
5. Do not calculate the position
direction of any object (except for the provided above). Just
use number directly based on your best guess.
6. You not need to make the robot do extra things not mentioned in the plan such as stopping the robot.
vi) Baseline: Code-as- Policiis Prompt for Dexterous Manipulator
19

---

We have manipulator and we want you to help plan how should move to perform tasks using the following APIs:
def und_effector__eee httpectorrtttpsistiooooo
position_obj is list of 3 float numbers [x,zz]
def si__effector_ppeee Open the end effector.
def sii_eeffecor_ccosse Close the end effector.
def [et_object_possitooooo]___
Given an object name, return a list of 3 float numbers [x,yz] for the object position. the object come from list of l"apple", "banana". "bowl". "box", "drawer handle", "faucet_handee handle", "drawer_center "rest_ position
def wwwrmalliedddddiieo htt.eleooointppssiiijiiiitttttee
Given an joint name, retur a float numbers x. the joint can come from a list of ("drawer", "faucet"] def reset()
Reset the agent.
Example answer code:
import numpy np
reset()
apple_pos get_ end seaffectoo_toaapple_______
Remember:
1. Always format the code code blocks.
2. Do not invent new functions or classes. The only allowed functions you can call are the ones listed
above. Do not leave unimplemented code blocks in your response.
3. The only allowed library numpy. Do not import use any other lihrary.
4. you are not sure what value use, just use your best judge. not use None for anything.
5. You do not need to make the robot do extra things not mentioned in the plan such as stopping the
robot.
6. Try your best to generate code despitc the lack of context.
A.6 Reward functions used in experiments
In this work we use a set of generic reward functions for each embodiment that the LLMs can modulate. More specifically, design set of residual terms as in Equation I that optimized to reach zero by internally converting them to a 12 loss. Thus given a residual term r(.) a reward term can recovered by in experiments for each embodiment.
-1 Below we describe the full set of residual terms For each term we select the weights for them to have about the same magnitude. The reward coder can adjust the parameters in each term and optionally set the weight to to disable term.
A.6.1 Quadruped
Table shows the residual terms used the quadruped tasks. Note that for the foot-related terms, they are repeated for all four feet respectively. Furthermore, LLMSs can optionally set the target foot positions fp directly or through periodic function mar(asinn1222) (c).+ where a the magnitude of the motion, b is the frequency, and is the phase offset.
20

---

Residual Term
Formulation
Defaull weight
position
Pay
Payl
CoM height
1.0
base yaw
Pyam
Pyaw
0.3
base pitch
Dyitch
Dpitch
basc roll
Proil
Prall
0.1
forward velocity
Du
R
sideways velocity
Dy
0.1
yow speed
Pyam yark
0.1
foot local position x
fp. fp:
foot local position y
fi. fp,
foot local position
fp. fp.
Table 3: List of residual terms used for the quadruped robot. p denotes the position and orientation of robot's torso. fp denotes the position of the robot's foot (in local space). (.) means the target value and means the time-derivative the quantity.
Residual Term
Formulation
Default weight
chji close ohi2
c1 xgc c2
5
move obj target X-Y position
C2 0.-0a_ 02
5
tarett beight
Czy Ezyl
move obj targel onemtation
Ooay 0
move joint to turgtt value
4-9
Table 4: List of reward terms used for the dexterous manipulator robot. с denotes the position of the object, denotes the orientation of the object, is the degree of freedom to be manipulated.
A.6.2
Dexterous Manipulator
A.6.3 Sim-to-Real residual term
As seen in the supplementary video, MuJoCo MPC can discover highly dynamic and dexterous manipulation skills that exceeds the capabilities of existing hardwares. To enable successful deployment on the hardware, we design a regularization term to help achieve steady motions on the robot. Specifically, we use the following residual term:
Pee 0,
if ||>0. otherwise
T 2rim2rool =3
q.
if ||l 0.7
0.
otherwise
t0.05paa
Poe 0. otherwise
Poby' if Poty <0.1
+0.1
+0.4 Agripper
1.0.
if Pec Pabil 0.1
Agripper:
otherwise
where Pee is the robot end effector velocity, q is the robot joint velocity, Potj the object velocity. The first two terms regularizes the joint and end effector velocities to encourage them to stay at a low-speed region to prevent jerky robot motions. The third and fourth term encourage objects to move slowly and match gripper speed. This is to discourage dynamic in-hand movements. The last temn encourages gripper to open while being away from objects and close when approaching objects. This term helps improve the grasping skill transfer by encouraging more clean grasp motion.
A.7 Detailed evaluation results for each task
Figure 10 and Figure shows the full evaluation results for individual tasks Section 4.4. For each task, we query the Reward Translator LLM 10 times. For each generated reward code, evaluate on MJPC for times and report the success rate. Therefore the results here shows the individual performance of Reward Translator and Motion Controller. We can that for most tasks Motion Controller achieves
21

---

near perfect success rate as long as the right reward is provided, while for tasks that require more dynamic motion or more precise manipulation (e.g. biped standing, and upright banana) Motion Controller shows more variance in performance.
apple_bol bowl
banana bowl
hip_bowl
3.0
1.0
3.4
Success rale
0.8
0.8
SUcceSS rate
0.8
5.
0.0
30
10
reward code id lift apple
reward code id banana
rewaar code id
lift banana box
3.0
1.0
3.0
Success rate
0.8
0.8
success rabe
0.8
0.6
6-2
0.0
0.0
20
reward code
reward code
reward code id
lift box
move apple box
move banana to
3.4
1.0
3.0
Success tate
2.0
Success rate
0.2
0.2
0.0
0.0
20
reward code
reward code
reward code id touch banana
open drawer
touch apple
3.0
1.0
3.0
success rote
0.8
SUCCOSS rate
0.2
0.2
3,2
0.0
0.9
300
reward code
reward code
reward code id turn faucet
touch bowl
touch box
3.0
1.0
3.0
SUCCeSS rate
0.8
SUCCESS
0.2
0.2
0.2
8.0
0.0
36
reward code
reward code upright box
rewaidd code id
banana upright_banana
3.0
1.0
SUCCeSS rate
0.1
0/2
0.2
8.0
0.8
Ours
Reward Coder only
Code-as-policies
Figure 9: Full evaluation results for the Dexterous Manipulator robot. Note that to better interpret the results, we order the generated reward code in the figures based on the mean success rate.
22

---

face sun in morning
face sunset in the afternoon
1.0
Suicress rate:
3.8
SUCCOSS rate
8.8
2.5
8-6
success
0.6
241
8.4
044
2.2
0-3
0.
3,3
8.6
0.9
lift one paw
one paw higher
roll over
1.0
SURcEOS rate
SUCCOSS rate
success rate
2.8
8.6
0.4
2.4
2.4
0.4
2.
0.2
0.2
3/0
6.
0.0
reward code
reward code
rewerd code
sit down
spin and
spinning
Sliceess rate
SUCCOSS rate
success rate
2.8
0.6
0.6
2.4
0.4
04
27
0.2
9.
24
80
60
reward code
reward code
reword
Ours
Reward Coder only Code-ac as policies
Figure 10: Full evaluation results for the Quadruped robot. Note that better interpret the results, we order the generated reward code in the figures based on the mean success rate.
A.8 Example outputs
To help understand the behavior of different methods discussed in this work, here show sampled responses for two tasks across different methods: Biped standing for quadruped, and Turn on faucet for dex- terous manipulator. For each method, sample 2-3 responses show both successful and failed results.
A.8.1
Sampled results for Biped standing task
User instruction: Make the robot stand upright on feet like a human.
Ours
Sample 1
import numpy as np
reset_rewarde set_torso_targets(t.
set_torso, (o.gettto..6.
np.degaraddgo), (p)d (e.e, 0.0), None, e, None)
0.65, None, None) set hil_feee__osppprrmeetststttcee left", 0.0, None, None) set lel_ffet_po_pppraaeeee ttaarrmmmtttessffotttrereeee 0.65, None, None) htr. set_feet_pos_parmmttesss ("back_biiii bach_right", 0.e, None, None)
set fffeet_pos_prrmmeerrssfffff
execute_plan()
23

---

Sample 2
import numpy as np
reset_rewarde
set_torso_ (oorso_targets(ee.5 np.degzrad(9e), np.degarad (e.0, 0.e), None, 0, None)
set frreee__possprrmtetesrrffoooooo
0.65, None, None)
set eet_ffet_.o_pprrreeerrssccadd "eb left", 0.0, None, None) 0.65, None, None)
set_ feet sarrreteoss_pprmmeeeesssss"s"s"e set het_feet_pos_e porpett_pos_ppraeeerese taw.aaateeeraaaaaraaaaaa "back_right", 0.0, None, None)
Sample 3
import numpy as np
reset_reward() set_torso__trgets(o..55 (or5ets(o.65,, np.deg2rad(99),, np.degarad(o), (0, o), None, o, None) )
set_torso_
set_ httpsppparrrmtersfooootttttoo 0.65, None, None)
oet_leee__os_.arrmmmttttccceceee 0.e, None, None) set ser.ffeet.os_oorrrmmtttssttttrrd seL_ffet.pe__erraeettteeebbeeeee 0.0, None, None)
0.65, None, None)
set [eg_ffee__teeepphg..pprrmeeteter _eatttt____t,_ 0.0,
0.0, 0.0, 0.0, False)
set_ffettsteppinnnpprrmmttttssss ("back_ 0.0, 0.0, 0.0, 0.0, 0.0, False)
set set_ffet_steepinggorrmmee ("front_right" 0.0, 0.0, 0.0, 0.0, 0.0, False)
set set_feet_stepping_orrammee right,, 0.0, 0.0, 0.0, 0.0, 0.0, False)
execute_plan()
Reward Coder only
Sample I
import numpy np
# Set torso targets to stand on back feet set_torso_ set_torso_targets (trrgettttrgetttorssoooooo
prrret_torso_picchee.seppp
target_torso rol.roll..,. trreet_torsa_rollo0.e
None, target_ torso ttt_ttorooovloittt_o_otote velocity_xyyleeeeeeeeeeeeeeereee None,
target_turninn pnget__urming_spee-Noor
Lift front feet off the ground sit__feet_pos_prrmeteess (eattrfeeeeeeaeeerrrttttt
lift_heiggh=e..3, (fett_naee""ffoo_rrrrrr
move soe_iniwad=NNoee
set stt_feet_pos_parametrsss wttpaaaaaaaeeeeeeeaaaaaaaaaaaaaa (feet_eam=="""""eeeeeeeee Lift_height=o.3, extend extend_forward forward-None, move inward=None)
Execute the plan execute_plan(plan_ )

---

Sample 2
import numpy
Set the torso targets to stand back feet set_torso_ ooroooo_tarretttttrrettooooooooo torso_roll=___.0 ceceee_ttrro_ppichhee target_torso trrse__tors__lcction..oo xy=None, target_torso carget_torso_velocittyyyy o),
harget_torso_st
harget_turning_speeeee
Lift the front feet off the ground set fieet_pos_paramtterse "ooss"ff____eft""
Lift__eightt=..3, extend_ forwar=Noee,, move_ )
set_feet_pos_parmetters wwtoeetteessrrrrririiree frrottrrigt'',,,
lift_height=e..3 SxtendfoowarddNons move inward=None)
# Execute the plan tatpee_plllpppll_uuattiiio
Code-as-Policies
Sample 1
set caaret_jointt Dooint_angles("frree front_left', [o, 1.5l)
set Latrget_joint_angleeeeeeee left", [o, 0.5, 0])
set_target_joiee songjiiinnnnllsesffroottt [e, 1, 1.5]) set joint__nggles"bacee [e, 0.5, o]) execute_plan()
Sample 2
set_ [e_taarge_joont_anggeeeee (arnglees"footeeeff"" [o, 0.5, 1.5])
set_ "taa [o, 1.2, 0.5])
[e, 0.5, 1.5))
set_ set_target.joint_nnglee http::joooonnnnnllletttc.cirrri ("ackks"baarkrrhhtt" [o, 1.2, 0.5l)
execute_plan()
25

---

Sample 3
import numpy as np
def set set_legs_ to_ posoeeeppss)::
set_target_joint _anglint___ll_(((c__ front_left", [o])
set_ set_target_joint_ angles( (agoless"bbc_eeeee" ("back_left", pose[a])
set_target_joint__ (ngles("front___))
pose[2]) [2])
set_target_joiit__nnee _angttt__nnssaaa ("back_right"" pose[3])
def execute_pose(pene duration=2):
puecute_plan(duration
initial pose initial _pose [
[o, 0, e], # front_lef [o, 0, 0], # back_lefi
[o, 0, el, # Front_right
[o, 0,
back_right
set_ get_legs_to_peee to__tooseeiniiill_pssss execute_plan()
set target pose target_pose =
[o, 0.3, 9.5], # front_leff left Jack_left
[o, 1.2, 1.5], [o, 0, ol, # Front_right [o, 1.2, 1.5] back_rige right
Nvecute_poseetargee_posss
A.8.2
Sampled results for Turn on faucet task
User instruction:
Turn on faucet.
Ours
Sample 1
import numpy as np
reset_ (settrrewar()
This is
task reset reward; otherwise don't need it
set set_l2 wel.dditteeeeeee set_obj_orientation orientation "eveentationreward((" "" np.degarad(9e))
cacpnnceeeeard"""ee "palm", "faucet "aucet_handle") set_obj [r__joint_fractionreeeeaa "aarrriioorraaadaaaaaeaee faucet", 2)
execute_plan(2)
26

---

Sample 2
import numpy as np
reset_ (eettrrewar(()
This
task so reward; otherwise don't need
Der_l_ddistaneerewarrr rewaeedwddaaa"",, set fr__jooin_frattio____o
1) # means open
execute_plan(2)
Reward Coder only
Sample I
import numpy np
reset_reward() fet_joint_fraciion_rewaddd 1)
execute_plan()
Sample 2
import numpy np
reset_rewarde # Reset reward a new task set itt_jonn_ffratiinn
fraction (aan__eeard""""" faucet""
1) Turn on the faucet by setting the joint fraction to (open)
execute_plan(2)
Code-as-Policies
Sample
reset()
faucet_ hww_haanddeeeeee sanditnnossiionn get_obbect_positione faucet_handle""
end_effector wtt.ttoaettaaanleee_ppp
end_effector_ clooee)
end_effeector_open(
27

---

Sample 2
#python
import numpy
reset()
faucet_ _ou get
[nd_eefeecto_oppen] end_effeccor__llsee(
normalizedd_fuuctt_posiit
get rire_normllizeeee
target_faucet_pooittieee
0.5, 0, 1)
end wwweeeetto_ttooaauuuceecc [tt
faucet Suncet_haandeepos[[3 faucet sauuet_handle_pos[22 e.1])
faucet_hande sat__handle__s(s),
end_ cnu_eeffectoropee(( buucet_hanlle_po(ta]]
A.9 Full outputs for the interactive examples
Below we show the output reward code from our system for the two interactive examples: tasking quadruped robot to perform moonwalk, and tasking dexterous manipulator robot place an apple in a drawer.
Quadruped moonwalk
Instruction 1: Make the robot stand upright on two back feet like human.
import numpy as np
reset_rewarde
set_torso, set_torso_targets(..
np.degaraddgo), (p)deg2rad()), (e.e, o.0), None, e, None)
0.65, None, None) 0.0, None, None) set fer.fee_poo_prrmmeeerssCCCC right", 0.65, None, None)
set_
seg_feett.ospparrmettrrr""bb
set_feet_pos_arrmmttess ("aatk"bbaagee" right", 0.e, None, None)
execute_plan()
28

---

Instruction 2: Good, you actually don't need to keep the front paws certain height, just leave it to the controller.
import numpy as np
reset_ reset_rrward() )
set_torss_targets((e targets
set_torso np.degarad(9o), np. np.degarad (0.0, 0.0), None, 0, None)
hea_ffet_pos_prramttereresss baabbaakkkk''''.ro"aaa left", 0.0, None, None) Let_feet_pos_ppramettee ("aackarihhhtt"""""" o.e, None, None)
Set stepping parameters for all feet to disable stepping motion
for ["front_left', "rear_left,, "front_right" "rear_righ"
set htw.sseepppinn_paaatteesssss 0.e, 0.0, 0.0, e.e, 0.0, False)
execute_plan() )
Instruction 3: Good, now make the robot do moon-walk while stand up like this.
import numpy as np
reset_reward()
set_ set_torso, catoootttrttttttoeoo 65, dp.degarad np.deg2rad(o), None, -0.2 0.0),
0, None)
set s___fet_pos_pprrmeterssee ("abaca ("back_lbeeeeeeeee _eek_", None, None, None) sarippoo___rramteess"ttaeae
None, None, None)
set set_feet_steppe srret_sseeping_pprameteeers ("ackessssbbaaleca"""" 0.5, 0.5, 0.5, 8.1, -8.2, True)
fifeet_stepping_Peee (aaasss""""ku"""eee
("back_right" rrihht, 0.5, 0.5, 0.0, 0.1, -0.2, True)
set feetffett___ee (eaammtterssffoo_tteue ("front_left", 0.0, 0.0, 0.0, 0.0, 0.0, False)
set htt.pparmmmteesooottettreeeeeeee ("front_right? 0.0, 0.0, 0.0, 0.0, 0.0, False)
execute_plan()
29

---

Instruction 4: Moon walk means the robot should walk backward while the feet swings as if they are moving forward. Correct your answer and also make it walk at a speed of 0.5m/s.
import numpy np
reset_rewand reset_reward() set_torso__ cetrsso_argettee.bee np.degarad(90), op.degarad None, -0.5 0.0),
None)
set feet____ppp paree_pesparmmeeerrs (aacersssaaaaaeeeeaeeeeaa None, None, None) set set_feet_pos_parametee hatarammmttessabaaarriiiige None, None, None)
set sos_feet__ttepinngppraaeeeee ("back__ left", 0.5, 0.5, 0.5, 0.1, 0.2 True)
see_feet_seepinngpprrattttsss ("back_right", 0.5, 0.5, 0.0, 0.1, 0.2 True)
set set_feet_stepping_aaaaaa ("front_left') 0.0, 0.0, 0.0, 0.0, 0.0, False)
set st__feet_stepinng_prrmeeeer ("roocerrssffrrrrrrrreee front_right",, 0.0, 0.0, 0.0, 0.0, False)
execute_plan()
Dexterous Manipulator place an apple in drawer
Instruction I: Open the drawer.
import numpy as np
reset_reward()
This a new task so reset reward; otherwise we don't need tersstaneereeaaree cepooit_fraction_ttt reeeerrewwrd"" "wward""rrwwrr,, 1) # means open
"drawer_har (trawer_handle))
execute_plan()
Instruction Good, now put the apple inside the drawer while keep open. Assume drawer on the floor no need to lift up the apple.
import numpy as np
reset_ reset_reward()
This a new task so reset reward; otherwise we don't need "apple") "drawer_cee (eater_center")
set set_l2_dis [i____ddstance____ brs__ddistaac_eewar(("ppp iiloiitffraction Beinnnrewardd"d reward("drawer', 2)
execute_plan()
Instruction 3: Good, now release the apple and move hand away.
import numpy np
reset_ (reetrrewar(()
This is task so reset reward; otherwise don't need it 12, ditttddettaceeeeeaard "palm", "rest_position")
execute_plan(4)
30

---

Instruction 4: Now close the drawer.
import numpy np
reset_reward()
# This is a new task so reset reward; otherwise we don't need it le_ll__ditancee_.__ dtw_stanceeeeaarr( "pam", "drawer_hande" handle")
oet_joint_fraction_te rewarr("draawaer"" o)
execute_plan()
31


Reviews:
[{"review_id": "k4gZph4dTzP", "reviewer": "Reviewer_8cwu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors propose using a two-step Code-As-Policy approach using LLMs. This two step approach first translates a natural language prompt into a set of rewards and then translates those rewards into a set of function calls to be executed. Under the hood, mujoco MPC is used to turn the rewards into a plan which is executed on the robot.", "summary_of_recommendation": "While somewhat limited, the work illustrates an interesting and potentially useful factorized prompting strategy for generating rewards from an LLM that are of sufficient quality to synthesize (simple) behaviors. That's a useful, if slightly incremental, finding.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- What LLM was used for the Motion Descriptor and Reward coder and was any finetuning performed?\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors hypothesize that producing rewards, instead of directly producing robot commands, is more natural for LLMs and is likely to make better use of their inductive bias. Their experiments on simulated quadrupeds, an arm with dexterous hand, and a real-world robot equipped with parallels jaw gripper seem to back this up. Results seem promising compared to a code-as-policy baseline and their implementation detail of making reward prediction a two step process also appears affective when compared to the baseline of Reward Coder only.\n\nThat being said, I have some questions, likely beyond the scope of this paper, about this approach scaling. The tasks used here were all fairly simple, which means they were amenable to solving via MPC. That's not to say that LLM reward generation wasn't useful, it certainly eased the design work of humans requesting tasks, but for more complex tasks finding good rewards that both sufficiently constrain behavior and allow efficient solutions is known to be difficult and this work does not address that difficulty.\n\nThe work is also lacking on details about the LLM used. It seems it was an off-the-shelf LLM with only in-context learning used.\n\nAs an aside, this seems highly related to the concurrent work VoxPoser (https://voxposer.github.io/). That's not itself a problem, but the authors should consider citing it as concurrent work.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "h_36pjSRdfs", "reviewer": "Reviewer_NCUb", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "In the present work, the authors explore a task to employ LLM in robotics. In their approach, LLM defines a task by generating a reward function which is afterward optimized by model predictive control methods. The reward function is generated in two steps. In the first step, LLM receives a description of the desired state and outputs the target position of the robot. In the second step, LLM receives a description of the target position and outputs the reward function. The authors demonstrate the capabilities of the proposed method in a simulated quadruped robot and a dexterous manipulator robot and on a real robotic arm. ", "summary_of_recommendation": "I recommend accepting the paper. It presents an interesting method to ground LLM for robotics tasks. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "How easy it would be to apply this approach to generate different motion styles like gallop, trot, walk? ", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Large language models have demonstrated remarkable capabilities in natural language understanding. Nevertheless the direct application of LLMs to control robotic devices is not possible. The authors investigate one of the possible approaches that generates reward functions from LLM given a textual description of a task. This approach seems to be quite generic since in a wide range of applications reward can be defined by simple equation. The paper clearly presents the developed approach and the evaluation results. The authors show that the proposed method performs better then code-as-policies baseline and baseline with only one LLM with returns reward directly from natural language task description. \n\nOne possible limitation of the presented approach is simplicity of generated reward function. For example to train a robust reinforcement learning policy to control quadruped robot on needs to design a reward function with multiple terms punishing big torques, velocities, non-smooth movements. However the punishment terms may be similar for different tasks and may be handcrafted and shared between them. \n\nAlso there are some sentences with missing spaces like \"... the Reward Translatorthat interprets the user ...\"", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "RWGtNg8-Vz", "reviewer": "Reviewer_ba4p", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper introduces a pipeline for using LLM to generate reward specification for a variety of RL tasks for a quadruped and serial robot manipulator. The authors design a two-part pipeline to prompt GPT-4 to create reliable and parameterized reward code used to train the motion controller using a MJPC planner. The work conducts an ablation study using different prompting contexts to demonstrate differences in the RL model’s success rate on each task.", "summary_of_recommendation": "While the results are interesting and the experiment selection was well designed, I believe the paper still lacks in several key areas. I do not feel that the paper is significantly novel, and there is a lack of real world quantitative results. As a result, I would recommend a weak rejection for this paper.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Please address the concerns in the review. No additional questions.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Significance:**\nThe authors propose a two-part prompt for GPT-4 to produce parameterized reward code. While this approach is interesting, I believe it lacks novelty. Much of the contribution seems to center around the prompt engineering to produce a reward parameterization. Afterwards, the paper uses a standard motion predictive controller. However, I believe that the paper does explore an important avenue within LLM + Robotic Learning / AutoRL research area.\n\nI believe the paper would significantly benefit from evaluating the method’s performance on a vaguer “motion descriptor prompt” and a “reward coder prompt” standardized across robot types. The current method, as-is, still requires extensive domain expertise and human intervention to generate reward code.\n\n**Quality:**\nThe experiments conducted by the authors were thorough and well-designed within the paper’s objective. The authors clearly explored two RL domains (quadruped robot, serial manipulator) through 17 total tasks. As an additional ablation study, I would be curious to see the RL performance when the LLM is given varying levels of context (ex: fewer variable definitions, fewer reminders, etc).\n\nI would also be interested in seeing results from the real-world experiment. Were the success rates for each task similar to the simulated rates? \n\n**Clarity:**\nThe paper was very well written and clear. The data was presented and explained in a direct and concise manner. The sample prompts and videos in the supplemental materials were well received. There were a few spelling and grammar errors that do not impact the overall message of the paper.\n- Figure 1 caption requires space between \"Translator\" and \"that\".\n\n\nAs summarized from the above discussion on significance, quality, and clarity...\n\n**Strengths:**\n- The paper was clear and concise. The supporting materials were helpful.\n- The experimental design allowed for a thoroughly exploration of the scope of the paper.\n- The exploration of interactive systems for reward tuning was a cool idea.\n\n**Weaknesses:**\n- No sensitivity study of the effects of varying the amount of context given to the LLM.\n- Although interesting, the methods in the paper only produce a incremental contribution.\n- Experiments largely focus on simulated results. Quantitative results on real world experiments should be included.\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses the problem of skill generation where an input language instruction must be translated to a reward model. The core contribution lies in the design of a reward interface that associates a high level instruction to parameters of a reward model. Once the reward model is known, a MPC is used to generate control. From the LLM perspective, the approach lowers the symbolic to real boundary by reasoning over reward parameters instead of pure symbols as used in task planning. 

During the rebuttal phase the authors provided clarifications on the experiments, engaged with closely related works and uploaded data on real world experiments. The authors are requested to update their manuscript based on the reviewer feedback.
Decision: Accept (Oral)