Text2Reward: Reward Shaping with Language Models for Reinforcement Learning
Authors: Tianbao Xie
Siheng Zhao
Chen Henry Wu
Yitao Liu
Qian Luo
Victor Zhong
Yanchao Yang
Tao Yu
Venue: ICLR 2024 spotlight (2024)
Keywords: reinforcement learning; large language models; robotics
Primary area: reinforcement learning
TL;DR: Reward shaping for reinforcement learning based on large language models

Abstract: Designing reward functions is a longstanding challenge in reinforcement learning (RL); it requires specialized knowledge or domain data, leading to high costs for development. To address this, we introduce Text2Reward, a data-free framework that automates the generation and shaping of dense reward functions based on large language models (LLMs). Given a goal described in natural language, Text2Reward generates shaped dense reward functions as an executable program grounded in a compact representation of the environment. Unlike inverse RL and recent work that uses LLMs to write sparse reward codes or unshaped dense rewards with a constant function across timesteps, Text2Reward produces interpretable, free-form dense reward codes that cover a wide range of tasks, utilize existing packages, and allow iterative refinement with human feedback. We evaluate Text2Reward on two robotic manipulation benchmarks (ManiSkill2, MetaWorld) and two locomotion environments of MuJoCo. On 13 of the 17 manipulation tasks, policies trained with generated reward codes achieve similar or better task success rates and convergence speed than expert-written reward codes. For locomotion tasks, our method learns six novel locomotion behaviors with a success rate exceeding 94%. Furthermore, we show that the policies trained in the simulator with our method can be deployed in the real world. Finally, Text2Reward further improves the policies by refining their reward functions with human feedback. Video results are available at https://text-to-reward.github.io/

Published as a conference paper at ICLR 2024
TEXT2RENH TEXTRREWARD: REWARD SHAPING WITH LANGUAGE MODELS FOR REINFORCEMENT LEARNING
Tianbao Xie
Siheng Zhao
Chen Henry Wu
Yitao Liu
Qian Luo
Victor Zhong
Yanchao Yang
Tao Yu
The University of Hong Kong
Nanjing University
O Carnegie Mellon University
Microsoft Research
University of Waterloo
ABSTRACT
Designing reward functiins a longstanding challenge in reinforcement learning (RL); it requires specialized knowledge domain data, leading to high costs for development. To address this, we introduce TEXT2REWARD, data-free frame- work that automates the generation and shaping of dense reward functions based on large language models (LLMs). Given a goal described in natural language, TEXT2REWARD generates shaped dense reward functions as executable pro- gram grounded in a compact representation of the environment, Unlike inverse RL and recent work that uses LLMs to write sparse reward codes or unshaped dense rewards with constant function across timesteps, TEXT RRWARRD produces interpretable, free-form dense reward codes that cover wide range of tasks, utilize existing packages, and allow iterative refinement with human feedback. We eval- uate TEXT2REWARD on tWO robotic manipulation benchmarks (MANISKILL2, METAWORLD) and two locomotion environments of MUJoCo. On 13 of the 17 manipulation tasks, policies trained with generated reward codes achieve similar or better task success rates and convergence speed than werrtee reward codes. For locomotion tasks, our method learns six novel locomotion behaviors with success rate exceeding 94%. Furthermore, in the simulator with our method can be deployed in the real world. Finally, TEXT2REWARD further improves the policies by refining their reward functions with human feedback. Video results are available at hutps://exttto-reearrd wwr.tre..g...g.gg....o github.io
show that the policies trained
INTRODUCTION
Reward shaping (Ng et al. 1999) remains long-standing challenge in reinforcement learning (RL); aims to design reward functions that guide an agent towards desired behaviors more efficiently. Traditionally, reward shaping is often done by manually designing rewards based expert intuition and heuristics, while it a consuming process that demands expertise and can be sub-optimal.
Inverse reinforcement learning (IRL) (Ziebart al. 2008 Wulfmeier et al.) 2016 Finn al. 2016)
and preference learning (Christiano et al.) 2017 Ibarz et al. 2018] Lee et al. 2021 Park et al. 2022
have emerged as potential solutions reward shaping. reward model is learned from human demonstrations or preference-based feedback. However, both strategies still require considerable
human effort data collection; also, the neural network-based reward models are not interpretable and cannot be generalized out of the domains of the training data.
This paper introduces a novel framework, TEXT2 to generate and shape dense reward code based on goal descriptions. Given an RL goal (e.g.. "push the chair to the marked position"). "), TEXT2REWARD generates dense reward code (Figure [ middle) based on large language mod- els (LLMss), grounded on a compact, Pythonic representation of the environment (Figure [ left). The dense reward code is then used by an RL. algorithm such as PPO (Schulman et al. (017) and SAC (Haarnoja et al. 2018)) to train policy (Figure [ right). Different from inverse RL, TEXT2REWARD is data-free and generates symbolic reward with high interpretability. Different from recent work (Yu al. 2023) that used LLMs write unshaped reward code with hand hand-designed APIs, our free-form shaped dense reward code has a wider coverage of tasks and can utilize established
Equal contribution. Work mainly done the University of Hong Kong. +Corresponding author.

---

Published as a conference paper at ICLR 2024
Feedback: keep standing
Dense reward function
def
action)s
Goal: push the chair the marked position
esproach
rp.ercomntc,.eo chatr(3))
Enwironment description
GPT-41 Codae
e Stage rowand
class
self.chate: artteolanedejeet
cheir, rEll 6.2 ro if sist_wwtte. swu.ee 0.22
Po
self robot: Oialerehando
Relcct
class entiovlatedejecti
pose: 0sjecthese
& gec the polnt cloud of the object
get_pca(ssir) ns.ndarrayl (N, builarafunda:
PULLET roward
positions gripper tingers
Envirommntt
seif.es_come >3 op (p.33)rray((433)
apertaatttttion
An overview TEXT2REWARD of three stages: Expert Abstraction provides an abstraction of the environment as a hierarchy of Pythonic classes. User Instruction describes the goal to be achieved in natural language. User Feedback allows users to summarize the failure mode or their preferences, which are used improve the reward code.
Figure 1:
coding packages (e.g. NumPy operations over point clouds and agent positions). Finally, given the sensitivity of RL training and the ambiguity of language, the RL policy may fail achieve the goal or achieve it in unintended ways. TEXT2REWARD addresses this problem by executing the learned policy the environment, requesting human feedback, and refining the reward accordingly.
We conduct systematic experiments on two robotics manipulation benchmarks (MANISKILL2 Guu al.] 2023), METAWORLD Yu et al. 2020)) and two locomotion environments of MuJoCo (Brock man et al. 2016), as cases. On 13 out of 17 manipulation tasks, policies trained with our generated reward code achieve comparable or better success rates and convergence speed than the ground truth reward code carefully tuned by human experts. For locomotion, TEXT2REWARD learns 6 novel locomotion behaviors with over 94% success rate. We also demonstrate that the policy trained in real Franka Panda robot. With human feedback of less than
the simulator can be deployed iterations, our method can iteratively improve the success rate of learned policy from to almost 100%, well as resolve task ambiguity. In summary, the experimental results demonstrated that TEXT2REWAL BRRWAARS can generate generalizable and interpretable dense reward code, enabling wide coverage of RL tasks and human-in-the-loop pipeline. We hope that the results can inspire further explorations in the intersection of reinforcement learning and code generation.
2 APPROACH
2.1 BACKGROUND
Reward code Reinforcement learning (RL) aims to learn a policy that maximizes the expected reward in an episode. To train policy to achieve goal, the key is to design reward function that specifies the goal. The reward function can take various forms such as a neural network or a piece of reward code. In this paper, observation and the action are represented as variables, such that the reward does not need to handle perception it only reasons about abstract variables and APIs in code.
focus the reward code given its interpretability. In this case, the
Reward shaping Reinforcement learning from task completion rewards is difficult because the reward signals are sparse and delayed (Sutton Barto 2005). A shaped dense reward function is useful since it encourages key intermediate steps and regularization that help achieve the goal. In the form of code, the shaped dense reward can take different functional forms at each timestep. instead of being constant across timesteps or just at the end of the episode.
2

---

Published as a conference paper at ICLR 2024
2.2 ZERO- SHRO-SHOTAAA AND FEW-SHOT DENSE REWARD GENERATION
In this part, we describe the core of TEXT2REWARD for zero-shot and few-shot dense reward generation. Detailed prompt examples can be found in the Appendix C Interactive generation is
described in the next subsection.
Instruction The instruction is natural language sentence that describes what we want the agent to achieve (e.g. "push the chair to the marked position"). It can be provided by the user, or it can be one of the subgoals for a long- horizon task, planned by the LLM.
Environment abstraction To ground reward generation in an environment, it is necessary for the model to know how object states are represented in the environment, such as the configuration of robots and objects, and what functions can be called. We adopt a compact representation in Pythonic style as shown in Figure I information in the list table format, Pythonic representation has higher
which utilizes Python class, typing, and comment. Compared to listing
all prrironmment-speccin level of abstraction and allows us to write general, reusable prompts across different environments. Moreover, this Pythonic representation is prevalent in LLMs pre-training data, making it easier for the LLM to understand the environment.
Background knowledge Generating dense reward codes can be challenging for LLMs due to the scarcity of data in these domains. Recent works have shown the benefits of providing relevant function information and usage examples to facilitate code generation (Shi et al. 2022: Zhou et al.. 2022). Inspired by them, we provide functions that can be helpful in this environment as back knowledge (e.g., NumPy/SciPy functions for pairwise distance and quaternion computation, specified by their input and output types and natural language explanations).
Few-shot examples Providing relevant examples as input has been shown to be useful in helping LLMs solve tasks. We assume access to pool of pairs of instructions and verified reward codes. The library can be initialized by experts and then continually extended by our generated dense reward code. We utilize the sentence embedding model from Su al. (2022) to encode each instruction. Given new instruction, we use the embedding to retrieve the top-k similar instructions and concatenate the instruction-code pairs as few-shot examples. We set the to I since context length limits and filter out the oracle code of the task from the retrieved pool to make sure that the LLMs do not cheat.
Reducing error with code execution Once the reward code is generated, we execute the code in the code interpreter. This step may give us valuable feedback, e.g., syntax errors and runtime errors (e.g., shape mismatch between matrices). line with previous works (Le et al.] [002]] Olausson et al. tool for ongoing refinement within the LLM.
2023), we utilize the feedback from code execution This iterative process fosters the systematic rectification of errors and continues until the code is devoid of errors. Our experiments show that this step decreases error rates from 10% to near zero.
2.3 IMPROVING REWARD CODE FROM HUMAN FEEDBACK
Humans seldom specify precise intent in single interaction. In an optimistic scenario, the initial generated reward functions may be semantically correct but practically sub-optimal. For instance, users instructing a robot to open a cabinet may not specify whether to pull the handle or the edge of the door. While both methods open the cabinet, the former is preferable because it is less likely to damage the furniture and the robot. In pessimistic scenario, the initially generated reward function may too difficult to accomplish. For instance, telling a robot "clean up the desk" results in a more difficult learning process than telling the robot to "pick up items on the desk and then put them in the drawer below". While both deseriptions specify the same intent, the latter provides intermediate objectives that simplify the learning problem.
To address the problem of under-specified instructions resulting sub-optimal reward func- tions, TEXT2REWARD actively requests human feedback from users to improve the generated reward functions. After every RL training cycle, the users are provided with rollout videos of task execution by the current policy. Users then offer critical insights and feedback based the video, identifying areas of improvement or errors. This feedback is integrated into subsequent prompts to generate more refined and efficient reward functions. In the first example of opening a cabinet, the
3

---

Published as a conference paper at ICLR 2024
user may say "use the door handles" to discourage the robot from damaging itself and the furniture by opening using the door edges. In the second example of cleaning a desk, the user may say "pick up the items and store them in the drawer" to encourage the robot solve sub-tasks. It is noteworthy that this setup encourages the participation of general users, devoid of expertise in programming or RI., enabling democratized approach optimizing system functionality through natural language instructions, thus eliminating the necessity for expert intervention.
3 EXPERIMENT SETUP
We evaluate TEXT2REWARD on manipulation and locomotion tasks across three environments: METAWORLD, MANISKILL2, and Gym MUJoCo. We use GPT-4" as the LLM to demonstrate our method and further includes an examination of other open-source models, as presented in Appendix F This is done not only to ensure reproducibility but also to clarify the complexity of the task at hand. We choose the RL algorithm (PPO or SAC) and set default hyper- parer--rameterss according to the performance of human-written reward, and fix that in all experiments on this task to do RL training. Experiment hyperparameters are listed in Appendix A
3.1 MANIPULATION TASKS
We demonstrate manipulation on METAWORLD, commonly used benchmark for Multi-task Robotics Learning and Preference-basee based Reinforcement Learning (Nair et al. 2022 Lee al. 202T Hejna III & Sadigh 2023). and MANISKILL.2, platform showcasing diverse range of object manipulation tasks executed within environments with realistic physical simulations. We evaluate a diverse set of manipulation tasks including pick-and-place, assembly, articulated object manipulation with revolute or sliding joint, and mobile manipulation. For all tasks, we compare TEXT2REWARD with oracle reward functions tuned by human experts (provided in the original codebases). We also establish baseline by adapting the prompt from Yu et al. (2023) to suit our reinforcement learning framework, as opposed to the Model Predictive Control (MPC) setting originally deseribed in Yu et al.) (2023) since designing the physics model demands a considerable amount of additional expert labor. For RL training, we tune the hyperparameters such that the oracle reward functions have the best results, and then keep them fixed when running TEXT2REWARD. The full list of tasks, corresponding input instructions, and details of simulated environments are found in Appendix B
3.2 LOCOMOTION TASKS
For locomotion tasks, we demonstrate our method using Gym MUJoCo. Due to the lack of expert- written reward functions for locomotion tasks, we follow previous work (Christiano et al. [017 Lee cct al. 202T) to evaluate the policy based on human judgment of the rollout vidco. We develop six novel tasks in total for two different locomotion agents, Hopper (a 2D unipedal robot) and Ant (a 3D quadruped robot). The tasks include Move Forward, Front Flip and Back Flip for Hopper, as well as Move Forward, Lie Down, and Wave Leg for Ant.
3.3 REAL ROBOT MANIPULATION
Unlike model-based methods such as model predictive control (MPC) (Howell al. 2022). which require further parameter adjustment, our RL agents-traineet trained in simulator can be directly deployed in the real world, necessitating only minor calibration and the introduction of random noise for sim- to-real transfer. To demonstrate this benefit, as well as verify the generalization ability of RL policy trained from our generated reward, we conducted a real robot manipulation experiment with the Franka Panda robot arm. We verify our approach on two manipulation tasks: Pick Cube and Stack Cube. To obtain the object state required by our RL policy, we use the Segment Anything Model (SAM) (Kirillov et al. 2023) and depth camera to get the estimated pose of objects. Specifically, query SAM to segment each object in the scene. The segmentation map and the depth map together give us incomplete point cloud, We then estimate the pose of the object based on this point cloud.
http:://htttoo...mmmiiiccooooo// This work mainly uses gpt-4-033 4-0314.
4

---

Published as a conference paper at ICLR 2024
3.4 INTERACTIVE GENERATION WITH HUMAN FEEDBACK
We conduct human feedback on challenging task for single-round reward code generation, Stack Cube, investigate whether human feedback can improve fix the reward code, enabling RL algorithms to successfully train models in a given environment. This task involves reaching the cube, grasping the cube, placing the cube on top of another cube, and releasing the cube while making it static. We sample 3 generated codes from zero-shot and few-shot methods and perform this task with two rounds of feedback. In addition, we also conduct experiments on one locomotion task Ant Lie Down, where the initial training results do not satisfy the user's s preference. The general user who provides the feedback can only see the rollout video and learning curve, without any code. The authors provide feedback as per the described setup.
Crucie
Teo shot
LINCUR
Open
PACK Cube
Cabinet
8.2
Figure 2: Learning curves on MANISKILL2 under zero-shot and few-shot reward generation settings, measured by task success rate. L2R means the baseline adapted from Yu al. (023); Oracle means the expert-written reward function provided by the environment; zero-shot and few-shot (k=l) means the reward function is generated by TEXTDREWARD from other tasks. The solid line represents the mean success rate, while the shaded regions correspond the standard deviation, both calculated across five different random seeds.
and retrieving expert-written examples
Drawer Open
Window Closs
Button Pross
04
86
02
62
400
500k
Sweep Info
Door Close
Hondle Press
98
0.5
03
Figure 3: Learning curves METAWORLDD under zero-shot reward generation setting, measured by success rate. Following Figure [ the solid line represents the mean success rate, while the shaded regions correspond to the standard deviation, both calculated across five different random seeds. Additional results can be found in the Appendix G
5

---

Published as a conference paper at ICLR 2024
(a) Hopper Back Flip
(b) Ant Lie Down
(c) Ant Wave Leg
Figure 4: Novel locomotion behaviors acquired through TEXT2REWARD under zero-shot reward
generation setting. These images are sampled by policy rollouts in Gym MuJoCo.
RESULTS AND ANALYSIS
4.1 MAIN RESULTS
This section shows the results of TEXT2REWARD for robotics manipulation and locomotion. Gener- ated reward function samples can be found in the Appendix D
TEXT2REWARD = expert-designed rewards on manipulation tasks. Quantitative results from the MANISKILL.2 and METAWORLD environments are shown in Figures and 3 In the figures, L2R stands for reward function generated by the baseline prompt adapted from Yu et al. (023)): Oracle means the expert-writles dense reward function provided by the environment; zero-shot and few-shot stands for the dense reward function generated by TEXT2REWARD without human feedback under zero-shot and few-shot prompting paradigms, respectively. On 13 of the 17 tasks, the final performance (i.e., success rate after convergence and convergence speed) of TEXT2REWARD achieves comparable results to the human oracle. Surprisingly, TEXT2REWARD can even outperform human oracle, in terms of either the convergence speed (e.g., Open Cabinet Door in MANISKILL2, Handle Press in METAWORLD" Cube in MANISKILL2, Drawer Open in METAWORLD), It suggests that LLMs have the potential to draft high-quality shaped dense reward functions without any human intervention.
4 of the 17 tasks, zero-shot and few-shot
the success rate (c.g., Pick
As demonstrated in Figure 2 the L2R baseline can only be effectively applied to two tasks of MANISKILL2, where attains results comparable those of the zero-shot setting. Nevertheless, L2R struggles with tasks involving objects that have complex surfaces and cannot be adequately described by a singular point, such as an ergonomic chair. In these tasks, the environment utilizes point cloud to denote the object's 's surface, representation that 1.2R fails to model. To more comprehensively evaluate the necessity of shaped and staged dense rewards, we introduce an additional baseline in Appendix E that circumvents the aforementioned limitation of the L2R formulation.
of the tasks that are not fully solvable, the few-shot paradigm markedly outperforms the zero-shot approach. This underscores the benefits of utilizing few-shot examples from our skills library in enhancing the efficacy of RL training reward functions.
Furthermore, as illustrated Figure 2
TEXT2REWARD can learn novel locomotion behaviors. Table shows the success rate of all six tasks trained with the reward generated under the zero-shot setting, evaluated by humans watching the rollout videos. The results suggest that our method can generate dense reward functions that generalize to novel locomotion tasks. Image samples from the Gym MUJOCO environment of three selected tasks are shown in Figure Corresponding full video results are available here
real robot. Figure 5hows the key frames of real robot manipulation on two tasks: Pick Cube and Stack Cube. Here, we use the same DoF Franka Panda robot arm as MANIŠKILL.2 simulation environment. Results suggest that the RL policy trained in the
Demonstrating TEXT2REWAN

---

Published as a conference paper at ICLR 2024
Table Success rate of locomotion tasks in Gym MuJoCo trained reward functions generated in zero-shot setting. Each task is tested on 100 rollouts, and task success is determined by the authors who reach an agreement after reviewing the rollout videos. Generated codes are in Appendix D.2
Hopper
Ant
Task
Success Rate
Task
Success Rate
Move Forward
100%
Move Forward
94%
Front Flip
99%
Lie Down
98%
Back Flip
100%
Wave Leg
95%
Pick Cube
Stack Cube
Figure 5: Sampled images for real robot manipulation Pick Cube (i.e., pick cube and move it to a predefined position) and Stack Cube (i.e., stack a cube onto another one).
simulator using dense reward function generated from TEXT2REWARD can be successfully deployed the real world. Full videos of robot execution are our project page
TEXT2REWARD can resolve ambiguity from human feedback. To demonstrate the ability of FEXT2REWARD to address this problem, we show one case in which "control the Ant to lie down" itself has ambiguity in terms of the orientation of the Ant, shown in Figure After observing the training result this instruction, the user can give the feedback in natural language, c.g., "the Ant's torso should be top down, not bottom up". Then TEXT2REWARE will regenerate the reward code and train a new policy, which successfully caters to the user's intent.
Instructdon: Control
Faodback: The Art's torso should up.
Interactive reward generation from human feedback. The original instruction is control the Ant to lie down, which is ambiguous in the orientation of the Ant. Our interactive framework allows the user to provide feedback based the rollout observation.
Figure 6:
TEXT2REWARD can improve RL training from human feedback. Given the sensitivity of RL training, sometimes hingle-turn generation can not generate good enough reward functions to finish the task. these cases, REWARD asks for human feedback on the failure mode and tries to improve the dense reward. In Figure 7 we demonstrate this the Stack Cube task, where zero-shot and few-shot generation in a single turn fails to solve the task stably. For few-shot generation, we observed that interactive code generation with human feedback can improve the success rate from zero to one, as well as speed up the convergence speed of training. However, this improvement is prone to the quality of the reward function generated in the beginning (i.e. iter0). For relatively
7

---

Published as a conference paper at ICLR 2024
low-quality reward functions (e.g. zero-shot generated codes), the improvement in success rate after iterations of feedback is not as pronounced as for few-shot generated codes. This problem may be solved in sparse-to to-se reward function generation manner, which generates the stage reward first and then generates reward terms interactively. We leave this paradigm for possible future work.
Iter0
Stack Cube-Interactive
Zero-shot
0.6
0.4
0.2
4N
0.8
Few-shot
0.6
Step.
2M
4M
Figure 7: Training iteration vs. success rate on Stack Cube with interactive generation. Oracle the reward code manually tuned by experts; iterO is generated by TEXT2REWARD without feedback; iterl and iter2 are generated after or 2 feedback iterations; zero-shot and and few-shot (k=1) stands for how the iterr code is generated. The solid lines represent the mean success rate, and the shaded regions correspond to the standard deviation, with three samples.
4.2 QUALITATIVE ANALYSTS
In this section, summarize the differences between the generated reward functions (zero-shot, few-shot) and the oracle reward functions. Due to space limitation, the reward functions we will refer are in Appendix D
Few-shot outperforming zero-shot Few-shot settings generally surpass zero-shot in terms of downstream performance. In the Lift Cube and Pick Cube tasks, few-shot generated code delineates stages approach, grasp, and lift more clearly using conditional statements. In contrast, zero-shot code stacks and sums different stage rewards in a linear manner, reducing effectiveness. This is also demonstrated by the Push Chair task, which requires nuanced commonsense steps that zero-shot code only partially captures. Since the stage reward format shown the few-shot examples, GPT-4 can generalize the pattern even we instruct it to do so, which may be improved by the next version of LLMs.
tasks. Currently, this ability cannot be fully unfolded with zero-shot
Zero-shot sometimes outperforming few-shot Our experiments also show that zero-shot learning occasionally outperforms few-shot learning, as seen in tasks like Turn Faucet and Open Doors. By checking the generated code, we found that the quality and relevance of the few-shot examples are the key. For example, in the Open Door task, fho-shot learning lags it omits key reward term the door's positional change- thus hindering learning. Similarly, in the Turn Faucet task, zero-shot rewards facilitate easier success by allowing one-sided gripper actions, which leads effectiveness.
Few-shot sometimes outperforming Oracle There are scenarios where few-shot learning even outperforms the oracle reward function crafted by experts. An analysis of the Open Door task reward function shows that few-shot generated code omits the final stage of stabilizing the door, simplifying the policy learning process improving the learning curve, and finishing tasks in a slightly different manner. In the Pick Cube task, few-shot and oracle codes are structurally similar, but the weight terms are different. This shows that LLMs cannot only improve reward structure and logic but also tune the hyperparameters. Although this is not our focus here, it can be promising direction.
8

---

Published as a conference paper at ICLR 2024
5 RELATED WORK
Reward Shaping Reward shaping remains a persistent challenge in the domain of reinforcement learning (RL). Traditionally, handcrafted reward functions are employed, yet crafting precise reward functions is a time-consuming process that demands expertise. Inverse reinforcement learning (IRL) emerges as a potential solution, where a non non-linear reward model is recovered from expert trajectories to facilitate RL learning (Ziebart et al., 2008; Wulffeeir et al. [0066] Finn et al., (2016). However, this technique necessitates large amount of high-quality trajectory data, which can be elusive for complex and rare tasks. An alternative approach is preference learning. which develops reward model based on human preferences (Christiano et al. 2017] [barz et al.) 2018 Lee et al. 2021] Park al. 2022 Zhu et al.) 2023). In this method, humans distinguish preferences between pairs of actions, upon which a reward model is constructed utilizing the preference data. Nonetheless, this strategy still requires some human- annootaed preference data which is expensive or even hard to collect in some cases. Both of these prevalent approaches to reward shaping demand extensive high-quality data, resulting in compromised generalizability and low efficiency. In contrast, TEXT2RE with limited (or even zero) data input and can be easily generalized to new tasks in the environment.
excels
Language Models in Reinforcement Learning Large Language Models (LLMs) have exhibited remarkable reasoning and planning capabilities (Wei et al. [022; Huang et al. 2022a). Recent works have shown that the knowledge in LLMs can be helpful for RL and can transform the data-driven policy network acquisition paradigm (Carta et al. 2023 Wu et al. 2023). This trend sees LLM- powered autonomous agents coming to the fore, with a growing body of approaches that use LLMs as
policy networks during the RL process, indicating promising trajectory in this field (Yao et al. 2022 Shinn et al., 2023: Lin et al., 2023) Wang et al., 2023; Xu et al. 2023 Yao et al., [023] Hao et al. 2023). Instead of directly using LLMs as the policy model the reward model, TEXT2
generates shaped dense reward code to train RL policies, which has an advantage in terms of the
flexibility of agent model type and inference efficiency.
Language Models for Robotics Utilizing LLMs for embodied applications emerges as a popular trend of research, and typical directions include planning and reasoning through language model
generation (Ahn et al., [022] Zeng et al., 2022 Liang et al.] 2022; Huang et al. 2022b; Singh et al.
2023; Song et al. (022). Recent works have harnessed the capabilities of LLMs to assist in the
learning of primitive tasks (Brohan et al. 2022 Huang et al. 2023 Brohan et al. 2023 Mu et al.
2023). by finetuning LLMs on robotic trajectories to predict primitive actions while TEXT2
generates reward codes to learn smaller policy networks. A recent work, L2R (Yu et al. (023). combines reward generation and Model Predictive Control (MPC) to synthesize robotic actions. Although such rewards work well for many tasks with well-designed MPC, our experiments show that RL training is challenging for unshaped rewards, especially on complex tasks. Different from them, TEXT2REWARD adopts more flexible program structures such as if-elss conditions and point cloud queries to offer higher flexibility and work for tasks that unshaped dense rewards tend to fail.
6 CONCLUSION
We proposed TEXT2REWARD, an interactive reward code generation framework that uses LLMs to automate reward shaping for reinforcement learning. Our experiments showcased the effectiveness of our approach, as the RL policies trained with our generated reward codes were able to match or even surpass the performance of those trained with expert-designed codes in the majority of tasks. We also showcased real-world applicability by deploying policy trained in a simulator robot. By incorporating human feedback, our approach iteratively refines the generated reward codes, addressing the challenge of language ambiguity and improving the success rates of learned policies. This interactive learning process allows for better alignment with human needs and preferences, leading to more robust and efficient reinforcement learning solutions.
a real
In conclusion, TEXT2R 2 REWARD demonstrates the effectiveness of using natural language to transform human intentions and LLMs knowledge into reward functions, then policy functions. We hope that our work may serve as an inspiration for researchers across various disciplines, including but not limited to reinforcement learning and code generation, further investigate this promising intersection of fields and contribute to the ongoing advancement of research in these areas.

---

Published as a conference paper at ICLR 2024
REFERENCES
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. as can, not say: Grounding language in robotic affordances. arXiv preprint arXii:2204.01000 2022.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint arXiv: 01540, 2016.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale, arXiv preprint arXiv:2212.06817 2022.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. R1-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.158/88 2023.
Thomas Carta, Clément Romac, Thomas Wolf, Sylvain Lamprier, Olivier Sigaud, and Pierre- Yves Oudeyer. Grounding large language models in interactive environments with online reinforcement learning. arXiv preprint arXiv:2332.006660 2023.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint urXiv:2107.03377 2021.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint 2xXi:21100144666 2021.
Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang. Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. Advances in Neural Information Processing Systems, 35: 18343-18362, 2022.
Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization, 2016.
Jiayuan Gu, Fanbo Xiang, Xuanlin Li, Zhan Ling, Xiqiang Liu, Tongzhou Mu, Yihe Tang, Stone Tao, Xinyue Wei, Yunchao Yao, et al. Maniskil12: A unified benchmark for generalizable manipulation skills. arXiv preprint arXiv:2302.046553 2023.
Tuomas Haarnoja, Aurick Zhou, P. Abbeel, and Sergey Levine. Soft actor-rriti:: Off-policy maximum entropy deep reinforcement learning with stochastic actor. ArXiv, abs/1801.01 881.01290, 2018.
Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model. arXiv preprint 2rXiv:2305.14999 2023.
Donald Joseph Hejna III and Dorsa Sadigh. Few-shot preference learning for human-in-the-lee rl. In Conference on Robot Learning, pp. 2014-2025. PMLR, 2023.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Joaquin Vanschoren and Sai-Kit Yeung (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1. NeurlPS Datasets and Benchmarks 2021. December 2021, virtual, 2021. URL wttpppps..cppppp//221111 htw.rraatttroood.hhtata..
round2.hte
10

---

Published as a conference paper at ICLR 2024
Taylor Howell, Nimrod Gileadi, Saran Tunyasuvunakool, Kevin Zakka, Tom Erez, and Yuval Tassa. Predictive Sampling: Real-time Behaviour Synthesis with MuloCo. dec 2022. doi:
10.48500urxi.2222.00551 Utthttps:// wttp//aaivvv... org/
harg/abs abs 2ww.22211 00541
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference Machine Learning, pp. 9118-9147, PMLR, 2022a.
Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint 2.X1:22207.06088, 2022b.
Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. arXiv preprint arXiv:2300.005973 2023.
Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari, 2018.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yee Lo, et al. Segment anything. arXiv preprint arXiv:2304.000 02643, 2023.
Hung Le, Yue Wang. Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu Hong Hoi. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. Advances in Neural Information Processing Systems, 35:21314-21328, 2022.
Kimin Lee, Laura Smith, and P. Abbeel. Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. In International Conference on Machine Learning, 2021.
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. arXiv preprint arXiv:2209.07753 2022.
Bill Yuchen Lin, Yicheng Fu, Karina Yang, Prithviraj Ammanabrolu, Faeze Brahman, Shiyu Huang, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. Swiftsage: A generative agent with fast and slow thinking for complex interactive tasks. arXiv preprint 23XIv:2305.13390 2023.
Yao Mu, Qinglong Zhang, Mengkang Hu, Wenhai Wang, Mingyu Ding, Jun Jin, Bin Wang, Jifeng Dai, Yu Qiao, and Ping Luo. EmbodiedGPT: Vision-language pre-training via embodied chain of thought, 2023.
Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhi Gupta. R3m: A universal visual representation for robot manipulation. In Conference on Robot Learning, 2022.
Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application reward shaping. ICML, volume 99, pp. 278-287. Citeseer, 1999.
Theo X Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar-Lezama. Demystifying gpt self-repair for code generation. arXiv preprint arXiv:2306.09890 2023.
Jongjin Park, Younggyo Seo, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. Surf: Semi-supervised reward learning with data augmentation for feedback-efficient preference-based reinforcement lcarning, 2022.
Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba. Virtualhome: Simulating household activities via programs. In Proceedings the on Computer Vision and Pattern Recognition, pp. 8494-8502, 2018
the IEEE Conference
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code llama: Open foundation models for code. arXiv preprint 2308 2tl.::23302/0000000 /2950, 2023.
11

---

Published as a conference paper at ICLR 2024
Julian Schrittweser,, loannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with learned model. Nature, 588(7833):6044609, 2020.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. ArXiv, 0e//17077063477 2017.
Freda Shi, Daniel Fried, Marjan Ghazvininejad, Luke Zettlemoyer, and Sida Wang. Natural language to code translation with execution. arXiv preprint arXiv:2204.11454, 2022.
Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang. World of bits: An open-domain platform for web-based agents. In International Conference Machine Learning, pp. 3135-3144. PMLR, 2017.
Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. arXiv preprint arXiv:2303.11366, 2023.
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10740-10749, 2020a.
Mohit Shridhar, Xingdi Yuan, Mare-Alcxandee Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning, arXiv preprint ar 2000:20000.03768 2020b.
Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. Progprompt: Generating situated robot task plans using large language models. In 2023 IEEE International Conference Robotics and Automation (ICRA), pp. 11523- 1530. IEEE, 2023.
Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M Sadler, Wei-Lun Chao, and Yu Su. LIm-planner: Few-shot grounded planning for embodied agents with large language models. arXiv preprint ar 12.008222 2022.
Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang. Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. One embedder, any task: inoruuctionfinnttnne text embeddings. 2022. URL http:///////aaav.ooro//r////////
Richard S. Sutton and Andrew G. Barto. Reinforcement leamnng introuttti.............. IEEE Transactions on Neural Networks, 16:285-228 286, 2005. URL http:::/pp.....mmatttttoooooeeee org/
CorpusID: 9166388
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini arXiv:2307.09288, 2023.
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305. 16291, 2023.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in 2022.
Neural Information Processing Systems, 35:24824
Yue Wu, Yewen Fan, Paul Pu Liang, Amos Azaria, Yuanzhi Li, and Tom M Mitchell. Read and reap the rewards: Learning to play atari with the help of instruction manuals. arXiv preprint 044.44330 2023.
Markus Wulfmeier, Peter Ondruska, and Ingmar Posner. Maximum entropy deep inverse reinforce- ment learning, 2016.
12

---

Published as a conference paper at ICLR 2024
Binfeng Xu, Zhiyuan Peng, Bowen Lei, Subhabrata Mukherjee, Yuchen Liu, and Dongkuan Xu. Rewoo: Decoupling reasoning from observations for efficient augmented language models. arXiv preprint arXiv:2305.18323, 2023.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models, arXiv preprint arXiv:2210.06629, 2022.
Weiran Yao, Shelby Heinecke, Juan Carlos Niebles, Zhiwei Liu, Yihao Feng, Le Xue, Rithesh Murthy, Zeyuan Chen, Jianguo Zhang, Devansh Arpit, et al. Retroformer: Retrospective large language agents with policy gradient optimization. arXiv preprint arXiv:2308. 02151,, 2023.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Neta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pp. 1094-1100, PMLR, 2020.
Wenhao Yu, Nimrod Gileadi, Chuyuan Fu, Sean Kirmani, Kuang-Huet Lee, Montse Gonzalez Arenas, Hao-Tien Lewis Chiang, Tom Erez, Leonard Hasenclever, Jan Humplik, Brian Ichter, Ted Xiao, Peng Xu, Andy Zeng, Tingnan Zhang, Nicolas Manfred Otto Heess, Dorsa Sadigh, Jie Tan, Yuval Tassa, and F. Xia. Language to rewards for robotic skill synthesis. ArXiv, abs/2306.08641 2023.
Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, al. Socratie models: Composing zero-shot multimodal reasoning with language. arXiv preprint arXiv:2204.00595 2022.
Victor Zhong, Austin W Hanjie, Sida Wang, Karthik Narasimhan, and Luke Zettlemoyer. Silg: The multi-domain symbolic interactive language grounding benchmark. Advances in Neural Information Processing Systems, 34:21505 21519, 2021.
Shuyan Zhou, Uri Alon, Frank Xu, Zhengbao Jiang, and Graham Neubig. Docprompting: Gen- erating code by retrieving the docs. In The Eleventh International Conference Learning Representations, 2022.
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, al. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13851 2023.
Banghua Zhu, Jiantao Jiao, and Michael I. Jordan. Principled reinforcement learning with human feedback from pairwise or w--iise comparisons, 2023.
Brian Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pp. 1433-1438. Chicago, IL, USA, 2008.
13

---

Published as a conference paper at ICLR 2024
Appendices
A HYPER-PARAMETER DETAILS
In this section, we provide the hyper-parameter details used for our reward function generation and reinforcement learning backbones. For reward function generation, we base on GPT-4 In the experiments of the main body, the temperature of sampling is set to 0.7 for each experiment. For rein- forcement learning training. we use open-source implementations of SAC and PPC algorithmm and list the apper-parameters in Table and Table respeetilyy. Their respective training environments are indicated in parentheses.
Table 2: Slyper-prameeee of SAC algorithm applied to each task.
plyper-parameter
Value
Discount factor
0.99 (Meta World), 0.95 (ManiSkil12)
Target update frequency
2 (MetaWorld) 1 (ManiSkill2)
Learning rate
3e -4
Train frequency
(MetaWorld), 8 (ManiSkill2)
Soft update
5e
Gradient steps
(Meta World), (ManiSkill2)
Learning starts
4000
Hidden units per layer
256
Batch Size
512 (MetaWorld), 1024 (ManiSkil12)
of layers
(Meta World), 2 (ManiSkill2) 0.1 (MetaWorld), 0.2 (ManiSkill2)
Initial temperature
Rollout steps per episode
500 (MetaWorld), 200 (Maniskil12)
Table 3: Hyper-parameter of PPO algorithm applied to each task.
Hyper-parameter
Value
Discount factor # of epochs per update
0.85 (ManiSkill2),, 0.99 (MuJoco)
15 (ManiSkill2), 10 (Muloco)
Learning rate environments
3e-4
Batch size
400 (ManiSkill2), 64 (MuJoco)
Hidden units per layer
256
Target KL divergence
0.05
(ManiSkill2), None (MuJoco)
layers
of steps per update
3200 (ManiSkill2), 2048 (Muloco)
Rollout steps per episode
100 (ManiSkill2), 200 (Muloco)
We utilize multiple g5. 4xlarge instances (1 NVIDIA A A10G, 16 vCPUs, and 64 GiB memory per instance) from AWS for RL training. The time required for training a policy approximately 0.5 hours per task for Meta World, and between 0.5 to 10 hours for ManiSkil12, varying with the task's s difficulty. It's worth noting that we use the default environments from MetaWorld and ManiSkill2 without any speed optimization. Further enhancements such as parallel computing can be made to improve the training speed for validating reward functions generated. An early evaluation of the code semantics before starting RL training modifying the reward during each RL training could be promising direction to reduce costs.
B TASK DETAILS
In this section, we provide full list of tasks within each simulation environment, accompanied by their corresponding language instructions. Across all tasks, we follow the default settings of their
http:///ppaioooooo....c//dddoaap This work uses qpt 1opt-4-00 4-331444 4.
n
14

---

Published as a conference paper at ICLR 2024
respective environments. Here, we will also briefly describe the observation space and action space characterizing these environments. For a thorough and detailed understanding, we encourage the readers to refer to the official manual associated with each environment
World In METAWORLD environment, we use a 7 DoF Sawyer robot arm with a fixed base complete tabletop tasks. For all tasks, the observation space a combination of the 3D position of the robot end-effector, a normalized measurement of gripper openness, the 3D position of the manipulated object, the quaternion of the object, all of the previous measurements, and the goal position. The environment adopts end-effector delta position control, which means the action space consists of the change of the end effector's 3D position, as well as the normalized torque the gripper should apply. For all tasks, the initial and target positions of the manipulated object and the initial joint positions of the robot arm are variable. The full tasks list and their corresponding instructions are in Table 4
ManiSkil12 MANISKILL2 environment uses a DoF Franka Panda as the default robot arm. For manipulation tasks (Lift Cube, Pick Cube, Turn Faucet and Stack Cube), we use robot arm with fixed base. For mobile manipulation tasks (Open Cabinet Door and Open Cabinet Drawer), we use a single robot arm with Sciurus mobile base. For the mobile manipulation task Push Chair, we use a dual-arm robot arm with Sciurus 7 mobile base. For all tasks, the observation space consists of robot proprioception information (e.g. current joint positions, current joint velocities, robot base position and quaternion in the world frame) and task-specific information (e.g. goal position, end- effector position). We use end-effector delta pose control mode for this environment, which controls the change of 3D position and rotation (represented as axis-angle in the end-effector frame). For all tasks, the initial and target positions of the manipulated object, the initial joint positions of the robot arm and physical parameters (c.g. friction and damping coefficient) are variable. The full tasks list and their corresponding instructions can be found in Table
Gym MuJoCo Gym MuJoCo offers series of simulation environments that contain Ant, Half Cheetah and so, powered by MuJoCo physic engine and pre-defined XML files, each provided with a written reward for a simple task. To evaluate the ability of TEXT2 locomotion skills, we chose to use two robotic agents: Ant (a 3D quadruped robot) and Hopper (a 2D unipedal robot). Their actions are represented by the torques applied at the hinge joints, while their observations consist of positional values and velocities of different body parts. We provide the tasks list and their corresponding instructions in Table
learn novel
Real Robot For the tasks Pick Cube and Stack Cube, we take the joint values of the robot arm, the pose of the end-effector and cubes as input. For real-world robot control, we use end-effector delta position control mode, which first predicts delta 3D position for the gripper to move to, then utilizes inverse kinematics to iteratively solve the target joint value. To obtain the object state required by our RL policy, we used the Segment Anything Model (SAM) (Kirillov et al. 2023) and depth camera to get the estimated pose of objects. Specifically, we query SAM to segment each object in the scene, The segmentation map and the depth map together give us an incomplete point cloud. We then estimate the pose of the object based on this point cloud.
C PROMPT DETAILS
Zero-shot prompt To ground LLMs into robotics simulation environments, we propose a novel Pythonic prompt, which can be abstracted by the experts who developed the environment. This class-like prompt is a more compact representation than simply listing all environment attributes linearly, which can delete redundant information and save more tokens, and this Pythonic prompt can also better bootstrap Python reward code generation. More specifically, our prompt leverages python class, class attribute, python typing and comments to recursively define the environment. Here we provide an example of the zero-shot prompt for MANISKILL2 manipulation tasks:
You are expert in robctics, reinforcement learning and code generatien. Ne are going to a Franka Pande robot to complete given tasks. The action space of the rebot is a normalized
'Official document of MANISKM.I22 is ficial document of Gym MuJoCo is at htttps
and of-
15

---

Published as a conference paper at ICLR 2024
Table Task list of METAWORLD.
Task
Instruction
Drawer Open
Open drawer by its handle.
Drawer Close
Close drawer by its handle.
Window Open
Push and open a sliding window by handle.
Window Close
Push and close a sliding window by its handle.
Button Press
Press a button in coordination.
Sweep Into
Sweep puck from the initial position into hole.
Door Unlock
Unlock the door by rotating the lock counter-clockwise
Door Close
Close door with revolving joint by pushing the door's handle.
Handle Press
Press a handle down.
Handle PressSide
Press handle down sideways.
Table 5: Task list of MANISKILL2.
Task
Instruction
Lift Cube
Pick up cube A and lift it up by 0.2 meters.
Pick Cube
Pick up cube and move it to the 3D goal position.
Turn Faucet
Turn on faucet by rotating its handle. The task is finished when gpos of faucet handle is larger than target qpos.
Open Cabinet Door
A single-arm mobile robot needs to open cabinet door. The task
finished when qpos of cabinet door is larger than target qpos.
Open Cabinet Drawer
A single-arm mobile robot needs open cabinet drawer. The task is finished when qpos of cabinet drawer is larger than target ypos.
Push Chair
A dual-arm mobile robot needs to push swivel chair to a target location on the ground and prevent it from falling over. Pick up cube and place it on eube B. The task is finished when cube A is on top of cube stably (i.e. cube A static) and isn't grasped by the gripper.
Stack Cube
you help write reward functioon for forcement learning. 1'11 give you the attributes of the environment. You can use these class altributes write zeward function.
[=1, 1, (3,), float32)
Typlcca ly, reward function
manipulation task conaisted thase following parta: 1. the cistance between robot's gripper and our target object
2. difference between current state of object and its goal state
regularisation
1. [optional] extra constraint of the Larget object, which is often implied by task instruction rebot, which
[optional] extra. constraint
implied by task instruction.
class DaseDnv (gya.aavi.
sel f.oubeA Rigi dobject cube A in the env' ronmert self.cubeB Rigid0b ject cabe In environment
0.02 & in meters
self.robot #endaRobos
class PandaRobot:
self.ee_pose ObjectPose 3D position and quatersion of robot'a end effector self.1finger jeck left finger of robot'a gripper
self.rfinges LinkCh zight finger robot's gripper self.qpoz : ap.adarray ( joint position of the robot
self.qval : nd. (7,)) ] joint velccity
robot
float $ openness of robot gripper, normallzed vange in [0, 1J
def check Steeck_graappsell ob] Union [Rigidbbeedt LinkCb max_angle=85) bool
* indicate whether robos gripper successfully grasp objecs
ObjectPose:
self.p : aderrayy (3,)] 3u position of the rigid abject
self.q : ndarrdl (4,)] [ quaternion of rigid ohject
def invile,,, ObjectPose return "ObjectPoae" class instance, which is the inverse of the original pose def to.transformation htt.rfrrratioonnnttiieeeeeeee (serf
(4, # return a [4, 4) numpy arrey, which is the transform matrix
class Rigidabfest:
pope abjectPose 30 position and quaternton the rigid object
16

---

Flost and Published as a conference paper at ICLR 2024
Table 6: Task list of Gym MuJoCo.
Task
Instruction
Hopper Move Forward
Control the Hopper to move in the forward direction.
Hopper Front Flip
Control the Hopper front flip in the forward direction.
Hopper Back Flip
Control the Hopper to back flip.
Ant Move Forward
Control the Ant to move in the forward direction.
Ant Wave Leg
Control the Ant to stay in place while waving its right front leg.
Ant Lic Down
Control the Ant lie down.
self, velf.velccity ap. idpreaar (3, # linear velocity of the rigid objec: self. wwwaaannneeeeeeeecoa np.ndarray (3, (3,) angular velocity the rigid object def check_static
bool + Indicale whether this rigid ebject is stalic or not
class Linkobject:
ael rope : abjectPose # 30 position and quaternior of the 1ink object velccvy ap.ndarcay
(3, (3, 1ink object joint
velecity the 1ink object
up. aeparaaraay/ (3,)) * angular velocity of the link object
position self.qvel float & velocity of the 1ink object joint
self.qpos :
celf.target..posffloo target position of the link abject joins def got__loal__pee (self,) => ap.ndarray (M, 3);
the point cloud of the link object surface
in the local frame def pel ap.ndarray ();
get point cloud of link object surface in the world frame
el ass Artioulatedb'eet:
ubjestPose # 30 position and qusternion of the articulated abject self. f. calty velottty nd.nraayyy( (3,1) lnnea calty of the articulated object aiI..angula__veloot
self.pose :
up. np.ndarray (3, (3,). angular velucity of the arciculated ooject
self.quoz : up. ip.adarray (8,) position
the artieulated objess jeint
self.qvel : np. ap.nndrrayll (8,) velocity of the articulated object jeint
def get faelf, -> nd.ndaryay (X, (X,31] paint aloud the articulated object surfaae
in the world frame
Additional knowledge:
1. A staged reward could make the training more stable, you can write them is nested if-else stetement.
2. 'dbjectPase" class support multiply operator 'x', for example: ce_pcso_wrt_cubat helffeubee.poce (nv() a wwl.f.oboc.eeeeeee
3. You can 'Cransforms3d, auaassoooomdd..aaeeenniooo 'qinverse (quat: np.ndarray [441)) * for inverse of quatarnion, 'quatt mp. (4, 1), quat2: ap.ndarray (4,) for multiply quaternios, "quatzaxangle tquat: np.ndarzayl44, for quaternion to angle.
package to quaternion calculation, for example:
I want you to fulfill the following task: ranttnnction)
1. please think step by step and tell ne what does this taak mean;
2. then write function that formats 'def sopputs_Genss__reaaslleeee action)
returna the 'reward flost" only. code, you can also add some commente your thoughts.
Few-shot example prompt In order to retrieve previously acquired skills and generate better reward codes, it is necessary to provide LLMs with few-shot examples. These examples take the format of instruction and reward code pairs, which then filled into the prompt template. Here is the few-shot example prompt template used for MANISKILL2 tasks:
An example:
Tasks to be 1led: (nnstraction)
Corresperding reward function:
' ppythor. (reward__ode)
Interactive feedback prompt Due to the inherent information bottleneck of natural language, the users' language instructions may be ambiguous, and their preferences may not be conveyed. On the other hand, single-turn reward code generation may not work for some complex tasks. So, in these cases, multi-turn generation with human feedback can be beneficial. One possible prompt a combination of the previous round generated code, a description of the training results (learning curve, rollout videos, etc.) and the feedback for improvement. This is the prompt template:
Ganerated cade shown as bolow: "pyyhho
17

---

Published as a conference paper at ICLR 2024
[generaced_code]
Feed this seward code into the environment, and the slgorithm to train the pollcy, After treining, can see from the robot that: (deceription)
make more accurate (feadnack)
better robot,
improvament is:
Re- imagine which
missed wreng.
Show ma the improved code as below:
D SAMPLES OF GENERATED REWARD FUNCTION
D.1 ZERO/ENW-SOOT/ORACEE MANIPULATION RESULTTS
In this section, to help better understand the generation ability and results of TEXT2REWARD and the difference between generated and oracle reward functions, we provide zero-shot generated, few-shot generated and expert-written (provided by the simulation environment) reward functions different tasks on MANISKILL2.
Sample of zero-shot generated reward function on MANISKILL2 Lift Cube task:
import mumpy =a np from salpy. apatial all import distance
compute_deneee desss_ oction) flosts Def Define reward weights
weight_dist 0.4
weight_lift 0.4
weight_grcop = 0.2
Initialize zeward reward 0.0
Stage 1: Appreach the cube ec_pos self.rcbot.es_pee cube_pos salf.cubet.pose dist_to_cud ng, leward_dist - -weight_dist dist_to_cube
cube_pos)
Stage 2: Grasp the cube
grasp_succers self.robot. www.rroooo.ccc...ppee cwwckk wtw.eecc...raatteeee.ebb..... max_angle=30) reward_greap weight__rraap grasp_euscess
SLage Lift
lift_amount = cube_pas[2] self seliggoalhheihht height
reward_lift weight_ ap.abs (mout_smoutt)
lotal reward
reward_dist reward_grasp reward_lift
Stage 4: Maintain eube gosl height if self.cubex.pccs.pp >e heil__hiihh:: zeward 0.1 coe...
self.goel_height)
Regulariza the robct's action Ne don't rewazd 0.01 www.iinnggnnoommeeee..ooo
very action,
negat reward
return reward
Sample of sew-shot generated reward function on MANISKILL2 Lift Cube task:
import nungy as
def compute_dde compute_dense_ste (sled action):
reward 0.0
check if cubeA is lifted 0.2 meter is_ebj_lifted self.cubes..osee pose.p[2] >= self.cube_half_stee self.goal_height chock if the robot in static
is_robot_statie ap.max laaaiippbbae reli..
cond are met,
'3 auccessficl
18

---

Published as a conference paper at ICLR 2024
success is_obj_lifted statoi
1f auccess: seward i-
return rewerd
calculate the distance between robot'a end-efrector and cubet
top_to_pbi__po self.csbck.pose.p self.rabot.eet robot.se__pos..p ccp_to_ab"__daa no..imiagg (topmtttooo_ob_ppo
calculate the zeaching reward, which encourages the robot to approach cubed reaching_reward
np.tana tcp_te_obj_dist! reward reaching_rexard
check if the robot has auccessfully grasped cuses is_grasped self.robot, chhckk sellf.cubs,, mox_angle=30) if graaped, add reward
zeward 1 is_grasped else 0.0
if is_grasped;
# calculate distance ob obi_to_gual_diat - ap. gaell.goall_eiiggeeee (self.cubeh, p2. self.cuber _ calculate lifting reward, which encourages robot lift up lift_reward op, tana obj_to_goa___iss zeward life_reward
cubeA target height
return reward
Oracle (expert-written) reward function MANISKILL2 Lift Cube task:
compute_ dense__eward(sell action): reward = 0.0
is_cbi_plsen self.cuber passee..oos......m pose. .[2] > self.goal_height Silf.eube_half_itt is ig_robot_statee static mp.mox inp (self, roboo..qeel)... <= 0.2
success placee and st_toi
if success: zeward +- 2.25 return reword
reaching
gcipper_pos - self.
abj_pos self.
dist cubs..ppee..oos..p
(ww.noommniippr_eppp.. - obi_pos)
reaching_reward tp dist)
reward reaching_rexard
is_grasped self.robot. www.:ccttt..ccce.gssssseeeeeeeee (self. max_angle=30)
grasp reward if is_grasped; zeward 0.25
lifting reward is_grasped; lifting_rewait = self, celf.cubea.pose. p[2] celficube cttt..cchhhhs__eee hsif_size lifting_reward min misglifttin__rooorr gel.f.oal_haaght 1.0) reward +- Lifcing_reward
reward
Sample of zero-shot generated reward function on MANISKILL2 Pick Cube task:
import numpy as
def compute dense_ (sa action): NornaLize action
action clippactionn -1,
gripper
gripper_pos =
cube_pos calf.aubet.pose.f
dist_oriiper_cube np. Inrll.. igriiper__opp oube_pos)
Calculate distance between cube and goal gool_pos salf.ggall_psitionn position dist_cube_goal np.
sube_pos)
Check the robot: graaping cube
19

---

Published as a conference paper at ICLR 2024
grasping_ self. self.robot. wtt.bbot.ccccck.gaapeeeeee......
Define reward component:
deward_dist_dd sisgg__ess_ggige__uuee 1.0 dist dist_gripger_culs
reward_dist dist__de__e__eeee -1.0 dist dist_cube_goot
reword_grssping_euue. 1.0 grasping_cube else -1.0
weights weight seigghhtsss___oo
components
0.3
weight_distt www.eittt__be_e_ee_______e_ 0,5
weightt_grespinn_eube 0.2
Calculate total reward
reward gripppprrrr cuber_cube reward - cattt__ddtt___be_______ csbe__goll saaaaddddtt___baeee qubon_eoal weigght_graspinn_cube reward_grasping_ouee
weight
Regularization action
reward - 0.01 laction ** 2)mmum()
retuzn reward
Sample few-shot generated reward function on MANISKILL2 Pick Cube task:
inport nungy as up
def
action):
0.0
Check if cube_at__ool. = ap. www.iiingg.nomim.... wew............oee...... is_robot_statie ap.max Inp. robt..
placed goal position and is static self.goal_positionn <n self.cube_half.cco 0.2
If the auba is placed at the goal and the robot is static, return high reward cube_at_goal is_robot_statio:
zeward i- 2.25
return
resching rewerd, encourages the robot to reach the cube gripper_pos relf.rchot.ce_pant
poi_pos sel cuubek..ooop
dist_to_obj np.linalg, www..iing....megriiie...... teg.llg.eengggge......... pbi_
resching - ap.tanai5 dist_to_dbj) reward reaching_rewast
graap reward, encourages the rosot to grasp the cube in_grasped self. wob.t chockk (self, cubsA, max_angle=30) if is_grasped;
placement reward, encourages
place at goal
if is_grasped:
dist_to_gosl
np.tanh dist_to_goall) polsiionll
placement_rexont
reward t- placement_rewacd
regularization term on robot's action action_reg -np.oum (rp.ouumpppoqurrr (action)) (sctiio) reward -- 0.1 action_reg
return reward
Oracle (expert-written) reward function of MANISKILL2 Pick Cube task:
import numpy
der setputt_denss_eeeeeee reward 0.0
action):
is_obl___ is_obee ap, norm gow.eeell.oa___soioiiiaooa cess. <- 0.025 is_robot_etatie ap.max inp.ai.. abs robe....t....)lllll
suscess ii__oi__plaseeeee la_robat_ataric
if success: reward +-
return reword
cop_to_abi_pon = top cop_to_ob"__da_ np. tooo
20

---

Published as a conference paper at ICLR 2024
reaching_rexard np. top_to_obj.dist) reward = rooching_roword
is_gsedped self.robot, www.ottcccck..eee.eee grasp cubee, max_angle=30) reward is_grasped else 0.0
if 18 se_grasped; oit_lo_goallisst ap. ip.linolg.nnrm (self..aa position self.ciheA.pase.p) place_reward = "p.tanh(5 obi_to_gooal_dee __tatt zeward i- place_rewazd
return reward
Sample of zeroshoot generated reward function MANISKILL.2 Turn Faucet task:
inport nunpy
from scipy. spetisll distance import edist
def seattte_deese_eewaadeeee
action): Here define the reward and penalty weights for different aspects the task sannllerrsch__otiee
graap_hanm saaap_handle_wee weight 0,2
rotation_weight actioo..ennllty_we.gh 0.01
lleee define the grasp grasp_succesa_rcxort = 8.1
reward
lleze I compute the distance between the robot's end effector and the [aucet handle handle_ped self.
1finger_cords ap.
11
rfinger_cords mp.erray dist_lfinner_hendle cdist (lfinger_cordas hardle_ped) .min(-1) [0]
11
dist dist_rfinger_handle cdist (=finger__ordde hardle_pad) (min(11) [0] dist_handle_resch (axddddsiiiinger_eee hancle_resch_
cist_rfiingr__ardle
handle_reach_wee_p
dist_handle_reach
Here : check if the robct has successfully grasped the faucet handle graep_handle
check grasp (tt.. hendll)
grasp_handle_rewarr grasp_ gradp_iandle_eeight gresp_handlle if grasp_handle else 0
caloulate cotation Larget joint position
based the difference
rotation_diff max10, sele.faucet.. hadcee.ttneeeeeeee..... handie.carget...p...... apos felcefaucet. handle..ea..
rotation_rexane rotatioo_ rotation_difs
caleulate the penalty for robot's action
action_penalty, aotion_penaltyy wewiioooenalttywegggtt... (nnmpp wor.gg.ooomttttt.
The totel reward is the sum of all the individual rewerds and penal:ies
reward handle_ reward grasp_handle
ratation_rawart action_penalty
if gresp_handle and rotation reward +s graap_aveose hatpppoooossssree__
zetura
Sample of few-shot generated reward function on MANISKILL2 Turn Faucet task:
import numpy
from scipy.spatial spatial.distanee Import edist
def pempute_densee htt_denneeeewardsslll action)::
0.0
check handle is sancell handla. >e f. ha target_apos is_hand e_turned_on sel raucet, iaasethhan check if the robot static is_robot_static
lab row.t..rrbot.ooo <e 0.2
success www.aanneeeeueeeeeeeeeee
ie_robot_etati:
acccess: reward t= 5
seturn zewerd
calculate the reword for reaching the handle
tcp_to_ sep_tto_handle_pa sels. faucet. handle. pose..p self.robot.ee_poze.
poo_to_handde_de = sp. ntw.lllgooommtp...ooooooo to__hooooe_ppooo reaching np.tanh sap_to_handle_dt
21

---

Published as a conference paper at ICLR 2024
reward reaching_reward
calculate the reward for succeasfully grasping the handle ss_gsasped celf.roboo caasst.ccec__geee. (ree (self. handle, maX_angle=30) reward is_grasped else 0.0
if 18_ ie_grasped; rewerd
turning
sandle_t__target_dla = self. self.faucet faucetttnnt e.target_ sel saaacct.handde hapole. turning_reward
ap.Lanh Landle_to_target_.ua seward +- turning_reward
zeralize actiona that could potential's danage the robot action action_penalty = np.aum (np, (nppanuaree aquare (action)) reward aation_penallye (quare(actioon)
return reward
Oracle (expert-written) generated reward function MANISKILL2 Turn Faucet task:
daf cempute_densa_rovee reward 0.0
action):
angle_diff Sauree.handde.... sargees_eerg self.faucet hanuueetaaddss.pee success angle_dicr <
if success:
""Coomuue
between tap and fiagers.
ped =
pedl self.robet. wttp::i..gggggg..eoootooeee
ped2 relf.robct.rfinger pft..:eeeieee.....ool...llll
distancel odiat (ped, pool]
distance2 edist (pod, pod2]
distance stddiistannel.minn).. distanco2.min())
reward np.tanh (distance 5.0)
turn_reward__ reward =- tura_reward
(anglengle__diie 2.01)
delta_angle anglo_difs self. lalf..aatt_ngle__d: if angle_diff 0:
turn_reward_2 -np.tanh (dclta_aadddaggg 21
else:
Luzn_reward_2 ap. Lanh adelaa_ngee
turn_reward_2 4= curn_reward_2
self. angle_diff
return reward
Sample of zero-shot generated reward function on MANISKILL2 Open Cabinet Door task:
compute_ det_e_denee____eeeeeeeeeeee (se1d action): Defina weights for different parts of the reward v_distance 0.4
M_goel 0.4
v_action 0.2
Calcuiate distance between rebot's gripper and the cabinet handle handle_ped
se_corda half.robot.gee get__ee_____e_eeeeeeeeeee get_ee_ncordd(] (). reshape () 3)
distance (ee_coreecood)) handle_ped).ning .nin)
distance_rewazd -w_distance distance Negative reward since want mininize the distance
curzent state cabinet drawes its goal Positive reverd since vant maximize the goal_diff self.cabiiet.hand h.a.......s..s self.cabinet. goal_reward M_goal goal_diff
Calcuiate
Add regularization scbot's action, penslize large actions actioo_reward acctio ap.linolg (action!
Check if the target drawer is static, if so, give large pasitive reward self.cabine. htw....addeeeeccccceeec...
22

---

Published as a conference paper at ICLR 2024
static_rexerd
else:
statie_reward 0,0
Cembine different parts of the zeward
reward = distance_reward gool_reward potton_rowardd
static_reward
return
Sample of few-shot generated reward function on MANISKILI.2 Open Cabinet Door task:
Irom scipy, spatial import distance sciat
def corpeto__demso_rroorlctlll action: mp.ndarray) fleat:
reward
Compute distence from end-effector to cabinet 12, 3]
door handle
ee_coorda robet gett se _ee__e_ccooeee co
handle_ped self.cabinet ceteeinet.haaddd.. www.hha.deee..........e....... word
[N, 3]
opproach handle
dist_ee_to_haaee tanddleeaaaeeeeeeeee - adist.cdist tee coordd, handle_pod [2, N] dist_ee_to_hande dist_ee_te han.aaedeeemm.. min
[2]
dist_ee_to_handle handee ()ean
log_dist_ee_
loo (edistdiseeeddee _ee__t__heeaiteeeeeeee le 5)
reward dist__e__t__handit cp.clpppcoo.ddatt.e..t....
-10,
Peralize action
Assume action 1s ivlative and normalised.
action_nezm ap..inalg..n..... gns.iinaag.ncmmmatiiiee nerm(action))
reward - action_norm le-5
Cabines door position cabinet_doos door__dddr_______ target_door_pt bell:.abbint.
wtw..cabbttt.aaeee hww.la. epos
Stage
atage_rawerd = -10
if cebinet_door__os terges_dcor_poss
# Cabinet dcor is opened
stage_reward += 5
Encourage the robot to centinue moving the cabinet doos stage_reward (cabinet_dcor target_door_pant
reward reward atage_reward
return reward
Oracle (expert-written) generated reward function on MANISKILL.2 Open Cabinet Door task:
def compute_dense_ieee dense (el,, action):
0.0
The end effector should be close the Larget
handle_pose sellicabinet. handle pose
ee_pose self.rabot.ee_pcee
Position
ee_coorda self, robet.gee ee ce 12, 10, 3]
handle_ped celf.cabinet. waw.inetthhaddee.. wtw..l...et..wwrd........
disp_ee_to_haidle sdist. edist
adist.edist (ee_coorda reshape 1, 3), handle_pod)
= disp_ee_ tandhalaeaeeeeeeeeeeeeeee
ht._handee..shaaaeee reshape (2, -1) miint-l [2]
reward_eo_to_hcndll -dist_ee_tee wawtttoohnnn_oeeeee mean
reward - @eaard_e__tohhaddll
Encourage grasping the handle
ee ee__cette_at__ort se__cords.mman [10, 3]
ce_center_at_kanda tcansform.pointst
inne..i wtt.ininn.oooootiiiiaoaaa www..totssoommttiooomee.....oee.
ee_center_at_work
Lisi_ee_centerrttobheee
self. self.cabinet.kene otl.d.dliii.t.aaaa....e.e..d.eed
[ast__oo_centr__tt to
to_trrrtt_hhodd. handle
dist_ee titr__o_ccnterrta_hhnlld...
swarrd_eeccette.c handle
toip__nnd_orrmllzze (dist_ce_contee conter tote_coott__t_hhadleee. -0.01, 4e-3)
23

---

Published as a conference paper at ICLR 2024
Rotation
Larget_gresp_ celf.cabinet.hanee htt.:e.hheee....eeeeeeee....e..
carget_gresp__ [handle_pcse x for x in target_grarp_posea
dinglledisttneeeee (on_ x) for sarget__raap_porse
ee ee_rot_ reward _arpee____t__rraa_______________ / np.pl A rewaad
Stage reward
coeff_qvel 1.5 joint velacity
coetr_apos 0.5 { joint position cistance
stage_raward
(coeff_qvn cooff_epos!
link_qo0z sel..cabinet.handiee..g
link_qvel upp.iinalg.nrmmann www.iet.....teee...aoaaaaaaaaeee catisseethannda... www.dhanndene...... velecityll
link_vei_nee Hink_ang_vel___ np. wn..iialg..oommee wtttorrmself..cbiooee haw.eeettaaddddaagg.e.. bar.agaaa__wellllyyy
ee_close_to_hance (dist tesst_ee_to_iiii he__oo_anddeema.. if tha stage_reward 0.5
dost_ee_ceneer_t
# Distance between current and target joint positions reward_qpos tolip_and_normallle (link_qpos, sels:. zeward i- reward_qpos
coeff_gpoa)
if Link_apos self.cabinet. handeee.aanee....aa Larget_gpos:
Encourage positive jeint velocity to increase jeint position
reward_aval clip_ond_nosmallee (link_qvel, -0.1, 0.5) cocff_gvel
reward +o reward_qvel
Add eceff_qvel for smooth transition of stagess stage_reward coeff_qvel reward_static -Lliak_ve__nom aom 0.5) reward reward_static
if Link_vel <= 3.1 and Linloongg <= 1:
stage_reward
stage_reeard
return reward
Sample of zero-shot generated reward function MANISKILL2 Open Cabinet Drawer task:
inport nunpy from opepyaa dis.apaa
distence import adist
def compute denee_deeeeeeeeeeeeeeeeeeeeeeee_ (seld action):
Define weights for different parts of reward v_distance 0.4
M_goa1 0.4
V_action 0.2
Calcuiate
grippez
cabinet handle
handle_ped self. wawinet.haadlee.........b....aee wtwplleeee_wwwrd_peeee ee_cords self.robot. cerbootgee_____cceeeeeeeee () (-shapee-1,, distance adist (se_cords, sandle_pcde .min
distance_reward -w_distance distance Negative reward since we want to mininize the distance
Calculate the difference between current state of cabinet drawer and its goal state Positive zeward goal_diff self, waw.eccattttttddee.... goal_reward N_goal gofl_diff
want
spos
self.cabinet. www.eet..annnlal.......... handle. www.ia..aagee......... targe:_gpos
Add reqularization of rebot's action, penalize large actions action_reward -M_ection ap.linalg, hor.all..ormmctiiiee.
Check if target if self.cebinet.
static, so, give large posicive reward
static_reward else:
static_reward 0.0
Combine different parts of the reward reward distance_reward reward sction_reward static_revard
return reward
24

---

Published as a conference paper at ICLR 2024
Sample of few-shot generated reward function MANISKILL2 Open Cabinet Drawer task:
saipy.apetio
import distance aa adist
inport numpy aa
def corpete_derrs_eevaccceell action: ap.ndarrsy):
reward =
Compute distance between ead effecto: and cabinel handle auzface
ee_coords np.array wow.t.oootttett...o.. getetggett_eeeeeeeeeeeee [ [2, 10, 3] [N, 3]
handle_ped self www.biintthandl..ee........
approach handle stst_eee_to__aee dist_eee_to__andde dist__essto_hanndee
cdist.cdist (ee,
(-1, 3), handle_pedi F [20, N)
dist_ee_to_ handle - dist_ees_to_ _e___sss_t_hhnleeee_ee__ee wtt.thhnneeeeeeaal seen
np. log (diss_ee__ttt te_e__tto ti__ttohhllaae 1e-3)
reward - _t___eete__amdie
10,
Peralize action
Assume sction 1a relative and norms lized
action_nozm we..inelg.nezm(acte...
reward - action_norm. le-5
Encourage qpos change apos_change self. wawsnet.hancle..e.. self, cabinet... www.hhhnnnnnett........ target__ reward qpas_change 0.1
Penalize velocity cabinet handle
handle_vel_noee (s cabl. can.....aaie.. vendiie.elooity)
reward handle_vei_noze 3.01
cabinet_vei_nemm wwg.linagg.nemiiie (self (atseef.cabiiet... cabinet. velocity)
reward - cabinat_vel_nom. 0,01
Stage reward stage_rewerd if
< 0.1:
# stage_reward += if calf.cabbint..andl...oo >= self.cabinet. haw.daa.arggttopoo... drawer open
aLage_reward 1-
reward stage_reward
return reward
Oracle (eiter)) generated reward function on MANISKILL2 Open Cabinet Drawer task:
def compute_dent dense_ (se1, action):
0.0
The end-effector should be close the target pose
hancle_pose cabinet. handeet pose ee se_pose sel robot.eeope
Positicn
ce_coords rebct.eet settcct.eet_____c_ee (c handle_pod self. wtw.f.ccattttt.ta.m wwwdeett..aaa....eeeee hwndha. sondllo.gt_world_podd
10,
disdee sdist. edist (ee_coords... reshape 3), handle_ped) jist_ee_tt_haaddi = staapee__to,
waaeee_tt__aadleeeeeeeeee handle, reshape (2, -1) -mint-11 [2]
dist_ee_to_heee reward - teward_ee__t_handll
teward_ee_to_katt
Fncourage graaping the handle
ee ee_center_al ee (0 [10, 3]
ee_center_at_hent hendle trensform_oinnts(
handle_pose. invooeee..iiii.. in htw..ttotttaaoooootoooooooe
ee_centerr_t__orcc
dist_oe_conter_to__ha
self.cahintt.hand local _ddd__ddiee of
ttw.:ceeter__t_handeee dist.ee_.ntterttthhnnd..
reward_ee_center handllee
clip_cnd_tormaliine oo_tttto___ottrrrrrrr..ooooo tocleerrtt___aeee handle,,, -0.01, 4e-3)
reward - roward_cc cc cat_ectttt__rhaedede
25

---

Published as a conference paper at ICLR 2024
Rctation
graap podea hawnonneeeaandltttoo.. wtt.ll.trrt...rasp.ooo
carget_ggaag__on poses [handle_ v for in
angles_ee__oo_re pe__pp__rrsspeeeeee______e sangle_distance pos_, for in srage__.rraspppeees]
ee_ -min ott.aaggeee.......aae......r. / np-pi +
reward re_roo_
Stage reward
coeff_gvel 1.5 joint velocity
cooff_apos 0.5 f joint position distance
stage_reward (coeff_qval coeff_spos!
link_qas self.cabinst.
link_qvel link_vei_nozn (w..innggnnommmel.f....o.. lon ap..iinslg.none www.eee.ww.ee....eeeeeeeeeeeeeee webinet.hannle........ www.dha..nnnnneeeeeeeeee ongglddee.gglrrrrllooo.yyy
waw.eee...eeeee.eeeeee.........e vandle.veloccty!
oe_close_t__anddle (dist_ hot__po_t_handleem if haaddlee_oo_aande:: stage_reward
and vaa__ee_eenter_t__haaee
# Distance between current and target joint positions rexard_apos aaiip_nn__oormaiiee (link_gpos, 0, www.iiee..innde.tttttaeao reward + reward_apos
coeff_gpas)
if Link_apon self.cabiaee c.b.aet.eeeteeeeee target__poes:
Encourage positive jeint velocity increase joint position
gevard_qvel anli_aad__ommmlie (Link_qvel, 0.1, 0.5) coeff_qvel
reward reward_qvel elsa:
Add _eaal_qual for amooth transitlan of stagess aLage_ieward coelf_qvel
reward_static -(liak_vel_ reward_static
ang_ 0.5)
if link_vel_nor <= 3.1 and Link_eng_wel_nort <= 1:
stage_reward +
zeward stage_reward return reward
Sample of zero-shot generated reward function on MANISKILL2 Push Chair task:
import numpy np from apipyaapstial... distance import cdist
def compute pempute_dense___e seaatte__eeeeeewarddeeee (slI,, action) flost:
Stage 1: Encourage the robot to move towards the chair the distance
roaot's gripper the chair chair_ped self. suff.chhir.ge.ppdd/ dist_to_chair cdist (gripper_coords chair (-i)..mmmmon
gripper_coorde =
Get the difference between the chair's eusrent and Lazget posilion chair_ta_target_dkk. = 1p.linalg. nar.m.n ff w.w.chh......ppp..... sal* stals.target_xy/
The smaller the distance, the larger the seward reward_dist -dist_to_chair
The closer the chair is to the target, the larger the reward
reward_target -chair_to_ttrgee__de
Stage 2: Encourage the robot to push the chair towards the target location the velocity chair_vel = wal...chir..elocityy/.
faster chair moves tewards the target, larger the reward_vel np.dot ichafr_vel, fael f.target___p wel..ohair.... (ooatrrpoos-p(22)) inp.:inaelg.nezatchhe...
chair__t__tasge__ddl
Stage 3: Prevent the chair from falling over Calculate t_axis_chess = self, chair_tilt. mp.arenes The snaller the tilt, the largez the reward reward_tile chair_tilt
angle
chair
[:3, 21
Regularization of the rcbot's a action reward_reg squ.ee (action) (sun()
Neighta stage
26

---

Published as a conference paper at ICLR 2024
M_dist 0.2
v_target = 0.2
M_vel 0.3
M_tilt 0.2
M_zeg 0.1
Final reward
M_dist reward v_target sexerd_target k_vel revard_ve" k_tflt reward_tils M_reg reward_reg
reward
Sample of few-shot generated reward function on MANISKILL2 Push Chair task:
Import numpy sa op
inport ppatial.disee distance sdist
def compute_dense_reee rew.deeeeeeeeaaddssllll action):
reward - 20.0
octor self.chair
ee_coorda = ap.array fs robot. ()t__e__oorda()
ee_mids np.asray ([ee_coords [:2] (0), se_coords [m:]] chair_ped self, cheir.get_.ccit
Σа approach chair
dist_ces_too_iie cheir adist. (ee_ coords, chair_pedl # [4, N]
dist_pea_to_ohas diat_eea Maat_eea_to_one wew__t__haarrmmma
[4]
dist_ tit__eee_oo_hiii_ee_ Log_dist_ee_to_chat ap. log idist_ te_________tttirreeeeee_eeeaee 1e-5) reward - cist
ees_to_challe
-10, 0)
Penalize acticn action_norm np. woa.al...crmmccttti.i action_nos 1e-6
Keep standing
*_oxig_xorla np.array( 3, 11)
Liaxis_cheir self. wel...chiir.pse..e
wtt..tttsssemmtttootttttiie [:3, 21
chair_tilt ap.areses (s_axis_chairrr
log_cha'r_ttils np. log sechairrtilttt + 1e-5)
-chairrtilt
Chair velscity
choir_val actor.velocity
chair_vel_nome mp._innllg.ommm@@iir...llll solf. salf.target__x.
stsp_chair_ta_targee walffehaair.pse...
chair_vel_dir cotistt [:2], diep_chair_ wwwtoccttttttteeeeeeeeeeooeeoeee to_target]
chait_ang_vei np. htw.://eeeeet.gggl.eeeeeeooogg
Stage
stage_reward
dlot_chair_ta_targe up.
hpp.linal.......
sal...t.are fal...trrtt_xy/
dist_ee_to_chair
stage_reward +=
atage_reward +-
reward (mp.exp( casir_vel_non 10)
chair_vel_nnn
sLage_reward 1- chair_ang_vee -_nrm 0.2: chair_tilt
ap.pi:
stage_reward -
vel veddir_vel_drr chair_wel_noze
reward 1- ap.clip (p.expl-seward -1, ap.in5) wtw.cciiirrttttrreteee
if chair_tilt 0.4 np.pi: stage_reward
reward reward stage_reward return reward
Oracle (expert-written) generated reward function on MANISKILL2 Push Chair task:
daf compute www.seennnsssddeeeccooeeeooeeeee action: mp.ndarray):
Compute distance between end-offactora chaf
27

---

is Published as a conference paper at ICLR 2024
ee_coords np.array (self.robo www.oroottt.ttteeee..o....e.eeee get__e__corrde ()) 14,
anoir_ped = self, self.choir.gas werhiirrget_oord__poo
[3, 3)
DЕ approach chair
_ist_eee_to_cheii sdist.cdist cee_coords, chair_ped) # [4, N]
dist_oce_to_chhiiz sist__cs_tt_choir.mm
dist _aat_ee_to_chairz
_e_____t______e_ee______________ r.
log_dist_oe_to_chr
ieg.logiist.... te_tte_chttss 1e-5) reward = -di -dlat__e_tt__haale
np.el t -10,
Keep cheir standing
z-axis of cheir should be apward "_oxia_cheir self-chair.pose. www.srrsoootttionncoooooomoooooo matriti [:3, chair_tilt ap.arcoos (2_axis__ii__ [__axie_chaii([]] reward -chair_tilt
Penalize action
Assume action is relative and normalized.
aation_norm. np, ni.linalg,, ina.aalg.nttmmattiioon.
actionnnorm 1e-5
Chair velocity
chair_vel self.chair. wel..chhai...elooolyyyt
chair_vel_nerm wt..inelg..omm//hhiii...
sisp_chair_to_targe s.ff.chair.psse..
sels. telf.target_xy
poa_chairr_el_ce to__trettttttteett = (dainn cataapppssr_ttttrrrreo chsir_vel)
chair_ang_vei_norn np.Linalg.nom oot.
Stage
stage_ceward
10
disp_chair_to_targee diat_choir_to_tarpee up.linalg.nome tt.. aetpp_chair_tootrrrreeeee.
(asaiirrppse.p.2)) self. self.target_wx
chair_till # Chair is standing
np.pi:
dist_pe_toochat
ΣЕ is close to chair stage_rexard += if dist_chairr_o_ toorr_to_ttreettttoo 0.15:
$ Chair is close to target stage_reward
t rry keep cheir static reward wpw.xpp-chhir_.el_.mme
is chais_vel_norm <- 0.1 and cosir_ang_vel_oom 0.2: stage_reward
Try to increase velocity along direction to the target Compute direccional velosity
(1 totair__ve__o_ttrrreee reward max (x)exp(x)) aist_chair_tta_arrge
Mhair_vel_nen
clse:
stage_reward
zewazd - zeward stage_zeward
return
D.2 NOVEL LOCOMOTION RESULTS
Sample of zero-shot generated reward function on MuJoCo Hopper Front Flip task:
def cospute_denss_reevaddlllll action) Flost:
Sorward movement
reward_x_ = sel..hopper.co...elccc
* 1.0
Reward for ma maintaining low body height to prepare for flip
Lov_hight abs abs htp..
0.5) 0.5
Reward for fast iositive angulaz velocity for top joint self. wtg.iiittaniieeeeeooooory..
tewerd_top_jointtwellisss jelf wtt.:jiitt.nnnsrr.eeeloiiir....t > else 0.0
Reward for faat negative angulaz velocity for leg and foot Joints Lewar__ee__ooot__eeloocyy
hep htt.g_oooiitt.ngnlrsess... anguler 0.5 if
else 0.0
loa += -aelf, www.hhoppr.ottt_teeccooeooooeeee wow..t.aggglar..aalll....ooo htt.pperrfoo...onn..aiiieeeeesss
0.5 if
Peralty not completing flip (1.8., top_joint angle 18 noz 360 degree or 2xpl vadiant
28

---

Published as a conference paper at ICLR 2024
genally_incooppete.e -abstael..ogeerreee angle 2-math.pi) 0.2
Summing up the rewards and penalties
reward veeeer leward_low_heippee Lewaard_op_jooo___eee
return reward
Sample of zero-shot generated reward function MuJoCo Hopper Back Flip task:
def compute_dense_rewe (eel action) -> ILoat:
Reward for rotating around the foot jeint, which means flip is happening rexard_flip ebs
Peralty touching ground something than the foot wtw.anggggoooudeeeelleee 0 if sel wtt.poorr..t..psttttoo.... 1,25 or whoppeerrtiieeoooottooooo angle or
Oz
souchingg_rreum__emsttt -103
Peralty for not maintaining a reasonable speed speed_penall
if abs (self. wtw.pper.te..eelciiie..........t speed_penalty -50
or abs (self. (self.hcpper..... cocspper.footjooiii... fot...o........g..........eeeeee velocityy <0.1:
Reward for moving in the x-ccordinate direction reward_x if reward_x < D:
xeward_x
Total reward reward reward_flip touching__rrund_eealyy spees_pensity reward_x
return reward
Sample of zero-shot generated reward function on MuJoCo Ant Lie Down task:
compute_ sereeeddenee___eeeee denss__eewrrdseef, action) float: Defina the weight constants
K_pos 1.0 weight for the torso's z-coordinate pesiticn
k_vel 0.1 weight for the torsa's volocity
R_act 0.01 # weight for the action (torques applied)
Calcuiate position zeward (the closes the torse's position is to 0, the higher the reward) pos_rexard pas self. walf.annt.tozo.ppo. torso.posiiiionn
Calculate the velocity penalty (the slower the torss's velocity, the higher the reward) penpleyyyyyy -el faelf.ant.tai...... telso.
+
wow..tt.ttroo...ott......o. self.ant.torso veo.rr....oolllyyvr... wtw.
calculate the actien penalty (the smailer the action values, the higher the reward)
action periootpenaltyyy -M_act np. (mp.orray (action).12)
Combine the zewards and peaslties to compute the final reward pos_reward vel_penalty act' on_perslty
return
Sample of zero-shot generated reward function on MuJoCo Ant Wave Leg task:
def compute_dente reense_rwward action) float: Define the initial reward
reward D.D
Keeping the eat in place penaliza any significant movement in x, y. position
reward - abs sellf.ant i...ttttoroo torso, ve vol 0.1
reward abs (self.aant Lor. Lorso..ooo...oeeeeeeoooo velocity_yl 0.1
reward ass(self. iiselffannt ant. wtt.tttoooooooeeoooooooo wwwso.oesiiion...........
0.75) 1.0
Keeping sat's
rotating
me penalizs any significant angular velocity
(self.ant reward wtt.eelfaatttrrraee woroo..oo........o.ooooooooooo wtw.oo..o..ul.la......oo angular_veisn velocity_
reward ass (self,ant ant torso. ingular_veoocity. 0.1 reward (sel. (tt.:e/e.ttttttto..nuuuueeeeeee.
0.1
Naving the right front log give positive reward - abs reward fsel..ant.
movement in right leg
0.2
29

---

Published as a conference paper at ICLR 2024
Keeping other logs stationary Ne penalize movement the other legs For leg_joint, ankle_joint in
wtt.aaa.ffooo_ii___eee__.. seo.f.aat.bark_loft__csss (sel hea.rrraantbboc__igt__eeee.
joitt) btw.
self www.antt.ccckkeeec
hotr.ant.boo_rrigh.ennk..ec
reward -- abs jotg_oot velooctty 0.1
zeward abs bankle_joint wttp.:ioitt.nallaa..eeoooooooooo 0.1
return reward
D.3 INTERACTIVE RESULTS
For the interactive generation results on MANISKILL2 Stack Cube task, provide the traces of the whole round which contains the initial generated reward function, the human feedback and the revised reward function in git diff format.
Instruction: Pick up cube A and place it on cube B. The task is finished when cube A is on top of cube stably (i.e. cube static) and grasped by the gripper.
Initial generated reward function:
import numpy 8s op
def ceapute_dense_rewadlttte action):
reward 0.0
Check np.linalg. men.ag..cemmsel..... cemnssel..ubb...... cubeed.oseepppp (self.cubeB.poor is_cbj_on_taaget np.
top
whether it 1a
np.array (10, 0, welf.cube_half (atee_aarr__seer]____ 0.025 is_obi_statie self.cubea. www.ceue..b..ckkeeeeeeeeeeeeeeee eteck_station)
is_grasped relf.robot... grap (self,
succeas ji_ohhjon_ttrgee and stabi_statii and not la_graaped
if success:
seturn rewerd
Compute distence from rabot'a end-effector cube cop_to_ob:__pe = self.cubeh, pose.p ae robot robot.ee_poe..e <sp_to_obi_dist ap. (trpp reaching_rewarz 1 np. tcp_to_obj_distt reward peaching_rewartt
Check if the robot has grasped cube A is_grasped else 0.3
if is_grasped;
# Compute distance from cube a the top of exbe
obi_to_goal pos wtf.f.ebb.pose... np.array (a, 8, haaffanne_half__fde self, cals.
dist = up. sp..inalg..nmm. manmlg.nnrmmooo.... (obj_to__oal___ goa____o__l__p place_reward tp.tanh obj_to_goal__ias zeward i- plave_reward.
return reward
Human description: Now the robot only picks up cube A, stucks it onto cube but does not release cube A afterwards.
Human feedback for improvement: The robot should release cube A after stacking it onto cube
Revised reward function (Interaction 1):
import numpy up
compute_dense (ew.ddnssaooeadssssr action): reward - 0.0
Cbeck if cube ig..inalg.ne ner.n..rmmmalll .aub...... ts__ggjon_targgt
of cube and whether it is stable
(uolf.cubbas.pco..f np.erray (10, self.cube celff.csb__half_iieee21 0.025 is_obi_stctic self culf.cubee.ce caubbk. staticct gg_gsed self. robot, cchck cceckkggrasppseo
30

---

Published as a conference paper at ICLR 2024
success is_obj_or_taarge and is_cbj_state and not is_grasped
success is_obj on_ and is_cbj_sted
if success:
reword
return roVErd
1E not is_grasped;
reward
FOCBEE reward
Compute distence from the robot's end-effector cube tep_to_ab: ab__pos pas = aelf.aubeA, pose-p - sel f. robot robot.en_pose.
csp_to_obi_die ap, minalg.norm(tog. posohi_pos) reaching_reward = 1 np.tana ctp_to_obj_citt/ reward reaching_reward
Check if the robot has grasped cube A reward 1 is_grasped else 0.3
grapeee
# Computa distance from cube A the top of cibee obi_to_goal__og celf,
na. 8, seif.cubed.pose.
obj_to_goaa_die dist = ap mp. Linalg.norm(obj.p. (otj__t__g_gll_pe plaae_reward mp.tanh cbj_to__gall_diat) reward +* place_reware
if is_ob___00 targett
incourage the robot to relesse the grasp when cube A is on of cube reward + if not is_grasped else
return reward
Human description: Now the robot sometimes completes the task successfully, but sometimes fails.
Human feedback for improvement: Please write denser reward to complete the task stably. Revised reward function (Interaction 2):
Import mumpy sa np
def dewpttt__deeee_eeaddeeeeee reward 0.0
action):
Check if cube A on tep of cube and whether it is stable in_obj_on_target ip. aig.linalg.aom nog...nommee. (sclf. (ac.m.muebb...pppp....mm ar.array (10, 0, cub.a. Seelf.cubos.poee self.cubed _heck_statict is_grasped self.robot. check (self.
<= 0.025
g=ipper_openness
success is_obj_on_targee is_obj_stati:
1f auccess:
reward i- if not is_graaped:
|-
retur: reward
Compute distance from the roboc's end-effector to cube sop_tt_obi__aee self.cube,, sep_to_obi_dist www.linsg...o aorm tto__to_c____pee reachiigg_eware 1 np, tana tep_to_obi_datt reaching_rewaed
rolo.robbteee_poo
Check the robot grasped cube reward -- 1 12 is_grasped else 0.3
if is_grasped;
# Computa distance from cube A to the top othe oba_to_g celf.c.beB.poze.p ap.array(10, self.cube hawiibe_hll___iee_e__ self, salf.,cubAA.pase _ist dist 1p.1inalg, norm jo_g__o__oal_ppoo
obi_to_goa
31

---

Published as a conference paper at ICLR 2024
place_reward ap.tanh obj_to_goaal_dar reward + reward
if ___oo__o__targer
Encourage the robot to release the grasp when cube is of cube reward t- 3 if net is_grasped else
if _e_0b__oo_ttagee end is_abj_static:
top of and stable Encourage robot release groap when
re release_ roward release__esse reword
1* "_ggaappdd pppr_ppennesss else -3
Pnoourage the robot to keep the greap if it's not on target
holding_rewa.ce i: is_grddeed else
reward holding_rewws
else:
1f not 1s_obj_on tontage
Encourage the rcbot to move closer to the target when not grasping
top_to_target pos self self.cubel, cubee. np.ärroy [0, 0, self.cube ha f_size+2]
self robct pose..
top_to_target, sist np. 1 1nal Aeinnag.nnnn www.llgnoorrcppp.... wwwpoontcp_t___aarree__e wwr_ttt__ttttttee
Larget_reeard (5. Le__oo__arree_iiii
rеward target_reward
return reward
E ADDITIONAL ANALYSIS OF EFFECTIVENESS OVER PREVIOUS WORK
As detailed in Section 4.1 of our work, modifications to the baseline method of Yu et al. (202)) (L2R) enable adaptation only two tasks of the MANISKILL2 environment, while L2R is not suited for tasks that involve complexly surfaced objects, which cannot be adequately described by singular point. To further comprehensively evaluate the significance of the shaped dense reward structure, particularly staged reward necessitating Python's 's if-else statements, we introduce additional bascline. This baseline adapts the oracle expert-written reward codes to mirror the L2R's s format (the sum of set of individual terms) by removing all if-else statements and only keeping functional reward terms, while disregarding the original L2R's s inability to utilize point clouds for distance calculations.
As illustrated in Figure g Oracle-L28 attains comparable outcomes to the zero-shot setting for three relatively straightforward and short-horizon tasks: Lift Cube, Pick Cube, and Turn Faucet. However, for the remaining three tasks, which are comparatively more difficult, the modified Oracle-1.2R fails address the challenges, underscoring the necessity shaped and staged dense rewards.
F EXPERIMENTS ON OPEN-SOURCE LANGUAGE MODELS
We conduct experiments using open-source Large Language Models (LLMs) our foundational model to ensure reproducibility and accessibility. Specifically, we employ the instruction tunt--tudddd with human feedback version of Llama-2 (Touvron al. 2023 and Code-Llama (Rozière et al. 2023 as representative strongest open- souncs models, establishing them as the bascline.
World tasks within zero-shot setting, adhering to the methodol- ogy described in the main body of our work. The resulting learning curves of the policies trained using code generated from these two open-source models, in conjunction with GPT-4, are illustrated in Figure 9
Experiments conducted
https://huggingffae coce.co/cco cooe/ Tama/ 34b- Instruct-hf
32

---

of different reward Published as a conference paper at ICLR 2024
Grade123
Zero-hoo
Rowedoo
LFTt Cubb
Turn Faucet
Cabinet Doar
3.5
0.0
500K
Figure 8: Learning curves on MANISKILL2 compared to oracle Yu et al. (2023) settings, measured by task success rate. Oracle-L28 L2R means that we modify the expert-written reward function provided by the environment into the format of Yu et al. (023),, which is only terms, without if-else statements and other possible components of Python; zero-shot and few-shor stands for the reward function generated by TEXT2REWARD w.o and w. retrieving examples from expert-written rewards functions examples for prompting.
The learning curves illustrate the evolution of policy performance on MetaWorld tasks under a zero- shot reward generation setting. It is evident that GPT-4 consistently outperforms the other models across most tasks, achieving higher success rates and displaying more stable learning progressions. The Code-Llama model exhibits moderate success, with notable variability tasks such as 'Window Close' and 'Door Unlock'. Llama-2, while lagging behind in tasks like 'Drawer Open' and 'Door Close", shows promise in 'Button Press' with a learning curve that approaches GPT-4's performance. The shaded regions indicate standard deviations, suggesting that GPT-4's policy training is not only more successful on average but also more reliable across different random seeds. These results underscore the sophistication of GPT-4's code generation in producing effective reward functions for policy learning in a zero-shot setting, and the effectiveness of proof-of-the-comm toree-----e--c-epptt exploration in this direction using it.
It is noteworthy that, despite many models approaching the performance of ChatGPT and GPT- 4 existing benchmarks (Cobbe al. 2021 Hendrycks et al., 202T Chen et al., (02T), there remains a significant gap in the reward function generation task, suggesting that this problem could potentially serve as one of the benchmark tests for large language models and that the benchmark has considerable room for improvement. Enhancements could simultaneously benefit both the natural language processing and reinforcement learning communities.
G ADDITIONAL RESULTS
Due to page limitations, we include in this section additional learning curves on METAWORLD bench- mark (Figure 10), as well as additional image samples from the MANISKILL.2 and METAWORLDD environments (Figure TT and Figure 12). Figure [1b shows a failure case the dual-arm mobile manipulation task Push Chair. Here Oracle means the expert-written reward function provided by the environment; zero-shot and few-shot stands for the reward function is generated by TEXR2REWARD w. and w.o. retrieving examples from expert-written rewards functions examples for prompting.
G.1 ERROR ANALYSIS ON GENERATED FUNCTION
We conduct error analysis the generated reward functions. We manually go over 100 reward function examples each generated by the zero-shot and few-shot prompting on 10 different tasks of MANISKH.I12, where each task has 10 different reward codes. These reward functions are generated
33

---

Published as a conference paper at ICLR 2024
GPT-4
Code-Llama
Llama-2
Open
Window
Button
MM
0.8
0.6
86
0.6
0.2
0.2
MWN
Step
52mp
Step.
903k
400k
Sweep
Handle
0.6
WW
0.6
0.0
0.2
0.2
600k
420k
Close
0.
opp
B00K
4002
100k
SCOk
Window
0.8
M
/
0.6
0.2
St4p
600
Figure 9: Learning curves on METAWORLD under zero-shot reward generation setting, measured by success rate, using the reward functions generated by different language models. The solid line represents the mean success rate, while the shaded regions correspond to the standard deviation, both calculated across five different random seeds.
specifically for our enror analysis and with no execution feedback to L.Mss. We classify them into 4 error types: Class attributes misuse; Attributes hallucination (referring to attributes that do not exist); Syntax/shape error; Wrong package. Table shows that the overall error rate is around 10%. Within these error samples, 30% of the errors are caused by the code syntax or shape mismatch, the rest are introduced during grounding from background knowledge context choose the existing yet right one, indicating there is still space to improve on understanding how to choose the right function and attributes for TEXT2REWARD direction, especially for code generation community.
H LIMITATIONS AND FUTURE WORK
Our work demonstrates the effectiveness of generating dense reward functions for RL. We focus the code-based reward format, which gives us high interpretability. However, the symbolic space may not cover all aspects of the reward. Furthermore, our method also assumes that the perception
34

---

Published as a conference paper at ICLR 2024
Dracle
Zero-shot
Door Unlock
Drawer Close
0.1
0.
0.2
2008
4024
cook
5004
200k
403k
600k
BECk
Handle Press
Open
58
5.6
84
84
Step
S15
Figure 10: Additional learning curves on METAWORLD, measured by success rate. The solid line represents the mean success rate, while the shaded regions correspond the standard deviation, both calculated across five different random seeds.
Table Error distribution across 100 generated reward code on MANISKILL2.
Type
Description
Zero-shot
Few-shot
Class attribute misuse Attribute hallucination
Use other classes' attribute wrongly Invent nonexistent attribute Incorrect program grammar or shape mismatch Import incorrect package function Execute correctly without error
6%
4%
3%
2%
Syntax/shape error Wrong package Correct
3%
3%
1
1 %
87%
90%
is already done with other off-the-shelf components. Future works may consider the combination of code-based and neural network-basse reward design that combines both symbolic reasoning and perception. Utilizing the knowledge derived from LLMs in creating such models showcases promising prospects and could be advantageous in several scenarios (Wulfmeier et al.) 2016 Finn
al. 2016 Christiano et al. 2017 Lee al. 2021).
Although our main method is simple but effective, it still has room for improvement by designing methods generate better reward functions, possibly leading higher success rates and the ability to tackle more complex tasks.
At present, motion, to illustrate this approach. In the future, this research may find broader applications in various
primarily concentrate robotics tasks, specifically manipulation and loco-
reinforcement learning related domains, including gaming (Brockman al. 20166] Schrittwieser
et al., 2020; Zhong et al. 2021 Fan et al. (022), web navigation (Shi et al. 2017 Zhou et al. (023)
household management
Puig al. 2018 Shridhar al. 2020a b).
35

---

Published as a conference paper at ICLR 2024
(a) Zero-shot: Open Cabinet Door
(h) Zero-shot: Push Chair
(c) Few-shot: Push Chair
Figure 11: Additional image samples from MANISKILL2. (a) shows successful case of zero-shot TEXT2REWARD for the mobile manipulation task OpenCabinetDoor. (b) shows a failure case of zero-shot generation for the dual-arm mobile manipulation task PushChair. (c) shows that few-shot generation can stably solve the PushChair task even though zero-shot can not.
36

---

Published as a conference paper at ICLR 2024
(a) Zero-shot: Button Press
(b) Zero-shot: Drawer Open
(c) Zero-shot: Window Open
Figure 12: Additional image samples from METAWORLD. All three rollouts show that zero-shot TEXT2REWARD can stably solve these tasks.
37


Reviews:
[{"review_id": "DHH1jLUgfp", "reviewer": "Reviewer_fmuH", "summary": "The paper proposes a method to generate dense rewards for continuous control RL tasks using LLMs and an API summary of the environment. The method runs either completely zero-shot on top of the prompt with environment information, or in a few-shot manner by concatenating the nearest neighbor task specification and expert reward into the context. In addition, after RL training, human feedback can be solicited, leading to further refinement of the reward code by the LLM, which leads to hopefully stronger RL. Across a number of tasks, the zero-shot dense reward code is competitive with the expert code, and in some settings few-shot generation can improve the results, though in others it is harmful.", "questions": "See weaknesses above, I'm most interested in analysis on the source of the improvements in each of the tasks, and how the language model goes about creating reward for a novel task", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- Interesting application of a new tool! Using LLMs to generate reward code seems like an easy way to simplify problems we may not already have solutions for, but can describe in language, and is a completely different way around the sparse reward problem\n- Zero-shot results seem pretty strong across all environments\n- Nice results on new tasks that (as far as I know) we don't have expert reward for (e.g. Hopper flipping)", "weaknesses": "- There is no qualitative analysis/discussion of what the source of the improvement is:\n  - Why does Zero-shot outperform Few-shot on Turn Faucet, Open Cabinet Door, Open Cabinet Drawer?\n  - Why does Few-shot fail to outperform Oracle even though Oracle is in context (Lift Cube, Turn Faucet, Push Chair, Open Cabinet Drawer)?\n  - In the cases that few-shot improves on zero-shot, what is the source of this improvement?\n- There are a lot of missing details in the experiments:\n  - There is only one example of human feedback (Figure 6) and it is in a schematic diagram for a task that does not have ground truth, I would like to see a few traces of the whole round (generation, feedback, generation, feedback, generation) in order to understand what exactly is happening\n  - For results in Table 1 and Figure 4 on novel behaviors, the standards for human evaluation and who conducted the evaluation (the authors, peers from the same institution, crowdsourced to a platform) are missing\n  - The experiments in Table 1 and Figure 4 are supposedly conducted in a zero-shot setting (caption in Figure 4), yet Figure 6 gives a schematic for ambiguity resolution which would imply a few-shot result for novel behaviors, an experiment which I do not see in the paper and appendices.\n  - I do not see the choice of $k$ for the number of neighbors that appear in the few-shot context\n  - For generated rewards in Appendix D.1 on Pick Cube, the few-shot vs. oracle code is almost indistinguishable except for 2 constants (cube placed at goal reward, grasped reward). Given this difference is so small, it seems important to know what the human feedback was: are we just getting lucky?\n  - As before, given that the few-shot vs. oracle code is so close on Pick Cube (the only example we have to judge), why is it the case that the few-shot generation is underperforming oracle generation in other settings (Lift Cube, Turn Faucet, Open Cabinet Drawer, Push Chair)?\n  - Is it always the case that the Oracle code for the task is put into the context for Few-shot?\n- Section 4.2 is about code errors that occur before any RL happens, and this seems like a necessary filtering step, but I think having an example of the generation/feedback process is much more important in the main body than Table 2.\n- How long does the iteration loop take? Each iteration requires training policies, so it is quite expensive, and it may be nice to think about early evaluation\n- It would be nice to include code examples for the novel behavior tasks to see what is happening. Given there is no baseline in this case, simply presenting a quantitative evaluation without any analysis is a little sparse...", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pCZXpMBSgf", "reviewer": "Reviewer_XwRM", "summary": "This paper introduces Text2Reward, an automated generation framework of dense reward functions based on large language models. T2R produces interpretable, free-form dense reward codes using a compact representation of the environment source code. T2R is able to outperform human-written reward codes on a range of tasks and allow iterative refinement with human feedback.", "questions": "1. There are 11 tasks in MetaWorld, but only 6 of them have results in the paper?\n\n2. Is there a reason few-shot examples are not used for MetaWorld?\n\n3. How robust is T2R to different random generations? The temperature used for the LLM generation is not shown. \n\n4. Can T2R work for non pick-and-place, opening/closing/pushing robotic manipulation tasks?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper studies the pertinent problem of automated reward design using LLMs. Given that reward design is a fundamental challenge in RL and that LLMs for decision making have largely been limited to high-level planning tasks, this paper offers a fresh perspective and a nice solution to the growing literature of LLMs for problem design and low-level robotic control.\n\nThis paper's method is novel and more flexible than a prior work (Yu et al., 2023) in that it does not require extensive manual templating per robotic embodiment or task and is capable of generating free-form reward functions. It incorporates a compact representation of the environment, background knowledge (e.g., available function APIs), and/or few-shot examples to successfully do so. Finally, this paper demonstrates interesting use case of the proposed method, such as real-world evaluation as well as learning from human feedback.  \n\nThe paper is well-written and free of grammatic errors.", "weaknesses": "1. The primary weakness of the paper is that most evaluation tasks are from benchmarks that have been released before GPT-4's knowledge cutoff date (September, 2021). Mujoco and Metaworld tasks have been extensively studied in the reinforcement learning literature; ManiSkill2, though released recently, have many overlapping tasks with ManiSkill, which was released in mid 2021; in particular, most of the tasks, to the best of my knowledge, were in the original ManiSkill benchmark. Given this, it is not clear whether the reward design capability of T2R can readily transfer to an unseen task in a new simulator. \n\n2. Relatedly, the \"novel\" behavior on the Mujoco locomtoin tasks have appeared in prior literature; for example, Hopper back flip is shown in Christiano et al., 2017. It's unclear whether T2R has benefited from that knowledge possibly being in the training set of the backbone LLM. \n\n3. Most manipulation tasks are of \"pick-and-place\" or opening/closing/pushing nature. These are also the most common types of manipulation tasks that the RL literature has studied. It is possible that GPT-4 is generally adept at writing reward functions for those task types. \n\n3. T2R appears to still work best with few-shot examples. In many tasks that do not belong to a family of tasks introduced by a benchmark, providing few-shot examples can still be difficult. \n\n3. For each task, only one reward function is reported. It is not clear whether T2R is robust to stochasticity in LLM generation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DrwOPh39nW", "reviewer": "Reviewer_VQaY", "summary": "The authors present a method to use LLMs to write dense reward functions to train policies from natural language instructions.", "questions": "Putting it here because this is not a “weakness” as this paper just came out: Eureka has recently come out (https://eureka-research.github.io/), and as concurent work, what do the authors think they contribute compared to Eureka?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "************************************************************Evaluation Comprehensiveness:************************************************************ The authors utilized 17 tasks in ManiSkill2 and Metaworld, 6 in Mujoco, and also a few real world experiments — this is quite comprehensive compared to the average ICLR RL paper.\n\n****************************Experiments:**************************** \n\n- Real world experiments are always nice!\n- Human feedback experiments are also great and demonstrate that on certain tasks the method can improve the policy learning performance — something that isn’t always straightforward even with humans redesigning reward functions to try to improve polichy learning\n\n******************Results:****************** I think results are solid compared to the oracle.\n\n**********Motivation:********** The paper motivation is nice — reducing human effort in scaling policy training by using LLMs for python/numpy reward function design.", "weaknesses": "************************************************************Contribution over prior work:************************************************************ I’m not too convinced on the major **********technical********** contribution over Yu et al. 2023 (Language to Rewards for Skill Synthesis). Compared to that paper, the main claimed novelty is dense vs sparse reward and the use of standard python code: “Different from recent work….our free-form dense reward code has a wider coverage of tasks and can use established coding packages.” But that paper also uses very pythonic code, furthermore utilizes sparse reward mainly due to using MPC. I’m not too convinced that the pythonic → python and MPC → RL are large technical contributions on their own. This should be clarified more specifically/clearly in the paper if there is another technical contribution over Yu 2023, and if not, then is one of the main reasons for my score.\n\n************************Experiments:************************\n\n- The authors should compare against Yu 2023, especially if claiming their dense free-form reward + use of established coding packages can result in superior performance. The comparison isn’t exactly 1-1 given the claims, but currently there is no comparison to any baseline to contextualize the performance of the method. In fact, I think a comparison with Yu 2023 + RL would be fairest, as Yu 2023 likely can use RL instead of MPC without change.\n- Open-source LLMs: Utilizing closed-source LLMs has obvious downsides, e.g., reproducibility (API backend can change at any time) and access to academic researchers (cost per token vs able to be used on a standard GPU setup). It would be beneficial to the community to demonstrate some results with some smaller open source models like LLaMa-2.\n\n**************************Minor Issues:**************************\n\n- 4.1: Appendex → Appendix\n- I think it’d be nice to have a few small examples in the **********main paper********** of generated reward functions (not full things, just a few lines). This makes the experiment section more readable without needing to jump around to the appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C81YNNAhLY", "reviewer": "Reviewer_bPzM", "summary": "The authors propose a method for leveraging language models to write code computing reward functions for simulated reinforcement learning tasks, given a textual description of the task.\n\nTheir method makes use of an expert provided environment “abstraction” (a specification) written as a sequence of pythonic class declarations, and combines it with a human provided textual instruction describing the desired trainable task. This is used to prompt GPT-4 into generating python code for computing rewards. Optionally, the prompt given to the language model can be augmented with few-shot examples of expert and previously-generated reward functions, retrieved from a database using embedding similarity between desired and stored instructions.\n\nWithin the proposed framework, it is also possible to iteratively improve the generated reward function by having a human provide written and descriptive feedback given a video from a policy learnt with the current generated reward. Such feedback is introduced in the language model prompt and used to generate a new version of the reward function code.\nIn a series of experiments over a variety of simulated environments, the authors demonstrate the effectiveness of the approach, while generally showing few-shot and iterative prompting to improve results. Overall, generated reward functions appear to be similar in performance to expert provided rewards.", "questions": "* Would it be possible to show how this method generalizes to environments that are sure not to have been included in the training data of the model? (e.g: a simple simulated environment designed specifically for this paper)\n* Did you try your method on open source or otherwise publicly available models that would allow for a snapshot of the results to be preserved? If so, how do they compare to GPT-4?\n\nEdit: I raised the score according to my latest comment.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper addresses the important problem of dense reward specification in reinforcement learning, using a method based on zero-shot and few-shot prompting of a language model, something not shown before except for concurrent work **[1]**.\n\nThe authors show that their method is competitive with human expert reward specification in a broad series of experiments. They moreover demonstrate one such policy to transfer successfully to real world execution.\n\nThe authors also include an analysis of the failure modes of reward specification, specifically for the cases in which the generated reward function code leads to a python runtime error.\n\n**[1]** Ma et al., Eureka: Human-Level Reward Design via Coding Large Language Models, https://eureka-research.github.io/", "weaknesses": "The paper essentially proposes prompting techniques for code generation with pre-trained language models accessible via API, specifically GPT-4. Despite its interesting conclusion and results, it consists at most of an interesting observational study over the capabilities of GPT-4, as the prompting techniques appear straightforward, fundamentally easy to execute manually by any user of the GPT-4 online api platform in a process of trial and error for reward design. I would not consider this a problem worthy of rejection per se, as similar “prompting techniques” results have been published before in machine learning venues, such as the famous “chain of thought” prompting technique. Still, it is my opinion that this limits the significance of the work, as GPT-4 simply fills in the shoes of an expert reinforcement learning reward coder on well known simulation environments (while “chain of thought” prompting was an innovation in how to query a model for more general purpose NLP benchmarks).\n\nMore importantly, to my understanding, no truly novel environments are tackled in this paper. For most of the shown tasks, GPT-4 can reasonably be expected to draw from training data containing countless reward function specifications from those environments, if not reward functions for the desired task itself.\n\nMoreover, the reliance of the paper on GPT-4 makes the results inherently irreproducible, as OpenAI does not have a policy of indefinitely supporting api access to specific snapshot versions of their models (specifically, GPT-4-0314 will be deprecated on June 13th 2024 at the earliest).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)