Eureka: Human-Level Reward Design via Coding Large Language Models
Authors: Yecheng Jason Ma
William Liang
Guanzhi Wang
De-An Huang
Osbert Bastani
Dinesh Jayaraman
Yuke Zhu
Linxi Fan
Anima Anandkumar
Venue: ICLR 2024 poster (2024)
Keywords: Large Language Models
Reinforcement Learning
Dexterous Manipulation
Reward Learning
Robotics
Primary area: reinforcement learning

Abstract: Large Language Models (LLMs) have excelled as high-level semantic planners for sequential decision-making tasks. However, harnessing them to learn complex low-level manipulation tasks, such as dexterous pen spinning, remains an open problem. We bridge this fundamental gap and present Eureka, a human-level reward design algorithm powered by LLMs. Eureka exploits the remarkable zero-shot generation, code-writing, and in-context improvement capabilities of state-of-the-art LLMs, such as GPT-4, to perform evolutionary optimization over reward code. The resulting rewards can then be used to acquire complex skills via reinforcement learning. Without any task-specific prompting or pre-defined reward templates, Eureka generates reward functions that outperform expert human-engineered rewards. In a diverse suite of 29 open-source RL environments that include 10 distinct robot morphologies, Eureka outperforms human experts on 83% of the tasks, leading to an average normalized improvement of 52%. The generality of Eureka also enables a new gradient-free in-context learning approach to reinforcement learning from human feedback (RLHF), readily incorporating human inputs to improve the quality and the safety of the generated rewards without model updating. Finally, using Eureka rewards in a curriculum learning setting, we demonstrate for the first time, a simulated Shadow Hand capable of performing pen spinning tricks, adeptly manipulating a pen in circles at rapid speed.

Published as a conference paper at ICLR 2024
EUREKA: HUMAN-LEVEL REWARD DESIGN VIA CODING LARGE LANGUAGE MODELS
Yecheng Jason Ma William Liang", Guanzhi Wang² 3, De-An Huang', Osbert Bastani", Dinesh Jayaraman', Yuke Zhu! Linxi "'imm Fan 1, Anima Anandkumar1
'VVIIA, UPenn. "Caltech, 4UT Austin; 'Equal advising https: research, github..hh....... i0
ABSTRACT
Large Language Models (LLMs) have excelled as high-level semantic planners for sequential decision-making tasks. However, harnessing them to learn complex low- level manipulation tasks, such as dexterous pen spinning, remains an open problem. We bridge this fundamental gap and present EUREKA, human-level reward design algorithm powered by LLMs. EUREKA exploits the remarkable zero-shot generation, code-writing, and in-context improvement capabilities of state-of-the- art LLMs, such as GPT-4, to perform evolutionary optimization over reward code. The resulting rewards can then be used to acquire complex skills via reinforcement learning. Without any task-specific prompting or pre-defined reward templates, EUREKA generates reward functions that outperform expert numan-engineered rewards. In a diverse suite of 29 open-soure RL environments that include 10 distinct robot morphologies, EUREKA outperforms human experts on 83% of the tasks, leading to average normalized improvement of 52%. The generality EUREKA also enables new gradient-free in-context learning approach reinforcement learning from human feedback (RLHF), readily incorporating human inputs to improve the quality and the safety the generated rewards without model updating. Finally, using EUREKA rewards in curriculum learning setting, we demonstrate for the first time, a simulated Shadow Hand capable of performing pen spinning tricks, adeptly manipulating pen in circles rapid speed.
INTRODUCTION
Large Language Models (LLMs) have excelled as high-level semantic planners for robotics tasks (Ahn
2022 Singh et al. 2023) but whether they be used to learn complex low-level manipulation
Corresponding authors: jasonyma@seas wwwpaeaeeeessseeeeeceeomm aopenneedu, dr, jim ai ggmail....... gmmil.com coom
Isaac (9 Tasks)
Dexterity (20 Tasks)
frlr
Pen Spinning (Trajectory Frames)
Figure 1: EUREKA generates human-leva reward functions across diverse robots and tasks. Combined with curriculum learning, EUREKA for the first time, unlocks rapid pen- pen-spinning capabilities anthropomorphic five-finger hand.

---

Published as a conference paper at ICLR 2024
Environment Code
Coding LLM
(GPT 4)
Q
anguel,
class SasicooforrPanspssseeWeoo
compute, wet.errtttoonaaaaa
Reward Candidate Sampling
Angular velority peselty moretoss angvel ponoltt torch
selT..cb rrs
selF.cck,, (00_001__nn0 Tinwel
or.nore 2.8, laren wwwpeeeppeerrrr...ee rers 2.41
selff.db), Lanovll
self.igt_rot
& GPU- Accelerated
Eureka
Query with Feedback
self.compate, wttppppte_uulllatttteee sttate)
compute St
tratred
previded reward furction cade... av ww.penalty: Lt: ["a.02', 18.05".
Task Description
18.05", 18.84".
-..]
11.57"
33.95",
Reward Reflection
Please corefully analyos the policy feedback and provide a new, improved reverd functton....
Isaac
To make the shednw hand spin the pen to a target ortentation
Gym
Figure 2: EUREKA takes unmodified environment source code language task description as context to zero-shot generate exccutable reward functions from coding LLM. Then, it iterates between reward sampling, GPU-accelerated reward evaluation, and reward reflection to progressively improve its reward outputs.
tasks, such as dexterous pen spinning, remains an open problem. Existing attempts require substantial domain expertise to construct task prompts or learn only simple skills, leaving a substantial gap in achieving human-level dexterity (Yu et al. 2023 Brohan al. (023).
On the other hand, reinforcement learning (RL) has achieved impressive results in dexter- ity (Andrychowicz al. 22200 Handa al.
2023) well as many other domains-if the human
designers can carefully construct reward functions that accurately codify and provide learning signals for the desired behavior; likewise, many real-world RL tasks admit sparse rewards that difficult for learning, necessitating reward shaping that provides incremental learning signals. Despite their fundamental importance, reward functions are known to be notoriously difficult design in prac- tice (Russell & Norvig 1995 Sutton & Barto 2018); a recent survey conducted finds 92% of polled reinforcement learning researchers and practitioners report manual trial-and-error reward design and 89% indicate that their designed rewards are sub-optimal (Booth et al. 2023) and lead to unintended behavior (Hadfiel- -Hadeelld-Menelll et al., 2017).
Given the paramount importance of reward design, ask whether it is possible to develop universal reward programming algorithm using state-of-the-ar coding LLMs, such as GPT-4. Their remarkable abilities in code writing, zero-shot generation, and in-context learning have previously enabled effective programmatic agents (Shinn et al. 2023 Wang et al. 2023a). Ideally, this reward design algorithm should achieve human-level reward generation capabilities that scale broad spectrum of tasks, including dexterity, automate the tedious trial-and-error procedure without human supervision, and yet be compatible with human oversight assure safety and alignment.
We introduce Evolution-driven Universal REward Kit for Agent (EUREKA), a novel reward design algorithm powered by coding LLMs with the following contributions:
Achieves human-level performance on reward design across a diverse suite of 29 open-sourced RL environments that include 10 distinct robot morphologies, including quadruped, quadcopter, biped, manipulator, as well as several dexterous hands; see Fig. [ Without any task specific prompting or reward templates, EUREKA autonomously generates rewards that outperform expert human rewards on 83% of the tasks and realizes an average normalized improvement of 52%.
2. Solves dexterous manipulation tasks that were previously not feasible by manual reward engineering. We consider pen spinning, in which a five-finger hand needs to rapidly rotate a pen in pre-defined spinning configurations for many cycles possible. Combining EUREKA with curriculum learning, we demonstrate for the first time rapid pen spinning maneuvers on simulated anthropomorphic Shadow Hand (see Figure bottom).
2

---

Published as a conference paper at ICLR 2024
3. Enables new gradient-free in-context learning approach to reinforcement learning from human feedback (RLHF) that can generate more performant and humam--lignnn reward functions based various forms of human inputs without model updating. We demonstrate that EUREKA can readily benefit. from and improve upon existing human reward functions. Likewise, we showcase EUREKA 's capability in using purely textual feedback to generate progressively more human-aligned reward functions.
Unlike prior work L.2R on using LLMs to aid reward design Yu et al. 2023). EUREKA is completely free of task-specific prompts, reward templates, well as few-shot examples. In our experiments, EUREKA significantly outperforms L2R due to its ability to generate free-form, expressive reward programs. EUREKA'S generality is made possible through three key algorithmic design choices: environment as context, evolutionary search, and reward reflection. First, by taking the environment source code as context, EUREKA can zero-shot generate executable reward functions from the backbone coding LLM (GPT-4). Then, EUREKA substantially improves the quality of its rewards by performing evolutionary search, iteratively proposing batches of reward candidates and refning the most promising ones within the LLM context window. This in-context improvement is made effective via reward reflection, a textual summary of the reward quality based on policy training statistics that enables automated and targeted reward editing; see Fig. 3 for an example of EUREKA zero-shot reward as well as various improvements accumulated during its optimization. To ensure that EUREKA can scale up its reward search to maximum potential. EUREKA evaluates intermediate rewards using accelercee distributed reinforcement learning on IsaacGym (Makoviychuk et al., 2021). which offers up three orders of magnitude in policy learning speed, making EUREKA A extensive algorithm that scales naturally with more compute. See Fig. F for an overview. We are committed open-soune sourcing all prompts, environments, and generated reward functions promote further research on LLM-based reward design.
2 PROBLEM SETTING AND DEFINITIONS
The goal of reward design is to return a shaped reward function for a ground-truth reward function that may be difficult to optimize directly (e.g.. sparse rewards); this ground-trune -ouun reward function may only be accessed via queries by the designer. We first introduce the formal definition from (Singh et al. 2010), which we then adapt to the program synthesis setting, which we call reward generation.
Definition 2.1. (Reward Design Problem (Singh et al. 2010)) A reward design problem (RDP) is tuple P = (M, R, TM, F). where M (S, TT) is the world model with state space S, action space A, and transition function T. R is the space of reward functions; AM (+) R + II is learning algorithm that outputs a policy
5 ((A) that optimizes reward R E R in the resulting Markov Decision Process (MDP), (M, Rt:: F : 11 + IR is the fitness function that produces a scalar evaluation of any policy, which may only be accessed via policy queries (i.e., evaluate the policy using the ground truth reward function). In an RDP, the goal is to output reward function R E R such that the policy π
AM (R) that optimizes R achieves the highest fitness score F(x).
Reward Generation Problem. In our problem setting, every component within RDP is specified via code. Then, given a string that specifies the task, the objective of the reward generation problem output reward function code such that F F(AM (R)) is maximized.
METHOD
EUREKA consists of three algorithmic components: 1) environment as context that enables zero-shot generation of executable rewards, 2) evolutionary search that iteratively proposes and refines reward candidates, and 3) reward reflection that enables fine-grained reward improvement. See Alg. T for pseudocode; all prompts included in App.
3.1 ENVIRONMENT AS CONTEXT
Reward design requires the environment specification to be provided to the LLM. We propose directly feeding the raw environment source code (without the reward code, if exists) as context. Given that any reward function is function over the environment's state and action variables, the only
3

---

Published as a conference paper at ICLR 2024
def conpute_
cespute_renarllooiees www.reeeedddee.... (ejerstojjest.rot, tot, object_angvel, object_pes, fingartip_pos):
Rotation renard
rot diff w...bbtttrcc..uilloeeeeee goal.rot, dime1)
rotation_rewrd reep 28.0
rotation,
Changing hyperparameter
rotatton_rewrd forch.cxp(-rotattee cerattton_eeoo_roee rot_diff)
Distance reward
nin_cistunce_ten wow.ch..orrr (ingerrip.nnn.eee objjct_pouss, Nonw], dham21, dimed). .valowa
min_distance torch.eing
disttiae
dispaee_iittaneerree reward attchaxp(--iin_ddee www.ddisttteeeeeee teep win, distanst
Luncapped
reward, 9.0,
Changing functional
total_ rotation renard distance_sewozd
Angular velocity penalty
Adding
angvel_nozv barch.oorafobjeet.nnvee dim-1)
angesl, threshold 8.5
papell_pnaelyy.ee teep
borch.ateee(amgeee..mo mngrel_threshopl
torch. (xpgggge peralp_elttyteee teep (angve) norn tngeel_.trreshold) threshold)), terch.zeros, wtw.rreerrrr.oreeeee (inneengee _amm)
rewerd 9.5 notation_eewar 8.3 resaad 9.2 angular, angular_ velooitty_pmmaly
roward conponenesnnnttss =
*rotation rotation_romact
"distance_rewae roward:: rattene__renerd,
"angular_ velbiirr_elecityypeellt.. vepllr..elestty.eelll
return rotal_reward crard__copponents
Figure 3: EUREKA can zero-shot generate executable rewards and then flexibly improve them with many distinct types of free-form modification. such as (I) changing the hyperparameter of existing reward components, (2) changing the functional form of existing reward components, and (3) introducing reward components.
requirement in the source code is that it exposes these environment variables, which easy to satisfy. In cases where the source code is not available, relevant state information also be supplied via an API, for example. In practice, to ensure that the environment code fits within the LLM's context window and does leak simulation internals (so that we can expect the same prompt generalize to new simulators), we have an automatic script to extract just the environment code snippets that expose and fully specify the environment state and action variables. see App. for details.
Given environment as context, EUREKA instructs the coding LLM to directly return executable Python code with only generic reward design and formatting tips, such exposing individual components the reward as dictionary output (for reasons that will be apparent Sec. 3.3
Algorithm EUREKA
Require: Task description environment code M, coding LLM LLM, fitness function F, initial prompt pr ompt
2: Hyperparameters: search iteration N, iteration batch size K
3: for N iterations do
4:
// Sample K reward code from T.I.M
Prompt and 3 in App. Remarkably, with only these minimal already
Ri, Rg 1.1.(1,, M, prompt))
6:
// Evaluate reward candidates
instructions, EUREKA
7:
= ((..).....555 = F(Rx)
zero-shot generate plausibly-looking rewards in diverse environments in its first attempts. An example EUREKA output is shown in Fig. 3 As seen.
8:
// Reward reflection
prompt := prompt Reflection Reestr shst))
where best = max, 81,..., 8K
10:
// Update Eureka reward
EUREKA adeptly composes over existing observation variables (e.g.. (ingertipppos) in the provided
11:
REureaa &Eumla (R )
if sh 8Eunkk
12:
Output: Rrareka
environment code and produces a competent reward code all without any environment-speccin prompt engineering reward templating. On the first try, however, the generated reward may not always be executable, and even if is, it can be quite sub-optimal with respect to the task fitness metric F. While doing so does not scale to new tasks and hinders the overall generality of our system. How can we effectively overcome the sub-optimality of single-sample reward generation?
improve the prompt with spsk-spec formatting and reward design hints,
4

---

Published as a conference paper at ICLR 2024
3.2 EVOLUTIONARY SEARCH
In this section, we will demonstrate how evolutionary search presents a natural solution that addresses the aforementioned execution error and sub-optimality challenges. In each iteration, EUREKA samples several independent outputs from the LLM (Line 5 in Alg. D Since the generations are i.i.d, the probability that all reward functions from an iteration are buggy exponentially decreases as the number of samples increases. We find that for all environments we consider, sampling just modest number of samples (16) contains at least one executable reward code in the first iteration.
Given executable reward functions from an earlier iteration, EUREKA performs in- in-context reward mutation, proposing new improved reward functions from the best one in the previous iteration. Concretely, new EUREKA iteration will take the best-performing reward from the previous iteration, its reward reflection (Sec, (.3c. and the mutation prompt (Prompt in App. [A] as context and generate K more i.idd reward outputs from the LLM; several illustrative reward modifications are visualized in Fig. 3 This iterative optimization continues until a specified number of iterations has been reached. Finally, we perform multiple random restarts to find better maxima; this is a standard strategy in global optimization. In all our experiments, EUREKA conducts independent runs per environment, and for each run, searches for 5 iterations with K 16 samples per iteration.
3.3 REWARD REFLECTION
In order to ground the in-context reward mutation, we must be able to put into words the quality of the generated rewards. We propose reward reflection, automated feedback that summarizes the policy training dynamics in texts. Specifically, given that EUREKA reward functions are asked to expose their individual components in the reward program (e.g., reward www..cooooenttt.cmmmmmmmmm in Fig. 3 reward reflection tracks the scalar values of all reward components and the task fitness function at intermediate policy checkpoints throughout training. For instance, consider the illustrative example in Fig. 2 where the snapshot values of av-penalty are provided as a list in the reward feedback. See App. G.T for full examples.
This reward reflection procedure, though simple to construct, is important due to two reasons: (1) the lack of fine-grained reward improvement signal in the task fitness function, and (2) the algorithm- dependent nature of reward optimization (Booth et al.] 2023). First, as we can query the task fitness function F on the resulting policies, a simple strategy is to just provide this numerical score as the reward evaluation. While serving the holistic ground-truth metric, the task fitness function itself lacks in credit assignment, providing no useful information on why a reward function works or not. Second, whether reward function is effective is influenced by the particular choice of RL algorithm, and the same reward may perform very differently even under the same optimizer given hyperparameter differences (Henderson et al. 2018 Agarwal et al., 2021). By providing detailed accounts on how well the RL algorithm optimizes individual reward components, reward reflection enables EUREKA to produce more intricate and targeted reward editing.
4 EXPERIMENTS
We thoroughly evaluate EUREKA on a diverse suite of robot embodiments and tasks, testing its ability generate reward functions, solve new tasks, and incorporate various forms of human input. We use GPT-4 (OpenAl 2023), in particular the gpt ept-4-031- 0314 4 variant, as the backbone LLM for all LLM-based reward-design waiggnalggooo algorithms unless specified otherwise.
Environments. Our environments consist of 10 distinct robots and 29 tasks implemented using the IsaacGym simulator (Makoviychuk al. 202I). First, we include original environments from IsaacGym (Isaac), covering diverse set of robot morphologies from quadruped, bipedal, quadrotor, cobot arm, to dexterous hands. In addition to coverage over robot form factors, we ensure depth in our evaluation by including all 20 tasks from the Bidexterous Manipulation (Dexterity) benchmark (Chen) ctt al. 2022). Dexterity contains 20 complex bi-manual tasks that require a pair of Shadow Hands to solve wide range of complex manipulation skills, ranging from object handover to rotating cup by 180 degrees. For the task description input to EUREKA, A, we use the official description provided in the environment repository when possible. See App. B for details on all environments. It is worth noting that both benchmarks are publicly released concurrently, or after the GPT-4 knowledge cut-off date (September 2021), so GPT-4 is unlikely to have accumulated extensive internet knowledge
5

---

Published as a conference paper at ICLR 2024
Success Rate
40 b0
Nommalzed
145
2.
0.5
1.4
2.6
Figure 4: EUREKA outperforms Human and 1.2R across all tasks. In particular, EUREKA realizes much greater gains on high-dimensional dexterity environments.
about these tasks, making them ideal testbeds for assessing EUREKA's reward generation capability compared to measurable human-engineered reward functions.
4.1 BASELINES
L2R YYu al. 2023) proposes a two-stage LLM-prompting solution to generate templated rewards. For an environment and task specified in natural language, a first LLM is asked to fill in natural language template describing the agent's ' motion; then, second LLM is asked to convert this "motion description" into code that calls manually defined set of reward API primitives to write a reward program that sets their parameters. To make L2R competitive for our tasks, we define the motion description template to mimic the original L2R templates. and we construct the API reward primitives using the individual components of the original human rewards when possible. Note that this gives L2R advantage it has access to the original reward functions. Consistent with EUREKA, conduct 5 independent 1.2R runs per environment, and for each run, we generate 16 reward samples. See App. C for more details.
Human. These are the original shaped reward functions provided in our benchmark tasks. As these reward functions written by active reinforcement learning researchers who designed the tasks, these reward functions represent the outcomes of expert-level human reward engineering.
Sparse. These identical to the fitness functions F that
evaluate the quality of the generated rewards. Like Human, these are also provided by the benchmark. On the dexterity tasks, they are uniformly binary indicator functions that measure task success; on Isaac tasks, they vary in functional forms depending on the nature of the task. See App. B for a description of the ground-truie truth scoring metric for all tasks.
4.2 TRAINING DETAILS
Policy Learning, For each task, all final reward functions are optimized using the same RL algorithm with the same set of hyperparameters. Isaac and Dexterity share well-tuned PPO implementa- tion (Schulman et al. 2017: Makoviichuk & Makoviychuk] 2021). and we use this implementation and the task-specific PPO hyperparameters without any modification. Note that these task hyperpa- rameters are tuned to make the official human-engineee engineered rewards work well. For each final reward function obtained from each method, we run independent PPO training runs and report the average of the maximum task metric values achieved from 10 policy checkpoints sampled at fixed intervals. In particular, the maximum is taken over the same number of checkpoints for each approach.
Reward Evaluation Metrics. For Isaac tasks, since the task metric F for each task varies in semantic meaning and scale, we report the human normalized score for EUREKA and L2R, Mached Sparse
Human- Saannn-pprrsee This metric provides a holistic measure of how EUREKA rewards fare against human-expert rewards with respect the ground-truth task metric. For Dexterity, since all tasks are evaluated using the binary success function, we directly report success rates.
4.3 RESULTS
EUREKA outperforms human rewards. In Figure 4 report the aggregate results Dexterity and Isaac. Notably, EUREKA exceeds or performs on par to human level on all Isaac tasks and 15 out of 20 tasks Dexterity (see App. for per-task breakdown). In contrast, I.2R, while comparable

---

Published as a conference paper at ICLR 2024
on low-dimensional tasks (e.g., CartPole, BallBalance), lags significantly behind on nigh-dimensional tasks. Despite being provided access to some of the same reward components as Human, L2R still underperforms EUREKA after its initial iteration, when both methods have had the same number of reward queries. As expected, 1.2R's lack of expressivity severely limits its performance. In contrast, EUREKA generates free-form rewards from scratch without any domain-specifie knowledge and performs substantially better. In App. F. we present results on additional evaluation metrics such as interquantile mean (IQM), probability improvement (Agarwal al. 2021), ), and the aggregate RL training curves; on all evaluations, we observe the consistent trend that EUREKA generates the most capable reward functions. Furthermore, we ablate GPT-4 with GPT-3.5 and find EUREKA degrades in performance but still matches or exceeds human-level most Isaac tasks, indicating that its general principles can be readily applied to coding LLMS of varying qualities.
EUREKA consistently improves over time. Fig. [5 we visualize the average performance of the cumula- tive best EUREKA A rewards after each evolution iteration. Moreover, study an ablation, EUREKA w.o. Evo- lution (32 Samples), which performs only the initial reward generation step, sampling the same number of reward functions as two iterations in the original EUREKA. This ablation helps study, given fixed number of reward function budget, whether is more advantageous to perform the EUREKA evolution or simply sample more first-attempt rewards without iterative improvement. As seen, on both benchmarks, EUREKA rewards steadily improve and eventually surpass human rewards in performance despite sub-par initial performances. This consistent improvement also cannot be replaced by just sampling more in the first iteration as the ablation's 's performances lower than EUREKA after iterations on both benchmarks. Together, these results demonstrate that EUREKA'S 's novel evolutionary optimization is indispensable for its final performance.
Dexterity
Isaac
Success Rate
Sceel
200
15
0.50
0.35
Curece Eatalors
Suraka Eamelton
Dureica
--
Earalia W.d Cudlation (32 Sampiie))
Figure 5: EUREKA progressively produces better rewards via in- context evolutionary reward search.
EUREKA generates novel rewards. We assess the nov- elty of EUREKA rewards by computing the correlations between EUREKA and human rewards on all the Isaac tasks; see App. Bpp.BBoor for details on this procedure. Then, we plot the correlations against the human normalized scores on a scatter-plot in Figure 6 where each point represents single EUREKA reward on a single task. As shown, EUREKA mostly generates weakly correlated reward func- tions that outperform the human ones. In addition, by examining the average correlation by task (App. D. we observe that the harder the task is. the less correlated the EUREKA rewards. We hypothesize that human rewards are less likely be near optimal for difficult tasks, leaving more room for EUREKA A rewards to be different and bet- In a few cases, EUREKA rewards are even negatively correlated with human rewards but perform significantly better, demonstrating that EUREKA can discover novel reward design principles that may run counter to human intuition; we illustrate these
Eureka Rewards vs, Human Rewards on IseacGym
1.0
Reward Correl Correlation
2.-
X
2.4
0.0
-0.4
Durita Whn Kount:33) sts icaur
-0.6
-08
Humen Normalized Score (Log Scale)
Figure 6: Eureka generates novel rewards.
EUREKA rewards in App. G.2
Reward reflection enables targeted improvement. To assess the importance of constructing reward reflection in the reward feedback, we evaluate an ablation, EUREKA (No Reward Reflection), which reduces the reward feedback prompt to include only snapshot values of the task metric F. Averaged over all Isaac tasks, EUREKA without reward reflection reduces the average normalized score by 28.6%; in App. F we provide detailed per-task breakdown and observe much greater performance deterioration higher dimensional tasks. To provide qualitative analysis, in App. G.1 we include several examples in which EUREKA utilizes the reward reflection to perform targeted reward editing.
7

---

Published as a conference paper at ICLR 2024
EUREKA with curriculum learning enables dexterous pen spinning. Finally, investigate whether EUREKA can be used to solve a truly novel and challenging dexterous task. To this end. test bed. This task is highly dynamic and requires Shadow Hand to continuously rotate a pen to achieve some pre-defined spinning patterns for as many cycles as possible; we implement this task top of the original Shadow Hand environment in Isaac Gym without changes to any physics parameter, ensuring physical realism. We consider a curriculum learning (Bengio et al. 2009) approach break down the task into manageable components that can be independently solved by EUREKA. Specifically, we first use EUREKA to generate a reward for the task of re-orienting the pen random target configurations and train a policy using the final EUREKA reward. Then, using this pre-trained policy Free-Trained, we fine-tune it using the same EUREKA A reward to reach the sequence of pen-spinning configurations (Fine-Tuned) To demonstrate the importance of curriculum learning, we also directly train a policy from scratch on the target task using EUREKA reward without the first-stage pre- pre-training (Scratch). The RL training curves are shown in Figure Eureka fine-tuning quickly adapts the policy to successfully spin the pen for many cycles in a row; see project website for videos. In contrast, neither pre-trained or policies can complete approach, we have also trained pen spinning policies for variety of different spinning configurations: all pen spinning videos can be viewed on our project website, and experimental details are in App. [D.]] These results demonstrate EUREKA'S 's applicability to advanced policy learning approaches, which are often necessary for learning very complex skills
propose pen spinning
single cycle of pen spinning. In addition, using this EUREKA fine-tuning
4.4 EUREKA FROM HUMAN FEEDBACK
In addition to automated reward design, EUREKA enables a new gradient-fres in-context learning approach to RL from Human Feedback (RLHF) that can readily incor- porate various types of human inputs to generate more performant and bulan-aligned reward functions.
Spinning
C Cycles
Spinninng
EUREKA can improve and benefit from human reward functions. We study whether starting with human reward function initialization, a common scenario in real-world RL applications, is advantageous for EUREKA. Impor- tantly, incorporating human initialization requires no mod- ification EUREKA we can simply substitute the raw human reward function as the output of the first EUREKA iteration. To investigate this, we select several tasks from Dexterity that differ in the relative performances between the original EUREKA and human rewards. The full results are shown in Figure
20
700 Rune
Scratch
Tandd
12
Sho
Figure 7: EUREKA can be flexibly combined with curriculum learning to acquire complex dexterous skills.
As shown, regardless of the quality of the human rewards, EUREKA improves and benefits from human rewards as EUREKA (Human Init.) is uniformly better than both EUREKA and Hu- man all tasks. This suggests that EUREKA'S 's in-context reward improvement capability is largely independent of the quality of the base reward. Furthermore, the fact that EUREKA can significantly improve over human rewards even when they are highly sub-optimal hints towards an interesting hypothesis: human designers are generally knowledgeable about relevant state variables but are less proficient at designing re- wards using them. This makes intuitive sense as identifying relevant state variables that should be included in the reward function involves mostly common sense reasoning, but reward design requires specialized knowledge and experience in RL. Together, these results demonstrate EUREKA'S 's reward assistant capability, perfectly complementing human designers' knowledge about useful state variables and making up for their less proficiency on how to design rewards using them. In App. G.3 provide several examples of EUREKA (Human Init.) steps.
Figure 8: EUREKA effectively improves and benefits from human reward initialization.
8

---

Published as a conference paper at ICLR 2024
Reward reflection via human feedback induces aligned behavior. So far, all EUREKA rewards are optimized against a fixed, black-box task fitness function F. This task metric, however, may not fully align with human intent. Moreover, in many open-ended tasks, F may not be available in the first place (Fan et al. 2022). In these challenging scenarios, we propose to augment EUREKA by having humans step and put into words the reward reflection in terms of the desired behavior and correction. We investigate this capability in EUREKA by teaching a Humanoid agent how to run purely from textual reward reflection; in App. G.4 we show the exact sequence of human feedback and EUREKA rewards. Then, we conduct a user study asking 20 unfamiliar users to indicate their preferences between two policy rollout videos shown in random order, one trained with human reward reflection (URREA-HHE) and the other one trained with the original best EUREKA reward; the details are in App. [D3 As shown in Fig. | despite running bit slower, the EUREKA-HFF agent is preferred by a large majority of our users. Qualitative, we indeed see that the HUREKA-HF agent acquires safer and more stable gait, as instructed by the human. See the project website for a comparison.
5 RELATED WORK
Reward Design, Reward engineering is long-standing challenge in reinforcement learn-
ing (Singh et al. 2010 Sutton & Barto 2018).
Eureka
The most common reward design method is man-
Reward Reflection
ual trial-and-error (Knox al.)
2023 Booth
Human
cet al.
2023). Inverse reinforcement learning
(IRL) infers reward functions from demonstra-
tions (Abbeel & Ng 20044 Ziebart et al.) 2008)
Method
Forward Velocity
Human Preference
Ho & Ermon 2016). but it requires expensive ex- pert data collection, which may not be available, and outputs non-interpretable black-box reward functions. Several prior works have studied au- tomated reward search through evolutionary al- gorithms (Nickum et al. 2010 Chiang et al. 2019; Faust et al. 2019). These early attempts limited to task rask-specific implementations of evolutionary algorithms that search over only parame- ters within provided reward templates. Recent works have also proposed using pretrained foundation
EUREKA
7.53
5/20
EUREKA-HF
5,58
15/20
Figure 9: EUREKA can incorporate human reward re- flection to modify rewards that induce safer and more human-aligned aligned behavior.
models that can produce reward functions for new tasks (Ma et al. 2022 2023 Fan et al.) 2022 Du
iet al.] [223a] Karamcheti et al.] 2023] Du et al. [0233b K Kwon et al. 2023). Most of these approaches
output scalar rewards that lack interpretability and do not naturally admit the capability improve or
adapt rewards on-the-fly, In contrast, EUREKA adeptly generates free-form, white-box reward code and effectively in-context improves.
Code Large Language Models for Decision Making, Recent works have considered using coding
LLMs (Austin al.] [202]|Chen et al.) [202]] Rozière et al., 2023) to generate grounded and structured
programmatic output for decision making and robotics problems (Liang et al. 2023 Singh et al.
2023
Wang et al. 2023b Huang et al., 2023 Wang et al. 2023a Liu et al.) 2023a; Silver et al.
2023 Ding et al. 2023 Lin et al. 2023 Xie al. 2023). However, most of these works rely on known motion primitives to carry out robot actions and do not apply to robot tasks that require low-level skill learning, such as dexterous manipulation. The closest to our work is recent work (Yu et al. 2023) that also explores using LLMs to aid reward design. Their approach, however, requires doman-specific task descriptions and reward templates.
CONCLUSION
We have presented EUREKA, a universal reward design algorithm powered by coding large language models and in-context evolutionary search. Without any task-specific prompt engineering human intervention, EUREKA achieves human-level reward generation on a wide range of robots and tasks. EUREKA'S 's particular strength in learning dexterity solves dexterous pen spinning for the first time with a curriculum learning approach. Finally, EUREKA enables a gradient-free free approach to reinforcement learning from human feedback that readily incorporates human reward initialization and textual feedback to better steer its reward generation. The versatility and substantial performance gains of EUREKA suggest that the simple principle of combining large language models with evolutionary algorithms are a general and scalable approach to reward design, an insight that may be generally applicable to difficult, open-ended search problems.
9

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
We are grateful to colleagues and friends at NVIDIA and UPenn for their helpful feedback and insightful discussions. We thank Viktor Makoviychuk, Yashraj Narang, Iretiayo Akinola, Erwin Coumans for their assistance on Isaac Gym experiment and rendering. This work is done during Yecheng Jason Ma's internship NVIDIA. We acknowledge funding support from NSF CAREER Award 2239301, ONR award N00014-22-1-2677, NSF Award CCF-1917852, and ARO Award W911NF-20-1-0080.
10

---

Published as a conference paper at ICLR 2024
REFERENCES
Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning, pp. 2004.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 34:29304-29320, 2021.
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do as can, not say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01699 2022.
Ilge Akkaya, Marein Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al. Solving rubik's cube with a robot hand. arXiv preprint 19X1:0991000711: 2019.
OpenAl: Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, al. Learning dexterous in an- manipulation. The International Journal of Robotics Research, 39(1):3-20, 2020.
Jacob Austin, Augustus Odena, Maxwell Nye. Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, al. Program synthesis with large language models. arXiv preprint arXiv: 07730773, 2021.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference machine learning, pp. 41-48, 2009.
Serena Booth, W Bradley Knox, Julie Shah, Scott Niekum, Peter Stone, and Alessandro Allievi. The perils of trial-and reward design: misdesign through overfitting and invalid task specifications. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 5920-5929, 2023.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv: 014.0000000 2016.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.158/8 2023.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 2021.
Yuanpei Chen, Tianhao Wu, Shengjie Wang, Xidong Feng, Jiechuan Jiang, Zongqing Lu, Stephen McAleer, Hao Dong, Long-Chun Zhu, and Yaodong Yang. Towards human-level bimanual dexterous manipulation with reinforcement learning. Advances Neural Information Processing Systems, 35:5150-511 5165505-56,..01 2022.
Hao-Tier Lewis Chiang, Aleksandra Faust, Marek Fiser, and Anthony Francis. Learning navigation behaviors end-to-end with autorl. IEEE Robotics and Automation Letters, 4(2):2007-2014, 2019.
Yan Ding, Xiaohan Zhang, Chris Paxton, and Shiqi Zhang. Task and motion planning with large language models for object rearrangement, arXiv preprint arXiv:2303.062: 2023.
Yuqing Du, Ksenia Konyushkova, Misha Denil, Akhil Raju, Jessica Landon, Felix Hill, Nando de Freitas, and Serkan Cabi. Vision- models as success detectors. arXiv preprint arXiv:2303.07280 2023a.
Yuqing Du, Olivia Watkins, Zihan Wang. Cédric Colas, Trevor Darrell, Pieter Abbeel, Abhishek Gupta, and Jacob Andreas. Guiding pretraining in reinforcement learning with large language models. arXiv preprint arXiv:2302.06663 2023b.

---

Published as a conference paper at ICLR 2024
Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang. Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar, Minedojo: Building open-ended embodied agents with intemet-scale knowledge. Advances in Neural Information Processing Systems, 35: 18343-18362, 2022.
Aleksandra Faust, Anthony Francis, and Dar Mehta. Evolving rewards to automate reinforcement learning. arXiv preprint
2019.
Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In 2017 IEEE international conference robotics and automation (ICRA), pp. 3389-3396. IEEE, 2017.
Dylan Hadfield-Menell, Smitha Milli, Pieter Abbeel, Stuart Russell, and Anca Dragan. Inverse reward design. Advances in neural information processing systems, 30, 2017.
Ankur Handa, Arthur Allshire, Viktor Makoviychuk, Aleksei Petrenko, Ritvik Singh, Jingzhou Liu, Denys Makoviichuk, Karl Van Wyk, Alexander Zhurkevich, Balakumar Sundaralingam, et al. Dextreme: Transfer of agile in-hand manipulation from simulation to reality. In 2023 IEEE International Conference Robotics and Automation (ICRA), pp. 5977-5984. IEEE, 2023.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning, Advances in neural information processing systems, 29, 2016.
Taylor Howell, Nimrod Gileadi, Saran Tunyasuvunakool, Kevin Zakka, Tom Erez, and Yuval Tassa. Predictive sampling: Real-time behaviour synthesis with mujoco. arXiv preprint arX1v:2212.00541, 2022.
Siyuan Huang, Zhengkai Jiang, Hao Dong, Yu Qiao, Peng Gao, and Hongsheng Li. Instruct2act: Mapping multi-modality instructions to robotic actions with large language model. arXiv preprint arXiv:2305.11 11176, 2023.
Siddharth Karamcheti, Suraj Nair, Annie Chen, Thomas Kollar, Chelsea Finn, Dorsa Sadigh, and Percy Liang. Language-driven representation learning for robotics. arXiv preprint 12766 2023.
W Bradley Knox, Alessandro Allievi, Holger Banzhaf, Felix Schmitt, and Peter Stone. Reward (mis) design for autonomous driving. Artificial Intelligene, 3 316: 103829, 2023.
Ashish Kumar, Zipeng Fu, Deepak Pathak, and Jitendra Malik, Rma: Rapid motor adaptation for legged robots. arXiv preprint 10x.v:2107.040::0 2021.
Minae Kwon, Sang Michael Xie, Kalesha Bullard, and Dorsa Sadigh. Reward design with language models. arXiv preprint arXiv:2303. 00001, 2023.
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp.
IEEE, 2023.
Kevin Lin, Christopher Agia, Toki Migimatsu, Marco Pavone, and Jeannette Bohg. Tex:2motion: From natural language instructions to feasible plans. arXiv preprint arXiv:2303. 2023.
Bo Liu, Yuqian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone. Llm+ p: Empowering large language models with optimal planning proficiency, arXiv preprint
arXiv:2004.11477, 2222,
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304.0844 2023b.
Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. Vip: Towards universal visual reward and representation via value- iipleii pre- pre- training.. arXiv preprint 200.v:22/0.000:30 2022.
12

---

Published as a conference paper at ICLR 2024
Yecheng Jason Ma, William Liang, Vaidehi Som, Vikash Kumar, Amy Zhang, Osbert Bastani, and Dinesh Jayaraman. Liv: Language-iman -mngge representations and rewards for robotic control. arXiv preprint arXiv:2306.0099 2023.
Denys Makoviichuk and Viktor Makoviychuk, rl-games: A high-performance framework for rein- forcement learning.
May 2021.
Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al. Isaac gym: High performance gpu-based physics simulation for robot learning, arXiv preprint
2tth::200..1077000 2021.
Gabriel B Margolis, Ge Yang, Kartik Paigwar, Tao Chen, and Pulkit Agrawal. Rapid locomotion via reinforcement learning. arXiv preprint (tt::2205.022222 2022.
Scott Niekum, Andrew G Barto, and Lee Spector. Genetic programming for reward function search. IEEE Transactions Autonomous Mental Development, 2(2):83-90, 2010.
OpenAl. Gpt-4 technical report, 2023.
Baptiste Rozière, Jonas Gehring. Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code Ilama: Open foundation models for code. arXiv preprint arXiv:2308. /2950, 2023.
Stuart J Russell and Peter Norvig. Artificial Intelligence: A Modern Approach. Prentice Hall, Englewood Cliffs, NJ, USA, 1st edition, 1995.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06:: 063477 2017.
Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: autonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:2303. 1366, 2023.
Tom Silver, Soham Dan, Kavitha Srinivas, Joshua B Tenenbaum, Leslie Pack Kaelbling, and Michael Katz. Generalized planning in pddl domains with pretrained large language models. arXiv preprint arXiv:2305.11014. 2023.
Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. Progprompt: Generating situated robot task plans using large language models. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 11523- 11523-11530. IEEE, 2023.
Satinder Singh, Richard L. Lewis. and Andrew G. Barto. Where do rewards come from? In Proceedings of the International Symposium on Al Inspired Biology A Symposium at the AISB 2010 Convention, pp. 111-116, 2010. ISBN 1902956923.
Laura Smith, Ilya Kostrikov, and Sergey Levine. A walk in the park: Learning to walk in 20 minutes with model-free reinforcement learning. arXiv preprint arXiv:2008.07860, 2022.
Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA, USA, 2nd edition, 2018.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RS. international conference on intelligent robots and systems, pp. 5026-5033 IEFE, 2012.
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291 2023a.
Huaxiaoyue Wang, Gonzalo Senzalez-Pumariegee Yash Sharma, and Sanjiban Choudhury. Demo2code: From summarizing demonstrations to synthesizing code via extended chain-of- thought. arXiv preprint arXiv:2305. 16744 2023b.
13

---

Published as a conference paper at ICLR 2024
Grady Williams, Andrew Aldrich, and Evangelos Theodorou. Model predictive path integral control using covariance variable importance sampling, arXiv preprint 1rXiv:1509.01 2015.
Yaqi Xie, Chen Yu, Tongyao Zhu, Jinbin Bai, Ze Gong, and Harold Soh. Translating natural language to planning goals with large-language models. arXiv preprint arXiv:2302.05//88 2023.
Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Jalung-Ching Lin, Zicheng Liu, and Li- juan Wang. The dawn of Imms: Preliminary explorations with gpt-4v (ision). arXiv preprint arXiv:2309.17420 17421, 9, 2023.
Wenhao Yu, Nimrod Gileadi, Chuyuan Fu, Sean Kirmani, Kuang-Huct Lee, Montse Gonzalez Arenas, Hao-Tien Lewis Chiang, Tom Erez, Leonard Hasenclever, Jan Humplik, et al. Language rewards for robotic skill synthesis. arXiv preprint arXiv:2306. 08647, 2023.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint 1rXSv22304.10592, 2023.
Brian D Ziebart, Andrew I. Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. Aaai, volume 8, pp. 1433-1438. Chicago, II., USA, 2008
14

---

Published as a conference paper at ICLR 2024
A FULL PROMPTS
In this section, we provide all EUREKA prompts.
Prompt I: Initial system prompt
reward
trying
reward funations reinnorpement learring
Lasks as efJective as possible.
Your goal
rewerd fanction
environment that will help agent Learn
task descrited in text.
Your reward funct fon should use variables
ronment inputs. example
the reward function signature can be: Storch. jit.script def cospute_rewardjca pos: corch.Tenso;, gosi_pos; Corch.Tensor Mupeehtorch.eeneoo (ict, 1enso.1enso:]]:
return reward, II
Since the rawerd function will be dacorated with @torch. jiriicript, pi ease make aure that the code is compatible with Torch Stript (e.g., use torch tensor instead numpy ar:ay],
Make sure any new tensor or varieble introduce is on the same device as the input tensers.
Prompt Reward reflection feedhack
We trained a RL policy using the provided seward function code and tracked the values of the individual components in the reward function as well aa global policy metrics such aucceas rates and episode lengtha after every (epech__req) epocha and the maximum, minimum values encountered:
<REWARD REFLECTION
reward function that can better salve the task. some helpful tips for anolyzing the policy feedback:
carefully analyze policy
provide new,
(1) If the success rates are always near sero, then you must rewrite the entire reward function
(2) If the values for a cerzain reward component are neer identicsi throughout, then this able optimiza this component it written. You may consider (a) Changing its scale or the value of its temperature parameter
Re writing
component
(c) Biacarding the reward component
(3) If some reward components! magnitude significantly lazger, then you must ze scale proper range
its value
Please analyze each existing reward component in the suggasted manner above first, and then writa the reward function code.
Prompt 3: Code formatting tip
output reward fuactioa should coneist of
(1) the total reward,
(2) dictionary
individusl
zomponent.
The code output should be formacted as python code string: "1*pyyhon 1..8
Some helpful tips for writing the reward function code:
(1) You may find it helpful to normalize the reward to a fixed range by applying
torea.exp the overall (2) If you choose to transform zeward compozent, then you must also introduce sempersture parameter
-ranstormations
compenente
transformation function; this parameter zust variable in the veward funatioon and it must not be an input varlable. Each transtormed reward component should have its own temperature varisble
(3) Make the type of each input varisble correctly specified; flost input variable should not be specifled as Tensor
(4) importantly,
zeward code's s input variables mus: conzein only attributes the provided environment class definition (namely, variables that have prefix self.!. cireumstance can you introduce new input variables.
B ENVIRONMENT DETAILS
In this section, we provide environment details. For each environment, we list its observation and action dimensions, the verbatim task description, and the task fitness function F. F is evaluated per-environment, and our policy feedback uses the mean across environment instances.
For the functions below, denotes the Tag norm, and denotes the indicator function.
15

---

Published as a conference paper at ICLR 2024
IsaacGym Environments
Environment (obs dim, action dim)
Task description
Task fitness function F
Cartpole (4, 1)
To balance pole on a cart so that the pole stays upright duration
Quadcopter (21, 12)
To make the quadcopter reach and hover near fixed position -cur_dist
FrankaCabinet (23, 9)
To open the cabinet door [cabinet_poo
0.39]
Anymal (48, 12)
To make the quadruped follow randomly chosen x, y, and yaw target velocities [linvel_erro) error angvel_error) error)
BallBalance (48, 12)
To keep the ball on the table top without falling duration
Ant (60, 8)
To make the ant run forward as fast possible
cur dist prev_dist
AllegroHand (88, 16)
To make the hand spin the object to a target orientation number of consecutive successes where current success is [rot dist 0.1]
Humanoid (108, 21)
To make the humanoid run fast possible
cur_dist prev_dist
ShadowHand (,11, 20)
To make the shadow hand spin the object to target orientation number of consecutive successes where
current success is [rst_dist 0.1]
Dexterity Environments
Environment (obs dim, action dim)
Task description
Task fitness function F
Over (398, 40)
This class corresponds the HandOver task. This environment consists of two shadow hands with palms facing up. opposite each other, and an object that needs to be passed. In the beginning, the object will fall randomly in the area the shadow hand on the right side. Then the hand holds the object and passes the object to the other hand. Note that the base of the hand is fixed. More importantly, the hand which holds the object initially can not directly touch the target, nor can directly roll the object to the other hand, so the object must be thrown up and stays in the air in the process
1 [dist < 03]
16

---

Published as a conference paper at ICLR 2024
DoorCloselnward 52)
This class corresponds to the DoorCloseInward task. This environment require closed door be opened and the door can only be pushed outward or initially open inward. Both these two environments only need do the push behavior, so it is relatively simple
1 Latoooz__annee_ddd
< 0.5]
DoorCloseOutward (417, 52)
This class corresponds to the DoorCloseuutward task. This environment also require closed door to be opened and the door only be pushed inward initially open outward, but because they can't complete the task by simply pushing, which need to catch the handle by hand and then open close it, so it is relatively difficult
1 [door_ha nd dist < 0.5]
DoorOpenlnward (417, 52)
This class corresponds to the DoorOpenlnward task. This environment also require opened door to be closed and the door can only be pushed inward or initially open outward, but because they can't complete the task by simply pushing, which need to catch the handle by hand and then open or close it, so it is relatively difficult
1 [door_ Sloor_handi haldle e _ddist > 0.5]
DoorOpenOutward (417, 52)
This class corresponds to the DoorOpenOutward task. This environment require a opened door be closed and the door can only be pushed outward initially open inward. Both these two environments only need to do the push behavior, so it is relatively simple
1 [door_hamee __dist < 0.5]
Scissors (417,55 52)
This class corresponds to the Scissors task. This environment involves two hands and scissors, we need use two hands open the scissors
[[dof_pos 3]
SwingCup (417,52)
This class corresponds to the SwingCup task. This environment involves two hands and dual handle cup, we need to use two hands to hold and swing the cup together
[rot dist 0 785]
Switch (417, 52)
This class corresponds to the Switch task. This environment involves dual hands and bottle, we need use dual hand fingers to press the desired button
1[1.4 4 (left_ssittee switth_withh_ +
2) 0. 0.05]
Kettle (417, 52)
This class corresponds the PourWater task. This environment involves two hands, kettle, and a bucket, we need to hold the kettle with one hand and the bucket with the other hand, and pour the water from the kettle into the bucket. In the practice task Isaac Gym, we use many small balls simulate the water
1[ bucket bettle_spone
0.05]
LiftUnderarm (417,50 52)
This class corresponds to the LiftUnderarm task. This environment requires grasping the pot handle with two hands and lifting the pot to the designated position. This environment designed simulate the scene of lift in daily life and is practical skill
[dist 0.05]
Pen (417, 52)
This class corresponds to the Open Pen Cap task. This environment involves hands and pen, we need to use two hands to open the pen cap
1[5 ppen_cap pen_ body|
1.5]
BottleCap (420, 52)
17

---

Published as a conference paper at ICLR 2024
This class corresponds the Bottle Cap task. This environment involves two hands and bottle, we need to hold the bottle with one hand and open the bottle cap with the other hand. This skill requires the cooperation of two hands to ensure that the cap does not fall [dist 0.03]
CatchAbreast (422, 52)
This class corresponds to the Catch Abreast task. This environment consists of two shadow hands placed side by side in the same direction and an object that needs to be passed. Compared with the previous environment which more like passing objects between the hands of two people, this environment is designed to simulate the two hands of the same person passing objects. so different catch techniques are also required and require more hand translation and rotation techniques [dist] 0.03
CatchOverUUnderamm (422, 52)
This class corresponds the Over2Underarm task. This environment is similar Catch Underarm, but with an object in each hand and the corresponding goal on the other hand. Therefore, the environment requires two objects to be thrown into the other hand at the same time, which requires a higher manipulation technique than the environment of a single object
1 [dist < 0.03]
CatchUnderarm (422, 52)
This class corresponds to the Catch Underarm task. In this task, two shadow hands with palms facing upwards are controlled to pass an object from one palm to the other. What makes it more difficult than the Hand over task is that the hands' translation and rotation degrees of freedom are no longer frozen but are added into the action space
1 [dist < 03]
ReOrientation (422, 40)
This class corresponds to the ReOrientation task. This environment involves two hands and two objects. Each hand holds object and need to reorient the object the target orientation
[rrot_ dist 0.1]
GraspAndPlace (425, 52)
This class corresponds to the GraspAndPlace task. This environment consists of dual-hande an object and a bucket that requires us to pick up the object and put it into the bucket
][]block bucket < 0.2]
BlockStack (428, 52)
This class corresponds to the Block Stack task. This environment involves dual hands and two blocks, and need stack the block as tower
[gooa dist_1 0.07 and goal dist_2 0.07 and
50 (0.05 2 2_dist_ 1) 1]
PushBlock (428, 52)
This class corresponds to the PushBlock task. This environment involves two hands and two blocks, we need to use both hands to reach and push the block to the desired goal separately. This a relatively simple task
1 [left_dist <= 0.1 and right_ <= 0.1]
0.5 * _iitt_distt <= 0.1 and right_diss 0.1]
TwoCatchUnderarm (446, 52)
This class corresponds to the UwoCCathUnnderarm task. This environment is similar to Catch Underarm, but with an object in each hand and the corresponding goal on the other hand. Therefore, the environment requires two objects to be thrown into the other hand at the same time, which requires higher manipulation technique than the environment of single object
1 [goal __dis__
0.06]
18

---

Published as a conference paper at ICLR 2024
C BASELINE DETAILS
Language-to-Rent Rewadds (L2R) uses an LLM to generate a motion description from a natural language instruction and set of reward API calls from the motion description. The reward is computed as the sum of outputs from the reward API calls. While the LLM automates the process of breaking down the task into basic low-level instructions, manual effort still required to specify the motion description template, low-level reward API, and the API's function implementations.
All three parts require significant design considerations and can drastically affect L2R's performance and capabilities. Unfortunately. this makes comparison difficult since 1.2R requires manual engineer- ing whereas Eureka fully automatic -ammatii-mmiiuitg thus arises from how much suman-tuning should be done with L2R's 's components. Nonetheless, we seek to provide fair comparison and base our implementation off two factors:
To create our motion description template, reference L2R's quadruped and dexterous manipula- tor templates. Specifically, our templates consist values and statements that relate two parameters. We also aim mimic the style of L2R's 's template statements in general.
of statements that set parameters to quantitative
The reward API is designed so that each template statement can be faithfully written in terms of API function. The functions implemented resemble their respective human reward terms from their environment; thus, L2R is given an advantage in that its components resemble the manually-tune human reward. In few exceptions where the human reward differs significantly from the L2R template style, we base our API implementation on the formulas provided in the L2R appendix.
L2R was designed to allow for agent in single environment perform multiple tasks. Thus, each environment has its own motion description template and reward API. Since our experiments range over many agents and environments, we have template and API for each Isaac task, and generalize all Dexterity tasks into one environment with all necessary objects.
For illustration, descriptor and coder prompts for the Dexterity experiments below.
Prompt 1: Dexterity descriptor prompt
have dextarous monipulatera (shadow hands) and you to help plan how it should move to perform tasks us using the fol terplate:
[start of descriptien]
objecti=[CKIICE: KINSERT овJЕсTS HERE>! should (CHDICE: to, for from) object2=
<TNSERT ORJECTS FERE>, natring).
[optional] object33
<INSERT овJECTS HERE>) should be (CHOICE: close to, far Irom] (2900CC, <TNSERT ORJECTS FRRE>, nothing).
[optional] ob'ect1 needs to have a rotation orientation similar to object2, [optional] object3
have zotation crientetion similar objects. <iNSERT OPTIOMAL AAND DESCRIPTIONS HERE>
[optional] doors needa to be (CнOТCE: spen, closad) (CROICE: inward, outward). [optional] aciasor needs to opened to [NUM: D.0] radians.
[optional] block2 needs to be stacked on top of blockl. description)
Rules:
1. If sae phrases like [NUM: default_walue) replace the encire phrase with numerical value.
2. II see phrases like (CHOICE; choicel, choice2, ..th it means should replace the entire
choices listed.
3. If you see [opticnal], it means you only add that line if necessary for the task, otherwise remove
4. The ervironment contains KINSERT OBJECTS HERE>, Do not invent new objacts not listed here. that want the manipulator to perform and you will 5. will tell you a behavior/skt if you may only need change [ew lines. Always stazt the provide the full plan,
description with Istart of plan] and end it with lend of pian]. capable doing anything,
can assume cask.
most chailenging
Your plan should es close to the provided template as possible. not include additional detaila.
Prompt 2: Dexterity coder prompt
19

---

Published as a conference paper at ICLR 2024
with palm to nanipulate objects we want you turn into the corresponding program with following Functions:
a plan
def Fe ...
name_obj_st
This term sets reward for minimizing 12 diatance between name_oBj_A and name_obi_a so they get closer to other.
name_obi_A and nama_obi_R are selected from [<TNSERT FTRLDS HERR>].
...
def set_max ...
(1 name_abj__i
This term sets reward for maximizing 12 diatance between name_obi_A and name_obi_B so they
get closer te other.
name_obi_A and name_obi_B are selected from [<TNSERT FIELDS HERB>].
...
def set ...
(nang_obb name_abi__i
This Lerm encourages the orientation of name_ob __ to be close to the orientation of neme_obj_B. c and same_obj_B selected [<INSEar ORIENTATION FIELDS HERE >1.
example plan:
sobjact1 should be close to obfect?=abjjatt__gee L_gal.
ject1
rotation orientation similar object2.
To perforn this task, the left manipulator's paln should move close to objectl.
Example code:
1i__iinn__ddstenee reward ("abjecti", "ebjecti__oal") set set_min_ www.dittteeeeeeeeeeeeeeeeee reward ("a "left_ pa Im*) sel_00)_orientationnne ("object1", gebje_l__oaal")
Remember:
1. Always format the code in code blocks.
wrap your code funation. Your output should only consist of function calls like the example above,
invent
functions classes. only
functiens you
are
ozes listed above, and do not implement them. Do not leave unimplemented code blocks in your respanse.
import 5. If you are not sure what value to use, just use your bast judge. Do not use None for arything.
4. The only allowed library numpy.
any other library.
6. Do not calculate the position or direction ion of any object lexcept for the ones provided abovel
number directly 1. You do not need to make the robct da extra things not mentioned in the plan such as stopping the robot.
your guess.
For the sections surrounded by angle brackets <>, we specify a list of valid objects for each Dexterity task. For example, ShadowHandPen'ss list of objects is defined as follows:
"shadow hand pen": ["ieft_pain", "zight "right_paia"" paln", foreflngee,, "left_ "iefe_aiddllfinnee""
beét_ringfinger"" "left_lietlefinger= "lest_thumb", "igaat_forefinger' "rightt_ringtingar" "right_lsttlefinnan "right_thunb" "pen_cap",
"pen*]
summary of terms and their implementations for each experiment is in Table 3 Note that many enviromments automatically randomize their target parameters during training after a reset or success criteria is met, which L2R cannot account for during the reward generation stage. Thus, while L2R's experiments define targets in terms of quantitative values, it's it's incompatible with our environments, and we define targets instead as relations between two parameters (usually the object and the object's target).
C.1 L2R REWARD EXAMPLES
Example 1: I.2R reward function on Humanoid, Human Normalized Score: 0.0
set_torso cat__oorsohheighttt heighttrewarrd reward 11.1) sat_torso, pocitt__ewarrdde
Bet_angle_to_ttaqee__eeee
20

---

Published as a conference paper at ICLR 2024
Reward Term
Formulation
Dexterity
Minimize distance
- 1l P2l12
Maximize distance
|p: P2lla
Minimize orientation
2 aarssin (anssimmmin((l(qq22)alaa 1))
AllegroHand
Minimize distance
|p - p2112
Maximize distance
Ip] -2lle
Minimize orientation difference
1/(/2 (r 1))
Maximize orientation difference
-1/([2 aresin(min([|luqqggg) 1))
Ant
Torso height
-1h - h1]
Torso velocity
|I|Vayy /| Uz
Angle to target
0 Ozl
Anymal
Minimize difference
exp - (x - x2t)
BallBalance
Ball position
1/(1 + ||p Pell2)
Ball velocity
1/(1 + |10 vell2)
Cartpole
Pole angle
(A 0₂)
Pole velocity
V Ut
Cart velocity
V U2.
FrankaCabinet
Minimize hand distance
|| P2112
Maximize hand distance
1P1 P2 1/2
Drawer extension
1 P
Humanoid
Torso height
-1h h1
Torso velocity
|||Vayli || Uz
Angle to target
0 - Opl
Quadcopter
Quadcopter position
1/(1 + ||| pel1))
Upright alignment
1/(1 [1 1 12
Positional velocity
1/(1+ + ||v vx13)
Angular velocity
1/(1+ ||w will3)
ShadowHand
Minimize distance
[|pi 12/12
Maximize distance
[p: - P2l12
Minimize orientation difference
1/(/2 arcsin (min(I) 1)) +c)
Maximize orientation difference
- ( aresin (min (wwlmm/mnnnnnnllllll 1))1 +e)
Table 3: L2R reward primitives and their implementations, v()) denotes the vector part of quaternion subscript f denotes target value, and denotes the normal vector (orientation). All components are weighed equally.
Example 2: L2R reward function on ShadowHHandKettl. Success Rate: 0.07
set sat_min_le htw.:iin____iiannn______ooo distannc_rrvvard ("ketsla ha_dae""" "busket handla")
set_min dit.__iaaanceeeeeeaeeeee "ketsle_spout", "bucket_handia httin____diitnn__________ ("Ieft_palet) hanckee_handee") set_min stl_disttnca__eee ("right_paaaaa lmt, "kattll__anddle")
21

---

Published as a conference paper at ICLR 2024
set_min wtw..iinnn__dictnceeeeeeeeee (" thumm=' "Sucket_ handlee) "kattla_handle")
set min_ (tt_n__2__ssstne_rreeo ("rightt (rrring
D EUREKA DETAILS
Environment as Context. In Isaac Gym, the simulator adopts a environment design pattern in which the environment observation code is typically written inside compute.observat function within the environment object class; this applies to all our enviromments. Therefore, we have written automatic script extract just the observation portion of the environment source code. This is done largely to reduce our experiment cost as longer context induces higher cost. Furthermore, given that current LLMs have context length limit, this task agnostic way of trimming the environment code before feeing it to the context allows us to fit every environment source code into context.
()
Example 1: Humanoid environment observation given to EUREKA.
Ilumanoid (VecTeek)
***gest of the envirenment definition onitted.
compute_ computo_abeervattie ons 1self)
self. well..gym.rrr.. www.gggmm.eeee...... sefrress_di__ttte___ee tonsor (self.sim)
(self.sim)
selr. fow..eeeeeehhooroee...oooo sensorr_enaor (self.
wtt....ww......o............ tewacceeetetrteeeeeeeeeeeeeeeees (self.
oblf.obs_buffi self salff.pttentiaaa(1))
self.prev.pctontiaasss... self.up_wec(:), self.
head ng_ ng_vac[:] compute compote_ humanoo d_tpsdddottttoooeoeooooooooo
self.obs_buf, self.roat_statte
self.targets, petenpoten
inffiiny_ttrt_rot self.dof__eo se f.dof_vel, felf.dof_ folr.dof_
self::of_limita_lover self.dof_limite,
delf.dof_
self. vet_vee__ensortteeaao self.actions self.dt, aaf.contart_fooceeceaa self. megulas_velocity.o halffbbsia_vvent self.basia bell._assl_vec"))
daf compute_ humanoid abservations buf, root_atates, targets, potentials, inv_stast_rot,,
dof_poS, dof dof_vel,
dof_[orce, dof_Himite_lower dof_ dof_vel_scale
censar_force_ttrrpisse actions, dt, contast_fosce_aant
angular_valoo wwlcivay........oom.. ootooccytylllly_ooaeeeep
basis_vecO, basie_vecl);
(Tenaor, Tensor, Tensor, float, Tenaor, Tensor, flost, float, float, Tenaor, Tensor] => Tuple Tensor, Tensor, Tensor, Teaser, Tensoz]
type:
Tengor,
Tenso", Tengor,
corso_positie position root_atates 0:3]
torao_rotation root_atateal 3:7]
velocity coot_statesl;, 7:10] ang_velocity root_states[;; 10:13]
to_terget targets torso_position
to_target(;, 2] 0
prev_potentiall_nne iotentialls.conee potentials toach.nasmlto.targee. p-2, dim--1) 1) /
torso_quat, up_proj, heading_proj, up_vec, heading vec compote_heddig_end
torso_rotation, inv_start_rot, to_target, basis_vecl, basis_vecl,
vel_lee, angvel_loc, roll, pitch, yaw, rigle_to_tarpe compute_rot/
torso_quat, velocity, ang_velocity, Largets, Lorso_posi.ion)
roll cormalize_onglaleee (roll) (=squeeze(-1)
yaw - aormalize (yaw). unsqueeze(-1)
angle_to_target aozmlliz (angle_to ttrggeetttttttttl.ccoo wtw.:taee.ttuuuueeeeeeeeee .ussquezze (-1)
dof_pos_scalad = unscale unscale poot dof_l'mits_lomee loker,
obs torch. cat (torso_posiiiontl vi.view(-i, 1), vol_ aoc anguior..vlloccty_ccal. yaw, roll, angla_to_target,
heading ursqueeezz
(-1),
dof pof_pos_scaled, daf_ve' ses_vee__lal,, def_force contact_ sottaaat__foo_ssall__aa
aanoo__orro_torggss..veer 121 Cortac=_foroo_de actions), dim
-)
return abs, potentials, prov_pctentiala_nee ve_vec, heading_vec
22

---

Published as a conference paper at ICLR 2024
EUREKA Reward History. Given that LLMs have limited context, also trim the EUREKA dialogue such that only the last reward and its reward reflection (in addition to initial system prompt) kept in the context for the generation of the next reward. In other word, the reward improvement is Markovian. This is standard in gradient-free optimization, and we find this simplification to work well in practice.
EUREKA Reward Evaluation. All intermediate EUREKA reward functions are evaluated using PPO run with the default task parameters. The final EUREKA reward, like all other baseline reward functions, are evaluated using 5 PPO runs with the average performance on the task fitness function F as the reward performance.
Human Normalized Score Reporting. Given that there are several significant outliers in human normalized score when EUREKA is substantially better both Human and Sparse on task, when reporting the average normalized improvement in our abstract, we adjust the score so that the normalized score must lie between [0, 3] per task before computing the average over all 29 tasks.
D.1 PEN SPINNING TASKS
The pen spinning environment builds on the original Shadow Hand environment in Isaac Gym without any changes to the physics parameters; the cube object in the original task is swapped out with the pen object, which is an unused asset in the Isaac Gym repository. In both the pre-training and fine-tuming stages, the task is to reorient a pen to a target. 3D pose, which provided as a goal to the policy. When the current target pose is deemed achieved (i.e., pose difference lower than certain threshold), then a new target pose is provided. In the pre-training stage, a random pose from SO(3) 3D orientation group is sampled uniformly; this mechanism is identical to the original benchmark cube reorientation task. During the fine-tuning stage, the target poses are a predetermined sequence of waypoints that specify the pen spinning patterns to be achieved. When the policy reaches the current waypoint, the target pose will switch to the next waypoint in the sequence. If the policy can consecutively reach all the waypoints, then it has accomplished relay continuously, allowing the policy to spin the pen for as many cycles as the policy is capable of until the episode length reached.
cycle of the pen spinning pattern. The waypoints will
D.2 EUREKA FROM HUMAN INITIALIZATION
In our human initialization experiments, we use Eureka to improve human-written reward functions. This can be done by modifying the first EUREKA iteration to the human reward in place of the LLM-generated one, thereby "assuming" that Eureka's 's first proposed reward is the human reward. To complete this iteration, we use the human reward in IsaacGym, compute feedback, and query the LLM to generate new reward functions based on the human reward and the reward reflections. Future iterations are identical to the default EUREKA setting.
To provide the human reward in the first iteration, we refactor the code slightly to be consistent with the EUREKA reward format, which exposes the individual reward components in dictionary. Furthermore, as human reward functions are often written in less interpretable fashion than EUREKA rewards (see App. G.5 for an example), we also strip away excess variables and parameters besides those needed for the actual reward computation.
D.3 EUREKA FROM HUMAN FEEDBACK
In our human reward reflection experiment, we investigate whether humans can provide reward reflection for desideratum such as "running with natural gait" that may be difficult to express via task fitness function. We repeat the EURFKA procedure with the following modifications: (1) We only sample reward per iteration, and (2) human textual input will replace the automatically constructed reward reflection as in the main experiment. To ensure that the human textual input do not require domain expertise, have used feedback that is as colloquial as possible; the full conversation is shown in App.
After the EUREKA-HH agent is trained, we have asked 20 unfamiliar users to indicate their preferences between two videos shown in random order, one depicting the EUREKA-HE Humanoid agent and the other depicting the original best EUREKA agent. These 20 users other graduate and
23

---

Published as a conference paper at ICLR 2024
Table 4: EUREKA Mujoco Humanoid environment. As Isaac Gym Humanoid, the task fitness function the Humanoid's forward velocity.
Task
EUREKA
Human
Mujoco Humanoid
7.68
5.92
undergraduate students that have wide range of familiarity in reinforcement learning, but are not involved with this research.
D.4 COMPUTATION RESOURCES
All our experiments took place on a single A100 GPU station. Each individual Eureka run took less than day of wall clock time our GPU station. Given that Eureka evaluates most 16 reward functions at a time, and each RL run on Isaac Gym takes at most 8GB GPU memory, Eureka can be on 4 V100 GPUs, which is readily accessible on academic compute budget.
E EUREKA ON MUJOCO ENVIRONMENTS
To fully test the generality of EUREKA with regard to code syntax and physics simulation, we also evaluate EUREKA the OpenAl Gym Humanoid environment (Brockman al.) 2016) implemented using Mujoco (Todooov et al. 2012). We make no changes to the prompts in App. except the formatting tip that instructs the LLM use numpy array instead of pytorch tensor when performing matrix operations in the generated program. The observation code for EUREKA context is displayed
in Example below. As shown, compared the observation code for the Isaac Gym Humanoid task (Example in App.D), this variant conveys the observation information in very different manner: commented block reveals the state and action spaces, and the variables themselves have to be indexed from monolithic observation array.
The comparison against the official human- reward function is in Tab. Despite vastly different observation space and code syntax, EUREKA remains effective and generates reward functions that outperform the official human reward function.
Example Mujoco Humanoid environment observation code given to EUREKA.
class IlueenoidEnv jocoEnn, utils. Btils.BaPickle) BePicklel:
en* *"*gest of envirenment definition onitted.
##4 Observation Space
Ooservations consiat positional values of different body parta Humanoid, followed by the velecities of those individual parts (their derivatives) with al: the positions
velecities.
Byy default, the observation la 'ndazray" with ahape 11375,) where the elements correspond to the follewing:
Observation
Min
Fax
Name (in corcesponding XML file) | Coins
z-coordinate of the torso (centre)
I -Inn Inf coot
Iree [
position (m)
x-orientation of the terso (centre)
-Inf
root
free
angle (rad)
2
ocientation of the terso (centre)
-Inf inf
root
free
angla (rad)
z-orientation of terso (centre) -Inf
Inf
root
free
argla (rad)
24

---

Published as a conference paper at ICLR 2024
k-orientation of tcrso (centre)
-Inf Inf root
free
I
angle trad)
angle of the abdomen (in lower_weist) -inf inf abdomen_s
hinge I
angla (rad)
6
yrangle of the aboomen (in Lower_Malse
-Inn
y
hänge
angle Irad]
x angle of the abdomen (in pelvia)
-Inf Inf abdonen_x
hinge
anglo (rad)
x-coardinate angle between palvis and right hip (in right_shight
| -Inf Inf right_hig_x
hinge
anglo trad)
2-coordinate of angle becween pelvis and right hip (in right_chight Inf Inf sight_hig_e
hinge
angle (rad)
19
y=coordinate of angle between polvis and right hip (in right_thight
-Inf Ter right_hip_y
hinge
angle (rad)
11
angle between right hip
right shin lin kigh:_knee)
-InE Inf kight_kkee
binge
angle !zad)
12
x-coordinate of angle between polvis and left hip (in left_thigh)
I -Inf Inf laft_hip_X
hinge I
angle !rad)
13
z coordinate of angle between pelvis and left hip (in left_thigh)
-ine inf lest_hip__
hinge
angle (zad)
14
y-<oordinate of angle between pelvis and left hip (in left_thigh)
-Inf inf lest_hip_y
hinge
angle (rad]
15
angle between left hip and Left shin le[l_knee)
-nn Inf lest_knee
binge |
anglo trad)
18
coordinate-1 (multi mulli--xiis] angle between torse and right (in right_upper__sng -inf inf right_shoulderi
hinge
angla (rad)
17
coordinate-2 (multi-axisl angle between torse and right azr: (in sight_upper_and Inf right_ shouldee2
hinge
angle (rad]
angla between right upper arm right_lower. lower_oerrr aarm
-Inf inf riiht_elbew
hinge
angla (rad)
coordinate-1 (multi-axis) angle between torse and left (In left_upper -Inf Inf sect__sholdee1
hinge
|
angle (zad)
20
coordinate-2 (multi-axis) angle between torse and left arm (in left_upper_an arm) -Inff
left_shoulder?
hinge
angle (rad)
21
angla between left upper arm and left__lowr_amm
-Inf Inf lect__ebow
hinge
angle (rad)
22
x-coordinate velocity of torso (centre) | -Inf Inf root
froe
velocity (m/a)
23
y-coordinate velocity of the
(centre)
| -Tnf Tnf | root
free
velocity (m/s)
24
z-coordinate velocity of the torzo (centre) -Ing
root
free
velocity (m/a)
25
cooordiinaee angular velocity the torso Icentre)
Inf Inf zoot
Free
anglular velecity (rad/s)
28 y coordinate angular velocity -InE inf
Lozse Icentre)
root
free
angluler velecity (rad/s)
27
7-coordinate angular vel ty of the torse Icentre)
Inf Inf zoot
Izee
angluler velecity (rad/s)
28
zocorddiaate of angular velcoity of the abdomen Lin lower_walat)
-Inf
abdonen_s
hinge
angluler velecity (rad/s)
29
y-coardinate engular velcnity the abdomen Hin lower_waist)
-In Inf abdoree_yy
hinge
angluler velecity (rad/s)
30
x-coordinate of angular velceity of the abdomen lin pelvis)
Inf | Inf abdonen_x
binge
|
asnglular velocity (rad/s)
25

---

Published as a conference paper at ICLR 2024
x-coordinate angular velocity of angle between pelvis right hip right_thight
-Inf Inf tight_hip_x
hinge
|
anglular velccity (rad/e)
32 coordinate of the angular velocity of the angle between pelvis and right hip (in Sigh=_thight
-ing inf right_hhp__
hinge
anglular velecity (rad/s)
33 y-coordinate of the angular veloctty of the angle between pelvis and right hip (In tiigt_thigh)
-InÉ inf right_hip_y
hinge I
anginlar velecity (rad/s]
34
angulaz velocity of the angle between right hip and the right shin tin tight_knee) hinge |
-Inf Inf right_knee
angluler velecity (rad/s)
35 x-coordinate the angular velocity of the angle betwean pelvia and left hip (in left_shigh)
I -Inf Inf lest_hip_X
hinge |
angluler velecity (rad/s)
36 2-coordinate of the angular velocity of the angle between pelvis and left hip (in left_shigh)
Inf Inf lest_hip_z
hinge
angluler velecity (rad/s)
37 y-coordinate of the angular velacity of the angle betwean pelvis and laft hip (in left_shigh)
-Tnf Ter laft_hip_y
hinge
anglular velecity (rad/e)
38
angular velocity of angle between
hip keet_knee
left shin (in leet_knest hinge
IInn Inf
angluler velecity (rad/s)
39 coordinate-l (multi-axis) of the enguler velesity of the angle between torso and hinge
right arm (In right_ upper___p__ -Inf I Inf right shot_llodddrr anglular velccity (rad/s)
40 coordinate 2 (malti exist of the anguler velcoity of the angle between toreo and hinge
right erm right__pper__rm) -inf I ind right_sheuider2 angluler velecity (rad/s)
41 angular velocity of the angle between right uppez arm and right_lower -Inf inf right_placw
hinge
angiular velecity (rad/s) I
42 coordinate (multi axist the angular velcoity left arm. (in left_upper_aral angluler velccity (rad/s)
angle Lozeo and hinge
-Inf | Inf shot__ao
43 coordinate-2 mmllti-axis) of the engulez veleeity of the angle between tozeo and hinge
left arm. (in left_upper_ass arm -inf I ing left_shoulder? anglular velecity (rad/s)
44 I angular velocitty of the angle between left upper arm and left_lowes_ Inf
left_elbow
hinge
angluler velecity (rad/s)
Additionally, efter all the positionsl and velocity based values in the table, the observation contains (in ardez):
scinerti Xasa and Inertla (this is an intermediate result of transition). It has shape 14x10 (vnbody x 10x) and hence adds enother 140 elements in state space.
single rig'd body relative
center
veveli> Center of mass based velacity. it has shape 14 (xnbody 6x) and hence adds another elements in the state space
=oftt_actuator:= Censtraint force genersted the actuator force, This has shape (23,1) (nv 11* and hence adds another 23 alements to the state spane. =efro_ext:= This la
center of 11 6 (onbody x 54) and hence adds to another 34 elements in the stace space. where <nbody- stands nimber of degrees of frendom las dim/qual)
external force on the body. =t shape
number bodies
robot enve stands
The (x,7,2) coordinates translational DOFs while the crientacions rotational noFs expreased as quaternions.
looe that the obs argument passed to the zeward function is concatenation of cbs across all enviroaments,
will have shape (n, 3767-
def _get_obs (selfi:
position = aelf.data. dal..ddaa...ss...... copy velocity self.data.gvel.ffdd.
com_inertiz mol.f.dtt.cinartt.laa..coom
com_velocity self ..t.
sctuatos_forces self data. www.__actttoo...lee... cilernal_ccottettffocees self.cata.cfre_ett flst.
if self htw.suud__uurrr___pttttooooeeeee position position(2:]
return ap. sp.oncatenatee
position, velccity,
26

---

Published as a conference paper at ICLR 2024
com inertia, com con_velocity, pecuuacor_foroes, externel__cotaee__eeeees
F ADDITIONAL RESULTS
Reinforcement Learning Training Curves. In Fig. 10 we present the aggregate RL training curves on the Dexterity benchmark. As shown, EUREKA reward functions enjoy improved sample efficiency compared to all baseline rewards. We do notice that EUREKA reward functions, average, exhibits less smooth trend than the Human reward functions. We hypothesize that this due to the fact that the underlying RL algorithm is not tuned for the EUREKA rewards; instead, they are tuned for the Human reward functions.
Dexterity RL Tra Training Curves
123
euroca
SuccossoRis Rats
0.3
022
01
0.0
1000
2000
2500
3000
Beeations
Figure 10: EUREKA reward functions enjoy improved sample efficiency compared to various baseline reward functions on aggregate over 20 Dexterity tasks.
Additional Evaluation Metrics. In Fig. 11 we present holistic evaluation metrics, such mean, median, and interquantile mean, as suggested by Agarwal al. (202T). All metrics and associated 95% confidence intervals are computed over the set of 100 RL runs from RL runs for each method's final reward function on all 20 Dexterity tasks. As shown, on all evaluation metrics, EUREKA is consistently effective and outperforms all baselines.
Furthermore, using the same set runs, we also compute the probability that EUREKA outperforms the baselines with 95% confidence intervals. The results shown in Fig. 12 As shown, our experi- mental results provide strong evidence that EUREKA'S 's improvement over baselines are statistically significant.
27

---

Published as a conference paper at ICLR 2024
Median
IQM
Mean
Sparse Human
L2R
Eureka
0.0
0.2
0.4
0.6
0.2
0.4
0.6
0.15
0.30
0.45
Success Rate
Figure 11: EUREKA reward functions enjoy improved sample efficiency compared to various baseline reward functions on aggregate over Dexterity tasks.
P(Eureka Y)
Sparse
Algorithm Y
Human
L2R
0.6
0.7
0.8
0.9
Figure 12: EUREKA reward functions enjoy improved sample efficiency compared to various baseline reward functions on aggregate over 20 Dexterity tasks.
Dexterity Performance Breakdown. We present the raw success rates of EUREKA, L2R, Human,
and Sparse Fig. 13
Succest bale
SUcc404 hate
0.00
0.2
Sacco812 8410
0/
1.25
0.9
342
037
02
0.17
1:35
Succest hate
0/44
C41
EIM
Figure 13: EUREKA rewards are less correlated with human rewards when the tasks are more high-dimensional and less common in the reinforcement learning literature.
Reward Reflection Ablations. In Fig. we provide a detailed per-task breakdown on the impact of removing reward reflection in the EUREKA A feedback. In this ablation, we are interested in the average human normalized score over independent EUREKA restarts because the average more informative than the max (the metric used in all other experiments) in revealing LLM behavior change
28

---

Published as a conference paper at ICLR 2024
aggregate. As shown, removing reward reflection generally has negative impact the reward performance. The deterioration is more pronounced for high sigh-dimnnsioal tasks, demonstrating that reward reflection indeed can provide targeted reward editing that is more instrumental for difficult tasks that require many state components to interact in the reward functions.
Eurcka Eureka Roward Reflection Human Normalized Score
30.33 6.37
namakkdd Soom
1.20
141
1,5
.92
Figure 14: EUREKA rewards are less correlated with human rewards when the tasks are more nigh-dimensional and less common in the reinforcement learning literature.
compare the performance of EUREKA with GPT- 4 (the original one reported in the paper) and EUREKA with GPT-3.5; specifically, we use gpt-3 3. to-tuurro-666 1eoo-16k-06111 in the OpenAI API. While the absolute performance goes down, we see that EUREKA (GPT-3.5) still performs comparably and exceeds human-engineee rewards on the dexterous manipulation tasks. These results suggest that the EUREKA principles are general and can be also applied to less performant base coding L.Mss.
EUREKA with GPT-3.5. In Fig. [5]
(GPT-3.5]
12:05
2:38
2.07
2/78
3.00
3.00
2.0
85
0,3
Qaskopter
deggroen
Figure 15: Using GPT3.5 observes performance degradation in EUREKA but still remains comparable to GPT- majority of the tasks.
Reward Correlation Experiments. To provide a more bird-eye view comparison against human rewards, we assess the novelty of EUREKA rewards. Given that programs that syntactically differ may functionally identical, we propose to evaluate the Pearson correlation between EUREKA and buman rewards all the Isaac task. These tasks are ideal for this test because many of them have been widely used RL research, even if the IsaacGym implementation may not have been in GPT-4 training, it is possible that EUREKA produces rewards that are merely cosmetically different. To do this, for a given policy training run using a EUREKA reward, we gather all training transitions and compute their respectively EUREKA and human reward values, which then be used to compute their correlation. Then, we plot the correlation against the human normalized score on scatter-plot. The resulting scatter-plot is displayed Fig. In Fig. [6, also provide the average correlation per task. As shown, as the tasks become more high-dimensional and harder to solve, the correlations exhibit downward trend. This validates our hypothesis that the harder the task is, the less optimal the human rewards are, and consequently more room for EUREKA to generate truly novel and different rewards.
29

---

Published as a conference paper at ICLR 2024
Average Eureka vs. Hurnan Rewerds Carreletion by Iseac Task
0.95
0.95
Correlation
0.47
0.0
Figure EUREKA rewards are correlated with human rewards when tasks are more high-dimensional and less common in the reinforcement learning literature.
G EUREKA REWARD EXAMPLES
In this section, we provide several unmodified EUREKA reward examples from various experiments we detailed in the main paper.
G.1 REWARD REFLECTION EXAMPLES.
We illustrate EUREKA reward reflection step by displaying the sequence of previous EUREKA reward, EUREKA reward reflection, and then the subsequent EUREKA reward.
Example I: EUREKA Reward Reflection on ShadowHand. Iteration
Score: 9.29, Iteration 2 Score: 10.43
Iteration 1: Buraks Output:
"pyyther
import torch
from typing import Tuple, Dict
itcrch. jit.
compute_rowarddcooee
corch, goal_rot: torch. Tensor) -> Tuple(sorch, Tensor, (atr, Arrch,Tensor]]]
rot_diff abrch (trrh rotation_reward torch.exp(-20 rot_d'
gosl_rot, dim=i) 11 /
Scaling rotation_temp 20.0
rocation reward
total_reward rotation_reward
reward_ cooerdd_ommooentsseo "totation rolation_reward
return total rewaa_, coward_compoeents
This reward function lon computes the difference between the currens object rotation and the rotalion_reward
Large:
absolate difference, scaled a temperature parameter (rotation_temp), The totel reward is the cotation_rewasee
It
*** #*****444444444444HHH
Iteration 1:
Reward Reflection:
Me trained a RL policy using the provided reward tunotion code and tracked the values of the individual components in reward function as
global pelicy metrics such success razes and episede lengths after every 300 epochs and the maximun, mean, minimun values encountared:
rotation soraion__raa:: ("0.03', 13.31', 30.30", r0.321, 10.34", 10.34', 10.34', r0.34r, 10.34',
0.32"], Max: 0.36, Mean: J.32, Min: 0.03
success_rate: 1'0.00', 10.83', '1.85', 12.891, <3.72', '4.94', '5.99", 6..90', (7,711, 13.83'
1, Max: 5.29, Mean: 4.81, Min: 0.00
seisode_lengther ['7.07', '364.30', '378.22', 1378.06', 116.11'' (315.13', (422.64, (412.50' 1384.10', 143,,24',, Max: 482.35,
Xeen: 396.02, Min: 7.37
Please carefully analyze the policy feecback and provide
improved reward function that can better selve the task. Some helpful tips for analyzing the policy feedbeck: rates always near
(1) the function
then you rewrite the entire reward
(2) values
near identicsi throughout, means RI is not able to optimiza this component as it is written. You consider (a) Changing its scale or value its temperature parameter
cercein reward component
30

---

Published as a conference paper at ICLR 2024
(b) se-writing reward component
(c) Biscording the reward compoment
(3) some reward component nagnitude La aigniticantly larger, then you must re re-acale value proper zange
Please analyze each existing revard component in the suggasted
above first, and then write the reward function code. The output of the reward tunction fon ahould conalst of two
items:
(1) the total reward,
(2) à dietionary of each individual reward component.
The code output should be formatted python code string: "1ypython "...
for writing the (1) You may Lind it helpful to nomaalize the reward to a fixed range by applying cronsformotions
halpful
function code:
terch.exp to tha overall reward (2) Tr choose tranaforn seward component, then must also intuoduce zemperature parameter inside the fornatioo functien; this pazameter must named veriable the zexard function and it
components
be input variable. Each transfermed reward component should have its own temparsture variable
Make the type variable should net be specifled as torch.Tensur (4) Host importently,
correctly apeci speci float
only atsributes provided environment class definition inamely, variables that have prefix self.), Under no circumsience can you introduce new input variables.
zeward
input
Iteration 2: Bureka, Output:
Based on the enalysis the policy feecback, the rotaticn_reward component has minimum, stabla. The suzcess,
maximum, and mean values within fairly close range hag increased gradually, indicating that the agent ia learning to aome degree. Rowever, episode_lengtie
quite some variance, which not consistently perform well during training.
indicate that agent
Since the reward
mostly adequate, I would recommend introducing another compenent, such as a penalty for large angular velocities, to reinforce stable spinning behavior. Here nigular__elooctty_penllll
funation that incorporates
"pyyhhen
import torch
frem typing import Tuple, Dict
Storch. jit.script
def compute_reward ject_rooeeertttteeeeeeoee forch.Tenso: goai_rot; Lorch.Tensor, ob ject_aneev Lozch. Tensor) Tuple(torch. Teaser, [stt, Tensoolens Retation reverd
rot_diff tbrch,abs (torch, sun (objact_ gos goal_rot, dim=l] 11 rotation_reward torch. (-20 rot_diff) rotation__tmmp 20.0
Angular velocity penalty
angvel_nezm torch.normicb bzzch.namiccbeett_nnge.. dimmi)
angwel_threshold 5.0
anguel_penalts__ee temp 13.0
angular_.ealoitty_eenaa
sorch.
angvel_threshold, torch.exp/-
angval argval_penalty_terg (angvel _aamm
torch,zeros__ Saaro__lkkee langval_norm)
total_cewerd = rotation_reward vegularr
reward_ components =
"rotation contatio__ewwadff reward"; cotatioo_rewartt
Vangula__veoocity_penna engules_welocity.penddlg
return tetal_reward, rewerd__omponents
The new zeward function applies penally for large angular velocities (lazger than the stgvei_threshold)
temperature parameter Langve__penally_eeny to scele the penally. The total reward now combines the rotation_reward with the axguler_velocityypenn
Example 2: EUREKA Reward Reflection on Humanoid. Iteration Score: 3.79, Iteration 3 Score: 4.37
Iteration. Bureks. Output:
sased on the provided feedback, it seens like the Btrack_reword componens is not contributing learning process as
are Therefore,
31

---

Published as a conference paper at ICLR 2024
=rack_ companent. will also add a new component to the reward function to encourage the agent maintain upright poature.
new
emphasizes 'velocity, reward" reward"
selow is the updated reward functicn:
""python
import torch
from typing import Tuple, Diet
3terch. Stcrch. siript def (orch.Tensor]]
torch.iensor, dt: float) +> Tuplejtorch. (ict(atr,
Calculate the forward velooity forward_velacte reot_statea[ ?]
Reward for forward velocity relority_eemerature 1.0
this temperature parameter adjust how much
weight the forward velceity gets
belooettyewaard torch.exp "'velotdd velloeity_temmeratus
Calcuiate height torso_height Ioot_atates(;,
humancid torso
Rexard for meintaining upright posture
ppature_tempastunn 1.0 $ Tune this temperature parametar to adjuss the weight of the posture reward
posture_reward =
posure_lenjeeztuure
Combine rewards
total_sewerd velocity_reward posture_reward
reward_componeed iyvelcalty_revard relard,y_reward, "postura_reward" posture_reward]
retura tetal_reward, reward_components
have removed the 'track_ reward' component and added a new ecmponent, 'posture_rewwrr to encourage humaroid maintair. upright posture. The posture roward is computed the expenential of the humanold's torao height divided by a temperature paraneter. Adjust tune the model's learning process.
cemperature parameters
components
Iteration 2: Reward Reflection:
trai polioy using provided veward function
values the individual components in the reward function as well as global policy metrics such as evezy epochs and meximun, mean,
success rates and episode lengthe values encountered;
vallocity_reaard ('1,05', '6.33', "30.09*, '66. 72", 1101.08r, 1133.44', 1148.37', 1175.901,
116.61, Min: 1.05 castura_rewarde (13.26', '3.31', 13.28', "3.29', *3.27', 13.25', 13.28', 13.28', 33.281, +3.27 "I, 3,33, Mean:
188.6a', '204.21<),
235.50,
Min: 3.02
consecutive_ succeeesess [ 0.05", '1.18', 388, '2.31', 12.95', 13.20', 13.45', 13.52', 13.627, 13.66 ' 13.67'1, Max: 3.79,
2.95, Min:
lengthat 94", 185.50', 1804.58', '123.72', 1128.38', 1130.28', (135.181, (13
1134.81', 1136.61"), Max: 142.11, Mean: 120.44, Mins 25.89
Please carefully analyze the policy feadback and provide
improved reward function that can better salve the task. Some helpful tips for analyzing the policy feedback:
always near seco, then you mus: rewrile
zeward
function
(2) values
cersain reward component are near means RI is not able to optimiza this component sa it is written, You consider
throughout,
(a) Changing its scale oz the value of its temperature parameter.
(b) Re writing the reward component
(c) biacarding the reward comsonent.
(3) some
cemponent nagnitude significantly larger, then you value to proper range
Please analyze each existing
auggasted mannor first, writa the reward runction code, The output of the reward runctiion ahould constat of two items:
component
(1) the total reward,
(2) a dictionary of each individual reward component.
output should
python code string: "1ylpttonn ... Y.
Same halpful. tipa for writing the
code:
(1) You find it helpful to normalized the reward to fixed
app-ying
acensformstions like terca.exp to the overell reward or its components
(2) If you choose transform. zeward component, then
also introduce
cenepezature paranete: inslde the tzansformetion function; this paramele: must be named
32

---

Published as a conference paper at ICLR 2024
variable reward
not input varisble. transformed
reward component ahould have its own terparsture variable
(3) Make sure the type of each Input variable correctly apecifiec; a float Input be specifled (4) most importantly, the reward code input varisbles must contain only attributes of the provided anvironment
verisble should
torch. Tensor
torch.Tensor
definition Inanely, variables that prefix self.), Under no circumttance can you introduce new Input varisbles.
Iteration 3:
Bureke. Output:
Based on the provided feesback, it seens that the (posture_reware cemponent has little impact laarning proceas, its values will modify the reward fuaction to better balance the "velocity_ reeaardd_eewa"""e and posture_rawer componerts, and also minimize the energy consumption of its action (1.e., "action_energy".
identical throughout the training.
new component to encourage the agent
Here the updated reward Sunczion:
"pyyhho
import torch
typing import Tuple,
Sterch. scr.tt
def corpute_reward
torch. actions: torch.lensor, float) Tuple/torch Tensor, Dict (atr, torch. (anchrTansor]] velooity
Calculate rcot_states[ 7]
veroird_
Reward Ior Sorward velocily velocity_temperauure = 5.0 increased the value from 1.0 to 5.0 to balance the impact of the veleaity reward.
vel oolayy = torch.exp ("orward_eellaa
ve vellorittyttmmaaatur
Calcuiate the height of the humancid torso torso height root_states(;,
Reward for meintaining upright pasture peature_temparae
13.0 Inereased the value from 1.0 to 10.0 balance the impact of the posture reward.
posture_reward torch.exp (torso_heihh postuze_tempezatuuee
minimizing energy consumption actions action_energy torch.norm(actionnn
p-2, dim--1)
Seargy_temperateee 5.2
anergy_reward reward -torch.explactton (actooppttononneeeeeeeee on_enery / temprrteem))))trre to mininise energy consumption.
Suboract this componert
Cembine rewards
total_reward velocity_reward postura_rewarr energy_reward
reward'; velocity_reward, 'postara_reward" postare_reward, 'energy_rewee reward's enerqy_reward:
goward_componente =
return reward, rewprd_components
increased the temperature
for both "velccity_ reward" and posturrerrewade components to balarce their ispact on the learning proceas. Add'tions added new component, "eaerqy_rewazd""
encouzage the agent minimize energy consumption in its actions. Experiment with different temperature parameter values i5 necessery to
the policy furcher.
G.2 NEGATIVELY CORRELATED EUREKA REWARD EXAMPLES
We display several negatively correlated but superior rewards that EUREKA discovers our experi-
ments.
Example Task: ShadowHand, Human Normalized Score: 1.45, Correlation
jttrch..it..crrpp
def compute_reward jecttct__ooo forch-Tensos goa__rot: torch. torch.Tensor, fingectip_pos torch.
Tensor, object_pos: torch. Tensort Tuple[terch.eene Dict (str, torch. Sensor]]: Compute the quaternion distance between the object's current orientation and the geal oriensstion
disi torch. sun [cbject_root goal_rot), dim--1)
g_dist = torch. mirch.minnggdieee di.dis, 1 g_dist) Make mure the dist in in the range [0, 1]
33

---

Published as a conference paper at ICLR 2024
Normalize quaternion distance using temperature pasemeter temp_rot a 0.5
rot_reward torch.exp tordh.exp--tepp_re e_dist)
Compute the distance between the fingertips and the object cencer Eingertips_ohjett_.di
north.no
cbjest__post; None], dim=-1)
Apply threshold for distance
distance_tho taaance_treehhed 0.1 close_enough
distance_threshone to (corch.
Normalize the distance between fingertips and object center using an updated temperature
temp_dist = 10.0
ststance_rewars torch.mean (terch.exp (etprchhhxppptmtpmepeee (-temp____dtt fingertipo__bjjet_dd.. close_enough) dim=-1)
Apply penalty the agent is close enough to the object distance_penaltyy 0.5 (1 (clone__ dim=-1))
Combine the seward componeats total_rewerd rot_reward distance_rewart distance_penalty
Store the reward cemponente in dictionazy rexard_companents, =
"rot_ reward"; rot_reward,
"distance_reward": distance_reward,
"distance_penally"" dissance_penallyy "cotal_rewerd": total_reward,
total @otal_reward, reward__amponents
Example 2: Task: FrankaCabinet, Human Normalized Score: 11.98, Correlation: 00.3
Storch. 2torch.jit.ac att..cippt
compute_rewarde saaika_graap_poee_ Lorch. Denaor, drawer_grasp_pox torch, Tenaor,
cabinet_dof__eat torch.Tensee Tensor,
franka Branka_l"inger_paa torch, torch.Tenaor, tranka_rtinger_pos sorch. Tensor] Tuple (torctt Tensor,, Dict[ste, torch, [ensor]];
calculate the distance between Franka greaping pasition and the cabinet graaping poaition
gzaep_dletang
torch..orm(.aankkeeee (raaaaaaaaaaiieeassppp..... drawer_grasp_poz
calcuiate the diatances between (ranke_Liinee__pp [ranka_ and drewer_graap__
diinger_diatance torot.nom//rrankaafff... rfinger_dlstanen
drawer_grasp_pe dims-1) drawer_graap__oo, dim=-1)
opening Calculate cabinet_dor_pos(1: 3]
drawer_opening
for transferming the reward components gwwsp_distante. wtw.ddtaanc__ccclaaae acaling torch,tensor (20.0)
Define temperature
hancle_graapi hespaplng___appppe weg_ttemerattue = Lensor (20.0)
transform the reword components
wzasp_ddatance_reeea 1.0 (1.3 + graep_dietanceeeeaallng grasp_distance
torch.exp htw.pp/thh.....ggg..o.....eeseea (lfinger_distance
Slinger_distanne wvawer_openingg_ee
txrca.exp boros.eppddeeee.oonnnnneeeeeeee
drawer_opening)
Compute reward
reward = www.aanoe_oowarttt..teee
handle_graopinggrewor
drawer_ drawer_opening_roonr
Create dictionary of individual reward
reward goarrd_cmpooente =
distogei
paaallee_gapiin__eweeed gaadle__raeping_reee
"ddower_oponing_ewwrrdd" drawer_opening_ren
reward, seward_oomponent
34

---

Published as a conference paper at ICLR 2024
G.3 EUREKA FROM HUMAN INITIALIZATION EXAMPLES
We display several examples of a single step in the EUREKA from Human Initialization setting. In these examples, the first reward (Iteration 0) is the original human-written task reward, and Iteration the best reward after one step of EUREKA improvement.
Example 1: EUREKA Human Initialization on Kettle. Human Success Rate: 0.11, EUREKA Success Rate: 0.91
Iteration
Initialization:
">python
import torch
jit.
compute_rent
heetlle_handl__pp
sackkl_hhandle
kelLls_epout__po
right_hand__ff_co right_haaddmf__oe
right_hand
right_hand__hhpps left_hane__f__pot
left_ left_hendLif_pos, hent left_hend
right_hand_ toreh..eemm/ettlllellandddd pig_t__and_____ps, p=ž, dim--1)
fiitt_han__ffgger____ Finger_ddstt (torch. (oorah.nom//kettllhhmaa. - right_hand fand_ff_pos, p=2, dims-1)
tight_hand p=2, dim-
torch.nomm/ee::eehhnnnn
-igitt_aai_____ps, p=2, dims-1)
left_hand finger__inee_iiii dist [oorch..ommmkellleeeladddddddddd
right_hand_thhe p=2, dim- 1))
lef=_hand ff p=2, dim--1)
torch.nerm (wacket_handll__pgee -eft__and_____pppa F=2, dim=-1)
+ ttorch.normbbucke. has___aandd_ppp
fii__hand_rf__oe p=2, deme-11
Left_haad_____ee p=2, dis-
+ torchh.ormmbuckke_haanee
- thfd p=2, dim--11)
right_hand st_reee fint__had__feeee______ _tstt jeft_hand_dist_ree feit_haad__ingee_ddee
ue_rew = torch, zeros_ lero_ (right werrighh_hnn__dde wwwhhhaann__eeeeee up_rew where (wtgghitggghhhiiiiineee
0.7,
torch.
where toreh haadd__haa____nnee_e__ee
0.7,
0.5 (orchh.orm(buakees handle_pos 'eettle_apout_poo.
=2, dim--1) 2, up_rew), up_rew]
reward up_rew right__and__d'st__ dawazd_componee - fup reward' up_rew,
ht___anndddatt_tre
htwthhhadd_d__teeeede_e serft_hand_diat_reeee jeft_hand_distt_ree
reward_oomponents
Iteration 1: Reward ReClection:
Me traf ned a RT. pol tay using the provided reward funation code and tracked the values of the function as global policy metrica such success rates and episede lengthe after every 300 epochs and the maximun, mean, minimun. values encountered:
individual components in
rewroddd:: ("0,00', "0.00', , 10.02', 1-0.01', 10.04', 1-0.02", 10.03', 1-0.011, r0.02', 10.04'
J. Max: 0.12, Mean: 3.03, Min: -0.06
1"0.93', '0.31', r0.401, 10.38', 10.36', '0.41' 10.43', r0.421, 00.33
r0. 42' Max: 0.96, Mean: 0.41, Min: 0.27
left
17', r0,87', *0.69', '0.89', 10.63', r0.811, 10.83', '1.02' r0.84'
1, Max: 3.57, Mean: 0.88, Kin: 0.39
cansecurive_aucceeeee
(00, r0.00", *0.00", *C.00', 10.00", '0.00', 10.00' *0.00" <0.cor
"0.00" Max: 0.01, Means 0.00, Min: 0.08
episode Lengttss 1'125.00', 1122.45', 1123.92', 1124.24' 124.22', '124.83', 1224.86',
124.42', 192..89', 132,, Max: 125.00, Mean: 123.61, Min: 111.56
Please carefully analyze the policy reecback and provide new, improved reward function that can better
Some helpful tips analyzing
pelicy
(1) If the success rates are always near zero, then you mus: rewrite the entire zeward
(2) If the values for certain reward component
near identical throughout, then this means BI is not able te optimize this component as it is written. You may consider
(a) Changing its scale the value of its temperature
(b) Re-writing the reward companent
(e) Discarding
zeward component
(3) If some reward component magnitude is significantly larger, then you must re-scole its a proper range
35

---

Published as a conference paper at ICLR 2024
analyze existing component suggested manner above first, and writa the reward function code. The output of the reward function should consiat of twe items:
(1) the total zeward,
(2) a dictionary of each individual reward compenent.
The code output should be formatted python code string: "1'python ".
Same halpful tipa writing the
code:
(1) You rind it helpful to normaliz the reward fixed range applying SzansCormation like Lorca.exp to the overall reward or its componente
(2) If you choose transform.
eiso introduce cempereture parameter inside the transformation functicn; this paramster must be a named funation
component, then
varisble the
he irput variable. Rach transformed reward component should have its own Lemperature variable
(3) Make sura the type
input varisble corractly specified; a input variable should not be specifled as torch.Tense Tensor
(4) Most inportantly, the reward code input variables must contain only att:lbutes of the provided environment class definition (nazely, veriables that have prefix self.), Under no circumstance car you introduce new input varistles.
34 #44T
Iteration. 2: Bureka Output:
Frem the feedback data you previded, it is evident that up_ceward has a rather small value and is always function. sirst, we can update the distance rewards by adjusting their acale and applying exponential tramstormation velues range beiween 0 and 1. We will slso slightly adjust the up_reward to get mere kettle
try modificstions improve the reward
temperature.
their
value
add some weights to reward components balance their contribution te the total reward.
rinally,
Hare the updated reward function:
Mpythhon import torch
Sterch. jit.script def compute_reward
gottllehhanda__pee
kettlle_ppou___e
Saant_hand_t__ppe
right_hand__f__os, right_hand_rr_eon right__and__i__ps
tight_bans_th.pos, left jeft_hand_ff_pos, left_ heft__annd_ffposs, left hf heft_hand_ 1f heft_haan_th_poe
dist_temp 10.
right_hand hand ft finger_dte (torch. norm/kktt satel__haal___ss___ night_handd pos, p=?, dim=-1)
becch.nommkketileeeeeeiitt torch.nommketttlehannndii - hight_hand_lf__ps,, p=2, dims-1)
right hait__hand____pss p=2, dim-
right_hand p=2, dim--1)
rightt_hadd pos, p=2, dima-1))
jeft_hand_finger.. dist ttoreh.nommmbceett....ll....
left_hand_ff__oo, p=2, dime-1) hirtt_hand______ F=2, dim=-1)
corch.nern (bucket_ htaccket_hondee_pat
left_hand_rt_ppg p=2, dim=-11 torch.nora (bucket haaddleet_haade_ppg left_hand p-2, dis-
torch.
left_hand__th_ps, p=2, dime-ll)
@iat__had_ddaae rew torch.exp (rr-hhaat-ddt_teemm right_ "indr_""ngrr_ddi jeft_handddiist_rew
torch.exp (-dist__oop left find___d___iige_ddttt
ug_rew Lorch, seros_ ceros_like (right hand (iiightt_aann_dss__eeee
rew - torch. where (tiigtt hand_haaddd_d_ddaeeaeieeiiaaaeee
0.7,
where (left_hand_ ftngg_aaadd_fge___dit_
/bw.h.n h handle_pos kectleeapout_ppm p=2,
din--1) 2, up_rew), (p_rew)
reward = up_rew
right_hanc_diat..om
feft_hand_dlat_ree
zeward components
"ap_reward'' up_rew,
siight_handddist_raee
sight_hand_diat_rroo
rewarrr weff_hand_dsst_reeee
return reward, compon_oomoeentts
Please test this updated funcsion and provide new policy feedback data for further analyais 15 needed.
36

---

Published as a conference paper at ICLR 2024
Example EUREKA Human Initialization CatchUnderarm, Human Success Rate: 0.33, EUREKA Success
Rate: 0.64
ILeration 1: Numan Initiallcation:
"Pyyther
import torch
tsaaogym.torch_ueel import quat_mul, gust_conjugate
storch. att.
compute reward!
pos, object_rot, posl_pos, gosl_rot,
goal_dist torch.norm(goal
object_pos p=2, dim=-1)
quat dift = qust_mul perfect_rrtt, quat cont_oonnonn jongat (gnal_oot) rrot)) -1.01:
2.0 torch.asin (Lorch, wtw.ccccc.catttooc.oooo..eeeese.
0:3), p=2, d_m=
dist_rea _dist reward torch. (xp.eeeeppp00.00 (-0.2x rewart_componetts
reist_rew 1ditt__e 50 rot_dist!)
"dist_reward"" dist_ "cot_reward"; rot_dist,
return reward, teward__omponents ...
Iteration Reward Reflection:
Me trained RI policy using the provided zeward funotion code and tracked the values of the
individual components in the reward function as well as global policy metrics such as and episcde lengths after every 300 epochs and the maximum, values encountered:
minimum.
dist_rewerd: ('0.41', '0.34', 10.34', 10.29', r0.26', r0.25', 10.231, 10.211, r0.211, r0.2111,
Max: 0.42, Kean: 0.26, Kin: 0.19
"ot_reward: [12,21', *2.21', 12.19', 12.22', 12,201, 12.21' 12.221 (2,211 12,201 12.20")
Max: 2.27, Mean: 2.21, Mia: 2.16
000000', r0.00', 10.00", 10.00', 10.04", r0.14', 10.25', 10.17', 10.14'
'0.03'1,
0.33, Mean: 0.08, Kin: 0.00
episoda_ episode_lengthh:
lengtha: 1"28.00', '59.83', '56.22", 151.10', 156.34', '57.48', 151.40', 664. 31, $6.47', 167.11',, Max: 75.00,
60.98,
28.00
Please carefully analyze the policy feadback and provide
improved reward function that can better selve the task. Sone helpful tips for analysing the policy feedback: (1) If the function
zates
always
then you rewrite the eatire seward
(2) the values for certain reward component
identiesl throughout, then this means RI la not able to optimize this component sa it is written, You may consider
(a) Changing its
of its tomparature parameter
(b) Re-writing the reward component
(c) Discarding the zeward comsonent
(3) If some reward
magnitude is significantly largez, then you must re-scele its
volue to a proper range
first, and then write the reward function code. The output of the reward function should conalst of two itema:
aralyze
existing reward component in auggaated
(1) the total reward,
(2) a dictionary of each individual reward component.
The code output should be formatted python code string: "1Upython 19
helpful
writing the
function
(1) You may find it helpful to no:nalize the zeward to a fixed range by applying cransformation
torch.exp to the overal reward or comporents
(2) If choose tranaforn seward comporent, then must also introduce zemperature paremeter inside the transformation functienr this paremeter must be named variable the reward function and it
be input variable. Each transformed reward component ahould have its own tempersture variable
(3) Make the type
correctly apecifiec;
Input
verisble should net be specified as terch.Tensor
(4) Most importantly, the reward code Input variables must contain only attributes of the provided enviroament class definition Inazely, veriables that have prefix self.), Under no circumstence can you introduce new input varisbles.
Iteration 2: Eureka Output:
on policy feedback, can observe success spisode lengths improving over time. Hewever, the values for 'rot_reward" identical throughout
37

---

Published as a conference paper at ICLR 2024
policy learning, which means
agent is not optimize cusrent rotation
reward component. We can charge the cacll//emperature paranster of the rotation reward and try to make it more senaltive to the changea rotation. We will keep the distance reward component it is because it te be eptimized during training.
updatad reward function for the given taak:
"pythor
import torch
from isaaogym,
torch issaogm..orchh.uti
Import quat_nul, qust_eon
Iterch. jit,
compute_reward)
oaject__pos: torch. Tensoz, object_rot: torch, Densor,
goal_pos: torch.Tenser, geal_ret: torch.Tensar Tuple[torch, Tensor, Dict (sur, torch. Tensor].:
goal_dist torch.norm(pall_pn abjest_pos, dimm-1)
quat_dirf quat_mul (object_rot, jon_ssonngatteeenee
rot_ - 2.0 Lorch htw.:chhcccciiillcccccnnntttteee =1.0):
0:3), p-2, d_m= 11, max
dist_rew goal_dist rot_rex_reng 0.1
row_rew ewwehheep-rrtrrewwccoo rot_ciati
reward - exp( 0.25 (-0.5 dist_rew 50) rot_rew
seward__ompoments
"dist_rewed rewardd" dist_rew, "rot_reward"; rot_dist,
return reward, seward_component.
The zeward function changes the scale the rotation reward cemponent introducing cempersture parameter #=ot_rev_temp and applying a transformaticn using 'torch. This ghould help the batter optimize the rotation reward component during policy learning.
G.4 EUREKA FROM HUMAN REWARD REFLECTION
We display the raw dialogue that includes all human reward reflection texts as well as the generated EUREKA rewards in EUREKA from human reward reflection experiment.
Example EUREKA from Human Reward Reflection
Iteration 1: Eurake Output:
The Lask is to make Uhe humancid
Iast possible.
The Therefore, the reward funet lon ahould encourage forward movement. This achieved by rewarding agent besed velocity forward direction. We can also penalize the agent for using too much energy, as this would encourage efficient movement.
Here a possible reward Fanction:
"python
befet_buf, progress_bui, actiens, up_weight,
weading_weight, pateatials, ptey_potentaal,, sciiens_csst_sclle energy_coat_scalle jointa_at_ ltt____t__mmt_ss________________ sest_scale, mox_motor_offott sotor_offorta, cermination_heigeee death_cost, wod_eppsodeelengtthhh
Lype: (Tensor, Tensor, Tensur, Tensos, float, foet, Tensoz, Tensor, float, float, float float, Tensor, float, float, float] Tuple[Tensor, Teaser]
Reward
forward velocity
reward = bbf:buf(;, 1]
Penalize energy energy_cost eaergy_cost_scedd torch. dim- reward energy_cost
Penalize joints
bointt_.t_llmittcooo = joints_at_limie _imit_coct__cccll torch. sum bufl;, 12:32] 0.99, dim--1)
38

---

Published as a conference paper at ICLR 2024
Soints_ar_liiitttt
Perallze for ralling
not_upright terch. abs obs_bufl;, 201) 0.25
reward torch. where uprighh,,,,,, terch.cnes (reward) * death rewerd)
Reset if fallen
reset torch,where inst upright, swrch.ones_iikee (reset_bat) reset_bufi
reset thre www.eheerrppgggeo iprogreapggn >4 max_epi sode_ length 1, torch,onea_ ikke (reset__bbf) reset:
return reward, reset
...
This function anccurages agent move as as possible, also encouráging efficient movement and penalizing the agent for falling or reaching Coint limits.
Iteration 1: Numan Reward Reflection: behavior resembles forward equat jump;
please revise the zeward Iunction that the behavior resemblea Torward running,
Iteration
Output:
To encourage running behavior, we can modify the zeward function to reward the agent for alterneting leg mevements penslize jumping
ground) We elso reward the for meintaining relatively upright coreo to resemble running posture.
Here is reviaed reward fonction:
"'yyhhen
seset_buf, progress_bus, sctiens, up_weight,
weighng poteatials, prev_potentiale, reciens_cost_scall energy_coat_scete saatta__t_iimtt_sst__ssss scale, mox_motor_effort motor_efforts, cermination_height death_anst max_epiaodo_
type: (Tensor, Tensor, Tensor, Tensor, float, flost, Tensor, Tensor, float, Float, float float, float, float, float] Tuple[Tensor, Teaser]
forward velecity
reward obs_buf(:, 1]
Peraliza energy usage energy_cost costey_cost_ecale torch.sun dim=-1) energy_cost
Peralize joints Daaiss_tt_iiittcoosseeee..d sum 12:32] 0.99,
joints_at_limit_cor dims-1)
reward jointa_attlliittcoos
alternating leg nevements
diffriffeee = torch.abs buf 12] buf_ 13))
reward fferenc
Peralize for jumping (hoth feet off the ground)
[eet_od[[grouud torch, bull; 54:56; reward - fest_cff_grrund
Reward for ma maintaining an upright torso upright_torso
torch.abs obe_buffi, 10]) reward - upright_torso
Penalize falling not_upright tarch. (1 obs buf_b 101) > 0.25
reward terch. whare (not_upright, torch.cnes_like (reward) death_ocst, reword)
Reset if fallen
worch.where inst_upright, torch, (zesettbbu)) reset_buf)
reset whree iprogresa_buf b- enw_epssodeee_egtl 1, vorchhonea__leee (re)et_but), reset:
return reward, reset
This reward function encourages the agent forward fast possible, while alse encouraging efficient movement, Alternsting leg movements, maiataining an upright Lorse, and penalizing agent jumping reaching joint
39

---

Published as a conference paper at ICLR 2024
#t+
Iteration 2: Numan Reward Reflectien:
The learned behavior nоw looks like duck walk; the indaed alternating the is very 1ox. Could you Improve the reward function for upright running?
Iteration 3: Bureke. Output:
Ta encourage z more upright running posture, increasa the reward for maintaining upright and panal the agent heavily for having Low torso position. also add reward mointaining certein height, which would discourage the "duck walk" sehavior,
Here revised reward function:
"python
def compute htw.eeuuuaannnneeeeeeeeeee
seset_ proqress_bui, actiens, up_weight,
heading_weight petentials, prev_potentials, acsions_cest_scete energy_cost__t jointa_at_iiiit_oottt
mex_motor_etforl, motor_efforts, termination_leightt height,,, death_cost, mak_spiscde_lennthh)
type: (Tensor, Tensor, Tensor, Tenses, float, flost, Tenser, Tensor, float, float, flont flaat, Tengor, float, fleat, float) +> Tuple(Tensor, Tensor]
Reward is based on forward velecity obs_buf(:, 11
Penalize energy usage
energy_cost energy_cost_ccn reward - enargy_post post
torch.oum/actions> dims-1)
Penalize joints at limit
joints_at liw.ttt_imttt_tt___eecoooo sarch.sumicbs_huee 12:32] 0.99,
dim=-1)
reward joints_ar_limit_cors
Reward for alternst Ing leg novementa leg_differrene torch.abs bubs_buf(;, 12] buff, 131) reward - leg_difference
Penalize for jumping (beth feet off the ground) feat_off_ground torch, (of 54:56; 0) feat_off_grours
mainteining an upright torse upright_torso abrch - abs_buÉ(;, 10])
reward upright_torgs Tharease reward upright corso
Peralize for low terso position low_torso abs_buflt, reward = forch.wheze (lox_toooo,,,, torch..nes 1.0, reward)
1.0
Reward for meintaining certain height height_rewas reward obs_bbfll 0]
reward - height_reword
Penalize falling
not_upright tarch, abs (1 obs bufl, 101) > 0.25
where (noihuprigh,, torch.cnes_lixe (reward) death_ccst, rewerd)
Reset if fallen
reset torch. where inst_upright, torch, like (reset_buu)) reset_buf) reset
iprogresa_but be www_eppoodeeeeetttie - 1, torch. ceeahoonea__lie like (reset_but),
return reward, reset
This seward function encourages the agent forward fast possible, while also
encouraging efficient movement, alternsting leg movements, maiataining an upright Lorse, meintaining
height, penalizing agent jumping, having low position, or reaching joint limits.
iteration Reward Reflectien:
The lescned behavioz has the robot hopping on one of its foot in ordez Lo move Iozward. function encourage upright running behavior.
40

---

Published as a conference paper at ICLR 2024
Iteration 1: Bureka Output:
ic discourage hopping and encourage a more belanced running behavior, add pancity
using both legs equelly. We can also increase the reward for maintaining an upright torse and add penalty for high anguler velceity,
which would di discourage spinning hopping.
Here revised reward function:
hor
def compute_humaniid
reset_ proqress_buit actions, up_weight, heading_waighh patentials, prey_potentialy nex_motor_ettort, motor_etforts, sermination_helght. deatl exk_episode__ennghi
cest____ss___eeeee anergy_cast__eee
type: (Tensor, Tensor, Tensor, Tensoz, float, flost, Tensor, Tensor, float, float, float. flaat, tengor, float, flaat, float] +> puple[rensor, Tenser]
Reward is based on Iorward velueity obs_buf(:, 11
Penalize energy usage energy_comt mergg__ostt_sett torch.aum(actiooss.. dimm-1) reward - maatgy_oost
Penalize joints at limit joints_ai_limit_
joints_at limiattttmmmmeettmmmm
torch. sunicss_uhib 12:321 0.99,
dim- 1)
reward joints_at_limit_coo
Reward for alternsting leg novements leg_difference Lbr bub[,buI[ 12] buf_b 13]) reward =- leg_difference
Penalize for using both legs equally
eeg_ussge_diieereee = torch. abs (torch. sum buf 12:14), dimm-1) 1.00 reward stt_jaaao_diifferreee
Penalize jumping
ground)
oeet_ooffground torch. reward feat_off_grrond
54:56; =- 0)
Reward for meintaining an upright torse upright_torsa torch.abs abe_buf(t, 10))
rewazd - upright_Lorso Incresse reward for upright c0r80
Penalize for high angular velocity
high_ang_vellee vyllccyy tbsch.abs toba teba_huf [t, 4:7.), sum((d===)))) reward torch. where anigh torch. lik -1.0, reward)
Peralize falling
not not_upright torch. abs obs bufl;, 101) > 0.25 where (poighp torch.cnes like (reward) death_cest,
Reset fallen
wherh not_upright, torch, inke_iike (resetbbbE), reset_buf)
reset corch,where whereeppooee iprogresa_hut be max_epi aadte_leeetltll - 1, torch.anea_] ikk (reset_buf),
return reward,
This seward function encourages the agent forward fast possible, while also encouraging efficient movement, Alternating leg movementa, maintaining an ighttht torse, and penalizing agent jumping, using legs equally, having high angular velccity, oz reaching joint limits.
Iteration 4: Ruman Reward Reflectien:
This zeward function resoved the penalty for low torse position that you added last Lime; could you just
#*
Iteration. 5: Bureke Output:
apologize for the oversight. edd back the penalty for low position to discouzage the "duck walk" behavioz. llere is the revised zeward function:
41

---

Published as a conference paper at ICLR 2024
'pyther
def teading_weight potentials, prev_potentialg pailons_oost_see wattatt_iiiitttt_____ee max_motor_effort, motor_efforts, tesmination_leightt death_cost, max_episode_lentth)
(abs_bj buf, progress_bus buf, actiona, up up_xeight,
snergy_cost_ssalle
type: (Tengor, Tensor, float, Tenaor, Float, Float, float1 -> Tuple(Tensor, Tensor]
Tanses, float, flost, Tengor, float, float, flont
Reward la based forward veleotty reward bbf 1]
Penalize energy usage energy_oost energy_cost_eent torch.sum (antionss dima-1) reward -- energy_cost
Penalize joints Limit
Bianss_at__iii___o Himt__oost_saaaall torch. sumteba 12:32] C.99,
dime-l)
reward - joints_or_
Rewazd foz alternsting Ing leg novements
deg_difference torch. (ubs_buf(;, 12] obs_buu(:; 131) reward =- leq_differeece
Penalire for using both legs equally sag_eaagedd = torch. abs (torch. oum (obs_buf 12:14), dims-1) 1.0)
Penalize jumping
off ground)
[eet_off_ground surch. (obs_buf(i, 54:56; reward feet_cff_ground
Reward for ma msintaining an upright torso upright_torso reward = 2 upright_torso Increase reward for upright zozso
obs_bufl;, 10])
Penalize for low terzo position low_torse = bušl; A] 1.0
torch, whare (low_toro,,, torch.ones_ 1k (roword) -1,0, reward)
Penalize high angular velocity
vegllcitgyy = torch. abs babs_bufll; sum(iir-)))) 1.0 -1.0, reward)
reward torch, (high_ang_
Peralize for falling not_upright torch.aba [t, 'C]) rewazd = Lorch, where upoi_up, torch.ones_ (reward) x death_ooss, reward)
Reset if fallen
reset torch, inot upright, tarch.ones_ (resett___of buff) reset buf) torch. where iprogress mox_episode_eenge 1, torch. liks_ (reset_bbb)) reset:
return reward, reset
..
This reward funct enocurages the agent move forward as fast as possible, e, while also alterneting mevements, maintaining upright
ancourzging effinient and penallzing the agent for jumping, not using both lega equally, having low torao position, having higa angulaz velocity, reaching limits.
G.5 EUREKA AND HUMAN REWARD COMPARISON
We display the human reward on Dexterity task verbatim and contrast it with EUREK --gnerr-e reward the same task. As shown, the human reward is difficult parse and has many commented- out blocks of reward components, suggesting history of trial-and-crror reward design. In contrast, EUREKA reward is clean and interpretable, amenable to post-hoc human inspection and editing.
Example 1: Human reward for PushBlock
Storch. jit.
def senpute_hend_reware
rew_buf, reset_buf, reset_gosi_buf, progress_buf, successes, censecutive. mnaeeutive_euscesses. max mxx_epiacde_length: float, object_pos, abjezt_ rot, left_target__pot pos, left left_ target_rrtt. right_targge__oz, tight_target_ret, block_right_handlee.oo leotk wew.eeee_eeeeeeeeee left_hand right_hand__p hf sight_hand__r__po po saght_handd=f_pp_
right_hand_] fos,, right_hand__h._p
42

---

Published as a conference paper at ICLR 2024
left_handd_ff_po left bfft_aand__f__p_ Left_haad__=f_po. hent_ head_ ard_saceeee floot, fot_troot__reee rot_raward_asall: float, rot_opo: float,
actions, petiioonpeealt___ee scally float, success_tolerance flost, reaciugosi__omme float, fall_dist flost, penalyyy float, mak_compeur-vv__ccceses int, sy_factor: float, ignore_R_ bool
bistance from the hand to the oaject left_goal_ torch.norm targgett right_goa _ont__gol_dtatt at goal_dist Larget_pos 2] pbjjct_pos[s, 2]
saaftt_aande_ereee p=2, dins-1) hww.rrggggttttnle__geee p=?, 61me-1)
right_ handd left_hand_dtse torch.norm htt.mm//ooe_lllff_aaallllll
posd p=2, dime-l)
left_hand_pon, p=2, 6ima-1)
right_hand. fingt
p=2,
--1 torch.norm (block_r wwwlooooriggttanneeeeeeeoee right__andie_pps hig_g_haha______poo p=ž, dim=-11
right_hand_:__pos p=2, dim right_hanee lapp_ p=2, dims-1)
--1) norch.normbbles. httpl/ookiiittthhiiiiiieeeeeeeee hatd_hhadee__ppp__
torch. hardd..ormmbbackkk.......ddd
higgtt_hnnd p=2, dim
--1)]
hand finger_ dist (baccconnnmmtt.eeeeeeeed..
haft_hand__f__a p=2, dim- 1) torch.norm (bechk jtteeet__andd__pp_______ Left_hand p=2, dim--1)
be p=2, dim
--1) torch. (block lelocc_left_haadde_ppp -fft__han__l_____eeeed pos, p=2, dim
borch..ormbbback..eftt..bo.
taf=_hand_th_pps,, p=2, dim
--1))
Orientation aligament quat_ diff = ( puat_conjugate
goal
(target__oo)
ret_dist terch.asia clamp now. wor.h..ommmuutt...ifff...... 0:31, p=2, dimm-1), max -1.0):
htt__hhod_dist__eeee 1,2-1+right_hhn Einger_dirs @eff_hand_dfat__ee 1.2=1+iiftt_hano_ffnga
sot_ 1.0/ (torch. abs (zot_dist) rot_eps) cot, ret_rdewad_
action_ torch. sum(actions
dim--1)
Total reward is: position distance orientation aligrment action regularization aucceas bonua fall peralty
torch.exp( 0.05 (ip_rew rest__elwrd corch.exp 0.05.
right handt__had_dist_rree pist_roward_scale)
seal_ expehhexpl-0.000 (lnff_
up_rew sereccceere_lleetighhh....mm
up_ = 5 Seleft_gooo_diat 3+rightt
reward exp( dee dist_rewardsscaee)
dist__eward i torch.expt [-0.1 (
reward = distt__aadddisst_rov left_hand_disttroo up_rew
torch.where faighht_aan_ffiggedddie reseta torch, whare (otterrdoootthnd___nneeee..eed >e 1.2, corch.ones_1ikeiraree reseta)
1.2, Likk
seset_but)
Find out which eavs hit the goal and update successes count (successes
successes
torch, where abs siitt_goal_dist) <= 0.1, there (torch. (right_goal_ddt _dist) 0.1, forch.onea_likee sicch.one_likccccceeeeee
0.5), successes), suecesses)
(progreas_bul max_episode_iengly Lorch.ones_ Liechoonss_Lkkeetteeesss.
goal_ real_resets zeros_ lekos_
rus_resets torch, Corchhsum (resets) finished_
sorch.sum resets.flos::))
cone successee reward successes
resets, consecutive meac
return reward, reaets, goal goal_resets, progress_bon buf, successes, cona_auocesses
Example 2: Eureka reward for PushBlock
Storch. jit. script
def compute__eward ject_pos: Tensor,
heft_hand_pos ri right_ post Tensor,
left_ggallpos: Tensor,
ri right_goal__pos: Tensor) >> Tuple [Tensor, Diet (atr, rensor]]:
43

---

Published as a conference paper at ICLR 2024
temperature parameters reward components praxxitiyy flost = 0.5
float 0,3
temp_iann__distence: float 0.1
Proximity reward for pushing the block the left goal left_paah_diatane diatance torch. worrh.normloajett_poo left_goal _oaa__ps, dim=1)
ieft_proximiityrewwnr tcrci.cxp wttppeppptttmeleeeppoooooeo
left_push_cisstance
Proxinity reward for pushing the block to the zight goal
dight__osh__iitancee torch (objettttojjcppppr goght_g dimri)
exp r-ttm__right__orattee
right_push_
Proxinity rewarda for the handa to be close to the block for better control
left_hand_to_blese norch.normlbajeer_pee. left_hand_reward terch.exp (-temteeppppppepeeeee corooheeyptttmm__haaaaaaaaaee cand_claaaaee left_hand_to_bloct
left_hand_pos, dimel)
right_hand block torch. aeceh.oommee (otjeeteeettppp sight_hannd_pss, dimmi)
right_hond__aacse = torch.exp wetch.exp--teeee hand >
Pinal reward as a weighttd sum of individual zeward components
reward (left_ left_hand_reward right_haan_rewwrdd
sight_proximitt_eeweeed 0.25
reward rexord_components
ploft_proximitt__raaaa
"right_proximity__eee right_proxinitt_rrwaad
"left_ heed_hhddd hend_ rewazd, "rightthandd_ right_hand
return reward, reward_ componetts
H LIMITATIONS AND DISCUSSION
In this section, we discuss several limitations and discuss promising future work directions to address these limitations.
One limitation is that EUREKA currently is evaluated on simulation bbased tasks with the exception of the preliminary real-robot experiment in App. which demonstrates real-world hopping behavior learned via EUREKA A reward in a Sim2Real pipeline. There are two promising directions in extending EUREKA to the real-robot setting. First, as demonstrated in App- F EUREKA can be combined with Sim2Real approaches to first learn policy in simulation and then transfer to the real-world (Akkaya) cct al. 2019; Kumar et al. 2021): ); this is a well-established approach in the robotics community. Isaac Gym, particular, has been used in many Sim2Real results (Margolis et al. 2022, Handa et al.) 2023). Therefore, we believe Eureka's 's strong simulation results on various Isaac Gym tasks bodes well for real-world transfer. Another approach is to instrument the real-world environments with sensors that can detect state measurements and then directly generate reward functions over these state variables, thereby enabling direct real-world reinforcement learning (Gu et al. 2017 Smith et al. 2022. Yu et al., (023). For either direction, progress in Sim2Real and state estimation will enhance the applicability of EUREKA to the real-world setting.
Another limitation is that EUREKA requires a task fitness function F to exist and easily definable by humans. While this assumption holds for wide range of robot tasks (including all our benchmark tasks) and is required for reward design problem to be well-defined (Definition 2.1). there are certain behavior that hard specify mathematically. such as "running in a stable gait". In those cases, we have presented EUREKA A from Human Feedback (Section 4.4 that uses human textual feedback to directly steer the reward generation, sidestepping having to specify F that accurately captures the human intent. A promising future work direction is to use vision- lssion-language models (VLMs) to automatically provide textual feedback by feeding the policy videos into the VLMs (Liu et al. 2023b
Zhu al. 2023
Yang et al. 2023).
Finally, our experiments currently take place on a single simulator and a fixed RL algorithm, with the exception of the Mujoco Humanoid experiment in App. Given that EUREKA does not take the identity of the simulation engine or RL algorithm as input, we believe that should be straightforward extend EUREKA custom simulators and optimizers. With regard to latter, it would be interesting
44

---

Published as a conference paper at ICLR 2024
to explore using model-predictive control methods (Williams al. 2015 Howell et al. 2022), which can potentially lead to faster behavior synthesis in terms of wall clock time.
45


Reviews:
[{"review_id": "zpptFKNqGs", "reviewer": "Reviewer_DCBe", "summary": "The paper presents a novel approach to leverage LLMs to plan/control complex low-level manipulation tasks in the form of an LLM-powered reward function generator. Its generation algorithm performs iterative improvements over an initially designed reward function without task-specific prompting nor a few short examples, by automatically calculating the fine-grained fitness of a policy in text over a sequence of executions on a target RL environment and using the LLM to improve the code. The presented approach is evaluated in varied RL environments and experimental results show it outperforms RL-expert humans on the majority of tasks. Moreover, the authors provide a case study of how the proposed approach enables higher complexity tasks where manual reward policy design is challenging.", "questions": "In view of the claim of generality, how to qualify/quantify the impact of different simulation platforms (and level of physics detail) or RL algorithms? Please also comment on the scenario of interfacing with the real world.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The submitted manuscript is very well written and presents a novel and interesting approach to automatically generate reward functions for simulated RL environments, which seemingly could be applied to different scenarios.\n\nIt presents a clever approach to leveraging recent LLMs' zero-shot code generation ability to both understand a simulation environment and to iteratively improve generated reward functions that would be hard to manually author and tune.\n\nMoreover, the described evolutionary search and reward relection components or the approach, while not groundbreaking, provide interesting insights into the problem and on better interacting with LLMs for code generation.", "weaknesses": "One of the main weaknesses of the submitted paper is the lack of a Limitations section/discussion, or such discussion throughout the text. While the authors claim the generality of Eureka, the proposed approach has only been evaluated on a single base simulator (Isaac Gym) and with a fixed RL algorithm. In other words, the claim seems to be overstated. \n\nAnother weakness is the experiment part, while the submitted text showcases different (and relevant) comparisons with human results, the human rewards are zero-shot and not tuned for many RL trials to further improve the performance. Therefore, I believe the comparison may be unfair. If you tune the human rewards in this baseline (e.g. search the weights for different reward terms) and train RL for many trials (same as the cost of the evolutionary search in Eureka ), some claims may not hold.\n\nA specific discussion I missed was about how the proposed approach handles the difference between optimizing for the internals of the simulation vs its sensing/acting interface. The former should be avoided in any general approach. The authors claim to use an \"automatic script to extract just the observation portion of the environment source code\", but this doesn't necessarily guarantee no leaks or that such observation code abstraction level leaks details.\n\nMoreover, as the proposed approach depends on feeding the environment code to the LLM, besides just claiming the \"the observation portion of the environment\", I believe a more in-depth discussion is needed on how Eureka could be adapted to a) more complex environments, which may be too large for the model context windows; and b) scenarios of interaction with the real world (actual robot control).\n\nParticularly for a), this is a critically important discussion. E.g., What would be the impact on the pen spinning demo with more detailed material characteristics and physics (friction, inertia, actuator latencies, etc.)?\n\nThe authors also claim only a \"handful\" of LLM output is enough. However, 16 is hardly a handful (<5). Intuitively the number of samples to obtain one fully executable policy will grow in proportion to simulator and environment complexity. However, again there is no discussion of such limitations of the approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M8mKWYcthK", "reviewer": "Reviewer_hi7d", "summary": "The main idea of the paper is to have a LLM provide the reward function for RL problems.\nTo achieve this, the LLM gets to see the code of the environment along with the plain text description of the task. The task for the LLM is to provide sample code for reward functions, which are in turn improved using an evolutionary search. The experiments section of the paper goes through a number of environments, where performance is shown to be better than performance using human-designed reward functions. For the examples provided, Eureka does not use task-specific prompts.", "questions": "My main questions are around limitations as mentioned above, and the types of problems this is expected to work well for (and where do you see this not working). \nTo what extent the approach makes use of information available at training time (eg about the simulator or environments). More directly about the approach I would find it interesting to hear about approximate run times from beginning to end, for some of the examples.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "I love the idea of using an LLM to provide initial versions of the reward functions, and to then improve it using evolutionary search. Moreover, the evaluation shows that the approach can deal with challenging environments, leading to good solutions or solutions for problems that have not been solved before. The work is also well motivated, and potentially lead to interesting advances in RL itself; it would be quite interesting to see this published and available for further research.\n\nThe work already contains inputs/outputs of the approach in the appendix, and code and prompts are expected to be released.", "weaknesses": "While the paper does do a great job in selling the idea, there's a frustrating lack of technical detail in the main part of the paper. One example to illustrate this problem: The subsection on evolutionary search provides no detail on what is the exact input, outputs, or about the specific method being used. This is one core aspect of the proposed approach, and would require more details to be understandable. I understand some parts of this appear in the appendix or will be clear from code release, but the main part should make the core parts of the approach more clear.\n\nIn the Eureka algorithm description, it is a bit unclear to me what the reward reflection part does; for example the algorithm specifies the reward functions and their evaluations are provided as inputs, but the text also talks about the choices of RL method that are important for this part. There's only little information in the text that tells me how it works.\n\nSimilarly, I like the idea of \"environment as context\", but it would be good to know what is considered here as environment (what does it mean), for example to what level of detail does the environment need to be described / down to what level do you have to go. The appendix describes that, due to limited context length, the only the observation context will be used, but for simulators different from Isaac, what information do you expect you need to provide for this to work.\n\nThis could be connected with a missing discussion of limitations of the approach, for example do you expect this approach to be useful when you do not have full access to the code of the environment but maybe just some API, or if the environment is the real world.\n\nMaybe more philosophically but I am also not quite sure about the classification of the generation as \"zero-shot\" as it is unclear what the LLM has actually seen during training, and it would be interesting to see further investigations of this (not necessarily in this work) and capabilities of transferring domain knowledge - the release of benchmarks after the cut-off doesn't necessarily mean there was no environment like this before.\n\nMost of the above weaknesses impact the presentation of the work; while formal presentation of the work is good overall, Figure 2 and Figure 3 could be improved for contrast and font sizes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eRaYz1TAV9", "reviewer": "Reviewer_XD65", "summary": "The paper introduces EUREKA, an LLM-powered method for reward design for reinforcement learning tasks. The proposed method utilizes the zero-shot generation and in-context improvement of LLM, enabling evolutionary optimization over reward code without task-specific prompting or pre-defined reward templates. The paper showcases EUREKA's performance against human-engineered rewards in a diverse range of robot tasks. Furthermore, the algorithm is able to integrate with human feedback for further reward improvement, enhancing the quality and safety of generated rewards without the need for model updating.", "questions": "While the pen spinning demonstration is impressive, it remains uncertain what is driving the task's success. Is it due to an effective reward design, which is the paper's main focus, or is it a byproduct of unspecified engineering efforts? Section 4.3 is not very clear and leaves room for several pertinent questions:\n\n1. The paper does not detail how the pre-trained policy was obtained. The statement, \"Specifically, we first instruct EUREKA to generate a reward for ... Then, using this pre-trained policy,\" leaves the readers wondering what exactly \"this\" references to. \n\nThe application of curriculum learning here appears to break the policy training into two main stages - an initial 'pre-training' stage and a 'fine-tuning' stage with the pre-trained policy. If this interpretation is accurate, clarity around the following is crucial:\n\n2. Is the training process – both pre-training and fine-tuning stages – guided by the reward functions derived using the LLM-powered reward design method proposed in this paper? Are these reward functions identical?\n\n -  If they are, there needs to be a detailed explanation of the distinctiveness of 'pre-training' and 'fine-tuning' when it comes to the optimization of policies under the same reward function. The reason how a dual-stage policy optimization can notably enhance policy performance remains under-explained. Additionally, if this is the case, it appears that the success of the task may be due to the dual-stage policy optimization rather than the reward design, casting doubts about the central argument of the paper.\n\n -  If they're not, clarity is needed on how these varying reward functions are generated, and what differences exist between them. Furthermore, how does this dual-stage reward function model vary from the primary algorithm proposed? How to ascertain the number of reward functions sufficient for a given task?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "While the idea of this paper is rather simple, it yields a surprisingly good performance, which reflects a well-structured system. Being able to bring an easy idea to such a complete and well-considered system is commendable.\n\nMoreover, this work brings insight to the reward design community by removing the dependency on collecting expert demonstration data. The study suggests that Large Language Models (LLMs) can serve as an cheap alternative to human expert demonstrations for acquiring domain-specific task knowledge.\n\nThe paper's presentation is clear; the authors make the content easily comprehensible. Their responsible practice of providing all relevant prompts and code offers an added advantage.", "weaknesses": "1. Unrealistic assumption of access to the environment source codes:\n\nThe reward code generation in this paper critically depends on having access to the source code of the MDP specification as context for the initial reward proposal. The authors have presented this as a benefit, allowing the LLM to exploit code structure to understand task environments. However, it makes an unrealistic assumption, as most reinforcement learning setups only require access to a black-box simulation. \n\nA significant limitation of this approach is that it may be infeasible for real-world robotic tasks where the transition function may either be unavailable or in different analytical forms. Given the heavy dependence on source code for task environment understanding, this method could be essentially restricted to simulated RL environments only.\n\n2. Strong assumption on the fitness function F(.)\n\nAnother weak point is the strong assumption on the fitness function F(.). The evolutionary search for the LLM generated reward function requires a fitness function capable of assessing the quality of each proposed reward function. In this work, the fitness function F(.) is implicitly assumed to have access to the ground truth reward function to evaluate the induced policies of the proposed reward functions. This limitation implies that the method is applicable only to tasks that come with known ground-truth reward functions. This weakens the method's practical applicability, restricting it mainly to idealized or known environments, hindering its usefulness for real-world, less predictable reinforcement learning tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QOk8GxxwqW", "reviewer": "Reviewer_JeMn", "summary": "This paper purpose a novel pipeline to harness the LLM to design reward for using reinforcement learning to do diverse tasks without any task-specific template. The pipeline use environment code and reward feedback as input to LLM, and let LLM to continously update the reward function. The comprehensive experiments demonstrate the generalization ability of the LLM for reward design and the effectiveness of the reward designed by LLM(comparable to human).", "questions": "1. Can this method used for task that object is diverse?\n2. If the task require image as input, it hard to run many environments simutenously, can this method still work?\n3. Will only give success reward as inital reward, can make the LLM to find more optimal reward?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. This paper purpose a general pipline for reward designing of RL, which is indeed a long standing problem for RL researches, this kind of pipeline may save a lot of time for human to shape the reward.\n2. The pipeline requires no task-specifiction template for LLM, which shows a great generalization to different tasks.\n3. This paper demonstrate the LLM with evolutionary is a effective approach for using LLM, has potential for other area.", "weaknesses": "Time cost and compute resource cost: Since each iteration, LLM will sample tens of reward sample, and we need to test all this reward function and get feedback, we need to simultanously run multiple experiments for each reward sample, it seems there will be a lot compute resource needed? And for each sampled reward, how many environments do you create to train? How do you decide when to terminate each running expertiment? Will it be possible that because the environment not create enough or did not train for a long time than miss a effective reward sample? What will be the total time of finding a good reward based on what kind of device?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)