REvolve: Reward Evolution with Large Language Models using Human Feedback
Authors: RISHI HAZRA
Alkis Sygkounas
Andreas Persson
Amy Loutfi
Pedro Zuidberg Dos Martires
Venue: ICLR 2025 Poster (2025)
Keywords: Evolutionary Algorithms
Reward Design
Reinforcement Learning
Large Language Models
Primary area: reinforcement learning
TL;DR: We propose an evolutionary framework for reward design for Reinforcement Learning, where LLMs act as intelligent operators, guided by human feedback, to generate and refine reward functions.

Abstract: Designing effective reward functions is crucial to training reinforcement learning (RL) algorithms. However, this design is non-trivial, even for domain experts, due to the subjective nature of certain tasks that are hard to quantify explicitly. In recent works, large language models (LLMs) have been used for reward generation from natural language task descriptions, leveraging their extensive instruction tuning and commonsense understanding of human behavior. In this work, we hypothesize that LLMs, guided by human feedback, can be used to formulate reward functions that reflect human implicit knowledge. We study this in three challenging settings -- autonomous driving, humanoid locomotion, and dexterous manipulation -- wherein notions of ``good" behavior are tacit and hard to quantify. To this end, we introduce REvolve, a truly evolutionary framework that uses LLMs for reward design in RL. REvolve generates and refines reward functions by utilizing human feedback to guide the evolution process, effectively translating implicit human knowledge into explicit reward functions for training (deep) RL agents. Experimentally, we demonstrate that agents trained on REvolve-designed rewards outperform other state-of-the-art baselines.

Published as a conference paper at ICLR 2025
REVOLVE: REWARD EVOLUTION WITH LARGE LANGUAGE MODELS USING HUMAN FEEDBACK
Rishi Hazra, Alkís Sygkounas, Andreas Persson, Amy Louthi & Pedro Zuidberg Dos Martires Centre for Applied Autonomous Sensor Systems (AASS)
Órebro University, Sweden
{rishi hazra, haasai alkis. alkis.sskkaann.. sygkounas, andreas persson,, amy.loutfi loutfii,
pedro. wwrroo/zuibbbrg--eee dostberr-oo---oeee mars-maatirsssoo se
https://rishthazra, gi Coub.io/REvee
ABSTRACT
Designing effective reward functions is crucial to training reinforcement learning (RL) algorithms. However, this design is non-trivial, even for domain experts, due to the subjective nature of certain tasks that are hard to quantify explicitly. In recent works, large language models (LLMs) have been used for reward generation from natural language task descriptions, leveraging their extensive instruction tuning and commonsense understanding of human behavior. In this work, we hypothesize that LLMs, guided by human feedback, be used formulate reward functions that reflect human implicit knowledge. We study this in three challenging settings autonomous driving. humanoid locomotion, and dexterous manipulation wherein notions of "good" behavior are tacit and hard to quantify. To this end, we introduce REvolve, a truly evolutionary framework that uses LLMs for reward design RL. REvolve generates and refines reward functions by utilizing human feedback to guide the evolution process, effectively translating implicit human knowledge into explicit reward functions for training (deep) RL agents. Experimentally, we demonstrate that agents trained on REvolve-designed rewards outperform other state-of-the-ee -frt---rtttttttt baselines.
INTRODUCTION
Recent successes applying reinforcement learning (RL) have been concentrated around areas with clearly defined reward functions (Silver al., 2018; Vinyals et al., 2019; Fawzi et al., 2022: Mankowitz et al., 2023; DeepMind, 2024). However, tasks with ambiguously defined rewards, such as autonomous driving, still present challenges due their complex and lifficult-to-spece et al., 2023). This echoes Polanyi's paradox, which states that we know more than we can tell (Polanyi, 2009). The notions of "good" behavior (i.e., aligned with human standards) are inherently subjective involving level of tacit knowledge that is hard to quantify and encode into reward functions. This is also referred to as the reward design problem and often leads to sub-optimal behaviors that exploit shortcuts in the reward function and that do not truly reflect human preferences (Kerr, 1975; Madffeld-Mened et al., 2017; Booth et al., 2023). This issue is underscored by the growing concerns about the misalignment between human values and the objectives pursued by agents (Turner et al., 2021; Omohundro, 2018; Russell, 2019).
goals (Knox
Recently, LLMs have shown impressive abilities in instruction following (Longpre al., 2023; Peng et al., 2023), search and black-box optimization (Yang et al., 2024; Zhang et al., 2023; Liu et al., 2024), self-reflection (Madaan et al., 2023; Shinn al., 2023), and code generation (Chen al., 2021; Roziere et al., 2023). Owing to their vast training and world knowledge (Wang et al., 2023; Hao et al., 2023; Hazra al., 2024). they can readily incorporate commonsense priors of human behavior. Together, these abilities make LLMs an ideal candidate for generating rewards from natural language (NL) task descriptions and refining them using language- based feedback loops. Indeed,
"Equal contribution.

---

Published as a conference paper at ICLR 2025
recent works like Language to Rewards (Yu et al., 2023) and Reward (Xie et al., 2024) have utilized LLMs for this purpose, More significantly, Eureka (Ma et al., 2024a) utilizes LLMs to design dense and interpretable reward functions, achieving improved performance in dexterous manipulation tasks. It does so by generating multiple reward functions and greedily mutating the best candidate over multiple iterations. As such, Eureka resembles more of a greedy iterative search than evolutionary search, as described in Ma et al. (2024a).
We identify this greedy approach as significant limitation that risks premature convergence and leads to a waste of resources (by preemptively discarding weaker reward functions). Moreover, Eureka relies on having access to fitness functions to assess the quality of learned behaviors and decide the best reward function. This leads to a thicken-and-ege -gg dilemma- - if designing good fitness measure was feasible, one could arguably design an effective reward function just as easily. This is particularly challenging for complex tasks, such as autonomous driving or dexterous manipulation.
To this end, we introduce Reward Evolve (REvolve), novel evolutionary framework using LLMs, specifically GPT-4, to output reward functions (as executable Python codes) and evolve them based on human feedback. This allows humans to serve as fitness functions by mapping their preferences and feedback into fitness scores.
In particular, REvolve offers the following novelties:
(1) Using Evolutionary Algorithms (EAs) for Reward Design. Traditional gradient- based optimiza- tion is unsuitable for reward design due to the lack a differentiable cost function. Instead, REvolve utilizes meta-heuristic optimization through EAs (De Jong, 2017) to search for the best candidates. This is done by evolving a population of reward functions using genetic operators like mutation, crossover, and selection employing LLMs as intelligent operators instead of traditional handcrafted methods. We empirically show that our evolutionary framework considerably outperforms iterative frameworks like Eureka all without incurring additional computational costs compared to Eureka. Through REvolve, we also demonstrate that EAs, combined with the full range of operations like mutation, crossover, selection, and migration, can be successfully implemented with LLMs.
(2) Utilizing Human Feedback to Guide the Search. Mapping human notions directly into explicit reward functions challenging; however, humans can still evaluate learned behaviors, effectively acting as fitness functions. Through REvolve, human preferences are directly mapped into fitness scores. Additionally, REvolve incorporates qualitative natural language feedback, offering GPT-4 high-level insights into which aspects to prioritize while refining the reward functions. Together, this ensures that the reward functions and thus the learned policies more closely reflect human notions.
(3) Eliminating the Need for Additional Reward Model Training. Unlike reinforcement learning with human feedback (RLHF) (Christiano et al., 2017), which relies on a black-box reward model - trained on massive amounts of curated human preference data align LLMs with goals like safety and reliability (OpenAI, 2022), REvolve requires no extra models. REvolve-designed reward functions interpretable (as executable Python code) compared to RLHF rewards, making it more suitable for safety-critical applications, such as autonomous driving (Christian, 2023). We also include an estimated data requirement comparison in Appendix D.
Figure 1, provides an overview of REvolve for autonomous driving within a simulated environment.
2 PROBLEM FORMALIZATION
Let us first formally introduce the reward design problem (RDP) (Singh et al., 2009), given by the tuple (M, R. F), where M = (S, A, T) is the world model consisting of state space S, action space A, and transition function T. R is the set of all reward functions where each function that maps a state-action pair to a real-valued scalar R : S x A + R. The tuple (M, R) constitutes a Markov decision process (Sutton & Barto, 2018). Furthermore, π : - A is a policy that maps state to distribution over actions, which is learned using reward function R.
R is a reward
Moreover, the RDP includes a function F that evaluates the fitness (i.e., quality) of the trained policies. Traditionally, F maps policy rollouts, i.e., rollouts of envnn-enn interactions, to realev scalars. Formally, we denote this by F ()(e,) + E. where e, is the random variable modeling the probability distribution of rollouts induced by л and ((O.)) is its sample space.
2

---

Published as a conference paper at ICLR 2025
Abstracted Environment
Output Reward Functions R
celtision, po, distarce:
cilfictien
action (steering, ttrottia]
soeed Feand rg Pzeggaapeec teep speed stffeverce
2ef statetinif];
aisition Into cell cel.erttettttoooo...... getcaettrre))
cxlliason coiitatonnrwer rwwrr
20,, at..rrannteemmjpllliaa..rr
rowerd compraara colitaaoonn
retur tstal, morend rourd, compoeort
cepute roserfaeerr cellisson, 30,, distanee::
fitness scores, NL feedback (a, A)
diseace rward re eageetee sects, 28 #letan001) tustenooresee
cettirinn roens cosperant
odbisiee solliatie daa
callpaarrsscciiiiiioo.... www.aacoccnnnneeeee.. roardd)) coiliatoo rwear
return tstal_ rovend rewnr__cmpoenrs
π
Human Feedback Interface
Policy Training in Simulator
4
policy rollouts
Figure 1: An overview of REvolve. Given the task (here, autonomous driving) and abstracted environment variables, a reward designer G (LLM) outputs population of reward functions R R. each used to train policy т T(R) in simulation. Then, we collect human preferences and natural language (NI.) feedback on pairs of policy rollouts (R(R) through a user feedback interface. Policy (and thus, corresponding reward function) fitness is calculated, and the fittest individuals, along with their NL feedback A, are refined by G. addition, the process leverages genetic programming for evolution.
For REvolve, we generalize this to also include natural language feedback. Specifically, we redefine F: [(O+)) R x L. where L is the set of all strings (including natural language feedback). Note that the fitness function is derived from human evaluations of the trained policies, thereby rendering non-differentiable.
In order to instantiate the RDP tuple for REvolve, we assume that we are given a reward designer G: L R that maps from the of all strings the set of all reward functions R. The objective now is to find a reward function R E R that maximizes the expected fitness score of the policy π trained on R, which we denote by (R). Formally, we express the optimization problem as follows:
R arg arg max Ee [F(0]]
RER
Here, 0 is trajectory sampled from xx()) We also define the optimal policy n² the policy trained with reward function R given as x² := (RR).
An implicit assumption for reward design is the knowledge of observation and action variables in the environment. For real-world problems, like autonomous driving, the assumption is supported by access to sensor data (such as speed from speedometers and distance covered from odometers) or by employing advanced vision- vision-based models (Kirillov et al., 2023; Cheng et al., 2024; Yuan et al., 2022).
REWARD EVOLUTION
As the fitness function F is based on implicit human understanding, it cannot be explicitly formulated and is, therefore, not amenable to gradient-basee optimization. Hence, we solve the RDP using a meta-heuristic approach through genetic programming (GP) (Langdon & Poli, 2013). Concretely,
3

---

Published as a conference paper at ICLR 2025
Algorithm REvolve
1: Require fitness function F, reward designer G
2: Hyperparameters evolution generations N, number of sub-populations 7, individuals per generation K, mutation probability Pr., crossover probability pe
3: Input: Task description
4: Initialization
> Section 3.1 b Initialize database
5: D = {(H,, ,
6: while not termination do
7: # Reproduction
Section 3.2 Initialize empty temporary database
8: Dummp = 0
9:
for K individuals do
10:
mutation ~ Bernoulli
Sample operation P ~ eht DD, (a, P1 1==1) Sample sub cub-population using average fitness
12:
mutation then
13:
Roew = Mutate(G,
Mutate individual in sub population
14:
else
Raew Crossover(G.
D. Crossover with pm probability
16:
end if
17:
Train Race)
Train policy on new reward Add to temporary database
18:
Dump Update( RRew,
19:
end for
# Selection
Section 3.3
21: for (R, P, Diemp do
22:
0~ 6
Sample rollouts from trained policy Compute fitness from human feedback b If individual fitness 2 sub sub-population fitness Add reward to database sub- uob-pooulation
23:
(a, A) = F(0)
24:
if a 2 o then
25:
[[P] + Tp,
26: end if
end for
28: end while
29: Termination
> Section 3.4 b Select tuple with the highest fitness score
30: (R", x=,"", ,0", arg max, D
31: Return
REvolve uses GP to iteratively refine a population reward functions (individuals) by applying genetic operations such mutation, crossover, and migration. In each iteration, new individuals are obtained and added to the population based on their fitness. This fitness is assessed through human evaluations of the policies trained with these reward functions thus guiding the evolutionary search.
Evolutionary algorithms, in general, are considered to be weak methods that perform blind searches and do not exploit domain-specific knowledge (De Jong, 2017). The initialization and perturbations are random, leading to scalability issues (O' Neill et al., 2010). In contrast, L.LMs can leverage their commonsense priors to suggest changes that are intelligent and better aligned with the feed- back (Gugerty & Olson, 1986). Therefore, we combine the evolutionary search of REvolve with LLMs by using them reward designers in the RDP.
We follow the island model of evolution (Cantá-Paz et al., 1998), where we maintain multiple (/) sub -obbououlltions (or islands) reward database D. Island populations evolve by locally reproducing (mutation and crossover). Periodically, individuals migrate from one island to another. The island model helps enhance genetic diversity across the population and prevent premature convergence in complex search spaces (Cantú- Paz et al., 1998). Following existing works in reward design with LLMs (Xie et al., 2024; Ma et al., 2024a), we represent the reward functions as Python programs that seamlessly integrate with the GP framework. In what follows, we describe the four stages (initialization, reproduction, selection, termination) of our evolutionary framework. We give the instead of π T(R).
formal description in Algorithm for brevity,
3.1 INITIALIZATION
We start by creating population of K reward function individuals using GPT-4. This leverages GPT-4's world knowledge (Hao et al., 2023) and its zero-shot instruction following abilities (Peng al., 2023). For each individual, train policy æ, compute fitness a, and obtain natural
4

---

Published as a conference paper at ICLR 2025
Hood
Tear Shard
Illustration of how GPT-4 applies mutation and crossover to reward functions. Mutation (left): shows the modification of the "smoothness reward" component. A red '-' sign indicates the line removed from the parent reward function, while green '+' sign indicates the line added to the new reward function. Crossover (right): demonstrates how parent reward functions are combined to create child reward function, incorporating the most effective reward components from each parent.
Figure 2:
language feedback À L. The fitness score calculation and feedback collection detailed in Section 3.3. Each individual and their respective policy, fitness score, and natural language feedback (R, n a. is then randomly assigned to a sub-population and stored in the reward database D.
3.2 REPRODUCTION
Each successive generation of K individuals is created by applying genetic operators (crossover and mutation), the existing reward database individuals. We use GPT-4 mutation and crossover operators with appropriate prompts. Due to space constraints, we provide prompt examples for mutation and crossover in Appendix E.
We begin by randomly selecting an operation - mutation with probability Pm or crossover with probability P.. 11-Pppg Prr; Next, we sample sub-population P via weighted selection based on the average fitness scores of each of the 1 sub-populations. Subsequently, we sample parents (one for mutation, two for crossover) from D[P) using weighted sampling based on individual fitness scores. These sampled individuals, along with their natural language feedback X, are used to prompt GPT-4 create new individuals. These prompts direct GPT-4 either modify a single reward component of the parent (Mutate(G, )) or combine best performing reward components from two parents (Crossover(C. (G, D(P)). Periodically, we enable the migration of individuals between sub-populations to enhance genetic diversity. Reproduction operations are illustrated in Figure 2.
3.3 SELECTION
Newly reproduced individuals (from the reproduction stage) are retained based on fitness scores, following survival of the fittest approach. A fitness function F, tailored to the specific problem, evaluates the quality of each individual. The process involves (1) training a policy x(R) until convergence, (2) sampling rollouts 0 ~ (x(R). and (3) using the fitness function to obtain a tuple of (real-valued) fitness score and (string) NL feedback A) = F(0). Individuals are retained if they contribute an increase in the average fitness of their respective sub- cob--opulations o 2 of, where P denotes the sub-population to which the individual is added!. This ensures that the average fitness of the sub-population monotonically increasing. what follows, we detail the process of obtaining the tuple (a, >) from the fitness function.
To compute the fitness score a, we ask human evaluators to judge policy rollouts from different policies in pairwise fashion. As illustrated in the user feedback interface in Figure 1, these rollouts are presented as video segments ss 30-40 seconds long. The evaluators choose their preferred segment (ties are also possible). We then apply an Elo rating system (Elo, 1978), which transforms pairwise assessments into fitness scores a. This avoids the need to learn a separate reward model,
"To ensure genetic diversity, add individuals based on surpassing the average and not maximum scores.
5

---

Published as a conference paper at ICLR 2025
ef. RLHF (Christiano et al., 2017). Asking human evaluators to provide preferences instead of quantitative scores for each video has the advantage of mitigating individual bias (Christiano et al., 2017). Details of the human feedback collection and Elo scoring documented in Appendix B.1.
The NL feedback À about each pair of video segments is, likewise, collected through the same feedback interface. Evaluators comment (select checkboxes) on aspects of the driving they find satisfactory or need improvements. This feedback is subsequently integrated into the prompt for GPT-4 when reward function individuals are sampled for reproduction.
3.4 TERMINATION
The evolutionary process repeats until reaches predetermined number of generations N, or when an individual attains a fitness score equivalent to human-level driving performance, as detailed in Section 5.3. The final output is the fittest policy x corresponding to the highest a in the database D. Note that n² is the best policy in the reward database according to human-assigned fitness scores but it is not necessarily the optimal policy.
A minimal input-output example of the working of REvolve is shown in Appendix F.
4 RELATED WORK
Reward Design with LLMs. Recent works have explored LLMs for complex search and black-box optimization problems like hyperparameter optimization (Yang et al., 2024; Zhang et al., 2023; Liu et al., 2024), reward design for RL (Kwon et al., 2023; Xie et al., 2024; Ma et al., 2024a), or robotic control (Yu et al., 2023). Kwon et al. (2023) use LLMs to directly output binary rewards by inferring user intentions in negotiation games, but this requires numerous queries to LLMs and limits the rewards to sparse binary values. Conversely, Language to Rewards (L2R) (Yu et al., 2023) and Text2Reward (T2R) (Xie et al., 2024) utilize LLMs to create dense and interpretable reward functions, improving them with human feedback. L2R, however, restricts its adaptability by employing manually crafted reward templates tailored to specific robot morphologies. In contrast, T2R outputs versatile reward functions in Python code. It generates a single reward function per iteration, checks for improvement from the previous one, and repeats this process, hence performing greedy search.
Eureka (Ma al., 2024a) further extends T2R by generating multiple reward functions and greedily iterating with the best reward function and discarding the rest. This means that by definition (De Jong. 2017), Eureka does not qualify as an EA since lacks crucial features. Namely, maintaining and evolving a population of candidates and employing genetic operators like mutation, crossover, and selection. Eureka only mutates the best candidate and discards the rest. In contrast, REvolve truly evolutionary, i.e., nopulation-basee and using the full range of genetic operators. By preserving genetic diversity, REvolve avoids the pitfalls of premature convergence without incurring additional costs (since both frameworks generate the same number of candidates per generation). Empirically, we show that utilizing the full range of evolutionary operators yields reward functions that lead to better performing agents. Consequently, REvolve significantly outperforms greedy search methods like Eureka and T2R.
Evolutionary Algorithms with LLMs. Implementing EAs with LLMs has been explored in recent works in the context of neural architecture search (Nasir et al., 2024), program search (Romera-Paredes al., 2024), prompt engineering (Guo et al., 2024; Fernando et al., 2023). as well as morphology design (Lehman et al.. 2024). In this work, we propose novel evolutionary framework for reward design with LLMs as intelligent genetic operators. Compared the above-mentioned works, which feature limited evolutionary settings, REvolve incorporates a wide range of operations like mutation, crossover, selection, and migration.
5 EXPERIMENTS
5.1 EXPERIMENTAL SETUP
We performed experiments across three different environments: autonomous driving, humanoid locomotion, and adroit hand manipulation. For autonomous driving, we used the high-fidelity

---

Published as a conference paper at ICLR 2025
Table 1: Summary of observation and action spaces, along with objectives and termination criteria for the different RL training tasks.
Task
Observation Space
Action Space
Objective & Termination
Autonomous Driving
1t [2807206
discrete, |A|
66
Navigate for 1000 steps
Humanoid Locomotion
12776
continuous, A E RR
Run upright for 1000 steps
Adroit Hand Manipulation
R:30
continuous, E TR:30
Open the door in 400 steps
AirSim simulator (Shah et al., 2018). For humanoid locomotion (Tassa al., 2012) and adroit hand manipulation (Rajeswaran et al., 2017), we used the MuloCo simulator (Todorov et al., 2012). As a reward designer G, we used GPT-4 Turbo (1106-rr))ww model, which was provided (a subset of) observation and action variables originating from the simulators as illustrated in Figure 1. The reward designer is prompted to provide a reward taking the functional form r = >. rs. We call the is reward components and the reward designer decides on their individual functional form, well as the number of reward components. This follows the protocol established by Ma et al. (2024a).
5.2 TRAINING SETUP
For the evolutionary search, we set the number of generations to N = 7 and individuals per generation to K = 16. This mimics the training setup of (Ma et al., 2024a). The number sub-populations was set to 1 = 13, and the mutation probability was set to pm. 0.5.
For each generation, 16 policies were trained (one per reward function). In the AirSim environment, we trained Clipped Double Deep Q-Learning (Fujimoto et al., 2018) agents for 5 x 10th training steps per generation per agent. For the MuJoCo environments, we trained Soft Actor- 2018) agents with x 106 steps per generation per agent. We summarize the different RI. tasks in Table and provide detailed descriptions of the environments and RI. setups in Appendix A.
(Haarnoja et al.,
Parallel training of single generation 16 NVIDIA A100 GPUs (40GB) consumed approximately 50 hours for the AirSim environment and 24 hours for the MuJoCo environments. Due to these extensive computational demands, report results using two random seeds.
In addition to REvolve (with human feedback), we also include a version in the experimental evaluation that we dub REvolve Auto. Here, the human feedback is replaced with automatically generated feedback comprising statistics on the reward components tracked during RL training. The fitness score for each agent now computed using a manually designed fitness function, which measures an agent's successful completion of the task. For instance, the fitness score for the humanoid locomotion task encodes that the agent should not fall over and that should attain high average velocity. We formally describe the fitness scores for the three different environments in Appendix B.2. This automated feedback mechanism adopted from (Ma et al., 2024a).
5.3 BASELINES
Expert Designed. For autonomous driving, we used an expert reward function based on best practices established in the literature (Spryn al., 2018). The total reward is calculated through a weighted aggregation of all the components, with the weights optimized via an exhaustive grid search conducted within the environments to find the reward function that produces the best policy. For humanoid locomotion and adroit hand manipulation, used expert rewards provided by Gymnasium and Gymmasium-Robotte respectively. Details on the expert-designed reward functions can be found in Appendix C.
Eureka. We also compare REvolve to Eureka, cf. Section 4. Akin to REvolve, we employed the human fitness function to refine the best individual with natural language feedback. Henceforth, denote it as Eureka. We also include the original version Eureka with automated feedback, which we denote by Eureka Auto. Training times are identical to the ones for REvolve.
2To prevent potential plagiarism through context memory (Carlini et al., 2021), we unique environment variable names and deliberately avoid mentioning the simulator in inputs GPT-4. Shttps:// // htttp::://ggmaassst// umm farama. ottpsttss: httppp:::///roorcccccceoo farama.
7

---

Published as a conference paper at ICLR 2025
REvens
BEcks
Purzan Drinig
Export Daignsd
Booii Ado
Randee
Eoot Ocaggned
1.0
1.00
0.75
Fitness score
0.8
Fitness score
Fitness score
0.50
0.25
1
5
0.00
3
5
Generation
Generation
Generation
We the fitness score over 2 random seeds of the best- rest-performing policy per generation for autonomous driving (left), humanoid locomotion (middle), and adroit hand manipulation (right), cf. Table 1. Each plot also contains the fitness score obtained using an expert-designed reward function. The fitness score for the human driving the autonomous driving setting (left) was obtained by observing the driving behavior of an expert human conducting driving sessions in a combined setting of AirSim and the high-fidelitg Logitech driving simulator. We treat this as the upper limit of performance. We give the formal definitions of the fitness functions for each of the three environments in Equations 2, 3, and 4, respectively (in Appendix B.2).
Figure 3:
5.4 EXPERIMENTAL QUESTIONS & RESULTS
As shown in Figure 3, REvolve exhibits continuous improvement over successive generations, ultimately achieving higher fitness score on the manually designed fitness scale. Furthermore, both REvolve and REvolve Auto outperform their Eureka counterparts, indicating that evolutionary search strategies yield better results than greedy search methods We also show that all methods outperform the Expert Designed reward function. Furthermore, we observe the benefit of providing human feedback, as both REvolve and Eureka with provided human feedback outperform their automatic feedback counterparts.
How does REvolve fair against the baselines?
We compare REvolve to the baselines using the task-specific fitness score described in Appendix B.2 and report the results in Figure 3. We observe that REvolve (with human feedback) consistently outperforms the other methods across all three environments and is only beaten by the expert human driver in Autonomous Driving, cf. Figure 3 (left). For the interested reader, we give the best (i.e., fittest) REvolve-designed reward functions R2 in Appendix G.
We also note that in the Humanoid Locomotion and Adroit Hand tasks REvolve Auto (i.e., with automated feedback) performs on par with Eureka (with human feedback), despite the former's evolutionary setting. This could be attributed to the limited and sparse nature of automated feedback (velocity for humanoid, steps to succeed for adroit hand) in contrast to the more detailed human feedback in Eureka. In contrast, automated feedback in Autonomous Driving is dense and incorporates multiple factors like collision, speed, and distance (Appendix B.2) - leading to bigger improvements during the evolutionary search. This also highlights that the combination of evolutionary search and human feedback is crucial for optimizing reward functions. can be observed that the improvements gradually diminish with each generation and stabilize, converging by generation 5.
How do humans judge REvolve policies?
We further compared REvolve and Eureka human evaluations. To this end, we generated roll-outs from the best performing policies (x)) from REvolve and Eureka. We paired them against each other and used the user feedback interface (Figure 1) for collecting preference data. This data was subsequently passed through an Elo rating system to rank each behavior, ef. Appendix B.I. As shown in Table 2, REvolve policies again outperform Eureka policies across all three tasks, and it is only the human expert driver beating REvolve in Autonomous Driving. Note, we used a fresh batch of
*Note that these results are not displayed on the human fitness scale because the Elo ranking system evaluates individuals relative to one another. Therefore, comparisons between individuals from different baselines are only feasible if we directly pair them against one another.
8

---

Published as a conference paper at ICLR 2025
Table 2: The table shows the best policies x* from each framework ranked based human preferences for each environment. Specifically, we use rollouts from the best policies of each framework and gather human preferences on the paired rollouts, described in Section 3.3. Then, we use the Elo rating system to map preferences to scores. All policies are initially assigned Elo score of 1500.
Environment
Rank
Baseline
Elo Score
Autonomous Driving
Human Driving
1586
2
REvolve
1575
3
REvolve Auto
15299
4
Eureka
1499
Eureka Auto
1411
6
Expert Designed
1397
Humanoid Locomotion
REvolve
1586
2
Eureka
1557
3
REvolve Auto
1514
4
Eureka Auto
1434
Expert Designed
1407
Adroit Hand Manipulation
1
REvolve
1594
REvolve
1524
3
Eureka
1522
4
Eureka Auto
1443
5
Expert Designed
1415
evaluators (disjoint from training) for this stage. We have submitted example rollout videos as part of the supplementary material.
5.5
ABLATION STUDIES
Table 3: Table shows the generalization scores in unseen AirSim environments. Fitness denotes manually designed fitness scores (mean std). Env 1 & Env 2 have 1500 & 1000 episodic steps, respectively.
1.0
Crossever
Mutation
Crossover
Mutation
Fitness score
0.8
Env
Baseline
Fitness
Episodic Steps
0.7
Expert Designed
0.71 # 0.03
710 # 199
0.6
Human Driving
0.95 = 0.02
1500 = 0
REvolve
0.86 + 0.03
1146 + 297
0.5
Eureka
0.73 = 0.04
759 167
0.4
Expert Designed
0.48 + 0.02
71 22
Generation
2
Human Driving
0.75 = 0.04
1000 = 0
REvolve
0.68 + 0.04
135
Figure 4: Comparison of REvolve's per- formance on Autonomous Driving using mutation only, crossover only, and both combined, over 7 generations.
Eureka
0.57 + 0.05
99 1
How do different genetic operators impact overall performance?
We evaluated REvolve's performance Autonomous Driving by running evolution search with three setups: using only mutation, only crossover, and the combination of both. As shown in Figure 4, REvolve with mutation crossover outperforms REvolve with only crossover, which in turn outperforms REvolve with just mutation.
Do REvolve-designed reward functions generalize to new environments?
Aside from our standard Autonomous Driving environment (seen in Figure 5), assessed the generalizability of the best REvolve-designed reward functions R" in two new (AirSim) environments. Specifically, trained policies from scratch using R' in new scenarios: (Env 1) featuring

---

Published as a conference paper at ICLR 2025
lanes and a completely altered landscape while maintaining similar traffic conditions environment, and (Env 2) characterized by increased traffic with multiple cars actively maneuvering, From Table observe that in both environments, REvolve outperforms Eureka and expert-designed rewards. However, the generalizability was lower for Env 2. This issue may stem from the temporal processing limitations of the RI. policy's visual module, which sometimes fails to detect oncoming cars. Impiementing more advanced models could enhance detection capabilities Hazra et al. (2023). Figure provides visual representations of these environments, while Figure 13 provides episodic steps on the Y-axis over total steps on the X-axis during training for both generalization environments. We did not conduct similar experiments on the MuJoCo simulator, as the Humanoid and Adroit Hand environments are limited to only single variant.
standard
5.6 LIMITATIONS
As stated in Section 5.2, due to its (dcep) RL nature, training multiple policies is rather computationally expensive and sime-consuming That said, the computational cost of REvolve is the same for Eureka with no resources wasted on discarding individuals REvolve does incur a slightly higher storage cost due to the maintenance of the reward database (see Appendix D). Another limitation of REvolve is its reliance on the closed source GPT-4 model. This was a necessary choice as existing works have shown that considerable gap exists in reward design abilities between GPT-4 and open-source LLMs (Xie et al., 2024; Ma et al., 2024a). With rapid advancements made by the open-source community, we believe this gap will be bridged in the near future.
6 CONCLUSION & FUTURE WORK
We formulated the RDP as a search problem using evolutionary algorithms for searching through the space of reward functions (Section 2). With REvolve, we proposed a novel evolutionary framework that leverages human preferences as fitness scores to guide the search process and integrates LLMss as intelligent operators to refine outputs through natural language feedback (Section 3). Our results show that this evolutionary search-based framework considerably outperforms the state of the art across three different simulation environments.
key future question for REvolve is its real-world applicability. One promising approach involves integrating REvolve with Sim2Real algorithms Allamaa al. (2022); Hu et al. (2024) by first learning a policy in simulation, using NEvolve-designed rewards, and then transferring it to the real world. Recently, Ma et al. (2024b) have demonstrated that such an approach is viable for Eureka by tackling a quadruped balancing task on a physical robot. Moreover, our framework opens up opportunities for future research in Human-Al interaction. In particular, exploring the impact of more open-ended feedback from various perspectives, such as egocentric or feird-pesoon views.
ETHICS STATEMENT
During human feedback experiments, we engaged 10 human evaluators per generation, each assessing 20 video pairs. Participants were recruited on volunteer basis and the preference learning task was clearly explained to them. No personal data was collected, and participants were compensated fairly per institutional guidelines. Our study adhered to ethical standards, followed institutional guidelines, and did not involve any potential risks related to privacy, security, or discrimination.
REPRODUCIBILITY
In the Appendix, we provide extensive details on our experimental protocol. Specific details can be found in the respective sub-appendices as detailed below. The reinforcement learning framework. including the algorithm, inputs, outputs, and hyperparameters, is outlined in Appendix A. The fitness functions are thoroughly discussed Appendix B. The expert-designee rewards are explained in Appendix C. Detailed prompt examples for GPT-4 can be found in Appendix E. Finally, list the best reward functions generated from REvolve in Appendix G. Our code is soun-sourced at https//gittub..com hww.:etthhhh...eeeeem wwwpoa...ooooooioiiaieeee Rishil Hazra laarra/Revove//tre//maai
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
The computations and data handling for this work were enabled by the Berzelius resource at the National Supercomputer Centre, provided by the Knut and Alice Wallenberg Foundation, under projects "Berzelius-2024-178 and "Berzelius-2023- 348". This research was supported by the Wallenberg AI, Autonomous Systems, and Software Program (WASP), also funded by the Knut and Alice Wallenberg Foundation. Additionally, this work is part of the NEST Project, multi-dimensional Alignment and Integration of Physical and Virtual Worlds (main). Finally, we thank Lue De Raedt for his valuable feedback in refining the manuscript.
REFERENCES
Jean Pierre Allamaa, Panagiotis Patrinos, Herman Van der Auweraer, and Tong Duy Son. Sim2real for autonomous vehicle control using executable digital twin. IFAC-PapersOnLiie, 55:385-391, 2022. doi: /attticlee/p//222559922223333332
10.314. URL https://www. sciencedirect.
htt//siienne/rrticc Automotive Control AAC 2022.
10th IFAC Symposium on Advances in
Serena Booth, W Bradley Knox, Julie Shah, Scott Niekum, Peter Stone, and Alessandro Allievi. The perils of trial-and-error reward design: misdesign through overfitting and invalid task specifications. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 5920-5929, 2023.
Erick Cantá-Paz et al. A survey of parallel genetic algorithms. Calculateurs paralleles, reseaux et systems repartis, 10(2):141-171, 1998.
Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ülfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pp. 2653.2650. USENIX Association, August 2021, ISBN 978-1-939133. 24-3. URL ptctps://wwiit /ttp:////wwwwuuii...cmmmmmm www.us..ni........ wwtp:/useni...ogcoooooo http:///nnnttrrrr//////sssssstta ieliiiiexxtaattnng
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6299-6308, 2017.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss. William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin. Suchir Balaji. Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained code. arXiv preprint arXiv:2107.03374 2021.
Tianheng Cheng, Lin Song, Yixiao Ge, Wenyu Liu, Xinggang Wang, and Ying Shan. Yolo-world Real-time open- vocebvocabulary object detection. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2024.
Jon Christian. Amazing "jailbreak" bypasses chatgpt's ethics safeguards. Futurism, February, 4:2023, 2023.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
Kenneth De Jong. Evolutionary computation: a unified approach. In Proceedings of the Genetic and Evolutionary Computation Conference Companion, pp. 373-388, 2017.

---

Published as a conference paper at ICLR 2025
Google DeepMind. AlphaProof: Al achieves melver- standard solving International Mathematical Olympiad problems. Jul 2024. URL https:///wwr wwtpiw.deeiiid...mm www.coom//bace boog//
Arpad E. Elo. The Rating of Chessplayers, Past and Present. Arco Pub., New York, 1978. ISBN 0668047216 9780668047210.
Alhussein Fawzi, Matej Balog, Aja Huang, Thomas Hubert, Bernardino Romera-Paredes, Moham- madamin Barekatain, Alexander Novikov, Francisco J R Ruiz, Julian Schrittwieser, Grzegorz Swirszez, David Silver, Demis Hassabis, and Pushmeet Kohli. Discovering faster matrix multipli- cation algorithms with reinforcement learning. Nature, 610(7930):47-333 2022.
Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rocktäschel. Promptbreeder: Self-referential self-improvement via prompt evolution, 2023.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pp. 1587-1596. PMLR, 2018.
Leo Gugerty and Gary Olson. Debugging by skilled and novice programmers. In Proceedings of the SIGCHI conference on human factors in computing systems, pp. 171-174, 1986.
Qingyan Guo, Rui Wang, Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yujiu Yang. Connecting large language models with evolutionary algorithms yields powerful prompt optimizers. In The Twelfth International Conference on Learning Representations, 2024. net/ forwmmid--ZGRaaaSa..
URL
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In Proceedings of the 35th International Conference on Machine Learning, pp. 1861-1870, 2018.
Dylan Hadfield-Menell, Anca Dragan, Pieter Abbeel, and Stuart Russell. Cooperative inverse reinforcement learning. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS' 16, pp. 3916-3924. Curran Associates Inc., 2016.
Dylan Moddfeld-Menelle Smitha Milli, Pieter Abbeel, Stuart Russell, and Anca Dragan. Inverse reward design. Advances in neural information processing systems, 30, 2017.
Shibo Hao, Yi Gu, Haodi Ma, Joshua Hong, Zhen Wang, Daisy Wang, and Zhiting Hu. Reasoning with language model is planning with world model. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 8154 8173, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 1/033.///222..mmlppppp 507. URL https: / / aal/aalllntoooooyyyoooooo
2023/ emnlp-main matp-mmii.50000
Rishi Hazra, Brian Chen, Akshara Rai, Nitin Kamra, and Ruta Desai. Egotv: Egocentric task verification from natural language task descriptions. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 15417-15 1447995429, October 2023.
Rishi Hazra, Pedro Zuidberg Dos Martires, and Luc De Raedt. Saycampay: Heuristic planning with large language models using learnable domain knowledge. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 20123-20133 2024.
John Houston, Guido Zuidhof, Luca Bergamini, Yawei Ye, Long Chen, Ashesh Jain, Sammy Omari, Vladimir Iglovikov, and Peter Ondruska. One thousand and one hours: Self-driving motion prediction dataset. In Conference on Robot Learning, pp. 409 409. PMLR, 2021.
Xuemin Hu, Shen Li, Tingyu Huang, Bo Tang, Rouxing Huai, and Long Chen. How simulation helps autonomous driving: A survey of sim2real, digital twins, and parallel intelligence. IEEE Transactions on Intelligent Vehicles, 9(1):593 612, 2024. doi: 10. 10.11009/TV.22233331777
Steven Kerr. On the folly of rewarding a, while hoping for b. Academy of Management journal, 18(4): 769-783, 1975.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo. Piotr Dollar, and Ross Girshick. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4015-4024 October 2023.
12

---

Published as a conference paper at ICLR 2025
W Bradley Knox and Peter Stone. Interactively shaping agents via human reinforcement: The tamer framework. In Proceedings of the fifth international conference on Knowledge capture, pp. 9-16, 2009.
W Bradley Knox, Alessandro Allievi, Holger Banzhaf, Felix Schmitt, and Peter Stone. Reward (mis) design for autonomous driving. Artificial Intelligence, 316: 10382,, 2023.
Minae Kwon, Sang Michael Xie, Kalesha Bullard, and Dorsa Sadigh. Reward design with language models. In The Eleventh International Conference on Learning Representations, 2023. URL https://opperrevew
William B Langdon and Riccardo Poli. Foundations of genetic programming- Springer Science & Business Media, 2013.
Joel Lehman, Jonathan Gordon, Shawn Jain, Kamal Ndousse, Cathy Yeh, and Kenneth O. Stanley. Evolution Through Large Models, pp. 331 Springer Nature Singapore, Singapore, 2024. ISBN 978-981-99-388 11--9999844-4- 14-8. doi: 10. 0111779788-81-9988111 98---9---88888888 11. URL https: http://ooo..org1/1//// org/10. 00007/978-88811 -811-993311-8-1111111-1--1---1--
Tennison Liu, Nicolás Astorga, Nabeel Seedat, and Mihaela der Schaar. Large language models to enhance bayesian optimization. In The Twelfth International Conference on Learning
Representations, 2024. URL wwwph::t///oo/ httb:::///ppeeeviert
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoe Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. In International Conference on Machine Learning, pp. 22631-22644 PMLR, 2023.
Yecheng Jason Ma, William Liang, Guanzhi Wang. De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eureka: Human-level reward design via coding large language models. In The Twelfth International Conference Learning Representations, 2024a. URL httt:::///ppeeeiieeee
Yecheng Jason Ma, William Liang, Hung-Ju Wang, Yuke Zhu, Linxi Fan, Osbert Bastani, and Dinesh Jayaraman. DrEureka: Language Model Guided Sim-To-Real Transfer. In Proceedings of Robotics: Science and Systems, Delft, Netherlands, July 2024b. doi: 10. 10.1 5607/55SSS..... 2T1.7//RSS.222..X.O0 2024.XX.094
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegr- effe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bod- hisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with seld-feed In Advances Neu- Curran Associates, wearr/flleeeeee//eeeeeee http:/ppprr222//E11/// /file/
ral Information Processing Systems, volume 36, pp. saapperrfillseeeeeeeeeeeeeeeeeee Inc., 2023. URL. https::/proceedings. wwwp://rrrrreiiiin...eoooo...pp. cc Paaerrrreeeeeeeeeeeeeeeeeeeeeee pdf.
Daniel J. Mankowitz, Andrea Michi, Anton Zhernov, Marco Gelmi, Marco Selvi. Cosmin Paduraru, Edouard Leurent, Shariq Iqbal, Jean-Baptiste Lespiau, Alex Ahern, Thomas Köppe, Kevin Millikin, Stephen Gaffney, Sophie Elster, Jackson Broshear, Chris Gamble, Kieran Milan, Robert Tung, Minjae Hwang. Taylan Cemgil, Mohammadamin Barekatain, Yujia Li, Amol Mandhane, Thomas Hubert, Julian Schritwieser, Demis Hassabis, Pushmeet Kohli, Martin Riedmiller, Oriol Vinyals, and David Silver. Faster sorting algorithms discovered using deep reinforcement learning. Nature, 618(7964):257-265 June 2023. URL https: org/ 10. 1038/ 0013/// 023-06004-9.
Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 1rXi5:1312.5602, 2013.
Muhammad Umair Nasir, Sam Earle, Christopher Cleghorn, Steven James, and Julian Togelius. Llmatie: Neural architecture search via large language models and quality diversity optimization. In Proceedings of the Genetic and Evolutionary Computation Conference, 2024.
Stephen M Omohundro. The basic ai drives. In Artificial intelligence safety and security, pp. 47-55. Chapman and Hall/CRC, 2018.
13

---

Published as a conference paper at ICLR 2025
OpenAl. Chatgpt, 2022. https://openai.cem http:///ppeaaa..cmmmmddd///aaggp
Michael O'Neill, Leonardo Vanneschi, Steven Gustafson, and Wolfgang Banzhaf. Open issues in genetic programming. Genetic Programming and Evolvable Machines, 11:339-363, 2010.
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4, arXiv preprint urXiv:2304.03277 2023.
Michael Polanyi. The tacit dimension. In Knowledge in organisations, pp. 135-146. Routledge, 2009.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. In A. Oh, T. Naumann, A. Globerson, K. Saenko. M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 53728-53777 7777-13741. Curran Associates, Inc., 2023. URL https: //poceeeeiccccccccc/oeoo newproeeeedinngs.nii........ppp a8 35554008666666474744333335566663 Paper Conference. pdf.
neurips. ccc.ppsccc//p/ cccccc//paaa paperp fameerffllsss www.s//plleeeeee /tlls//pprr//223/////// /fllf//
Antonin Rafin,, Ashley Hill, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, and Noah Dormann. babble-baselines:: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22:1-8, 2021.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv: /709. 10007 2017.
Bernardino Coomera-Paredes Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan $ Ellenberg, Pengming Wang, Omar Fawzi, et al. Mathematical discoveries from program search with large language models. Nature, 625 (7995):468-477 2024.
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308. 12950, 2023.
Stuart Russell. Human compatible: Al and the problem of control. Penguin Uk, 2019.
William Saunders, Girish Sastry, Andreas Stuhlmüller, and Owain Evans. Trial without error: Towards safe reinforcement learning via human intervention. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS 18, pp. 2067-2069, Richland, SC, 2018. International Foundation for Autonomous Agents and Multiagent Systems.
Stefan Schaal. Learning from demonstration. Advances neural information processing systems, 9, 1996.
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint 17X052257. 2015.
Shital Shah, Debadeepta Dey, Chris Lovett, and Ashish Kapoor. Airsim: High-fidelity visual and physical simulation for autonomous vehicles. In Field and Service Robotics: Results of the 11th International Conference, pp. 621-635, Springer, 2018.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: language agents with verbal reinforcement learning. In Advances in Neu- ral Information Processing Systems, volume 36, pp. 8634-8652. Curran Associates, www..ps.cccppppe.........peee
Inc.. 2023. URL https: neuriin cc cwpppperpppppppeeeeepeeeeeeeeeep /aa.ee/ppaer/223//// /file/ 1b44b878bb782cm hotp:
David Silver, Thomas Hubert, Julian Schrittwieser, Ioamnis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419): 1140-1144, 2018.
14

---

Published as a conference paper at ICLR 2025
Satinder Singh, Richard L Lewis, and Andrew G Barto. Where do rewards come from. In Proceedings of the annual conference of the cognitive science society. pp. 2601-2606. Cognitive Science Society, 2009.
Mitchell Spryn, Aditya Sharma, Dhawal Parkar, and Madhur Shrimal. Distributed deep reinforcement learning on the cloud for autonomous driving. In Proceedings of the 1st International Workshop on Software Engineering for Al in Autonomous Systems, pp. 16-22, 2018.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Yuval Tassa, Tom Erez, and Emanuel Todorov. Synthesis and stabilization of complex behaviors through online trajectory optimization. 2012 IEEF/RSJ International Conference on Intelligent Robots and Systems, pp. 1906-4913. IEEE, 2012.
Andrea L. Thomaz and Cynthia Breazeal. Teachable robots: Understanding human teaching behavior to build more effective robot learners. Artificial Intelligence, 172(6):116-377, 2008. ISSN 0004- 3702. doi: https://doi.org// wttp:/oii.rr10.0.......... 101.1011....... artint
0ttg://..001//...rtt..0001.....0 200.009.0000000000000000000 .09.009. URL hhttps://wwir sciencedirect..
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pp. 5026-5033, IEEE, 2012.
Alex Turner, Logan Smith, Rohin Shah, Andrew Critch, and Prasad Tadepalli. Optimal policies tend to seek power. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances Neural Information Processing Systems, volume 34, pp. 23063- 23074. Curran Associates, Inc., 2021. URL httpp::/ppptceeeeee wewpriisnnnneui...eeommmmm. courips.ccppape... fi
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
Oriol Vinyals, Igor Babuschkin, Wojciech M. Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John P. Agapiou, Max Jaderberg, Alexander S. Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom L. Paine, Caglar Gulcehre, Ziyu Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Chris Apps, and David Silver. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575 (7782):350-354, 11 2019. doi: 00.1038//41586-000 1724-z. URLhttts::/ http://ooooorgg///// 1038/ 541586-019 1724-2.
Fan Wang, Bo Zhou, Ke Chen, Tingxiang Fan, Xi Zhang. Jiangyong Li, Hao Tian, and Jia Pan. Intervention aided reinforcement learning for safe and practical policy optimization in navigation. In Conference on Robot Learning, pp. 410-421. PMLR, 2018.
Yi Wang, Jiafei Duan, Dieter Fox, and Siddhartha Srinivasa. NEWTON: Are large language models capable of physical reasoning? In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 9743-9758, Singapore, December 2023. Association for Computational Linguistics. doi: (0.18653/v1/2000 http:2/...n.n.nnn..eeee madings-emmlp.6652 URL https: httpss:///claaathoooooo .wwpogg.orr//// wrtp/023...o./ 2222./223322 eeaehnngs-emnl.... 652.
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Mare Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In International conference on machine learning, pp. 1995-2003. PMLR, 2016.
Tianbao Xie, Siheng Zhao, Chen Henry Wu, Yitao Liu, Qian Luo, Victor Zhong, Yanchao Yang, and Tao Yu. Text2reward: Reward shaping with language models for reinforcement learning. In The Twelfth International Conference on Learning Representations, 2024, URL https: //openreview htt://furmmiddttt3000
15

---

Published as a conference paper at ICLR 2025
Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoe Le. Denny Zhou, and Xinyun Chen. Large language models as optimizers. The Twelfth International Conference on Learning Representations, 2024. URL https://openrrview wwtp:/eetnn///... /ttr/f/mmdd--bAVGWWEEL
Wenhao Yu, Nimrod Gileadi, Chuyuan Fu, Sean Kirmani, Kuang- Huei Lee, Montserrat Gonzalez Arenas, Hao-Tien Lewis Chiang, Tom Erez, Leonard Hasenclever, Jan Humplik, brian ichter, Ted Xiao, Peng Xu, Andy Zeng. Tingnan Zhang, Nicolas Heess, Dorsa Sadigh, Jie Tan, Yuval Tassa, and Fei Xia. Language to rewards for robotic skill synthesis. In 7th Annual Conference on Robot Learning, 2023. URL. https:://opperreiiewww www.eeew.eeeeeeeeeeeeeeeeeeeeeee www.:eew.nneeeettteeeeee hoee/foouumiidssSyyyyee......
Wentao Yuan, Chris Paxton, Karthik Desingh, and Dieter Fox. Sornet: Spatial object-centric representations for sequential manipulation. In Conference Robot Learning, pp. 148-157, PMLR, 2022.
Michael Zhang, Nishkrit Desai, Juhan Bae, Jonathan Lorraine, and Jimmy Ba. Using large language models for hyperparameter optimization. In NeurlPS 2023 Foundation Models for Decision Making Workshop, 2023. URL https://opeereviiie net/ foret/ffrmm?? f d=FUdZ6HEOre.
16

---

Published as a conference paper at ICLR 2025
APPENDIX
The appendix is organized as follows: Section A Reinforcement Learning Preliminaries, where elaborate on the reinforcement learning framework for all environments; Section B Fitness Function, where outline the fitness automatic feedback approach; Section C Expert Designed Reward Function, which provides details of the expert-designed reward and grid search parameters; Section D Cost Analysis, where we compare the computation and data requirement cost of REvolve with Eureka and RLHF on autonomous driving, well as elaborate on the differences between REvolve and other existing approaches for autonomous driving; Section E Prompts, which presents system and operator (mutation, crossover) prompts for GPT-4 for REvolve and REvolve Auto; Section F Human Feedback Examples, which illustrates how human feedback allows GPT-4 to refine the reward function of individuals; Section G Additional Results, which lists the best reward functions generated from REvolve, and some additional results on RL convergence times and generalizability of the best reward functions.
computation from human feedback, well outline Eureka's
We also include our code and demonstrations of policies trained on best REvolve-designed reward functions in the supplementary material.
A REINFORCEMENT LEARNING PRELIMINARIES
A.1 ENVIRONMENTS
A.1.1 AUTONOMOUS DRIVING
Autonomous Driving experiments were conducted in the Neighborhood and Coastline environments of the AirSim simulator, high-fidelity simulator developed for autonomous vehicle research (Shah al., 2018). This urban simulation includes static features such as parked cars and dynamic obstacles like cars and moving animals, enabling thorough evaluation of the RL agent in varied scenarios.
Figure 5:
AirSim simulator environment used for our autonomous driving experiments. The egocentric camera view displayed on the bottom left serves the observations for the autonomous driving agent.
Observations. The observation space for the policy includes a combination of visual and sensor data. The visual input consists images concatenated along the channel axis, resulting in dimensions of 100 x 256 x 12. The sensor data includes readings from yaw, pitch, linear velocity in the and directions, speed, and steering angle. This structure allows the network to perceive temporal dynamics by integrating sequential frames into single input tensor.
17

---

Published as a conference paper at ICLR 2025
Figure 6: AirSim generalization environments used for assessing the generalizability of REvolve- designed reward functions. The left and right pictures visualize Env and Env 2, respectively.
4able DDQN Hyperparameters used during training all AirSim environments the Autonomous Driving task.
PARAMETER
VALUE
LEARNING RATE
0.00025
OPTIMIZER
ADAM
GAMMA
0.99
EPSILON INITIAL
1 0.01
EPSILON
EPSILON DECAY
1 x 10
BATCH SIZE
32
TAU
0.0075
REPLAY BUFFER SIZE
50000
ALPHA (PER)
0.9
FREQUENCY STEPS UPDATE
5
BETA
0.4
ACTION SPACE STEERING (SIMULATOR)
[-0.8, 0.8]
ACTION SPACE THROTTLE (SIMULATOR)
[0.1]
ACTION SPACE (REAL STEERING)
IMAGE RES.
100 256 12
CONY LAYERS
(16-256 FILTERS)
DENSE AFTER FLATTEN
256 > [128 x 2] > 66
SENSOR INPUT DIMENSIONS
6 (YAw, PITCH, Va, Vy, SPEED, STEERING)
The input shape is (100, 256, 12), and it outputs 66 discrete Q-vuues. capture temporal dynamics, frame stacking aggregates four consecutive frames [fri, fr2, frs. frai along the channel dimension into a single state. For subsequent states, oldest frame replaced by a new one, resulting in an updated stack [fr2, fra, [T,
Action Space. The action space for the autonomous car is a 2D vector the first dimension represents the discrete steering angles in the range , in 2 increments, the second dimension is throttling option {0, 13. This results in 66 possible actions.
Objective. The agent's task is to navigate an urban environment for 1000 consecutive time steps, maintaining safe driving behavior by balancing speed and avoiding collisions.
Termination. The episode successfully terminates when the agent achieves the objective. Alternatively, collisions or exceeding speed limits will also prematurely end the episode.
Training Algorithm. Due the high dimensional observation space and convergence time, adopted the Clipped Double Deep Q-Learning approach (Fujimoto et al., 2018), a refined version of Double Q-Leamnng (Van Hasselt et al., 2016). This method uses two separate neural networks to reduce overestimation bias by calculating the minimum of the two estimated Q-values, thus providing more reliable and stable training outcomes.
18

---

Published as a conference paper at ICLR 2025
In addition to Clipped Double Deep Q-Learning, our framework integrates the Dueling Architecture (Wang et al., 2016), which improves the estimation of state values by decomposing the Q-function that estimates the state value and another that calculates the advantages of each
into two streams: action. Such structure allows the model to learn which states are valuable without the need to learn each action effect for each state.
Furthermore, we utilize Prioritized Experience Replay (PER) (Schaul et al., 2015) to focus on significant learning experiences. PER prioritizes transitions with higher Temporal Difference errors for replay, which accelerates learning and increases the efficiency of the memory used during training. Table lists the RL model hyperparameters. Finally, inspired by foundational research, have adapted our architecture to incorporate vision based methods and the Dueling Architecture from Mnih al. (2013) and Wang et al. (2016), enhancing temporal dynamics and action-value precision.
Figure 7: Humanoid Locomotion: A bipedal locomotion task where a simulated humanoid robot must maintain balance and walk fast possible (along the x-axis).
Figure 8: Adroit Hand: A complex robotic ma- nipulation task requiring the dexterous control of a five-fingered robotic hand rotate the latch and pull the handle to open door.
A.1.2 HUMANOID LOCOMOTION
The Humanoid Locomotion environment complex bipedal locomotion task within the MuJoCo simulator (Todorov et al., 2012).
Observations. The observation space for the humanoid environment includes joint positions, joint velocities, and other information such as the center of mass and contact forces. The data crucial for controlling balance and forward locomotion. The observation shape is 376D vector, representing the state of the humanoid at each timestep (Tassa et al., 2012).
Action Space. The action space is 17D continuous vector representing the torques applied the humanoid's 's joints, including those in the abdomen, hips, knees, shoulders, and elbows. This allows finer control of walking or running movements.
Objective. The goal is to walk or run forward without falling.
Termination. The episode ends either when the humanoid falls or 1000 consecutive time steps have passed without falling (success).
Training Algorithm. We employed the Soft Actor-Critic (SAC) algorithm (Haarnoja al., 2018), utilizing the implementation provided by the Stable Baselines3 (SB3) library (Raffin et al., 2021). SAC is an off off-policy actor-critic method that optimizes stochastic policy in an entropy- regulaizeeellrrizeed reinforcement learning framework. By maximizing a trade-off between expected return and policy entropy, SAC encourages exploration and prevents premature convergence to suboptimal policies, making it well-suited for nigh-dimensione continuous action spaces characteristic of MuJoCo tasks.
We leveraged the default hyperparameters provided by SB3 for SAC, which have been empirically validated across various environments. Key hyperparameters included learning rate of 3 x 10-4, batch size of 256, discount factor (Y) of 0.99, and an entropy coefficient (a) automatically adjusted during training. The policy network employed the MlpPolicy architecture from SB3, consisting of multilayer perceptrons designed to process the nigh-dimensional state observations in MuJoCo environments effectively.
19

---

Published as a conference paper at ICLR 2025
A.1.3
ADROIT HAND
The Adroit Hand environment is another MuJoCo simulation task, which concerns dexterous robotic manipulation (Rajeswaran et al., 2017).
Observations. The observation space in the adroit hand environment consists 97D vector. includes joint positions for the hand, latch position, door position (hinge rotation), palm position, handle position, and the relative position between the palm and the handle. Additionally, the space contains joint velocities and forces acting on the hand. The binary door open indicator is also included to signify whether the door is fully opened or closed.
Action Space. The action space is 30D continuous vector that controls the absolute angular positions of the joints the hand and arm. This enables dexterous manipulation tasks like unlatching and swinging open a door.
Objective. The agent's goal is to manipulate the door latch and open the door, overcoming friction and bias torque forces.
Termination. The episode ends when the door successfully opened or 400 consecutive time steps have passed without opening (fail).
Training Algorithm. Same as humanoid locomotion.
B FITNESS FUNCTIONS
In this section, we provide details of (1) REvolve fitness function based on human evaluations, and (2) the automatic feedback (Auto) approach proposed in Eureka.
B.1 REVOLVE FITNESS
As stated in Section 3.3, REvolve employs user interface to gather human feedback on behavior t sampled from a trained policy. Human evaluators review pairs of videos, submitting their preferences. A fitness function then maps this data to real-valued fitness score and natural language feedback ie. (a, A) = Ft)). We start by introducing the Elo rating system that maps the preference data to a fitness score. For two reward function individuals and B with fitness scores A and a respectively, the expected scores EA and En are calculated using the logistic function:
1
E
EB 1 + 10 99)/400
1 10 444400
We then update the individual fitness based on the expected scores.
a = = K. (f(A, EA)
f(X, Y) be defined as:
if the X wins against Y,
f(x. Y)
0 if the X loses Y, 0.5 if the match results in tie.
Here, K 32 determines the magnitude fitness change. Intuitively, higher-ranked individuals lose to those with lower rankings, they incur a more significant penalty, while the reverse scenario leads to a greater increase for the lower-ranker individual. Conversely, when two similarly ranked individuals compete, the change in fitness scores resulting from win loss is smaller. All individuals start with an initial rating of 1500.
For the natural language feedback X, the checkboxes selected by evaluators indicating aspects of the behavior that are satisfactory or require improvement are stitched together using a natural language (NL) template. This final string serves as input for GPT-4.
The process greatly simplifies the feedback interface, as participants only need to choose their preferred video from pair rather than providing absolute rating. Additionally, this method
20

---

Published as a conference paper at ICLR 2025
reduces individual biases that could emerge from various factors, such as the need to adjust ratings at different stages of evolution from the initial phase, where many policies underperform, to later stages, when performances improve. To overcome this, compare individuals not only within the same generation but also across different generations, updating the fitness scores of all individuals the end of each generation In direct rating-based systems, participants must repeatedly rate all generations, adjusting their scales each time. In contrast, our preference-base rating system allows us efficiently utilize all existing preference data across generations to assign new fitness scores, thus requiring minimal data collection for the current generation.
B.2 AUTOMATIC FEEDBACK
Autonomous Driving. The fitness function is primarily designed to evaluate the performance of autonomous vehicle by assessing its behavior in speed regulation, path following, and collision occurrences. It assesses how well the vehicle adheres to speed regulations and how accurately it follows its designated waypoints while penalizing collisions and any deviations from critical operational thresholds. The function is structured as follows:
collision penalty, speed penalty, distance penalty, score(v, d),
if collision occurs
if < Uminn limit or Ummx.limit
d > diai otherwise,
where Cadj_min min = Umin Udhreshold, Cadj max Umax + Uth, with onh small threshold speed allowing for small deviations from the target speed range.
Here, penalties are assigned for: collision penalty= -1 for collisions; speed penalty is calculated based on the deviation from the adjusted range [Vadjusted mins Vadjusted max.. decreasing as the speed moves further away from this range, with the speed score set to 0 if the speed is less than Umin limir or greater than Jimit and distance penalty decreases as vehicle moves further from the intended waypoint, with the distance score set to O if the deviation is above daai- Otherwise, the agent receives score(u, d)-
speed score( distance score(d)
The speed score is calculated as:
if Vadj min S S otherwise
Umax
speed score(v) = max(0, min(le- [lu- Vaginial MM- )
Maljaras Valjanio
The speed range is |8.0, 11.5] m/s. The distance score is calculated as:
[1, max(0, 1
if d S dmax otherwise
distance score(d)
d=duss) ) dhaux
with dmaz = 0.5m being the maximum tolerance for deviation from the waypoint. Both functions are shown in Figure In Table 5, we summarize the fitness function components used for evaluating RL models.
Humanoid Locomotion. The fitness function for the Humanoid environment evaluates the agent based on its ability to maintain forward locomotion without falling over an episode of Tima 1000 steps.
Σ
T nos
if T max
Tax 0.
(3)
if T < T Trax
*This approach is akin to comparing chess grandmasters Gary Kasparov and Magnus Carlsen, each dominant player in his respective era. However, a direct comparison would require them to be matched in their primes; otherwise, the basis of comparison would he skewed and unjustified.
21

---

Published as a conference paper at ICLR 2025
Table 5: Fitness Function Hyperparameters used for calculating the fitness score for AirSim environment for Autonomous Driving task.
Hyperparameter
Description
Value
collision_penaie
Collision penalty
-1
Umin
Minimum speed target Maximum speed target
9.0 m/s
Umax
10.5 m/s
Jimi
Minimum penalty limit
2.5 m/s
limit
Maximum penalty limit
15 m/s
Vadjusted min
Adjusted minimum speed
8 m/s
Uadjusted max
Adjusted maximum speed
11.5 m/s
0.
Threshold Speed
m/s
dmax
Waypoint deviation tolerance Maximum tolerance for deviation from the waypoint
0.5
drail
4 m
1.0
Fireet
14
litret
Speed fiznas
Datance max
Cadtaaadd mi
Lotiooiil max
Finees Score
Filness Scate
22-
0.-
20
0.0
3.0
10.0
12.5
13.0
Spredd (v)
Distance from Waypoint (d)
Figure 9: Visual representation of speed and distance fitness scores for autonomous driving in the AirSim environment.
where, oi is the forward velocity along the x-axis time step T is the total number of steps the agent remained upright. This function rewards agents that achieve higher forward velocities while also surviving the entire episode.
Adroit Hand. The fitness function for the Adroit Hand environment rewards agents based on the number of steps taken successful task completion. As shown in Figure 10, completing the task in fewer steps results in higher fitness score E [0.5, 1.0]. If the task is not completed, the fitness score
Let steps be the number of steps taken to complete the task (50 S steps S 400), Ti
50 be
the minimum possible steps, Tmax 400 be the maximum allowed steps, 1 b 75
700'
70
constants defining the linear relationship. The fitness score a is defined as:
a x steps + b, if the task is successfully completed the task is not completed
0,
This score measures the efficiency of the Adroit Hand in completing the task, awarding perfect score of for completing it in 50 steps or fewer, with linear decrease to 0.5 for completing the task in the maximum allowable 400 steps, and score of if the task is not completed.
Feedback. Following Eureka, we track the statistics of all reward components at intermediate policy checkpoints throughout training. We refer the interested reader to the original paper (Ma et al., 2024a).
22

---

Published as a conference paper at ICLR 2025
Fitness Score
Adroit Hand Fitness Score
0.8
0.7
50
100
150
200
250
300
350
400
Number of Steps Complete Task
Figure 10: Visual representation of fitness score for Adroit Hand when completing the task.
Table 6: Grid Search Weights: Values and Corresponding Fitness Scores. The quadruple of weights yielding the highest fitness score is highlighted in bold.
FITNESS ScoRE
0.85
0.46
0.5
0.53
0.496
0.25
0.25
0.25
0.25
0.603
0.81
0.83
0.33
0.13
0.606
0.35
0.67
0.7
0.67
0.576
0.57
0.26
0.57
0.27
0.396
0.22
0.16
0.24
0.85
0.556
0.67
0.3
0.78
0.77
0.549
0.45
0.11
0,256
0,72
0.93
0.74
0.65
0.415
0.63
0.4
0.39
0.78
0,284
0.44
0.76
0.98
0.48
0.653
0.28
0.64
0.602
0.12
0.52
0.91
0.592
0,94
0.86
0.85
0,2
0.571
0.51
1.0
0.64
0.34
0.630
0.98
0.56
0.12
0.95
0.600
0.88
0.38
0.59
0.591
C EXPERT DESIGNED REWARDS
C.I AUTONOMOUS DRIVING
Following the methodologies advocated in Spryn al. (2018), the total reward at each time step. Rooat, is constructed from a weighted sum of speed consistency, smooth navigation, and collision avoidance behavior of the agent. In what follows, describe how the weights were calculated.
Human Expert Designed (HED): Utilizes combination of position, speed, sensor, and smoothness rewards.
Intal
HED
Cpos Tpos + Csm
Cspeed X Tspeed + Csensor X Tsensor
POSITION REWARD: Encourages the vehicle stay close to the target position, rewarding proximity to the desired path or waypoint.
8 ( 20)
Tpus
23

---

Published as a conference paper at ICLR 2025
Table 7: Hyperparameters used in all reward components designed by Expert AirSim in Autonomous
Driving task.
Hy Hyperparameter
Description
Value
Position reward decay factor
0.2
Position reward bias
0.25
Umin
Minimum target speed
9 m/s
U Umax
Maximum target speed
10.5 m/s
dis safe
Safe distance threshold
6 m
Smoothness penalty coefficient
0.5
SPEED REWARD: Promotes maintaining an optimal speed. penalizes the vehicle moves too
slowly or too quickly, encouraging it to stay within a specified speed range.
Ispeed Speed--Man Paiil
if speed < Mmin
Vnin
T "peed
speed Uyuel speed
if speed >> Umax otherwise
SENSOR REWARD: Encourages the vehicle to maintain a safe distance from obstacles using sensor data. If the distance measured by the sensor is less than safe threshold, it applies penalty. The sensor measures the distance from the vehicle to nearby front objects.
(disute distance)
distance < dissate otherwise
T Sennsor
diéute
0.5
SMOOTHNESS REWARD: Penalizes abrupt changes in steering to encourage smooth driving. It uses the standard deviation of the last four steering actions measure smoothness.
sm - std(action_lis)
Human Expert Designed Tuned (HEDT): We used grid search to optimize the values of the hyperparameters Cpos: Camy Cspood, Csensor-
7 total
HEDT opt pos
X Tpos
opt Can
X Tam +
opt speed Tspeed +
opt semsor
T Tsensor
These configurations allow for tailored training of autonomous driving agents, emphasizing different aspects of driving performance based on specific needs and expected driving conditions. All configurations use weights of Cposs Camy Cspeed, Csensor 0.25, except for the optimized version which uses customized weights.
C.2 HUMANOID LOCOMOTION
In the humanoid environment, the agent is tasked with moving forward while maintaining healthy state. The reward function is designed to encourage forward movement, staying upright, and minimizing control effort. The total reward R is computed
R Riarward Riealthy
Reontrol
Where:
Rforward 2.5 X Wodd x.velocity
This term provides a reward proportional to the agent's forward velocity, incentivizing fast forward movement, with Wf 1.25.
24

---

Published as a conference paper at ICLR 2025
Rhealhyy is reward for maintaining healthy state, where the agent's height is within predefined range:
15
if the agent is healthy
Rheatthy
if the agent is unhealthy
Reontrol = 0.5 x Weut x ||action|
A penalty applied based on the magnitude the agent's actions, discouraging excessive or inefficient control efforts with Weat 0.1.
The total reward encourages the agent to move forward efficiently while staying upright and minimizing unnecessary control actions. The episode terminates if the agent becomes unhealthy (e.g.. by falling) or after maximum number of steps.
C.3 ADROIT HAND
In the Adroit Hand Environment, the agent's task is to manipulate door handle and open door. The reward function is designed to incentivize both reaching the handle and successfully opening the door. The total reward R is combination of multiple components:
R Raundle + Ratorr Rielocity + Resoous
Where:
Rhandle = 0.1 x ||palm_pp pos - handle pos
This term rewards the agent for minimizing the distance between its palm and the door handle.
Rdoor = 00.1 x distandit - 1.57)²
This component provides reward for moving the door hinge closer to the desired goal position (typically when the door open).
Rueloctty -1 x 10-3 x Σ qvel?
A penalty applied for excessive joint velocities (qvel), encouraging smooth and efficient movements.
Rhonus is set of bonus rewards based on how much the door has been opened:
+2 if goal distance > 0.2
Raonus
+8 if goal_distance 1.0
+10 if goalloooataae distance 1.35 (goal achieved)
The episode terminates when the door is successfully opened (i.e., goal distance > 1.35) or after 400 steps. A sparse reward version of the environment provides a reward of +10 for achieving the goal and -0.1 otherwise.
D REVOLVE COST ANALYSIS & COMPARISONS
REvolve = Eureka. Both REvolve and Eureka output 16 reward functions and train 16 policies in each generation.
Computation cost:
Data requirement: RLHF >> REvolve: RLHF requires massive amounts of curated data to train reward model. Note that the input to a reward model is video (rollout of policy), and the output preference score, requiring heavy neural model. As reference, the common video classification model I3D trains on the Kinetics dataset, which has 650,000 video clips (Carreira & Zisserman, 2017). IRL >> REvolve: Similarly, Inverse Reinforcement Learning requires human demonstrations and interventions. In contrast, REvolve uses 10 human evaluators per generation, each assessing 20 data samples (video pairs) hence, for 7 generations 10 20 x 1400 curated samples only. Compare this to the Lyft self-driving dataset (Houston et al., 2021) used for motion prediction which has more than 1000 hours of data collected over 4 months by 20 cars.
25

---

Published as a conference paper at ICLR 2025
finness scores
scalar rewards
π
R
offline data
(a.) Expert Designed reward function
F
π
rollouts
demonstrations
(d.) RLHF
interventions
fitness
(b.) IRL (LfD Lfi)
+ NL feedback
G
R
ARR
rewards
R
II
rollouts
rollouts
(c.) Human reward function
(e.) REvolve
Figure Comparison of REvolve with existing frameworks used train autonomous driving agents. (a.) Expert Designed reward function R to train the autonomous driving agent; (b.) learning from demonstrations (LfD) and learning from interventions (Lff), where the agent is trained to imitate the human; (c.) humans acting as reward functions within the training loop, by assessing policy rollouts and providing scalar rewards, which requires significant manual effort; (d.) Reinforcement learning from human feedback (RLHF), where human preference data is used to train additional (black-box) reward model. This reward model is used as a proxy for human rewards to train the autonomous driving policy; (e) Proposed REvolve, which uses GPT-4 G and evolves them based on (minimal) human feedback on the rollouts sampled from the trained models. This feedback is directly incorporated into the reward design process. REvolve outputs interpretable reward functions, thereby avoiding learning an additional reward model. Here, т E II is a trainable policy in the of policies II. The trainable blocks are denoted by the symbol of the flame.
reward function generator
Storage Cost: REvolve > Eureka. The slightly higher cost is due to maintaining a population of best candidates rather than one candidate like Eureka. However, compared to the LLM memory cost (e.g.. the biggest open-source LLM Llama 3.1 takes up 810 GB of memory), the storage cost is negligible for every generation, the storage increases by 2.5GB (which includes stored policy weights and other metadata) 17.5 GB after generations. Also, REvolve does not require expert feedback.
Comparison with existing approaches Reinforcement Learning and Human Feedback Due to its complex and hard-to-specify goals (Knox et al., 2023), autonomous driving presents a challenging problem for reinforcement learning. Autonomous vehicles must maneuver through complex environments and interact with other road users, all the while strictly adhering traffic regulations. Common techniques to overcome these challenges include learning from demonstrations (LfD) (Schaal, 1996) or learning from intervention (Lff) (Saunders al., 2018). However, they do not always ensure that the learned behaviors align with human values such as safety, robustness, and efficiency (Hadfield-Menell et al., 2016). An alternative is integrating human feedback by using humans as the reward function in the learning process directly (Thomaz & Breazeal, 2008; Knox Stone, 2009). However, this rather resource-intensive as each rollout needs be evaluated by human. REvolve directly integrates human feedback (on the trained policy) into the reward design process. Instead of using humans as reward functions directly, human values, preferences, and knowledge can also be incorporated via so-called human-in-ther
(HITL) RL. Several techniques exist, for instance, human assessment (Thomaz & Breazeal, 2008; Christiano et al., 2017: Knox & Stone, 2009), human demonstration (Schaal, 1996), and human intervention (Saunders al., 2018; Wang et al., 2018). They are particularly beneficial in scenarios like autonomous driving. where the reward function is challenging, and the nature of the desired outcome is complex. More recently, reinforcement learning from human feedback (RLHF) (Christiano et al., 2017: Rafailov et al., 2023) has been used successfully in training models like ChatGPT (OpenAI, 2022), providing guidance and
26

---

Published as a conference paper at ICLR 2025
aligmment from preferences. A further comparison of autonomous driving approaches with and human feedback is presented in Figure 11, which illustrates how REvolve uses human feedback to align learned policies with human driving standards without requirements for extra reward models, such as in the case of RLHF.
PROMPTS
Here, we present the different prompts used for GPT-4. It includes the following:
System Prompt Boxes 1, 10 outline the system prompts (i.e. task description and expected input from the environment, complete with formatting tips to help guide GPT-4) for generating suitable reward functions for Autonomous Driving, Humanoid Locomotion, and Adroit Hand, respectively.
Abstracted Environment Variables Boxes 2, 8, 11 provides list of (abstracted) observation and Python class, for all three environments. This
action variables in the environment, formatted informs GPT-4 of variables it can use in reward design.
Operator Prompts like crossover (Box and mutation (Box 4). Additionally, we provide prompts for REvolve Auto baseline (Boxes 5, 6).
Box 1: System Prompt: Autonomous Driving
You are reward engineer trying write reward l'ucction to create an automomonu car agent learn to navigute the environmentt descrihed in text.
goal
reward function for enviromment help agent
Task description:
The objective is
the environment. Specifically, performance metric for the learning agent is to successfully drive without crashing. while maintaining smooth driving behavior and a speed betwren 9.0-10., 0-10.. episade terminaaees successfully agent for 1000 consecutive steps crashing. Otherwise, episode terminates in failure. The controlled actions are the steering and the speed. The car must avoid staying inactive and stay as close as possible ensure optimal performance.
agent
Your reward function should use useful variahles from the environment as inputs. An example reward function signature can be: "python
def conpptterrwwadddbeeeeeeeeoee
type, goal_pos: type) -> Tuple [type Dict[str, type]]:
return reward,
{
The ourput of the reward function should consist of two items:
1. The total reward,
A dictionary of each individual reward component.
The codc oupput should be formarted as a Python code string: """python... 10s
may it helpful to normalize fixed range 1.1 0, applying transformations np.exp0 the overall roward and/or its roward components.
2. If you choose to transform a reward component, then must intruduce temperature parameter inside the transformation function
function. This parameter must
not he input
Make type of each inpot variable correctly specified.
Most importantly the reward's code input variables contain only attributes of envireament. Under cireumstance can you introduce now input variable and assume variables that are not available.
Your outgui must always be the def function code with the outpul being the total reward.
sure you don't give contradictory reward components.
27

---

Published as a conference paper at ICLR 2025
Box Abstracted Environment Variables (as Python class): Autonomous Driving
from scipy spatial. transform inport Rotation inport numpy as np
class Env:
def Sinit.--self
self.client CarClient
actian [CarControls() .steering Cartontrols(..throtttt
def state(self))
position_info self.client, getCarState kinematics_estinated
curr-x position_info wtarronniin..ppittio....... position.x_vvl curr position info. postiioo. val
yau, pitch self. linear_velocity htt..ppc...caaatiioeestttoooooo
vx, vy Linear_.elocityy...all linear..ellcity..y.V.
angular velocity angular seguutarvelocityyyy. veguuir..ellocty...
position_info position.ings fiteeanes.stttttt..ggl..eeeeeeee position speed position_info.spe
collision [steering_valuel, steering.value?,
hasscclliee
action_list
steering value3, steering steering.valued]]
min_pos = get_min. position_info)
total counter
spisode_step. steoodd_ste_ccuuter
distance default range
is 20
def get orientation(self, pesition info): quaternion (assray(positionninet www.ntaaiinn......... v_val.
position wtt..noo.eeaatiio......o
ceatiio_info.orieenttttee
position_info. wrientation..
euler_angles
(quatrrrnnn)) as_euler((2gy) degrees=
return euler_ angles(]] (0], euler_angles
def
get get_mint pit__mnn_ossssett min_dis float('inf"
iaiitien_ifoo:::
for in data: stx, sty i[0], i[1]
dis eucl_dis(stx, sty, curr_I, curr_y)
if dis < min dis:
return min_dis
def [raelare_fooot_ddsaaeeeeeee_lll distance_data self. cleen distance distance_dar disteeddattdddtteee # nax sensor distance is 28
28

---

Published as a conference paper at ICLR 2025
Box 3: REvolve Crossover Prompt: Autonomous Driving
set of reward functions used train automomous driving agents. The reward lunctionn have fitness that reflect the agent' 's driving performance higher scores indicale superior driving behavior. For each reward function, we collected human feedhack what aspects of the agent's driving satisfactory and what aspects need improvement Additionally, tracked the values of the individual components in the roward function after every <eplsodes> opisodes and noted the maximum, mean, minimum observed. <EXAMPLES>
You
Your task:
Combine high-performing reward components from different reward functions based on human feedback and the data tracked,
the agent's driving capability further. Clearly explain how you intend combine and your rationale behind improving the driving performance.
Write the combined reward Function code.
Output of the reward function should consist of:
I: The total reward.
2: dictionary of each individual reward component.
Code output should be formatted as a Python code string: for reward function design:
"python...
1. Normalize to a fixed reward range like I1,1 or 0, by applying transformations like np.exp) the merall reward or its components.
transforming reward component, introduce temperature parameter inside transformation function. parameter must be named variable in the reward function and must not be an input variable. Each transfoemed reward component should have own temperature variable and you must carefully set value bused on its effect overall reward.
Ensure the type of each input variable correctly specified.
The reward's is code input variables must contain only attributes of the environment. Under no circumstance can introduce new inpur variable.
Always output function code with output being total reward.
Avaid contradictory reward companents.
29

---

Published as a conference paper at ICLR 2025
Box REvolve Mutation Prompt: Autonomous Driving
You reward lunction used train aulonomous driving agents. The reward functions has litness that rellects the agent's is driving performance higher scores indicate superior driving behavior. We also collected human feeihhcck on what aspects agent's driving sfactory and what aspects need improvement Addirionally, tracked the values indi- vidual components in the reward function after every <episodes> episodes and noted the maximum, mcan, minimum valoes observed.
<EXAMPLES>
Your lask sterate the reward function mulating single compament enhance the agent's driving capability further. Some helpful tips analyring the reward components: the values for a certain reward component are nearly identical throughout, then this means the training not able optimize this component
written. may consider:
1. Rescaling it to a proper range or the value of its lemperature parameler
Re-writing reward component
Discarding the component
The mutation process involves following steps:
First, select reward component based the guidelines, human leedlback, and values of individual reward components.
Next, clearty explain how you intend to mutate selected component and your rationale behind improving the driving performance. This could involve adjusting its scale, rewriting its formula, other modifications.
Finally, write the mutated reward function code.
ourput
consist
The total reward.
A dictionary of each individual reward component.
The code oupput should be formatted as Python code string: """ythoo....
Some helpful tips for writing the reward function code:
may find helpful normalize reward fixed range exp to the overall reward or its components.
applying transformations
2. If you choose to transfoem reward component, then you must introduce temperature parameter inside the transfermation function. This parameter must named variable the reward function must not an input variable. Each transformed reward component should have its lemperature variable and must carefully value based its elfect in the overall reward.
sure type
inpat
correctly specified.
importantly. reward's input variables must only attributes circumstance can you introduce new inpul variable.
Under
30

---

Published as a conference paper at ICLR 2025
Box 5: REvolve Auto Crossover Prompt: Autonomous Driving
driving agents. The reward functions have litness that reflect the agent's is driving performance higher scores indicate superior driving hehavior. We also tracked values of the individuul components in the reward function after every <epi sodes> episodes and noted the maximum, minimum values observed. <EXAMPLES>>
You given set of reward lunctions used to irain
Your task:
Combine high-performing reward components from different reward functions to enhance the agent's driving capability further.
Clearly explain how you intend combine and your rationale behind improving the driving performance.
Write the combined reward Function code.
Output of the reward
consist
2: dictionary cach individual reward component.
Code output should be formatted as a Python cude string: "python
Tips for reward function design:
Normalize to a fixed reward range like I or 0,1 by applying transformations like np.exp() to the overall reward andior its components.
If transforming reward companent, introduce temperature parameter inside the transformation function. Ensure the type of each input variable correctly specified.
4. The reward's 's code input variables must contain only attributes of the environment.
Always output the def function code with the outpur being the total reward.
Avoid contradictory reward components.
31

---

Published as a conference paper at ICLR 2025
Box REvolve Auto Mutation Prompt: Autonomous Driving
driving agents. The reward function has fitness that reflect the agent's is driving performance higher scores indicate superior driving hehavior. We also tracked the values of the individ- ual components the reward function after every sodeso episades and noted the maximum, and minimum values observed.
You given reward function used train
<EXAMPLES>
agent's driving capability further. Some helpful lips analyzing the reward components: the values for ceriain reward component nearly identical throughout, then this means the training is not able to optimize this component as written, You may consider:
on
mutating, single component
Rescaling it proper range the value of temperature parameter
2. Re Be-writing the reward component
Discarding the reward component
The mutation process involves following steps:
1. First, select reward component hased on the given guidelines.
Next, clearty explain how you intend to mutate selected component and your rationale behind improving the driving performance. This could involve adjusting its scale, rewriting its formula, other modifications.
3. Finally, write the mutated reward function code.
ourput function consist
1. The total reward,
A dictionary of each individual reward componeut.
The code auiput should be formatted us Python code string: ** "python....
Some helpful tips for writing the reward function code:
You may fidd helpful to normalize the reward lixed reward range like tfexxp) to the overall reward or its components.
applying transformations like
If you choose to transform a reward compunent, then you must introduce lemperature parameter inside the tramffommation function. This parameter be named variahle the reward function and not input variable. Each transformed reward componcnt should have its own temperature variable, and you must carcfully set its value based on its effect on soward.
Make the type of each input variable correctly specified.
Most importanily the reward's code input variables contain only altributes of the environment. Under circumstance can you introduce new input variable.
32

---

Published as a conference paper at ICLR 2025
Box System Prompt: Humanoid Locomotion
You reward engineer trying write reward function for humamoid robot Your to write reward function for the environment that will help the agent learn the task described in text.
Task descciption:
z-position falls outside The episade ends if the robot becomes unhealihy after 1000 timesteps, with success measured by speed. Ageni Incomotion should like.
humamoid agent
becomes unhealitty
range.
Your reward function should use useful variahles from the environment as inputs. An example reward function signature can "python
def comptte_reward(objeet...oo type, goal_pos: type) -> Tuple (type, Dict[str, type]]:
return reward,
{
The ourput of the reward function should consist of two items:
1. The total reward,
A dictionary of each individual reward component.
The codc output should be formatted as a Python code string: """pyhhon... -00
may find it helpful to normalize fixed range the overall reward and/or its roward components.
0, applying transformations np.exp0
2. If you choose to transform a reward component, then must intruduce temperature parameter inside the transformationn function Make type of cach input variable correctly specified.
function. This parameter must
not he input
Most importanil. the reward's code inpul variables contain only attributes of environment. Under cireumstance can you introduce now input variable and assume variables that are not available.
Your outgul must always be the def function code with the outpul being the total reward.
sure you don't give contradictory reward components.
33

---

Published as a conference paper at ICLR 2025
Box Abstracted Environment Variables: Humanoid Locomotion (Part 1)
class (utils. EePickle):
#8 Action Space
The action space is (Box(-1, 1, (17,), float32)', An action represeats the torques applied
hinge joints.
## Observation Space
Observations consist of positional values of different body the Hunanoid, followed by the velocities of those individual parts (their derivatives) with all
the positions ordered before all the velocities.
By default, observations do not include the and y-coordinates of the torso.
These may
be included passing exccuud_current_ppsitttoo orr.rttoon_.ooooeeeerrttiionaia during construction.
the observation space will 'Box(-inf, Inf, (378,), float64)* where the first two observations represent x- and y-coordinates Regardless of whether 'exclude 'exclude_current wwwrent.positionn.. from observation set true or false, the x- and y-coordinates will be returned in 'info' with keys ""x_position"! and ""y-osiitoon', respectively.
that
torso.
Homever, by default, the observation is (Box(-Inf, Inf, (376,), float64) ' The elements correspond to the following:
Nun Observation
Min
Nax
Name (in corresponding XML
file) Joint Unit
2-coordinate the torso (centre)
-Inf Inf
root
free
pasition (n)
x-orientation of the (centre)
-Inf Inf
root
angle (rad)
y-orieetaaionn of the torso (centre)
-Inf
Inf
root
free angle (rad)
z-orientation of the torse (centre)
IInf
Inf
free angle (rad)
... ....
43 coordinate-2 (multi axis) of the angular velacity of the angle between torso and left arm (in uepe_
-Inf Inf left shoulderr
hinge anglular velocity (rad/s)
44
angular velocity of the angle between left upper arn and left_loxer_arm -Inf Inf left_elbow
hinge anglular velecity (rad/s)
excluded x-coordinate
(centre)
-Inf Inf roat
free pasition (n)
excluded
y-coordinate of the (centre) -Inf
Inf root
position
Additionally,
pasitional velocity the observation contains (in order):
table,
*cinert:" Mass and inertia of a single rigid body relative to the center of mass (this is intermediate result transition). It has shape 14*18 (*nbody * 10*) and hence adds to another 148 elenents in the state space. shape
*cvel:"
based velocity.
("abody a 6*)
hence
in "'frc_actuator;" Constraint force generated the actuator force. This has shape '(23.)' *(nv 1) and hence adds another 23 elements to the state space. **ffrc_ett"* This is the center of based external force the body. It has shape
another
space
elenents in where "nbody" stands for the number of bodies in the robot and *y stands for the nunber degrees freedon (*a (in(e)))))
("abody 6*)
space.
34

---

Published as a conference paper at ICLR 2025
Abstracted Environment Variables: Humanoid Lomoton (Part 2)
def .get. (sellf)))
device torch. decibddevieeeeeueee if torch. cuda. iaaaialllll else "cpu")
position = self. data. qaata.qpo...fll flo..ffaa.ccoy
velocity delf.data.qell.fla...c
con inertia self,data cinert. copy con_velocity self. wwt.ddatt.cell.la..cooo..
actuator_forces self. alttaaafrcaactteeeeeeee qfrc_actuuaoo flat. copy O
external_contact_foreee self data. flat. ()
observation up, concateeentt
(position, velocity, con_inertia, con_velocity, actuator_forcee extermal_contctt_forrr
return observation
Box 10: System Prompt: Adroit Hand
You are a reward engineer trying to write reward function for the environment that will help the agent learn task described in text.
Task description:
involves using anthroponorphic must manipulate the handle elfectively, overceming the friction and resistance of the doar mechanism. The episode ends after 400 steps opened the successfully.
handle open
as possible. robot
Your reward function should use useful variahles from the environment as inputs. An example reward function signature can "python
def robbjj type, goal_poss: type) -> Tuple (type, Dict[str, type]]:
return reward,
{
The ourput of the reward function should consist of two items:
1. The total reward,
A dictionary of each individual reward component.
The codc oupput should formarted as Python code string: >""python 10s
may it helpful to normalize fixed range the overall poward and/or its reward components.
0, applying transformations np.exp0
2. If you choose to transform a reward component, then must intruduce temperature parameter inside the transformation function. This parameter must
function
not he input
Make type of each inpot variable correctly specified.
Most importantly the reward's code input variables contain only attributes of envireament. Under cireumstance can you introduce now input variable and assume variables that are not available.
Your outgui must always be the def function code with the outpul being the total reward.
sure you don't give contradictory components.
35

---

joint velocities and Published as a conference paper at ICLR 2025
Box 11: Abstracted Environment Variahles: Adroit Hand
class (EEPPcklee: #d Action Space
The space "Box(-1.0, (28.). float32)*. The control actions absolute angular positions of the Adroit hand joints. The input of the control set range in radians.
by scaling real
angle
Observation
observation space is of type 'Box(-inf, inf. (97.). fleat64)'. contains information about the angular pasition of the finger joints, the pase of the palm hand, forces.
of
Joint (in corresponding file) Site Name (in corresponding INL file) | Joint Typel Unit
Observation Min Max
|-----I- -I--------- ---------------
Angular position of the vertical arm joint
Inf ARRX
hinge
angle (rad)
def (sel))))
# opos for hand # for obj
# xpos for target apos
handle_pos = daff.data.site xpos sself handle handle_site
site id] .aveel ()
paln_pos
sit__sposssseeeellllessssss grpssse... grasp_site_idd raveeelll
door_pos ar.aaray((sel) datal.ddat.... deddaaapssssso spos[self. f. f.doorr_iingn (dor_h)nge_adérs]]
joint_velocities self. aasta.qve..rave. ()
jeint joint_forces forces self. actuaato torce.ravel0
actuator
if door_pos 1.4:
doer_
else:
door_open
latch_pos apos [-1]
return concatentn)
[p:-[[1-2],
[latch_pos],
door_pos,
palm.pos,
pain pos handle.pos. pos. [door_open]
36

---

Published as a conference paper at ICLR 2025
F
HUMAN FEEDBACK EXAMPLES
Box 12: Mutation with Human Feedback
Human feedhack on the antonomous driving agent's performance: Positive: collision avoidance, keeping, managing consistent speed, intersoction and turn handling. Negative:
Smooth Steering.
feedback and analysis reward components, modification. The feedback indicates issues with smooth steering. which suggests that the existing smoothness component isn't feedback specifically pointed deficiencies this
Based the
identify that the "smooth driving bonus" needs
elfectively promoting desired steering suggests the measure (variance of steering actions) might not he adequately sensitive or scaled improperly.
Intended Mutation: The mutation will focus revising the "smoath driving bonus" hetter capture and reward smooth steering. The current mothod uses the standard deviation of steering actions to compate penalty, which may not effectively discriminate between slightly varying highly
respensive approach might involve squaring the deviation before scaling, which penalizes larger variations more severely and provides sharper gradient for the learning algorithm to optimize against.
steering actions.
Rationale By squaring the deviation, small steering adjustments are less penalized, allowing for natural minor corrections while driving. Larger. more erratic movements result significanty higher penaltics, pushing the training to favor smoother transitions and stable steering behavioes.
def wtt.coopute_rwaareeeruee
float, curr_y: float, speed: fleat, collision: bool, action list, stt___ste___coutteeeee int, episode_ ttt_ssttep__ceeee..eeeeesssssese counter: int, min_pos: float. distance: float) -> fuple[float, Dict[str, float]]:
steering_variance snothhesss_facorr std
Steering variance 6.1
reward_components[ [' smoothness_ factor / wwe_steering.varineee
np. sp.exp(ttem__snnn cpp-teen_smootbness *
def conpute_reward (crr x: float, curr_y: float, speed: float, collision: bool, action_list
int, siisode_step_countee int, min_pos: float, distance: float) >> ffpleffloat, Dict[str, float]]:
Updated mutation smooth driving
max variance 0.1 smoothness_ factor ap.std(action_list)) reward_comy compondtompoentt( ('antss'''morrto smooth_ wiiidddvvinngooeee np. e-temeteppssooottmes smoothness #ootthness_factor=2 / stx_rteering wereeringvvaiaaneee....... variance**2))
G
ADDITIONAL. RESULTS
G.1
BEST PERFORMING REWARDS
G.1.1 AUTONOMOUS DRIVING
In Box 13, present the berf-perf reward from REvolve with fitness score of 0.84 for the AirSim environment. Similarly, Box 14 displays the best-performing reward from REvolve Auto with fitness score of 0.81 for the same environment.
37

---

Published as a conference paper at ICLR 2025
Box 13: Best reward function from RÉvolve: Autonomous Driving
def compute_rewarddcrrt.t float, curr_y: float,
speed:
collision: bool,
non_pos: float.
action List[fleat]] Tuple[float, Dict[str, float]]:
Parameters tweak the importance different reward components
collision.penalty = -10
inactivity_pematty
reead_ 2.0
position reward weight 1.0 smoothness_revard.
0.5
Adjusted temperature parameters for transformation speed temp 0.5 & Increased temperature for speed, reward oesition_temp smoothness 0.1
revard_components
"collision_penalg"" 8,
"laactivity_eenalty":
"speed_reward reward"; 8,
"position_reward"" 8,
"smoothness
snoothness_rewardd: reward":
Penalize for collision
collision: reward_ comeonetnm p'clllisioonpemalty]] collision_pemalty
Penalize for inactivity (speed close zero)
speed 4.5: Adjusted threshold for inactivity
inactivity_penalty
maintaining optimal speed range if 9.0 < speed <= 19.5:
abpe 9.75) around the average of optinal speed range else:
speed_score
Centering normalizing
speed_score
Use a signoid function to smoothly transforn the speed score and constrain it componetts""spedddrcriroeeee speed_score speed_temp)))
speed_ weight / np.exp(-
Reward for being close the of the road position_score up. expee---min_pe / position_temp) positiio_.eward.eeiggh position_score
driving (small variations
steering actions)
-np.std(action_list
ceward_.omponentss snoothmess_rewwrde Smoothness.revard.weihhh ex.exp( steering_smoothnesss smoothness_temp)
Calculate reward, giving precedence penalties
total_ reward =
"wrarddconponentts"coloiiii total_reward
elif rewardd.omponents.. innactvvyy pew.tvitttttyppttyllllll
total_reward [ hww.aionnnnnniimiiiiittooaaaaa
else:
total sun(reward components ( v))ues())
within reasenable range reward np. (p.clipptoral_ -1,
total_reward, reward_components
38

---

Published as a conference paper at ICLR 2025
Box 14: Best reward function from RÉvolve Auto: Autonomous Driving
def conpute_revardccurt.x: float, curr_y: float, speed: float, callision: bool, action_list: list, total_step.counter int, satsode_ste_.couteee int, min_pos: float. distance: float) Tuple Dict[str, float]]:
Define temperature parameters for reward transformations
temp collision = 5.0
temp inactvvity 10.8
temp_centering 2.0
temp-speed semp_smoothmee 1.0
temp.proxinity 2.0
Initialize the total reward and reward components dictionary reward_ eeward_compneents
'inactivity_peeally's 8,
collision__enalty:
"lane. centering bonus': 0..
speed_regulation_bor
'smooth "soooth.driving,
"front_ distttdddttteeebbeee
Penalty collisions
reward sovard_componente coaaaaeentssslolllllleeoaaaaaaae collision collision_penalty" penalty'] ep.exp(-tempp exp((teetp.clllision) if collision else
Penalty for inactivity
anactivity_threshol.
if speed < inacttvity_threeshlld and not collision:
reward components l'inactvvity penalty") exp(-temp inactivity senetivitt__hreshheee speed))
Lane centering bonus max max_ frr.amcece_ronceeeeerrereeeee 2.0 adaptable based on road width
reward_compnentt""annn. haaaantttnllannccttrrnnn..ooooo
exp(-teempcentering min_pos)
Speed regulation bonus
ideal_speed = (9.0 10.5) 2
speed_range
reward_compenents snperd_reuulation_bane '] gp.eep(-tenn_spee)
(abe
ideal_spred) speed_range))
Smooth driving bonus max full bonus
configured acceptable steering
snoothness_factor (istt(aacion_list)
reward_ compoedtcmmpoents iiapetttssssoootddiiiiiiiii smosth_driving
snoothness_factor / max_sterring.vrriace))) exp(-teep_snootheess
Front distance bonus max www.taiietvvieee reward_components[ (frpanntttttoootiiistttaaa.. front_distance_pee 6, 1)
np. cl.ccipddsstaace / mar_distanco_vier,
Sun the individual rewards for the total reward
sun(reward components.valuee
penalties precedence if collision or speed inactivity_threstte
retard
Apply nax negative reward for collision or inactivity
Normalize the total reward to lie between -1 and ap. cp.ccipptotall reward, -1,
return total_reward, roward_components
G.1.2 HUMANOID LOCOMOTION
In Box 15, we present the best-performing reward from REvolve with an average velocity of 5.67. In Box 16, the best-performing reward from REvolve Auto has average velocity of 4.71.
39

---

Published as a conference paper at ICLR 2025
Box 15: Best reward function from REvolve: Humanoid Locmotion
def compute_reward (obsevvati::: ap.array) >> (float, dict):
Constants and Parameters
speed_temp 0.1 orientation health_ temp 1.0
Extract relevant components from the observation speed [aservation[22] # x-velocity component for reward r_position observation[4] z-coorinnate of the torse orientation = observation
orientation of the torso (x, y, 2, M)
Reward Components
speed_reward up. up.exp(speed speed_temp)
Encourage
novement,
exponentially
orientation p -np. amp.var(orientatio))) orientation_temp Penalize variance orientation encourage stability
healthh.eenale = 0 if 1.0 < z.position 2.8 else -np op.eep(-abs22 1.5)
health_temp) Penalty unhealthy z-position
Total Rexard Calculation total_reward reward orientaciio. health.penalty
remord_components
sseed reward'; speed speed_ reward,
'erientation 'rrientation..enaltyy.sss orientation_penally.
"health_penalty' health_penalty
return total_reward, compretsss
Box 16: Best reward function from REvolve Auto: Humanoid Locomotion
def ceneete_eerrrddbsrevatttttt ndarray))y) >> (float, dict):
Constants
reeed_eeaad 8.1
health..enalty_-enn 0.01
orientation_reward_eee 0.1
moolth.thresboler 0.8 # Assumed threshold for the health component before aggressively
penalties
Extracting necessary components observation computation
p.position observation * Assuming first value represents torse z-position speed [itervation[2]] Assuning this represents the forwards velocity
Speed Reward Component: Encourages forward novenent speed_reward speed appeep(spped__ewaad,d temp speed)
Health Penalty Component: Penalizes unhealthy states less aggressively
is. unhealthy 1 if posoiiio 1.4 e.position 2.0 else is_unhealthy (2.0 2_position) if
health_penalty -np. exp www.rrrhhaleh..enaly..eeeeo z-position threshol else health.penally__tem
is_umbealthy 0.5)
Orientation Reward Component: Encourages maintaining upright
Lorientation, y_orientation obsorvatton[1]] observation[2] # Assumed orientation metrics
orientation reward (1 orientation"" oriennaatinn"")) expe orientation_reward.tem (1 =0rientation o y=orientation**)))
Total Reward Calculation
total_reward speed_rexard health orientation_reward
Reward Components Dictionary
reward rwward__copperrreeeeeeeeeeeeeeee components
reward' speed_
'health penalty': health penalty,
Prrientation_rewarre orientation_reward
return total reward, reward components
40

---

Magnitude of vector from Published as a conference paper at ICLR 2025
G.1.3
ADROIT HAND
"In Box 17, present the best-performing reward from REvolve with a fitness score of 0.81. In Box 18, the best-performing reward from RÉvolve Auto has a fitness score of 0.64."
Box 17: Best reward function from REvolve: Adroit Hand
compute compute_rewardCCne (observation, joint velocities joint. forces):
Example observables
Using door pos as an approximation for the goal goal_.pos pos observation
position
paln_to_handle = up mp.linalg. norm (cbservvation
palm handle
door_hing angle observation [28]
completion. threshold
radians, per
definition
Paraneters
dist_temp 10.0 + Temperature paraneter for distance transfornation
hinge_temp 5.0
Temperature paraneter for hinge reward transformation
Components autation: Distance cost modification with exponential decay dist_cost exp.expp-pal_tte handle dist_temp)
Door hinge reward: reward for rotating the door handle
door boor_hinge_revaee if door_hinge_angle < completion__hreshole else 1.8
ep.exp (ddoo__hinge_ang
completion_threesolla) hinge_temp)
Checking 1f the door is sufficiently opened to consider the episode success mmmpletio__treshaldd
door hii_hhing_gggee
completion_honus 108.0 Signi ficant bonus for completing the task
else:
completion_honus 0.0
Total calculation
total_ reward dist_cost dour hewae_ sampletion_bonue
Returning the total reward and the individual components as dictionary return total_reward, ('dist_cost'; dist_cast, "door_i door hoor_hiing_reward, "completion_hone" completion_bonus)
Box 18: Best reward function from REvolve Auto: Adroit Hand
def compute (observation: ndarray) (float, dict):
Defining temperature parameters for the transformations
position 0.1 # Temperature for position reward transformation
temp_door_open 0.05 Temperature for deor open reward transformation
Extract necessary components fron the observation position_rewane ohservation[-2] Assuming positional info door, coor_open_rewarr observation[-1] Assuming last value indicates door being open
correspends
Apply transformations with temperature paraneters
trans www.ooreed..psiionoooooo reward пр np.exp (temp- position.reward) Subtracting doo.sooodd....oopp..... (ten. dooe_p_doo_pper loor_open_reward)
Calculate total reward
total_reward transforred.positionn.e
transfered_.borr.ee__rrra
Constructing the dictionary of individual reward conponents reward serard_compenente = : transforned position. "door_ reor__open_ewward" transformed door_open open_rewart
total_reward, componets
41

---

Published as a conference paper at ICLR 2025
300
Rivdla Expan Detigred
370
Revelee Auto
320
50sodi:
220
150
200
120
100c 200c 330k
400k
500k
2M
Thirling Stepp
mingg Steps
Figure 12: Compares policy training with REvolve, REvolve Auto and expert-designed reward functions episodic steps from the start of the training for the Autonomous Driving [Left] and Adroit Hand [Right]. Note, that episodic steps in Autonomous Driving denote how long an agent drives (i.e. higher is better). In contrast, episodic steps in Adroit Hand denote the number steps the agent takes to solve the tasks (lower is better).
3806
Figure 13: Episodic Steps during training for Env [Left]] and Env scenario [Right].
Do REvolve reward functions lead higher success rates? Figure 12, we see that REvolve- designed rewards converge to higher number of episodic steps compared to expert-designed rewards, signifying better success rate per episode higher is better for Autonomous Driving and vice versa.
42


Reviews:
[{"review_id": "acHMOvKlr7", "reviewer": "Reviewer_zYk2", "summary": "This paper presents a framework that combines human feedback with genetic algorithms to guide LLMs in designing complex reward functions for agents. The authors conducted experiments on three tasks: Autonomous Driving, Humanoid Locomotion, and Adroit Hand Manipulation, and performed a comprehensive ablation study analysis.", "questions": "- The first sentence of the paper mentions \"Recent success...\", but the cited articles are from 2018 and 2019. Is that really considered recent?\n- How long does the entire pipeline process take? Are there cases where the reward function does not converge due to difficulties in summarizing it from human-provided language feedback? I think this is an important issue because the pipeline proposed in this paper seems overly complex and involves a lot of uncertainties.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- Well written, the technical approach is very clear, especially Figure 1 and Algorithm 1, which can help readers quickly understand the technical route of the paper.\n- The experiments involve a wide range of task types, covering multiple dimensions of continuous and discrete action spaces and observation spaces for RL agents in virtual simulation environments.\n- The authors have compared their work with a variety of baseline algorithms.", "weaknesses": "The paper does not clearly describe the foundational setup of the experiments and the comparison metrics. In fact, I do not understand the role of the \"fitness score\" mentioned in the text. In sections 2 and 3, it is used as a supervisory signal to guide the genetic algorithm in generating the reward function. However, in section 4, it becomes an evaluation metric for the experimental results of the paper. I am unsure whether this practice is appropriate because it raises the following concerns:\n- (1) If the fitness score can effectively measure the performance of agents in the current task, then why not use it directly as the reward function to train the agents?\n- (2) If the fitness score serves as a signal to guide the generation of the reward function, is it fair to use it as an evaluation metric for the experiments? This is because the REsolve method receives fitness supervision during training, whereas other baseline methods do not.\nThe insights provided by the paper are limited. When the RLHF paradigm was introduced, people saw that using human feedback could enable agents to perform challenging actions like backflips, which are certainly beyond the capabilities of rule-based reward functions. However, in the tasks and experimental setup implemented in the paper, all I understand is that the authors have used very costly human feedback to achieve a performance superior to the baselines, without offering additional insights or analyses to justify the necessity of this approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7miTvP4fcP", "reviewer": "Reviewer_CReE", "summary": "REvolve is an evolutionary algorithm for LLM reward function generation for RL tasks in robotics. Authors first point out that Eureka is not in fact an evolutionary algorithm (despite claiming to be so), because it does not preserve populations and do mutation and crossovers. REvolve takes Eureka and implements two main changes: make it an actual evolutionary algorithm, and use human labelers as a fitness function (instead of a manually-specified one as is done by Eureka). Authors demonstrate REvolve surpasses Eureka on 3 sim tasks: autonomous driving, humanoid locomotion, and adroit hand door opening.", "questions": "1. What proportion of the LLM-generated reward functions were executable? Eureka mentioned many generated reward functions could not be executed.\n\n2. Each human provides feedback consisting of a pairwise preference assessment and natural language feedback (extracted from a selection of checkboxes). This seems somewhat demanding. How important is each of these two components, relatively? Ablating either of the two components would have been informative.\n\n3. Did authors try allowing humans to provide open-form natural language feedback, not constrained to a checklist?\n\n4. Were the human evaluators for the policy’s final performance in Table 2 the same as the human evaluators that provided feedback during training? If so, I would take issue with that, because then the non-auto methods are given an unfair advantage over the auto methods since they were evaluated on the same distribution of human evaluators as during training.\n\n5. During crossover, how are the “most effective” reward components determined? Through human natural language feedback?\n\n6. Perhaps show an evaluation comparison as training proceeds through the N=5 generations. This would tell us whether REvolve consistently outperforms Eureka, or whether REvolve learns faster with increasing generation index.\n\n7. What is the feasibility of REvolve on real robot tasks? Authors mentioned sim2real as a promising approach, but given the difficulty in simulating high DOF robots (such as humanoids), training on simulation data may not necessarily yield satisfactory results. However, only training with real data would be expensive with REvolve especially with the high demand on human feedback.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Paper is written very clearly and is easy to understand.\n\n2. Results included good ablations that tested the effect of human ratings as fitness functions for both REvolve and Eureka.\n\n3. Human evaluations were conducted and full Elo scores reported between REvolve, Eureka, and their respective ablations.\n\n4. Appendix contained detailed experimental setup information.", "weaknesses": "1. Natural language feedback to the LLM is extracted from a series of checkboxes that the user ticks. This requires a human to predefine the specific characteristics that the agent must exhibit, which may bias the prompting of the model by constraining the natural language feedback to this small discrete list of checkboxes. So while the authors argue that no manual engineering is required for providing the fitness function, there is manual engineering required to provide a good list of attributes for the human labelers to choose from, and it seems like these attributes must be tailored for each task.\n\n2. A more in-depth analysis that would be helpful to the paper would try to quantify the total amount of human effort needed in REvolve vs Eureka auto: amount of time to define the natural language feedback checklist, provide all pairwise ratings and natural language feedback throughout training in REvolve, vs time to define a fitness function in Eureka. Since it requires hundreds of human labels over N=5 generations, I suspect REvolve involves much more human effort than Eureka auto, but the promising part of this work is that REvolve auto outperforms Eureka auto on all three tasks.\n\n3. Presumably there is additional effort involved in REvolve vs Eureka, from a ML practitioner standpoint, in needing to tune additional parameters related to the mutation and crossover operations (such as p_m, or number of islands).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J2ZBe5Ad8a", "reviewer": "Reviewer_vtL6", "summary": "This paper introduces a reward design framework. It proposes a fitness function that involves human feedback and natural language. It applies evolutionary algorithms to evolve a population of reward functions by LLMs, where the prompt is the output of the fitness function.", "questions": "(1) In Figure 3 (left), it seems auto grows faster than human feedback. In Figure 3 (right), auto grows exponentially but others grow linearly. What will happen given more generations? Will the auto surpass human feedback? I would suggest to run more generations to see a clear trend. \n\n(2) A follow up question for (1). If the auto surpass human feedback, or even human driving, would human evaluators still prefer human feedback to auto?\n\n(3) To solve reward design problem, REvolve is a good solution for explicit reward function generation, but how does this method compare with implicit reward models, i.e., RLHF, in terms of performance, interpretability, computational requirements?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "REvolve uses the evolutionary algorithm with genetic operators like mutation, crossover, and selection to overcome the disadvantage in greedy search in Eureka and present good performance. \nREvolve leverages LLMs to generate reward function explicitly and employ human preference to guide the reward function generation implicitly, which enhances the interpretability and aligns with humans.", "weaknesses": "No major weaknesses under the topic of explicit reward function generation, but it lacks the discussion of the comparison to an implicit reward design, for example, a reward model from human preference.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)