R*: Efficient Reward Design via Reward Structure Evolution and Parameter Alignment Optimization with Large Language Models
Authors: Pengyi Li
Jianye HAO
Hongyao Tang
Yifu Yuan
Jinbin Qiao
Zibin Dong
YAN ZHENG
Venue: ICML 2025 poster (2025)
Keywords: Reward Design
Reinforcement Learning
Primary area: reinforcement_learning->deep_rl

Abstract: Reward functions are crucial for policy learning. Large Language Models (LLMs), with strong coding capabilities and valuable domain knowledge, provide an automated solution for high-quality reward design. 
However, 
code-based reward functions require precise guiding logic and parameter configurations within a vast design space, leading to low optimization efficiency.
To address the challenges,
we propose an efficient automated reward design framework, called R*,
which decomposes reward design into two parts: reward structure evolution and parameter alignment optimization. To design high-quality reward structures, R* maintains a reward function population and modularizes the functional components. LLMs are employed as the mutation operator, and module-level crossover is proposed to facilitate efficient exploration and exploitation.
To design more efficient reward parameters, R* first leverages LLMs to generate multiple critic functions for trajectory comparison and annotation. Based on these critics, a voting mechanism is employed to collect the trajectory segments with high-confidence labels.
These labeled segments are then used to refine the reward function parameters through preference learning.
Experiments on diverse robotic control tasks demonstrate that R* outperforms strong baselines in both reward design efficiency and quality, surpassing human-designed reward functions.

R Efficient Reward Design via Reward Structure Evolution and Parameter Alignment Optimization with Large Language Models
Pengyi Li
Jianye Hao
Hongyao Tang'
Yifu Yuan
Jinbin Qiao
Zibin Dong'
Yan Zheng'
Abstract
mann et al., 2023a), robotic locomotion (Radosavovic et al., 2024), manipulation (Aguinaco et al., 2023; Yuan et al., 2025), navigation (Zhu & Zhang, 2021), and protein struc- ture design (Lutz et al., 2023). Although these achievements demonstrate the potential DRL, its learning process of- ten suffers from instability, leading to suboptimal policies even policy collapse (Arulkumaran al., 2017). A key factor for this problem is the low quality of reward sig- nals (Hare, 2019; Silver et al., 2021; Eschmann, 2021), which are often sparse and deceptive. Designing high- quality reward signals typically requires extensive domain- specific knowledge, resulting in significant design costs (Sut- ton & Barto, 1998). Even with carefully hand-crafted re- ward functions, remains difficult guarantee efficient guidance for policy learning (Booth et al., 2023a). How to provide high-quality reward signal guidance remains an open challenge (Menell et al., 2017; Icarte et al., 2022).
Reward functions are crucial for policy learning. Large Language Models (LLMs), with strong cod- ing capabílities and valuable domain knowledge, provide automated solution for high-quality reward design. However, code-based reward func- tions require precise guiding logic and parameter configurations within a vast design space, lead- ing to low optimization efficiency. To address the challenges, we propose an efficient automated re- ward design framework, called R*, which decom- poses reward design into two parts: reward struc- ture evolution and parameter alignment optimiza- tion. To design high-quality reward structures, R* maintains a reward function population and mod- ularizes the functional components. LLMs are employed as the mutation operator, and module- level crossover proposed facilitate efficient exploration and exploitation. To design more effi- cient reward parameters, RR first leverages LLMs to generate multiple critic functions for trajectory comparison and annotation. Based on these crit- ics, a voting mechanism is employed to collect the trajectory segments with high-confidence labels. These labeled segments are then used to refine the reward function parameters through prefer- ence learning. Experiments on diverse robotic control tasks demonstrate that R* outperforms strong baselines in both reward design efficiency and quality, surpassing human- human-designed reward functions.
To address the challenge of reward design, many meth- ods are proposed from different perspectives, which can be broadly classified into three categories: Inverse RL (Arora & Doshi, 2021), RL from Human Feedback (RLHF) (Kauf- mann et al., 2023b), Reward Function Generation (Ma et al., 2024). Inverse RL begins by collecting expert demonstra- tions and then learns the reward function based on these demonstrations. This approach relies heavily on the cov- crage and quality of the expert data (Ng & Russell, 2000; Ho & Ermon, 2016). RLHF-related methods incorporate human feedback into RL by comparing trajectories, assign- ing preference labels, and training reward model (Lee et al., 2021). This process typically requires the continu- ous involvement of human experts to provide preference annotations (Lee et al., 2024). The final category focuses on directly constructing reward function codes. The most straightforward approach manually design reward func- tions through trial-and-error (Booth et al., 2023b; Knox et al., 2023). Some early methods employ Evolutionary Algorithms (EAs) (Bäck & Schwefel, 1993; Bäck, 1996; Vikhar, 2016; Zheng et al., 2019) with manually designed operators for reward function design (Niekum et al., 2010; Faust et al., 2019). However, these methods still heavily depend on expert priors.
1. Introduction
Deep Reinforcement Learning (DRL) (Sutton & Barto, 1998) has shown remarkable success in various sequen- tial decision-making problems, such as drone racing (Kauf-
'College of Intelligence and Computing, Tianjin University, China. Correspondence to: Jianye Hao hjjjnne.hao@tju..o wtw..cdu.cn.........
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Recently, Large Language Models (LLMs) have demon-

---

R Efficient Reward Design via Reward Structure Evolution and Parameter Alignment Optimization with LLMS
strated extensive domain knowledge and remarkable coding capabilities (Singh al., 2023; Ichter et al., 2022; Wang et al., 2024a;b). Some works use LLMs to directly gen- erate the code of reward function, significantly reducing human labor (Xie et al., 2024; Zeng et al., 2024). How- ever, relying solely on LLM-generated reward functions still faces challenges such as large search spaces and lim- ited robustness. Concurrently, Evolutionary Reinforcement Learning (ERL) (Drugan, 2019; Sigaud, 2022; Bai al., 2023; Li et al., 20-22)0222iiiii combines EAs with Ra-has achieved significant improvements in exploration efficiency and learning robustness (Khadka & Tumer, 2018; Zheng et al., 2019; Gupta et al., 2021; Hao et al., 2023; Li et al., 2023; 2024b;c) across various domains. Accordingly, Eu- reka (Ma et al., 2024) applies the Synergistic Optimiza- tion (Li et al., 2024a) idea in ERL for reward generation: EAs evolve population of LLM-generated reward func- tions, while RI. is employed to train policies for evaluation to select best reward function. Although Eureka demon- strates state-of of-the-art performance across a range of tasks, two limitations stand out: 1) Greedy Exploitation: Eureka improves solely based on the best reward function at each iteration, which may lead to suboptimal results on reward design. Other potentially valuable candidates are discarded without being fully utilized. 2) Suboptimal Parameter Assignment: Reward functions often include many parame- ters, such as weights or coefficients. Eureka relies on L.MSs to set them directly, which is often suboptimal. As a result, the reward functions may fail to provide precise guidance.
ing low-quality labels. Others utilize VLMS for com- parisons (Liu al., 2024), but existing VLMS struggle to perform reliable analysis of visual inputs. Moreover, quality differences typically exist only in specific segments of a trajectory, and the above methods struggle to effectively ex- tract these fine-grained segments. To solve above problems, we propose the critic-based step-wise voting mechanism. This method uses LLMs to generate critic functions that evaluate trajectories the state level. The evaluation re- sults are aggregated through population-based voting. Next, trajectory segments are extracted by grouping consecutive steps with consistent labels. Finally, we optimize reward function parameters within the population using pairwise preference loss. Experiments across 8 robotic control tasks demonstrate that the reward functions designed by R* out- perform those generated by other strong baselines in terms of both final performance and convergence efficiency.
We summarize our contributions below:
We propose a novel LLM-based reward design frame- work RR with two key components: reward structure evolution and parameter alignment optimization.
We design modular-based reward functions and em- ploy LLMs and module-level crossover operations for efficient structure search.
We propose the critic-based step-wise voting mecha- nism for data collection and employ pair-wise prefer- ence loss for parameter optimization.
To address the above challenges, we propose R* novel LLM-driven evolutionary framework that decouples the re- ward function design process into structure evolution and parameter optimization. To optimize reward function struc- tures, R* employs L.LMs to generate modular- reward functions and maintains a reward function population for evolutionary refinement. The reward function fitness is defined by the performance of the policy that the reward function guides, i.e., success rate. R leverages the LLM reflection mechanism to improve reward functions through modular modifications, deletions, and additions. To fully ex- ploit high-quality reward functions, R# maintains a reward function archive and selects parent candidates based on their fitness. By employing the module-level crossover opera- tor, RR promotes efficient exploration of the reward design space and fully leverages the reward function candidates. To address the challenge of parameter configuration, we propose parameter alignment optimization, which aligns the pairwise trajectory rankings induced by the reward function with ground-truti preferences. This enables more efficient parameter optimization. To achieve this goal, a reliable la- beled trajectory dataset is necessary. Some previous works rely on LLMs for numerical comparisons (Zeng et al., 2024), but they often suffer from severe hallucination issues, lead-
We empirically show that R* outperforms other strong baselines and surpasses human-designed reward func- tions in various tasks
2. Background
Reinforcement I Learning can be formalized as a Markov Decision Process (MDP) (Puterman, 1990) which can be defined by tuple AS P, R, y, T, p), where S is the state set, A is the action set, P 8 x A S R is the tran- sition function, R S A - R is the reward function, 7 E [0, 1) is the discounted factor and T is the horizon. p represents the distribution of the initial state. The agent interacts with the environment by performing its policy
ा : S + A. RL (Sutton & Barto, 1998) completes the task by maximizing the expected discounted cumulative reward J Ea (osr as,,,)) D rrr.. where r. R (st, a1) and 66 ~ p.
In this paper, we aim to design reward function in code form that guides policy learning to maximize task success rates. To utilize the reward function codes, we need the vari- able information as function inputs. To simplify notation, we assume that these variables are included in 8.
2

---

R Efficient Reward Design via Reward Structure Evolution and Parameter Alignment Optimization with LLMS
Preference Learning. In our work, we employ preference learning for the optimization of reward function parame- ters. We define segment as sequence of step-indexed states (Sk....,St+t S with length H. Given a pair of segments (o".o') ..). we provide a label indicating which seg- ment is preferred, so the preference Neomp could be indicated 8º > o², o² > o". We discard data with equal preferences, and the remaining labels are encoded as binary vectors, i.c., Doump E (1,0), (0, 1)), and stored as tuples (a at, |ccop)) in the trajectory buffer Dr. Subsequently, we use the an- notated data to optimize the reward function Fe,, where t/ represents the parameters extracted from F, such as the pa- rameters within the reward module and the weights between modules.
nism to improve both efficiency and accuracy (Sharma et al., 2022; Guan et al., 2023; Zhou et al., 2025). Besides, some methods manually design reward functions through trial and error (Booth et al., 2023b; Knox et al., 2023), while others optimize reward functions using EAs with predefined opera- tors and templates (Faust et al., 2019; Niekum et al., 2010), These methods typically depend on domain knowledge from human experts.
With Large Language Models (LLMs) demonstrating strong domain knowledge and coding abilities, several works try to generate reward function code through LLMs. For example, Rexr2 (Xie et al., 2024) generates reward functions by calling pre-defined interfaces through LLMs based on task descriptions. Eureka (Ma et al., 2024) constructs a reward function population with LLMs and proposes an evolutionary framework for iterative improvement. While these works achieve impressive results, the vast design space leads to inefficient reward function optimization and limited learning robustness. In addition, there are some works on reward shaping (Ladosz et al., 2022), such as curiosity- driven mechanisms (Burda et al., 2019), which typically focus on designing intrinsic rewards to enhance exploration. In contrast, our work focuses on generating reward functions from scratch. Our work, like Eureka, also falls within the Synergistic Optimization (Li et al., 2024a) branch in ERL: EAs are used to iteratively optimize the reward functions, while RL handles policy learning for population evaluation.
Following the Bradley-Terry model (Bradley & Terry, 1952), model the preference based on Fy as follows:
exp (s)))
Polo' > 001 - - (iE(o.1) (x, (s))))
(I)
where o² > rḥ denotes the event that segment is preferable to segment j and (s), at) E o². To align the reward function with the preferences, The update of F6 is transformed into
minimizing the following cross-entropy loss:
C(b,, Dr) E ,0' Dr (000) log P P o']
+ y11 log P 10t > o
(2)
where the terms y00 and v(1) refer to the values at index 0 and 1 of the label vector Yeomp, respectively.
4. Method
This section presents our framework, R for efficient re- ward function design. The key idea of R* is to decompose the reward design into two parts: reward structure evolu- tion and parameter alignment optimization. The former is responsible for searching the functional components that should be included, while the latter focuses on efficiently optimizing the parameters within and across modules.
3. Related Work
Various methods are proposed from different perspec- tives to construct high-quality reward signals. IRL learns the reward function from the provided expert demonstra- tions (Adams et al., 2022), including methods such as max- margin methods (Ng & Russell, 2000), Bayesian meth- ods (Ramachandran & Amir, 2007), and maximum entropy methods (Ziebart al., 2008). In addition, GAIL (Ho & Ermon, 2016) uses GAN (Goodfellow et al., 2014) to ap- proximate reward function for policy learning. However, these methods are often sensitive to the quality and distribu- tion of the data.
4.1. Framework Overview
We first present the overall framework of R$ to gain a holis- tic understanding. The optimization flow is shown in Fig- ure 1, which consists of two stages: the initialization stage and the evolutionary improvement stage.
The initialization stage consists of two key steps: (1) Ini- tializing the reward function population, which provides the foundation for subsequent reward function design; (2) Initializing the critic population, which is used to collect trajectory segment pairs with high-quality labels.
RL from Human Feedback (RLHF) (Kaufmann et al., 2023b; Dong et al., 2023; Yuan al., 2024) leverages human feedback guide the learning process of RL agents. PB- RL (Christiano et al., 2017) collects preference data through trajectory pair comparisons with human experts, which is then used to learn a reward model that guides RL learning. PEBBLE (Lee et al., 2021) combines unsupervised learning to pretrain the agent and integrates off- policy RL. Addition- ally, some works focus on improving the feedback mecha-
1. Reward Population Initialization. R* leverages LLM to generate the initial reward function popula- tion P PReward by providing the task description I. and
3

---

R Efficient Reward Design via Reward Structure Evolution and Parameter Alignment Optimization with LLMs
LLM
Reward Function
Population
Reward Function
Task Description
Enw Code
Evokastion Criterion
Reward Function
Speed Module 1
You are trajectory Your godl
The python environment
2. The distance
following
Code. Fornat:
Imprevement Tips
coreett
dom ProfeaieleeneeeTca
Current Best
Reward Function
Grasping Module n
LLM
Reward Function
Best Info
Logical Structure Search
Summory
Update
Reward Function
Logical Crossover
Sampled trajectories
Critic Population
Archive
function
Gritic Function
performence: 200
Speed
Diatunce Rotetion Madulee
Wedill
Remord function
Critic Function
performance: xoox
Grasping Module
Grosping Module n
Madule loss: xox
Vote-based labeling
Step-wise Label
RL Parallel Training
Parameter Optimization
100 11000-
Reward Function
Buffer
0
RL Policy
1 1 1
Alignment
Ry. R2
1, O
Rank Label
Figure 1.
The optimization flow of R* The left side depicts the main reward optimization process, while the right side illustrates trajectory annotation for parameter optimization. (Left) First, generate reward population using LLM. Then, select superior parents from the reward function archive for module-level crossover. Next, perform the parameter optimization. Following this, train RL parallelly. After training. update the reward function archive, and the training results are summarized to select the best individual for LLM reflection. (Right) Generate the critic function population using LLM. The individuals perform step-level comparisons the sampled trajectories. voting mechanism is then used to annotate and extract trajectory segments with high-confidence labels for paramcter optimization.
environment code information C. To facilitate more ef- ficient structure search in subsequent stages, we design the prompt Preward to guide the LLM in generating re- ward functions composed of multiple reward modules.
using Isaac Gym (Makoviychuk et al., 2021). During training, we record the success rates of the learned policies, along with other relevant statistics, such as the mean, maximum, and minimum values of cach reward module. The reward functions, along with their fitness values, are then stored in the reward function archive Dr. The success rate of the policy serves as the reward function fitness.
2. Critic Population Initialization. In addition to the reward population, RR also needs to generate critic population |Prritic with each individual represented as code function. These critics are used to compare trajectories and provide step-wise comparison labels. Similar to the reward population, R* takes task descrip- tions P. environment code C, and the prompt P Periti as inputs to generate the critic population Feritic. A de- tailed introduction will be provided in subsection 4.3.
2. Reward Population Evolution. We select the re- ward function Frest with the highest fitness, along with its associated learning information, to serve as feed- back. Through the LLM reflection mechanism, we perform evolution operations such as modifying, delet- ing, or adding modules improve the reward function, thereby creating a new improved reward function. To fully exploit the high-quality reward functions discov- ered, we select parent reward functions from the reward function archive DF based on their fitness. Module- level crossover is then applied to these parents to gen- erate new individuals, facilitating further exploration of the reward design space.
After completing the initialization stage, we proceed to the reward function evolutionary improvement stage, which consists of three steps: population evaluation, evolution, and parameter optimization.
1. Reward Population Evaluation. For each reward function F. in the reward population Preward we train PPO (Schulman et al., 2017) in parallel solve the task
Reward Parameter Optimization. Using the critic

---

R Efficient Reward Design via Reward Structure Evolution and Parameter Alignment Optimization with LLMS
population PCrine we extract trajectory segments and apply comparison labels the segments, creating labeled trajectory buffer Dr. The reward functions are then optimized based on Dr. allowing them to better evaluate segment quality and guide policy learning.
reducing unnecessary exploration. This paper primarily pro- vides initial exploration of the method. Given that LLMs require additional token overhead, we leave this aspect for future work.
4.3. Parameter Alignment Optimization
The initialization is performed at the beginning of the algo- rithm, followed by an iterative process of reward function evolutionary improvement. In the following subsections, we provide a detailed introduction of the two key components.
In the previous subsection, we generate the initial reward population PReward- However, configuring parameters within these reward functions remains challenging. Since the fune- tion parameters are provided by LLMs, this black-box con- figuration approach makes it difficult to guarantee accurate and efficient guidance, leading to issues in both in-module parameter settings and inter-module weight coordination. To address this challenge, we propose ranking-based param- eter alignment optimization, which consists of two key steps: automated trajectory annotation and parameter alignment optimization. Automated trajectory annotation is used to col- lect trajectory segments and assign comparison labels, while parameter aligmment optimization optimizes reward param- eters with the labeled data. Below, we first describe how to extract trajectory segments and assign high-confidence labels.
4.2. Reward Structure Evolution
Reward functions are typically composed of multiple reward modules to guide policy learning. For example, in robot grasping tasks, the reward function should include the re- ward for guiding the robot arm closer to the target object, the reward for moving the target object, and reward for the arm's 's stability. These modules serve different objectives. How to design and combine these modules to create high-quality reward functions remains a challenging problem.
Inspired by the previous work (Ma et al., 2024), we first propose the modular-basee reward function, where each reward function F is composed of multiple modular compo- nents [M1 Mm J. The design of high-quality reward functions depends both on the internal structure of mod- ules and how these modules are integrated. To achieve efficient optimization, we leverage LLM to improve the reward functions with following three main operators: 1) In-module Improvement. Adjusting the reward calculation within modules, such as applying scaling transformation to distance-guided rewards. 2) Removing Module. Based on feedback from each module during the learning process, removing ineffective reward modules from the reward func- tion. 3) Adding Module. Introducing new reward modules to improve the reward function.
To collect data automatically, previous methods typically rely on LLMs or VLMs to perform comparisons based on numerical states or images. However, these methods face three key problems: 1) Numerical comparisons rely on man- ually specified state features, and the instability of LLMs often leads to unreliable outcomes; 2) VLMs frequently pro- vide incorrect labels due to their limited capabilities and the inherent complexity of the task; 3) It is difficult to directly assess the relative quality of two trajectories, as one may outperform the other in some segments while underperform- ing in others. Thus labeling the entire trajectory as a whole can result in low-quality annotations.
To address the above challenges, we propose the critic-based comparison voting mechanism, which constructs popu- lation of rule-based critic functions for step-wise compar- ison. Specifically, we input the task description L, the en- vironment information C, and the prompt Peniii into the LLM, allowing to automatically generate the critic func- tions. The input of the critic functions consists of two trajectories to be compared, which contain information such as variable names and their corresponding values, e.g., bblockrrghtt whwgett (osenddd.ppp--........ 0.5, 0.0)". Using above in- formation, the LLM can quickly understand the input and construct critic function. Regarding the function details, we require it to compare the corresponding-ster information of the two trajectories and assign labels of 1, O, or -1, We require that the function assigns label of 1 or -1 only when all metrics of the current step of trajectory A are either better or worse than those of trajectory B; otherwise, it assigns a label of O. To further enhance reliability, we generate a
The above process relies LLMs but does not fully lever- age the high-quality reward functions discovered. To ad- dress this, we design a selection operator and a module-level crossover operator. Specifically, we first maintain a reward function archive Dr to store the high-quality reward func- tions discovered. We select two parents based on the fitness and then perform a crossover operation by inserting mod- ule from one parent into the other, forming a new reward function. We formulate the operation below:
Fnow {M,,,, Ms.m 3 } = (Fossover(Fr,,F)) F)).
(3)
By leveraging the inherent randomness of the crossover op- eration, we can fully leverage high-quality reward functions to facilitate a more comprehensive exploration of the design space. The above process can be enhanced using LLMs in two ways: by generating reward functions to replace random module swaps, and by filtering out unreasonable ones- thus
5

---

R Efficient Reward Design via Reward Structure Evolution and Parameter Alignment Optimization with LLMS
population of critics rather than relying on single critic function. The final label is assigned only when least half of the critics reach agreement. After obtaining the final step-wise labels, we group consecutive steps with the same label (i.e., or -1) into segments. Segments with a length greater than 5 are added to the buffer Dr for subsequent optimization.
Algorithm 1 R Framework
1: Require: Task description I,, environment code C, re- ward function prompt Preward. critic function prompt Partic coding L.M LIMM, reward function archive Dr, trajectory buffer Dr
2: Hyperparameters: Evolution iteration K, crossover individual number ne. reward population size n, critic
population size c.
After obtaining the labeled segments, we conduct parame- ter alignment optimization for the reward functions within PReward The data is then split into 70% for training and 30% for validation, and we optimize the function parame- ters with preference loss in Equation 2. Through parameter optimization, each reward function can accurately evaluate the trajectory quality, facilitating efficient policy guidance.
Stage I: The Initialization Stage
Initialize reward and critic populations:
P Reward L.MMM. (T., C, Preward)
P Penie LLMMLL, C, Patte)
Stage II: The Evolutionary Improvement Stage
4: for each iteration do
5: # Modular Crossover
6: Sample parents Fr Pp2 ~ DF
4.4. R$ Pseudocode
Construct new functions through crossover
To provide a clear overview of R*, we present the pseu- docode Algorithm 1. Specifically, we begin with the Initialization Stage, where the task description L, environ- ment code C, and prompts Partic and Prwa provided as contextual inputs to the LLM, which then generates two populations: Friic and Rewward (line 3). Next, we proceed
J" = Crrsssovrr (Fpp. Fpp))
8: Add new functions into PReward
9:
# Parameter Alignment Optimization
10: Sample trajectories and label them using Pritie
11:
Add labeled data into Dr.
12:
Optimize parameters based on Dr using Eq. 2 Reward Population Evaluation
to the Evolutionary Improvement Stage. We first sample par- ents Fp, and Fpz from Dy and apply the crossover operator to generate multiple offspring. which are then added to the reward function population l'Reward (line 6-8). Subsequenly, sample new trajectories and annotate them using Critie incorporating the labeled segments into Dr. Based on the collected data, we optimize the reward parameters accord- ing to Equation 2 (line 10-12). We then conduct population evaluation, update the reward archive Dr and obtain the best reward function Feett (line 14-15). Finally, we employ the LLM to refine the reward functions within the popula- tion (line 17). The above shows the complete optimization process. In the next section, we conduct comprehensive experimental evaluation of R
13:
14: Evaluate with parallel PPO training
15:
Summarize training result, update the reward func- tion archive De and get the best reward function Fbest
16:
# Reward Population Evolution with LLM
17: Improve reward population with LLM reflection
Rewaard = Reflection[L, C, Preward, Frest)
18: end for
iterations is set to 5, The key difference lies in how individu- als in the population generated. Specifically, among the 16 individuals, 12 are generated or improved by the LLM, and 4 are generated through the module-level crossover. To guarantee that each experiment fits within a 40 GB GPU memory budget (e.g.. using two NVIDIA 3090 or 4090 GPUs), we adjust the number of parallel environment in- stances for certain tasks, which significantly increases the task learning difficulty. Detailed configurations are listed in Appendix A. To ensure fair comparison, all other hyperpa- rameter settings and related prompts remain consistent with Eureka. RL training is conducted using PPO under identical configuratioss, as detailed in Appendix A. All statistics are obtained from 5 independent runs. We report the average with 95% confidence interval.
5. Experiments
5.1. Environment Setting
We compare R* and the baseline methods on diverse robotic manipulation tasks to evaluate their reward generation ca- pabilities. All methods used GPT-4o the backbone LLM for generation. The test tasks are derived from the Isaac Gym (Makoviychuk et al., 2021) and the Bidextrous Manip- ulation (Dexterity) benchmark (Chen et al., 2022). These tasks cover manipulation tasks using robotic arms and single dexterous hands, as well as tasks requiring dual dexterous hands to perform complex operations, ranging from object handover rotating cup by 180 degrees. Our implemen- tation is primarily based on Eureka (Ma et al., 2024), the population size is set to 16, and the number of evolution
5.2. Baselines
Eureka (Ma et al., 2024) is a of LLM-based method for reward design. It constructs reward functions
6

---

R Efficient Reward Design via Reward Structure Evolution and Parameter Alignment Optimization with LLMs
FaiikkaCahnnet
Handdow-Hand-wwing..cc
Handow-Hand-OOer
Studow-Iunnd-Seison
1.0
0.7
Sacccs Rate
0.6
0.4
0.4
0.4
Onsee
0.2
Sponee
0.0
0.0
0.0
250
400
600
800
1000
1200 1400
1000
2000
5000 Dlettion
4000
5000
1000
2000
1000
4000
5000
5000
1000
2000
9000 Derations
4000
5000
6003
Iterations
Deeations
Allegro-Innee Hand
Shadow Hand- Door Open- Oent-Outwrrd
Shudow Haaee--and---aa Hann--Ketteee Kettle
Shadow Shadow-Handd-Pn Hand--anndeee Fiand-PPen
0.7
0.3
0.
0.5
0.6
0.4
0.3
0.1
0
2000
5000
0.0
3000 Itention
4000
$000
4000
5000
0.0
1000
2000
50.0
6000
Deritions
Figure 2. Performance comparison various manipulation tasks. R significantly outperforms other baselines and surpasses human- designed rewards most tasks.
from scratch without incorporating any human prior knowl- edge. Similar to our method, Eureka maintains a reward population constructed by an LLM and improves the re- ward functions within the population through a reflection mechanism.
Shadow- Hanedov-aann--verr
Franka
1.00
0.5
0.5
0.4
0.2
Oracle: In these tasks, reward functions manually crafted by human experts are used to guide policy learning. These reward functions represent expert-level human reward engi- neering. We use these manually designed reward functions for policy training as baselines for comparison.
000
0.5
Shadow- Han------n------ Swiinnd-wwinn---- Cup
0.5
1.0
0..0
0.4
Sparse: Based on the success criteria of the tasks, sparse reward is provided, where a reward of is given only when the task is successfully completed, and 0 is returned at all other times.
0.2
0.6
0.0
Esclation
Figure 3. Comparison of the best policy success per genera- tion between R and Eureka. R achieves higher evolutionary efficiency.
5.3. Performance Evaluation
We first evaluate R# and baselines on the diverse robotic manipulation tasks with different characteristics. The ex- perimental results are shown in Figure 2. We observe that the reward functions generated by R* significantly outper- form the strong baseline Eureka in guiding policy learning. Furthermore, R* exhibits more stable learning performance, whereas Eureka suffers from policy collapse in some tasks. In addition, R* achieves superior results compared to the oracle reward function in most tasks, indicating that the reward functions designed by R* can match or even surpass those designed by human experts. Sparse rewards gener- ally fail effectively guide policy learning and are unable to yield effective policies in almost all tasks, further high- lighting the importance of high-quality reward guidance. Furthermore, we perform experiments under the original parallel environment settings used in Eureka, where R* also demonstrates a significant performance advantage. Detailed
results are presented in Appendix A.
Since both R* and Eureka are evolutionary-basse reward function generation methods with identical settings, we com- pare the best rest-performing individuals with the highest suc- cess rates in the population at each generation, shown in Figure 3. The experimental results demonstrate that R* achieves higher success rates at earlier generations, indicat- ing a more efficient evolutionary efficiency. This further validates the efficiency of R
5.4. Analysis and Ablation Study
In this subsection, we further analyze R# to answer the following questions:

---

R Efficient Reward Design via Reward Structure Evolution and Parameter Alignment Optimization with LLMs
Shadow-Hand-Ohe
Honddon-aand-Kerte
Lill
Consoee LLM Ceserted
0.8
0.7
0.7
0.5
Protubiilts
0.5
0.4
0.4
0.3
Al Cott
weo Al
0.1
0.2-
0.0
1000
2000
5000
1000
5000
6000
0.0
1000
3000 Vomiion
4000
6000
6.0
Hond Over
Frank at Cahiaee
Swing-Cup Cup
Hand-PPan
Hand Katte
Doot-Oppon -hhrward
Silsuuss
ARgrondond
Figure 6. Ablation study on the crossover operator and alignment optimization.
Figure 4. Performance comparison on various manipulation tasks. Individuals generated by crossover are more likely to achieve better performance.
Omes
Aonracy
VLM
0.6
Shaddon-Haand-Ovr
Shadow-HH Haato--HHaa--llli
08
0.6
R44
0.5
83
096
0/
0.0
FranlaCablnet
Shadow-Hnud Beeoo--Hnd--eeee
Backa vi Uneanerid)
0.2
Dasta Coosmmntt)
Dank
6.1
Figure 7. Accuracy achieved by different labeling methods. Our method achieves the best accuracy in data annotation, while VLM has the lowest accuracy.
0.0
bamiiin
Figure 5. Performance comparison across different numbers of crossover individuals. high proportion crossover individuals can lead to performance degradation.
viduals further enhance performance? To explore this, we increase the number from 4 to 8. The experimental re- sults in Figure 5 show that rather than improving perfor- mance, this adjustment leads to decline, sometimes even underperforming the baseline. The primary reason is that crossover primarily integrates and refines existing high- quality reward functions, relying on the LLM -MM-geeertted reward functions as foundation. Reducing the number of individuals generated by the LLM significantly weakens the algorithm's exploration capability, hindering its ability to discover high-quality reward functions and ultimately resulting in degraded performance.
Q1. Does the module-level crossover in R* significantly enhances algorithm performance? How does its impact compare to that of the LLM in reward function design, and which component plays more critical role?
Q2. Can the critic population constructed by R* provide high-quality data annotations? Additionally, does the pa- rameter optimization in RR further improve the quality of the reward function?
To answer Q1, we compute the ratio of generations in which the best policy originates from crossover, and compare it to the ratio of generations which the best policy originates from L.LM-driven refinement. The experimental results in Figure show that the probability of best policies originat- ing from crossover exceeds 50% in most tasks, with some tasks surpassing 80%. Notably, only 4 individuals are gen- erated through crossover, whereas 12 individuals are refined and generated by the LLM. This demonstrates that leverag- ing prior superior reward functions for further optimization facilitates a more efficient search for high-quality reward functions. Additionally, we conduct ablation study crossover. As shown in Figure 6, removing crossover leads to decline in the performance of R*, further demonstrating the effectiveness of module-level crossover.
To answer Q2, we first compare several different data an- notation methods, including: I) Using images as inputs to GPT-4o (VLM); 2) Providing key information as context to GPT-4o (LLM); Utilizing our proposed critic population (Ours). We begin by collecting 100 paired comparison sam- ples for both the FrankaCabinet and Shadow-hand-kent tasks, which are manually labeled by human annotators based on numerical comparisons. The accuracy results in Figure show that our method achieves 100% accuracy, whereas the L-a-baseed approach performs worse, particu- larly when the numerical differences are small. Moreover, the VLM usually fails to make correct distinctions and out- puts O. These results demonstrate that the critic-based voting mechanism provides more reliable data annotation.
Furthermore, as shown in Figure 6, removing parameter alignment optimization leads to a significant performance decline in R* This demonstrates that directly using L.MM-
Based on the above experiments, we pose a new question: Does increasing the number of crossover-generated indi-
8

---

R Efficient Reward Design via Reward Structure Evolution and Parameter Alignment Optimization with LLMs
generated parameters may result in suboptimal policies, further demonstrating the effectiveness of the proposed pa- rameter alignment optimization.
wider range of real-world scenarios, making a promising direction for future research.
We also provide the generated reward functions and critic functions for several tasks. Due to space limitations, please refer to Appendix B for details.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
6. Conclusion
We propose an efficient framework R* for automatic re- ward function generation, which consists of two key com- ponents: Reward Structure Evolution and Parameter Align- ment Optimization. To design high-quality reward function structures, R* leverages L.MSs to generate a population of modular-based reward functions. The reward function fit- ness is defined as the policy performance it guides. We leverage the LLM's reflection mechanism and the module- level crossover for efficient exploration and exploitation of reward functions. To address the parameter configuration challenge, we introduce a critic-based voting mechanism for step-level labeling to collect trajectory segments. We then optimize the reward funetion population using pref- erence loss based on the collected data. Experiments eight diverse manipulation tasks demonstrate that R* sig- nificantly outperforms other strong baselines and surpasses reward functions.
A Akknowledgments
This work is supported by the National Natural Science Foundation of China (Grant Nos. 62422605, 92370132). We would like to thank all the anonymous reviewers for their valuable comments and constructive suggestions, which have greatly improved the quality of this paper.
References
Adams, S. C., Cody, T., and Beling, P. A. A survey of inverse reinforcement learning. Artif. Intell. Rev., 2022.
Aguinaco, I. E., Muñoz, A. S., Chrysostomou, D., Hidalgo, I. I., Bøgh, S., and Arexolaleiba. N. A. A review on rein- forcement learning for contact-rich robotic manipulation tasks. Robotics Comput. Integr. Manuf., 2023.
human-expeen
7. Limitations & Future Work
Arora, S. and Doshi, P. A survey of inverse reinforcement learning: Challenges, methods and progress. Artif. Intell., 2021.
In this paper. the experiments primarily focus on simulation- based manipulation and dexterous hand control tasks, where low-level information can be directly accessed. For real robot control, a real2sim2real paradigm can be adopted. Since real-world information from the robot and its envi- ronment are available, training can proceed by aligning the information from the real robot with that simulation. This enables direct deployment of the trained policies in real- world scenarios. For tasks where low-level information is inaccessible-sus such non-invasive control tasks only visual observations (i.e., images) are typically available. We believe that the key to applying R* in such cases lies in effective information extraction. In these scenarios, a detec- tion model (e.g., YOLO) can be used identify and extract relevant features from visual input. Once these key features obtained, the subsequent reward generation and policy training processes consistent with those used in tasks where low-level information is accessible.
Arulkumaran, K., Deisenroth, M. P., Brundage, M., and Bharath, A. A. A brief survey of deep reinforcement learning. arXiv preprint, 2017.
Bäck, T. Evolutionary Algorithms in Theory and Prac- tice: Evolution Strategies, Evolutionary Programming. Genetic Algorithms. 1996.
Bäck, T. and Schwefel, H. An overview of evolutionary algorithms for parameter optimization. Evol. Comput., 1993.
Bai, H., Cheng, R., and Jin, Evolutionary reinforcement learning: A survey. arXiv preprint, 2023.
Booth, S., Knox, W. B., Shah, J., Nickum, S., Stone, P., and Allievi, A. The perils of trial-and-error reward design: Misdesign through overfitting and invalid task specifica- tions. In AAAI, 2023a.
Existing methods, including Eureka, Texi2Reward, and our proposed R*, do not support image-based inputs and are applicable only when all low-level information is fully ac- cessible. This constitutes key limitation of current ap- proaches. We helieve that incorporating image-based inputs can improve generalizability and enable application to
Booth, S., Knox, W. B., Shah, J.. Niekum, S., Stone, P., and Allievi, A. The perils of trial-and-error reward design: Misdesign through overfitting and invalid task specifica- tions. In AAAI, 2023b.

---

R Efficient Reward Design via Reward Structure Evolution and Parameter Alignment Optimization with LLMs
Bradley, R. A. and Terry, M. E. Rank analysis of incom- plete block designs: I. the method of paired comparisons. Biometrika, 1952.
Ichter, B., Brohan, A., Chebotar, Y., Finn, C., Hausman, K., Herzog, A., and Others. Do can, not as I say: Grounding language in robotic affordances. In CORL, 2022.
Burda, Y., Edwards, H., Storkey, A. J., and Klimov, O. Exploration by random network distillation. In ICLR, 2019.
Kaufmann, E., Bauersfeld, L., Loquercio, A., Müller, M.. Koltun, V., and Scaramuzza, D. Champion-level drone racing using deep reinforcement learning. Nature, 2023a.
Chen, Y., Wu, T., Wang, S., Feng, X., Jiang, J., et al. To- wards human-level bimanual dexterous manipulation with reinforcement learning. In NeurlPS, 2022.
Kaufmann, T., Weng, P., Bengs, V., and Hüllermeier, survey of reinforcement learning from human feedback. arXiv preprint, 2023b.
Christiano, P. F., Leike, J., Brown, T. B., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. In NeurlPS, 2017.
Khadka, and Tumer, K. Evolution- guided policy gradient in reinforcement learning. NeurIPS, 2018.
Dong. Z., Yuan, Y., HAO, J., Ni, F., Mu, Y.. ZHENG, Y., Hu, Y., Lv, T., Fan, C., and Hu, Z. Aligndiff: Aligning diverse human preferences via behavior-customisable diffusion model. In ICLR, 2023.
Knox, W. B., Allievi, A., Banzhaf, H., Schmitt, F., and Stone, Reward (mis)design for autonomous driving. Artif. Intell., 2023.
Ladosz, P., Weng, L., Kim, M., and Oh, H. Exploration in deep reinforcement learning: A survey, Inf. Fusion, 2022.
Drugan, M. M. Reinforcement learning versus evolutionary computation: survey on hybrid algorithms. Swarm Evol. Comput., 2019.
Lee, H., Phatale, S., Mansoor. H.. Mesnard, T., Ferret, J., Lu, K., Bishop, C., Hall, E., Carbune, V., Rastogi, A., and Prakash, S. RLAIF vs. RLHF: scaling reinforcement learning from human feedback with Al feedback. In ICML, 2024.
Eschmann, J. Reward function design in reinforcement learning. Reinforcement learning algorithms: Analysis and Applications, 2021.
Faust, A., Francis, G., and Mehta, Evolving rewards to automate reinforcement learning. arXiv preprint, 2019.
Lee, K., Smith, L. M., and Abbeel, Pebble: Feedback- efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training In ICML. 2021.
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. NeurlPS, 2014.
Li, P., Hao, J., Tang, H., Zheng, Y., and Fu, X. Race: Im- prove multi-agent reinforcement learning with represen- tation asymmetry and collaborative evolution. In ICMI., 2023.
Guan, L., Valmeekam, K., and Kambhampati, S. Relative behavioral attributes: Filling the gap between symbolic goal specification and reward learning from human pref- erences. In ICLR, 2023.
Li, P., Hao, J., Tang, H., Fu, X., Zhen, Y., and Tang, K. Bridging evolutionary algorithms and reinforcement learning: comprehensive survey on hybrid algorithms. TEVC, 2024a.
Gupta, A., Savarese, S., Ganguli, S., and Fei-Fei, L. Embod- ied intelligence via learning and evolution. Nat. Commun., 2021.
Li, P., Jianye, H., Tang, H., Zheng, Y., and Barez, Z. Value- evolutionary-basse reinforcement learning. ICML, 2024b.
Hao, J., Li, P., Tang, H., Zheng, Y., Fu, and Meng, Z. Erl-re?: Efficient evolutionary reinforcement learning with shared state representation and individual policy representation. ICLR, 2023.
Li, P., Zheng, Y., Tang, H., Fu, X., and Hao, J. Evorainbow: Combining improvements evolutionary reinforcement learning for policy search. In ICML., 2024c.
Hare, J. Dealing with sparse rewards in reinforcement learn- ing. arXiv preprint, 2019.
Liu, J., Yuan, Y., Hao, J., and Others. Enhancing robotic manipulation with ai feedback from multimodal large language models. arXiv preprint, 2024.
Ho, J. and Ermon, S. Generative adversarial imitation learn- ing. In NeurlPS, 2016.
Icarte, R. T., Klassen, T. Q., Valenzano, R. A., and Mcllraith, S. A. Reward machines: Exploiting reward function structure in reinforcement learning. J. Artif. Intell. Res., 2022.
Lutz, I., Wang, S., c. Norn Courbet, A., Borst, A. J., Zhao Y., Dosey, A., Cao, L., Xu, J., Leaf, E. M., et al. Top- down design of protein architectures with reinforcement learning. Science, 2023.
10

---

R Efficient Reward Design via Reward Structure Evolution and Parameter Alignment Optimization with LLMs
Ma, Y. J. Liang, W., Wang, G., Huang, D., Bastani, O., Jayaraman, D., Zhu, Y., Fan, L., and Anandkumar, A. Eureka: Human-level reward design via coding large language models. In ICLR, 2024.
Wang, Y., Xian, Z., Chen, F., Wang, T., Wang, Y., Fragki- adaki, K., Erickson, Z., Held, D., and Gan, C. Robo- gen: Towards unleashing infinite data for automated robot learning via generative simulation. In ICML, 2024b.
Makoviychuk, V., Wawrzyniak, L.,, Guo, Y., Lu, M., Storey, K., Macklin, M., Hoeller, D., Rudin, N., et al. Isaac gym: High performance GPU based physics simulation for robot learning. In NeurIPS, 2021.
Xie, T., Zhao, S., Wu, C. H., Liu, Y., Luo, Q. Zhong, V., Yang, Y., and Yu, T. Text2reward: Reward shaping with language models for reinforcement learning. In ICLR, 2024.
Menell, D. H., Milli, S., Abbeel, P., Russell, S. J., and Dragan, A. D. Inverse reward design. In NeurlPS, 2017.
Yuan, Y., Hao, Ma, Y., Dong, Z., Liang, H., Liu, J., Feng, Z., Zhao, K., and Zheng, Y. Uni-rlhf: Universal platform and benchmark suite for reinforcement learning with diverse human feedback, 2024.
Ng, A. Y. and Russell, S. Algorithms for inverse reinforce- ment learning. In ICML, 2000.
Yuan, Y., Cui, H., Chen, Y., Dong. Z., Ni, Kou, L., Liu, J., Li, P., Zheng, Y., and Hao, J. From seeing doing: Bridging reasoning and decision for robotic manipulation, 2025.
Nickum, S., Barto, G., and Spector, Genetic program- ming for reward function search. IEEE Trans. Auton. Ment. Dev., 2010.
Puterman, M. L. Markov decision processes. Handbooks in operations research and management science, 1990.
Zeng, Y., Mu, Y., and Shao, Learning reward for robot skills using large language models via ceff-alignment In ICML, 2024.
Radosavovic, I., Xiao, T., Zhang, B.. Darrell, T., Malik, J., and Sreenath, K. Real-world humanoid locomotion with reinforcement learning. Science Robotics, 2024.
Zheng, Y., Xie, X., Su, T., Ma, L., Hao, J., Meng, Z., Liu, Y., Shen, R., Chen, Y., and Fan, C. Wuji: Automatic online combat game testing using evolutionary deep reinforce- ment learning. In ASE. IEEE, 2019.
Ramachandran, and Amir, Bayesian inverse reinforce- ment learning. In IJCAI, 2007.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint, 2017.
Zhou, X., Yuan, Y., Yang, S., and Hao, J. MENTOR: guiding hierarchical reinforcement learning with human feedback and dynamic distance constraint. IEEE Trans. Emerg. Top. Comput. Intell., 2025.
Sharma, P., Sundaralingam, B., Blukis, V., Paxton, C., Her- mans, T., Torralba, A., Andreas, J., and Fox, D. Correct- ing robot plans with natural language feedback. In RSS, 2022.
Zhu, K. and Zhang, Deep reinforcement learning based mobile robot navigation: A review. Tsinghua Sci. Technol, 2021.
Sigaud, O. Combining evolution and deep reinforcement learning for policy search: survey, arXiv preprint, 2022.
Ziebart, B. D., Maas, A. L., Bagnell, J. A., and Dey, A. K. Maximum entropy inverse reinforcement learning. In AAAI, 2008
Silver, D., Singh, S., Precup, D., and Sutton, R. S. Reward is enough. Artif. Intell., 2021.
Singh, I., Blukis, V., Mousavian,
A., Goyal, A., Xu, D., Tremblay, J., Fox, D., Thomason, J., and Garg, A. Prog- prompt: Generating situated robot task plans using large language models. In ICRA, 2023.
Sutton, R. S. and Barto, A. G. Reinforcement learning an introduction. 1998.
Vikhar, A. Evolutionary algorithms: A critical review and its future prospects. ICGTSPICC. IEEE, 2016.
Wang, L., Ling, Y., Yuan, Z., Shridhar, M.. Bao, C., Qin, Y., Wang, B., Xu, H., and Wang, X. Gensim: Generating robotic simulation tasks via large language models. In ICI.R 2024a.

---

R Efficient Reward Design via Reward Structure Evolution and Parameter Alignment Optimization with LLMS
A. Implementation Details
All prompt designs in R* for reward function design are based on the prior work Eureka (Ma et al., 2024), without any additional modifications. The specific design details can be found in the original Eureka paper and the corresponding code repository 1. The reward function population has a size of 16, while the critic population has size of 5. The number of crossover individuals is 4 for all tasks. If the reward function generation by the LLM encounters runtime failures (e.g., due to compilation error), the failed individuals are supplemented with those generated via crossover. For instance, if 2 out of 12 reward functions generated by the LLM fail to compile. the modulc-level crossover will be used to generate 2 additional functions as supplement. No extra API calls are required in this process.
Since there no archive during the first iteration, crossover cannot be applied in the first iteration. Thus we adopt a two-step iteration approach: first evaluate the individuals generated by the LLM,, then generate individuals via crossover for evaluation, and finally aggregate all evaluation results to select the best individual. This method does not increase the computational workload, but it does require more time. For the modularization of code, we use the Abstract Syntax Tree (AST) to decompose the reward functions based on the returned reward dictionary. This process returns the code blocks corresponding to each reward component, which are then used for subsequent module-level crossover.
In Eureka, some tasks use a large number of parallel environments, requiring at least 4 RTX 4090 GPUs to run. We reduce the number of parallel environments to ensure that the program can run with only 40GB of GPU memory. However, this reduction significantly increases the learning difficulty for the algorithm. The specific configuration of the number of parallel environments is shown in Table 1. All experiments conducted under the same configuration. Moreover, we conduct
Table I. Number of parallel environment instances per task
Franka-Cabinn
Swing-Cup Hand-Over
Hand-Scissors
Allegro-Hand
Door-Open-Outward
Kettle
Pen
Env number
4096
256
512
128
1024
2048
128
256
additional experiments using the original setting for the number of parallel environments. The corresponding results are shown in Table 2. We observe that R* also outperforms Eureka under the original setting.
Table 2. Success rates of Eureka and R across different tasks under the original setting.
Franka
Swing-Cup
Hand-Over
Kettle
Scissor
Door-Open-Outward
Eureka
33%
53%
83%
70%
100%
98%
R*
73%
96%
93%
95%
100%
100%
Here, we provide a detailed explanation of several key aspects. First, regarding the population voting mechanism, each critic function evaluates and ranks the quality of different trajectories. The final output is then determined based on the aggregated results of the five critics. To obtain the final labels, we gradually relax the voting criteria from strict more lenient. Specifically, at the beginning, state pair is labeled only if all five critics reach consensus. We collect 20 trajectory segments, each with a length of at least 5. If the required 20 segments are not collected, the criteria are labxeed-lbeling occurs if 4 out of 5 critics agree, and if necessary, further relaxed to 3 out of 5 critics until 20 labeled samples are obtained. Based on the collected data, we optimize parameters according to Equation 2. For the crossover operator, we use a softmax function based on the success rates to select parent individuals. A reward module from one parent is incorporated into another function to generate new individual. This process is repeated until the required number of individuals is generated.
For parameter optimization, we split the collected data into two parts: 70% as the training and 30% the validation set. We perform 1000 iterations of optimization on all training data, evaluating accuracy on the validation set after each training iteration. Finally, we select the parameters that achieve the highest accuracy on the validation set as the final optimized parameters.
To construct the critic function, we use the task description, available variables, and success criteria as inputs to guide the L.M in generating the reward function. After the reward functions are generated, we conduct an additional round of reflective optimization. Details of the prompt design are provided later.
12

---

R Efficient Reward Design via Reward Structure Evolution and Parameter Alignment Optimization with LLMs
Prompt 1: Critic Function Task Description
You trajectory annotator, tasked with distinguishing the quality of two robot control trajectories.
Your goal is to write data extraction function that extracts fully correct sub-trajectories from two given trajectories and assigns quality labels to them.
The data extraction function can be defined as follows:
{Critic-Fummn function-formatt
Make sure any new tensor variable you introduce on the same device as the input tensors. Here is the list of keys for dict: (keys}
!!! Please ensure that all the keys used exist in the list mentioned above.
The output of the function should consist of label list: label list contains the quality labels for the states at the corresponding indices in the trajectory. If the state in a is better than the state in b at given index, the label is 1; if the state in b is better than the state in a, the label is -1; otherwise, the label is O. The code output should be formatted as a python code string: """python... .*.
Some helpful tips for writing the function code:
(I) Comparison Criteria for States at the Same Index: The evaluation criterion for comparing the states at the same index follows: state can only be considered better than state b if is superior b in all metrics. For example, if two robotic hands are performing task, and a's left hand is better than b's left hand, but b's right hand is better than a's right hand, the label should be 0.
(2) Task Decomposition into Phases for More Accurate Evaluation: The task needs to be divided into multiple phases to allow for more accurate evaluation. For example, in dual-hand operation task, neither hand has grasped the target yet, then comparing the distances to the target is more appropriate. However, if both hands have grasped the target, then the comparison should focus on the distance to the target itself.
(3) Do not set thresholds; directly perform numerical comparisons on the information, for example, if < b for all distance, better.
(4) The provided information consists of raw, unprocessed data. Distance information and other metrics need to be calculated.
(5) Typically, you need to consider two distances: the distance between the manipulator or robotic arm and the block to manipulated, and the distance between the manipulated block (cube) and the target. Or other distances, such as the distance the drawer pulled open, the distance between two objects, and so on.
13

---

R Efficient Reward Design via Reward Structure Evolution and Parameter Alignment Optimization with LLMs
Prompt 2: Critic Function Design Tips
The Python environment {task_obe _oisk_obs_ur code code_ string). desskckescriition)
Write comparison function for the following task:
The conditions for determining task success First, the task needs to be analyzed, and based on the task, the objects that need to be manipulated should be identified. Then, the variables required to complete the task should be inferred. For example, we need to determine the specific positions where the left and right hands should approach. Simply stating that the hands should approach the object is not efficient.
follows: (task goal}
Please provide an analysis before starting, clearly defining the metrics be calculated and whether there are more appropriate metries that could replace them. Finally, check if the code achieves the intended purpose.
The following three types of distances need to be considered:
(1). The (five) distances (not sum) between the five fingers (if existing) of the left hand and the target (left) block or cube.
(2). The (five) distances (not sum) between the five fingers (if' existing) of the right hand and the (another right) target block or cube.
(3). Based on the task success criteria, the (rotational) distances between the objects (object and the target positions), as required.
The three distances mentioned above should be treated three separate metrics and should not be combined pairwise into single metric.
1. Please ensure that your final success distance metric is correctly represented, and avoid fabricating non-existent target information.
2. In addition, use traj a[key][:33] to get values instead of traj a[key][3], use the numerical distance metrics to compare between trajectories instead use bool value, as directly judging success metric (success a dist_a < 0.2) at the start of the trajectory will always return False, resulting in invalid comparison.
3. Typically, it is unlikely for the left hand operate the right side the right hand operate the left side.
4. If the key does not exist in the list, do not create the variable.
Prompt 3: Critic Function Reflection Tips
Reflect on the correctness of the above codes: Does the code include the following components, and does the code adhere the given recommendations?
1. verify whether any components missing, do not sum any components.
(1). The (five) distances (not sum) between the left five fingers (if existing) of the left hand and the (left) target
block or cube.
(2). The (five) distances (not sum) between the right five fingers (if existing) of the right hand and the (another right)
target block or cube.
(3). Based on the task success criteria, the (rotational) distances between the objects (object and the target positions),
as required.
2. Verify each criterion the key in the list and follow the code format. Key list: (keys}
3. Use the numerical distance metrics compare between trajectories instead of use bool value, as directly judging success metric (success.c a dist < 0.2) at the start of the trajectory will always return False, resulting in invalid comparison.
4. Reasonableness check: is unlikely for the left hand to operate on the right side or the right hand to operate on the left side.
Check the code against the four aspects mentioned above one by one and analyze whether they are satisfied. If there are no issues, please output the original code; if there issues, please output the modified code following the same format CCritic-Function--or
14

---

R Efficient Reward Design via Reward Structure Evolution and Parameter Alignment Optimization with LLMs
Critic Function Format
def get_correct sub traj and labels (trajectory__) trajectory.b)
Both trajectory_a and trajectory_b are lists, each containing multiple states, and each state is represented as dictionary.
The Label_list is a list of indices that corresponds to the stares in the sub- trajectory where is definitively better or worse than В. The label indicates the quality of the sub-trajectory:
is better, the label Former"; if B is better, the label is "Latter".
If no vaiid sub-trajectory can be found between the two trajectories, labol list will be empty 1ist (II)
traj_a_iength len (trajecctoyy_)) traj_b_length - len (trajecctory_b) assert araj_a_length -- tranth_
label []
for state_index in (tajettraaalleenghhh:
state_a_info trajectory.allstao_lnneddd
metric_l_a metric_l_b and metric_ metric_2_b and metric_2_4 metric_2_t and ...::
Label_list.
elif metric_l_a <= metric_1_b and metric_2_a <= metric_2_ and metric_2, __a <=
metric_2_b and ....
Labe__
else:
label 1aatlllisttapp (0)
return label label_lis 1ist
B. Generated Function Examples
B.1. Reward Function Examples
We provide the reward function generated by R* below. It can be observed that the rewards given by R* are reasonable, with optimized parameters, unlike other methods that only provide integer or single-decimal values.
B.2. Critic Function Examples
We provide the LLM-generated critic reward function for Franka-Cabine and Shadow-Hand- -att-Kell--ttlee as follows. Upon observation, we find that while the LLM's overall judgment criteria are generally correct. However, this does not necessarily lead to better performance. For example, proximity alone does not imply that one state is superior to another. We consider further exploring this issue in future work.
15

---

R Efficient Reward Design via Reward Structure Evolution and Parameter Alignment Optimization with LLMs
The Reward Function For Franka-Cabinet
from typing import Tuple, Dict import math
import torch from torch import Tensor jiorch.jit.sct
def comptte_reward (drawer_grase) pos: T
Tensor, franka_grasp_poss torch.
Tensor, cabinet_dof_ pos: torch. Tensor, cabinet_dof_ve vel: torch, Tensor) ->Tuple( Tensor, Dict (str, torch.Tensory
Reward weight hyperparameters
sellsstane_weiggee 1.5603480339050228
drawer_ sereer__pee_weee weiie 0.100000000149011112 velocityy wwlooctttyeiiieeee
197826731204998677
Temperature parameters for trans
distance temp 3. 1.867816686630249 open temp 0. 0.10000000149011611
velocity_tent temp = 4. 4.029726982116699
Compute the distance from the hand to the drawer band_to Mamn_tto_drawer_dsttaan franka_grasp_pos, dim=-1)
Corch.nomn (draver_grasp_pos
Reward for minimizing the distance to the draver
distance_rewand reward exrch.exp (-distaacce_tenp tondwte_ddwweddee
* pps_distance_weige
Reward for opening the door, with normal pemalized scaling open_reward torch, exp (open_temp torch, abs (cabinet_dof_pos [:, (3)]
cabinet_doffposl:: (3) clamp (max-1.0)) drawer_open_weiight
$ Reward for positive velccity movements towards opening the drawer velocity_rexard torch.exp (velocity_temp clamp (min=0.0)) velocity_weight
booinet_dof_vell__ (3)).
$ Total reward is a combination of the individual rowards = (2. 10652207226644 distance_reward + 1.0 spen_reward +
1.9782673120498657 velocityyreward)
Individuat reward components stored in a dictionary for analysis reward (' distancee I"distance_reward" reward'' distance reward, rewar_r
open reward, 'velocity "velocity_rewaraa" reward'; relocity_rewarc]
return total_reward, reward_dict
16

---

R Efficient Reward Design via Reward Structure Evolution and Parameter Alignment Optimization with LLMs
The Reward Function For Shadow-Hand-Ketie
from typing import Tuple, Dict
import math
import torch from torch import Tensor jtorch.jit.sct
def hompptt__rowrrdrrgghhh...dddddd
torch. Tensor, left_hand_pos: torch.
Tensor, kettle_hand @eatle_handle__pee Tensor, kettle_spout_poss forchrTensor, pucket_handle_ron torch. te object_pos: torch. Tensor, object_
Forch.Tensor. pucket_handle, torch.
Tensor, object_angvel:
torch. goal_pos: torch. goal_rot: torch. ->xuplettozchh.eense [str, torch. torch.Tensor]]: distarce_kettie_graap torch torch..nommmiiiga_aaaa.... dim=-1)
ketdle_handle_por,
grasp_kettie_rewarr - 1.0 (1.0 distanca_etttle_grrraa
grasp_kettle__rwaard 1.277540922164917 / (0.6857131719588923) torch.
norm pos kettle handle pos, dim=-1))
grasp pestle_rrwarke 0.9725923538208008
www.bbuuccettrrteeeeee grasp_
1.305189833522888 / 10.6645537614922388 + torch.
torm form[leftthand__os - bucket_handle_ppn
dim=-1))
grasp_ grasp_bucket_rcwardd 0.99533392332633655
eligmment_tolcranco: 0.39174232298172
goal_ pos, dim=-1)
spott_aiggment__rewaddd-
spptt__oo_hoceee__ddssaaeeeee
21,
C.01000000474974113, None)
None)
* 2. 2.22683620452888888
1.30611877033358
reaaa + 0.829009532928666
pourzing velociiyyyreeaaa 1.277540922164911
grasp_ kettle_rewadd)
reward reward_componde
= ("grass httrrasp_ettleereeeeeeeeeeeeee
grap_biccer__evvrdd: grappppbceeeeeeeeeeeeaa.
spoit_alligmenttrrowddd''' otgp://aooocopooaaaaaaaaooaooaar
pouzinng.velociit.rroaaad pouzinggvvooocityyyooottd
grasp_ktttle_rowarde grasp_kettlerrowardl
return total_reward, reward_components
17

---

R Efficient Reward Design via Reward Structure Evolution and Parameter Alignment Optimization with LLMs
The Critic Function For Franka-Cabinet
def calculate distance (vec_a, a, vec vec_b):
return [(vec_a
2) (axis=1) sqrrt
def get cet_ccorectt.o sub traj sraj_and Labels (trajectory. trajectery_b)
traj_a_length 1en (crajectory_a)
traj_b_length - len (trajectoor__b)
assert tracha_ =- trag_b_length label_list
for state_index in [angetttaa.__llenghhh
state_a_info = seajactory_alsttae__nnees index
state_b_info = trajectory__llttte__neee
Caicuiate the manipulation distances
(stataa___ntttttta__ana franka Lringe__iingee__os ),
F 1,
sireer_dist_a___
olallaaasssllcaaaatecistttaeeees (state infol' franka tranka_rfinger_pog 1,
state a state___infol ptt.aaaaaaaaaaaaaaaaarrapaapa/ea
rfinger_die rfinger_dist_b dii__isttbbb___ calcullaee_cssttnceee (state_ b inffffo'' franka_ rainkkr__fingge__oo__ ),
Cabinet Dor opening comparison (critical for success)
cabinet_dof_a state_a_infol" cabinet_dof_pos! [[;, 3]
cabinet_dof__ state_b_infol cabinet_dof_pos ] 3]
$ Check which trajectory has better performance at this index
if all iifinger_dist_a <= ifinger_dist_D and all (rfinger_dist
<=
rfinger_dist_b) and all (cabinet_dof__ >= b) Labee_
A better
elif all (11iinger__iitt... dist__rr_ddd_______ >= itinger_diss dist_ b) and all (rfingerr_di fiirr_ddst____ a >=
rfinger_dist_by and all (cabinet_dof. <= cabinet_dof_b)
label label_list.appeme
$ is better
else:
label 1ist append
return label list
18

---

R Efficient Reward Design via Reward Structure Evolution and Parameter Alignment Optimization with LLMs
The Critic Function For Shadow-Hand-Kettle
def set_correctt _______u_________ sub_ttubbat_rt pepp_rajj_aem ardj_and labels (trajectory trajectory_b)
import torch
traj a length 1en (trajecctory_
traj_b_iength - len (Crajectory_b)
assert tra]_a_lengh -- traj_b_length
label_list
for state_index in (ttaa
state_a srajectory_a
state_b srajectory_blstate_nnee
Calculate distances for both trafectories
right haatt__haad__iieerr
===ight_hand__f_ppo_"
'right_hand mf pos'
right__han__r__ps! /right__aad__f__oos",
/right_hand_th_pps
@oft_haaddfingers_2 - "riight_hand_ff_po''
fright_hand__f_pon /rightt_han__c__ps!
right__and__f_post /right__aad____pos_,,
left_hand fingers_a
left hand
"ieef_hande =2 pos",
'left_hand in saand____poo',
heft_hand rf rr___rr_______________________ 'left_hand 1f pos'
'left hand
hht.:aandd_____aae_eeeeee__e tb mand_tt_ppos'
left left_hand Seft_hand_fingers__ = ['left_ hand = ________p_______________________ _e_dd____po''
hand__han______ jeft__and_rf__os$ "Left_hand_lf_pos" "eeft_hand_th_pos"
Calculate the right hand's distances
kettle handle
right_hand
(torch.norm tensor (state_a [ffnger)) terch. tensor [ htt__hand_fingersee right_hand diadaddist
]), p=2) for finger in
Coorch.norm(teee (torch. tensor (state_ bttssttttttttteerlllll torch.
Sensor(state_] aaabor/ssttteoooktttteeeeeeeaee hat.keteeeadeeeepss 1), p=2) for tinger tight_hand Soggtt_band_fingers___
$ Calculate the left hand's distances to the bucket handle [torch. norm (torch. tensor (state_a[fingen) ttesor tensor httetttteeaaaaccccctaaaaleeeeeea
left_ ffft__annd_distancess
torch.
}), p-2) for finger in
left left_hand [iiggersan] Beft__hand_figgrs__al
left seft_hand bband_distaccee_b = [torch.norm (twrch.tttctteee tensor (state b(finger))))) torch.
Censor bucket wwucke__handle_peee }), p=2) for finger in Jeft_haand_finerrs_1
Caicuiate the distance between bucket handle and kettle spout
boout_buccet_dlttanneeeeo = torch.norm (oorcoootttttttttntoonnn tensor (state_al buw.allutttttanllle_..o
'1) - (arrch.ttemsrrrttttttaaaaeetttee
pos' }), p-2)
spoct boott_bocket_ddittnee_bb = (troohhh tensor (state_b( bbucke_hhandee p-2)
'1) Berrch.ensorrsstttt__Brk
Determine the better trajectory at this state
if all (ra <= rb for ra, rb in zip (righthht_haad.com hwn..aaittanness.....
right hand distances_pi [b) and all (la lb for la, 1b in zietleft hand distances left__aan__distancee_bbb
and
<= apout_bucket_disttace.
Label_list.appent elif all (ra >= rb for rb in htrriight_hand_cisttneeeee right_hand band_disttaces__b)
$ A is better
and
all (la >= for la, 1b in aip(teet_haadd_distannee,
and
>= ppout_bucket_distance.
label 1ist lpp.
is better
else:
labe lst append (0) $ Neither is better
return label list
19


Reviews:
[{"review_id": "fPklJHXmyf", "reviewer": "Reviewer_CcRx", "summary": "This paper proposes LLM-based reward function generation to train models for tasks such as robotic hand manipulation.\n\nThe presented method can be decomposed into:\n1. generation of modular reward functions using an LLM\n2. augmentation with new reward functions based on modular mixing of the functions from step 1\n3. Generation of trajectories and ratings from an LLM-generated critic population \n4. Reward function parameter optimization based on the labeled trajectories \n5. PPO-based training of agents for the end task \n6. LLM-based reflection based refinement of the reward functions.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback and for a positive assessment of our work. For the reviewer's questions, we will respond to them one by one as follows.\n\n1. **[The novelty of the proposed ideas is low, and seem to be a collection of multiple ad-hoc steps]**\n\nOur work focuses on reward function construction and enhancement from two key perspectives: **reward structure evolution** and **parameter alignment optimization**. The former aims to design meaningful reward components using LLMs and evolutionary principles, while the latter focuses on optimizing the parameters within those reward components.\n\nIn **reward structure evolution**, we introduce code-level crossover operations to fully exploit existing reward function implementations and address potential suboptimality issues.\n\nIn **parameter alignment optimization**, we first use LLMs to construct state-level critic functions. To mitigate potential errors from relying on a single critic, we build a critic population and use it to generate a preference dataset. We then make the numerical values in the reward function code differentiable and apply preference learning to efficiently optimize the reward function.\n\nOur method is primarily designed and optimized based on the challenges associated with reward generation, aiming to construct more efficient reward functions.\n\n2. **[the paper is focused mainly of robotic hand manipulation tasks, and there is no mention of this until the experiments section. The paper should do a better job of providing context and motivation.]**\n\nThank you for the reviewer’s suggestion. We will improve the presentation in the revised version.\n\n3. **[It seems to me that the critic population annotation is more important to the success of the method than crossover-based reward function generation. Is this correct?]**\n\nThe goal of crossover-based reward function generation is to ensure the effective utilization of existing reward functions, which can  facilitate the discovery of well-structured designs. Since parameter optimization does not modify the underlying structure, it is generally most efficient to apply parameter tuning only after a reasonable structure has been identified.\n\nIn many tasks, the initial parameter settings provided by the LLM are already fairly reasonable. In such cases, parameter alignment may have limited impact, whereas crossover can continuously explore and recombine high-quality existing reward functions to uncover improved designs. However, when the LLM provides suboptimal parameter settings, parameter optimization becomes crucial for constructing more effective reward guidance.\n\nBased on our experimental results, parameter alignment is generally more efficient in most cases.\n\n4. **[What is the intuition behind the crossover-based design? I am unable to get a feel for why it leads to such a big improvement in performance, and why the LLMs cannot generate such functions in the first place.]**\n\nEureka continuously improves the reward function by providing the best-performing individual to the LLM, while other reward functions are directly discarded. When the best individual is actually suboptimal, reflecting and improving based on it can lead the entire population to converge toward a suboptimal solution, ultimately resulting in low-quality outcomes.\n\nIn contrast, the crossover-based design aims to fully leverage existing superior reward functions without introducing any additional LLM overhead. By performing crossovers between different reward modules in superior reward functions, it enables thorough exploration and helps avoid being trapped in suboptimal solutions.\n\nAs shown in the results of Figure 4, the reward functions discovered by EA have a significantly higher probability of being the best in the population (the probability of best policies originating from crossover exceeds 50% in most tasks,with some\ntasks surpassing 80%), which further demonstrates EA's ability to effectively explore existing rewards and discover better reward functions.\n\n---\n\n**We would appreciate it if the reviewer can confirm that the concerns had been addressed and, if so, reconsider the assessment. We’d be happy to engage in further discussions.**", "claims_and_evidence": "Claims made: the proposed method outperforms existing SOTA methods for reward function design", "methods_and_evaluation_criteria": "Yes, the proposed methods and evaluation criteria are standard for the premise of the problem considered.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experimental design follows standard methodology and is very similar to existing methods that address reward function design.", "relation_to_broader_scientific_literature": "The paper improved upon previous SOTA methods (https://arxiv.org/pdf/2310.12931) that also uses LLMs for designing reward functions.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n\n- The paper's results are very strong and are significantly better than the existing SOTA methods. \n\nWeaknesses:\n\n- The novelty of the proposed ideas is low, and seem to be a collection of multiple ad-hoc steps \n- The writing can be improved. For example, the paper is focused mainly of robotic hand manipulation tasks, and there is no mention of this until the experiments section. The paper should do a better job of providing context and motivation.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. It seems to me that the critic population annotation is more important to the success of the method than crossover-based reward function generation. Is this correct?\n\n2. What is the intuition behind the crossover-based design? I am unable to get a feel for why it leads to such a big improvement in performance, and why the LLMs cannot generate such functions in the first place.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fjIKTiHD6o", "reviewer": "Reviewer_38wQ", "summary": "This paper introduces R* which designs reward function by utilizing LLMs to construct a set of reward functions, 'evaluating' these rewards by training PPO agents to maximize the rewards, and then improving the reward functions based on voting mechanism followed by preference-based learning. Ablation study shows that the proposed idea of using crossover operator (for better exploration of reward design space) and parameter alignment optimization (based on voting and preference learning) are indeed effective. Experiments are conducted on IsaacGym and Dexterity benchmarks and the main baseline is Eureka that is based on the same high-level idea of generating rewards with LLM, evaluating rewards, and improving rewards based on reflection.\n\n## update after rebuttal\n\nMy score has been updated from 2 to 3 during the rebuttal process. My acceptance recommendation is conditional on the authors' promise to largely update the manuscript (especially Introduction) for improving the positioning of the paper.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback. For the reviewer's questions, we will respond to them one by one as follows.\n\n---\n\n1. **[The performance of PPO with oracle reward seems very low to me -- it's much lower than the performance reported in Eureka tasks. Maybe I'm missing something here but I'm not sure why this happens.]**\n\nThe main reason lies in our environment configuration, which limits the number of parallel environments. In the original Eureka paper, some tasks use a large number of parallel environments, requiring at least 4 RTX 4090 GPUs to run. We reduce the number of parallel environments to ensure that the program can run with only 40GB of GPU memory. However, this reduction significantly increases the learning difficulty for the algorithm.\n\nHowever, to ensure a fair comparison, all experiments are conducted under the same configuration.\n\nThe specific configuration of the number of parallel environments is shown in the table below.\n\n|  | Franka-Cabinet | Swing-Cup | Hand-Over | Hand-Scissors | Allegro-Hand | Door-Open-Outward | Kettle | Pen |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Env number | 4096 | 256 | 512 | 128 | 1024 | 2048 | 128 | 256 |\n\n2. **[One weakness of this paper is that writing is very verbose and some paragraphs are really long. In particular the ones in the introduction.]**\n\nThank you for the valuable suggestions. We will work on improving the overall clarity and expression throughout the manuscript in the revised revision.\n\n3. **[What's the fundamental limitation of Eureka and how is the proposed idea addressing the limitations? Why should the proposed algorithm work better than Eureka?]**\n\nThe limitations of Eureka are mainly reflected in two aspects:\n\n1. **Insufficient utilization of existing knowledge**: Eureka relies on the best-performing individual during each reward function update. However, the individual may represent suboptimal solution, and continuously optimizing based on it can lead to poor results. The introduction of a crossover mechanism aims to fully leverage existing knowledge without introducing any API calls, enabling integrated exploration across different modules of the superior reward functions and ensuring more efficient exploitation.\n2. **Inefficient parameter optimization**: Reward functions often involve numerous parameters, such as weights within and between reward components. Iterative optimization of these parameters using LLM is inefficient. To address this, we propose a preference learning-based approach, which makes the gradients of the reward function parameters differentiable. By collecting preference data from the critic population, we optimize these parameters more effectively.\n\nTo solve above problems, we propose the **reward structure evolution and parameter alignment Optimization**. The former focuses on efficient structure search to avoid getting trapped in suboptimal designs, while the latter performs effective parameter optimization. By combining both components, our method enables more efficient reward design compared to Eureka.\n\n---\n\nWe would appreciate it if the reviewer can confirm that the concerns had been addressed and, if so, reconsider the assessment. We’d be happy to engage in further discussions.", "claims_and_evidence": "In terms of evaluating the efficacy of the newly proposed techniques, the paper provide an ablation study that supports it. But this paper is not making that much of a claim with regard to 'hypothesis that explains why this method should work better than previous algorithm'.", "methods_and_evaluation_criteria": "Eureka also used IsaacGym tasks so it makes sense. Some qualitative/quantiative analysis on the quality of learned reward is missing.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The performance of PPO with oracle reward seems very low to me -- it's much lower than the performance reported in Eureka tasks. Maybe I'm missing something here but I'm not sure why this happens.", "relation_to_broader_scientific_literature": "Key contributions of this paper will be very technical contributions around improving many parts of Eureka. But motivations for introducing these components are very weak so it is not clear if this is of interest to broader literature.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "One weakness of this paper is that writing is very verbose and some paragraphs are really long. In particular the ones in the introduction.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "- The biggest weakness of this paper is that it's not properly positioning the contributions of this paper with regard to existing works, and just say that \"reward design is challenging and we did a bunch of things to do that!\". But the very high-level idea of this work is very similar to Eureka's. Using LLMs to generate reward functions, training policies, and improvement based on reflection. So what's the fundamental limitation of Eureka and how is the proposed idea addressing the limitations? Why should the proposed algorithm work better than Eureka?\n- Would it be possible to provide quantitative/qualitative analysis that compares the quality of learned rewards, instead of only showing the downstream performance?\n- Why is performance with Oracle reward function worse than the ones reported in Eureka paper?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L73TDjZ1LD", "reviewer": "Reviewer_268i", "summary": "This paper proposes a new method for designing reward functions with LLMs, R*. R* uses LLMs to generate modular reward function components, and maintains a population of reward functions. These population of reward functions are evaluated based on how well they guide the agent to the sparse reward. Based on their fitness, these reward functions undergo mutation using the LLM. Furthermore, they do parameter alignment of the parameters in the reward function using a pairwise preference loss.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback and for a positive assessment of our work. For the reviewer's questions, we will respond to them one by one as follows.\n\n1. **[they do not discuss Eureka in depth, so it is really hard to estimate the novelty and contribution of this work. ]**\n\nEureka maintains a population of reward functions and iteratively improves them using an LLM. The improvement process involves feeding the LLM with the best-performing reward function discovered in the current population, along with feedback from its RL training process. Based on this information, the LLM adjusts both the reward function logic and its parameters to enhance performance.\n\nThe limitations of Eureka are mainly reflected in two aspects:\n\n1. **Insufficient utilization of existing knowledge**: Eureka relies on the best-performing individual during each reward function update. However, the individual may represent suboptimal solution, and continuously optimizing based on it can lead to poor results. \n2. **Inefficient parameter optimization**: Reward functions often involve numerous parameters, such as weights within and between reward components. Iterative optimization of these parameters using LLM is inefficient. \n\nTo address the two issues mentioned above, we propose **reward structure evolution** and **parameter alignment optimization**. The former fully leverages high-quality reward functions without introducing any API calls, enabling thorough exploration through crossover among components within superior reward functions. The latter adjusts parameters via preference learning, where a key challenge lies in constructing a reliable preference dataset. To tackle this, we utilize the LLM to generate critic functions—a process that does not involve human participation. Moreover, to ensure the accuracy of preference labels, we build a population of critics and annotate preferences between states through a voting mechanism. Finally, we optimize the reward function parameters based on the learned preferences, providing more accurate reward signals.\n\n1. **[The writing is not that well structured. For example paragraphs 2 and 3 are very long and hard to read.]**\n\nThank you for the reviewer’s suggestion. We will improve the presentation in the revised version.\n\n---\n\n**We would appreciate it if the reviewer can confirm that the concerns had been addressed and, if so, reconsider the assessment. We’d be happy to engage in further discussions.**", "claims_and_evidence": "The authors claims about prior works in the introduction were not fully supported. For example, in line 53 the authors say that the works they build off face instability and inaccuracies in parameter configuration. However, this is not explained in depth and it seems to be a crucial part of their motivation.\n\nThe authors claims about superior performance compared to eureka are well supported by many experiments across different settings.", "methods_and_evaluation_criteria": "Yes the proposed methods and evaluation criteria make a lot of sense. Their work tries automatically design reward functions, and conducts experiments in many popular RL environments to evaluate their approach. Their evaluation based on success rate makes sense as well. In general they try to improve upon Eureka, and use a very similar setting to that work.", "theoretical_claims": "na", "experimental_designs_or_analyses": "The experimental design seems quite sound to me. In all experiments they compare with SOTA baselines (eureka). They also conduct experiments over 5 random seeds and report the mean and standard deviation in their learning curves. Finally, they keep the hyperparameter setting to be very similar to Eureka, which makes it likely to be a fair comparison.", "relation_to_broader_scientific_literature": "This paper does not do a good job discussing their place in the broader literature. To me it seems like they make novel improvements to Eureka, which is a popular framework. However, they do not discuss Eureka in depth, so it is really hard to estimate the novelty and contribution of this work.", "essential_references_not_discussed": "This paper does not really discuss in depth their differences with Eureka (Ma 2024). I think this is necessary, as they seem to directly build upon this work. This makes the contribution of this paper difficult to ascertain.", "other_strengths_and_weaknesses": "Strengths:\n- The methodology is seemingly novel, although that is hard to judge.\n- The experimental results are comprehensive, and their method shows clear improvements.\n\nWeaknesses:\n- The writing is not that well structured. For example paragraphs 2 and 3 are very long and hard to read.\n- This paper mainly seems like a follow up work on Eureka. However, the authors do not clearly state their contribution compared to eureka, or state the benefits of their method. They only use vague language such as prior works have “instability and inaccuracy” (line 54) but do not say concretely how they improve upon it.", "other_comments_or_suggestions": "Typo in line 337.", "questions_for_authors": "Can you provide a more in depth comparison to Eureka?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EsMfHkyMrs", "reviewer": "Reviewer_apPp", "summary": "The paper introduces R*, an efficient framework for automatic reward function generation in reinforcement learning. R* addresses the challenge of designing high-quality reward functions by leveraging LLMs through two key components: reward structure evolution and parameter alignment optimization. The framework uses LLMs to generate modular reward functions, refines them with module-level crossover, and optimizes parameters via a critic-based voting mechanism for step-wise trajectory labeling. Experiments across eight robotic control tasks show that R* significantly outperforms existing methods, including Eureka and human-designed rewards, in both final policy performance and convergence speed.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback. For the reviewer's questions, we will respond to them one by one as follows.\n\n1. **[How are the parents selected, and how is it determined which reward module to insert into another parent?]**\n\nWe maintain a buffer of reward functions with a fixed size of 5 (Only the top-5 best-performing ones are retained), where each reward function is associated with the success rate of the RL policy it guides. We apply a softmax over these success rates and sample two reward functions based on the resulting probabilities.\n\nWe then randomly select reward modules for crossover. The main advantage of this approach is that it does not require any API calls. A potential improvement would be to leverage an LLM to guide the module selection process; however, this typically incurs additional LLM API usage.\n\n2. **[Which reward functions are used for training in Figure 2?]**\n\nThe reward function that achieves the highest RL success rate across all generations is selected for final evaluation, which is consistent with the setting used in Eureka.\n\n3. **[Are the trajectories for parameter alignment optimization sampled using the initially randomized policy?]**\n\nThe trajectories are collected using the trained policies guided by different reward functions within the population.\n\n4. **[Are the generated reward functions only suitable for PPO?]**\n\nThe generated reward functions are applicable to various RL algorithms. To demonstrate this, we conducted experiments on Metaworld, using the generated rewards to guide SAC for policy learning (max success rate and the required steps). The results are as follows:\n\n|  | **drawer-open** | **button-press** | **window-close** |\n| --- | --- | --- | --- |\n| Expert Reward | 100% \\| 33600 | 100% \\| 399000 | 100% \\| 220500 |\n| R* | 100%  \\| 36720 | 100%  \\| 421200 | 100% \\| 772050 |\n\nWe observe that **R*** achieves a 100% success rate on these tasks, matching the performance of the expert-designed rewards.\n\n5. **[I am curious whether the authors have considered strategies to address this issue, or at the very least, whether these limitations should be explicitly discussed in the manuscript.]**\n\nFor complex visual tasks, we believe a feasible direction is to obtain the relative positions of targets through object detection and prediction, and associate them with variables—this typically requires additional training. However, our current method does share the limitation mentioned by the reviewer: it cannot be applied to tasks where low-level information is inaccessible.\n\n6. **[The reported Eureka performance seems significantly lower than that reported in the original Eureka paper.]**\n\nThe main reason lies in our environment configuration, which limits the number of parallel environments. In the original Eureka paper, some tasks use a large number of parallel environments, requiring at least 4 RTX 4090 GPUs to run. We reduce the number of parallel environments to ensure that the program can run with only 40GB of GPU memory. However, this reduction significantly increases the learning difficulty for the algorithm. \nBesides, we use GPT-4o for our experiments, which may also introduce some differences.\n\nHowever, to ensure a fair comparison, all experiments are conducted under the same configuration.\n\nThe specific configuration of the number of parallel environments is shown in the table below.\n\n|  | Franka-Cabinet | Swing-Cup | Hand-Over | Hand-Scissors | Allegro-Hand | Door-Open-Outward | Kettle | Pen |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Env number | 4096 | 256 | 512 | 128 | 1024 | 2048 | 128 | 256 |\n\n7. **[typos: In line 6 of the pseudocode]**\n\nThank you to the reviewer for pointing out this typo. We will correct it in the revised revision.\n\n---\n\n**We would appreciate it if the reviewer can confirm that the concerns had been addressed and, if so, reconsider the assessment. We’d be happy to engage in further discussions.**", "claims_and_evidence": "yes", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "There is not theoretical claim in this submission.", "experimental_designs_or_analyses": "This work conducts experiments in Isaac Gym and the Bidextrous Manipulation (Dexterity) benchmark, which are commonly used benchmarks previously.", "relation_to_broader_scientific_literature": "This work builds on existing LLM-based reward design methods and introduces technical enhancements to improve their performance.", "essential_references_not_discussed": "no", "other_strengths_and_weaknesses": "Strengths\n--\n\n1. The paper is well-structured and the idea of parameter alignment optimization makes sence.\n2. The implementation details and prompt designs are clearly and thoroughly described.\n3. The experimental results are impressive.\n\nWeaknesses\n--\n\n1. The proposed method heavily relies on several assumptions about the environments, including the need for environment code for LLM-based coding and detailed state information for state ranking. These requirements make it difficult to extend the method to more complex and real-world scenarios, such as image-based tasks. I understand that Eureka, the foundation of this method, has also been criticized for these limitations. I am curious whether the authors have considered strategies to address this issue, or at the very least, whether these limitations should be explicitly discussed in the manuscript.\n2. The reported Eureka performance seems significantly lower than that reported in the original Eureka paper.", "other_comments_or_suggestions": "typos: In line 6 of the pseudocode, are $F_{p1}$ and $F_{p2}$ sampled from $D_T$ or $P_{reward}$?", "questions_for_authors": "1. I am uncertain about the crossover operation: How are the parents selected, and how is it determined which reward module to insert into another parent?\n2. Which reward functions are used for training in Figure 2?\n3. Are the trajectories for parameter alignment optimization sampled using the initially randomized policy?\n4. Are the generated reward functions only suitable for PPO?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The submission proposes an effective automated reward design framework, R*, which improves upon existing methods, in reward structure evolution and parameter alignment optimization. Reviewers praised the strong empirical results and comprehensive experiments conducted on robotic control tasks. Initial concerns regarding clarity, novelty, and experimental setups were thoroughly addressed during rebuttal. The authors provided additional experiments and improved manuscript structure. Given these improvements and the paper's clear contributions, the AC agrees with the reviewers to accept the submission.
Decision: Accept (poster)