LaRes: Evolutionary Reinforcement Learning with LLM-based Adaptive Reward Search
Authors: Pengyi Li
Hongyao Tang
Jinbin Qiao
YAN ZHENG
Jianye HAO
Venue: NeurIPS 2025 poster ()
Keywords: Evolutionary Reinforcement Learning
Primary area: reinforcement_learning

Abstract: The integration of evolutionary algorithms (EAs) with reinforcement learning (RL) has shown superior performance compared to standalone methods. However, previous research focuses on exploration in policy parameter space, while overlooking the reward function search.
To bridge this gap, we propose **LaRes**, a novel hybrid framework that achieves efficient policy learning through reward function search. LaRes leverages large language models (LLMs) to generate the reward function population, guiding RL in policy learning. The reward functions are evaluated by the policy performance and improved through LLMs.
To improve sample efficiency, LaRes employs a shared experience buffer that collects experiences from all policies, with each experience containing rewards from all reward functions.
Upon reward function updates, the rewards of experiences are relabeled, enabling efficient use of historical data. 
Furthermore, we introduce a Thompson sampling-based selection mechanism that enables more efficient elite interaction.
To prevent policy collapse when improving reward functions, we propose the reward scaling and parameter constraint mechanisms to efficiently coordinate reward search with policy learning.
Across both initialized and non-initialized settings, LaRes  consistently achieves state-of-the-art performance, outperforming strong baselines in both sample efficiency and final performance.
The code is available at https://github.com/yeshenpy/LaRes.

LaRes: Evolutionary Reinforcement Learning with LLM-based Adaptive Reward Search
Pengyi Li
Hongyao Tang
College of Intelligence and Computing Tianjin University
College of Intelligence and Computing Tianjin University
1 Lipengyi jtjuu edu.
tanghongyao@tjii edu.
Jinbin Qiao
Yan Zheng
College of Intelligence and Computing
College of Intelligence and Computing Tianjin University
Tianjin University
jinbin@tju. cetiiju..ddccom edu.
yanzheng@t.j..ee edu. cn
Jianye Haot
College of Intelligence and Computing Tianjin University
j iayye waaaa.eaoot).. hao@tju edu.
Abstract
The integration of evolutionary algorithms (EAs) with reinforcement learning (RL) has shown superior performance compared to standalone methods. How- ever, previous research focuses exploration in policy parameter space, while overlooking the reward function search. To bridge this gap, we propose LaRes, novel hybrid framework that achieves efficient policy learning through reward function search. LaRes leverages large language models (LLMs) to generate the reward function population, guiding RL in policy learning. The reward functions are evaluated by the policy performance and improved through LLMs. To improve sample efficiency. LaRes employs shared experience buffer that collects experi- ences from all policies, with each experience containing rewards from all reward functions. Upon reward function updates. the rewards of experiences are relabeled, enabling efficient use of historical data. Furthermore, we introduce Thompson sampling-basse selection mechanism that enables more efficient elite interaction. To prevent policy collapse when improving reward functions, we propose the re- ward scaling and parameter constraint mechanisms efficiently coordinate reward search with policy learning. Across both initialized and non-initialized settings, LaRes consistently achieves state-of-the-a performance, outperforming strong baselines in both sample efficiency and final performance. The code is available at https:// //github...
thub. com hemm//esseeyy///ee SaaienpyyLLReee... LaRes.
Introduction
Reinforcement learning (RL) [1] a class of learning methods that excels at handling sequential decision
decision-making making problems [2]. Through trial and error and gradient-based based optimization, RL approxi-
mates value functions and provides policy gradients for policy learning [3]. It has been applied in various fields, including robotic control [4]. game AI [5], and recommender systems [6]. In contrast,
Corresponding authors: Yan Zheng and Jianye Hao
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Evolutionary Algorithms (EAs) [7-9] are heuristic optimization methods inspired by Darwinian principles, typically employing gradient-free approaches to solve problems and have shown remark- able performance in fields like circuit design [10] and scheduling optimization tasks [11]. Previous studies have revealed complementary characteristics between these two approaches [12, 13]. RI. excels at utilizing fine-grained information, such as states and actions, offering high sample efficiency and strong local optimization capabilities. However, it faces exploration challenge and is prone to suboptimal solutions [14]. In contrast, EAs are strong in global optimization but suffer from weak local optimization and sample inefficiency [15, 16]. Given these complementary strengths, many works have explored the integration of EAs and RL for policy learning, demonstrating superior performance compared to each approach individually [17-20].
The ultimate goal of task solving is to learn an efficient policy, which depends on two key factors: the algorithm's 's search capability in the policy parameter space, and the quality of the task's 's reward function, which directly impacts policy learning performance and efficiency [1]. Previous works primarily focus on integrating EAs and RL to improve the policy search capabilities [13], with little attention given to the reward functions. Early works aim to improve reward functions through heuristic operators, but these methods struggle to scale to complex tasks [21]. With the advancement of large language models (LLMs) that demonstrate strong coding capabilities and valuable domain knowledge, the generation of complex reward functions using LLMs has been explored preliminarily. For example, Text2Reward [22] uses LLMs to construct reward functions based structured environment representation. Eureka [23] constructs a reward function population and improves the reward function based evolutionary principles. SA [24] integrates CoT and hyperparameter optimization. R* [25] further improves Eureka's 's performance through structural evolution and parameter optimization. However, these works overlook sample efficiency, key evaluation metric in RL. leading to excessive environment interactions. In contrast, LaRes focuses on sample-efficient policy learning from both the policy and sample perspectives.
To solve these problems, we propose an LLM-based adaptive Reward search hybrid framework (LaRes) for efficient policy learning. In LaRes, we use LLMs to generate population of candidate reward functions, while RL learns corresponding policies. The LLM then iteratively refines the reward population based the performance feedback from these learned policies. If a designd reward function is available, it can be included in the context to reduce the difficulty of reward search. To improve sample efficiency, we maintain shared replay buffer that stores experiences from all policies. The key difference from previous works is that each experience includes multiple rewards from the reward function population, rather than single reward. RL then optimizes multiple policies based on the corresponding rewards. Besides, when the reward functions are improved, we (1) relabel the corresponding rewards in the replay buffer to enable historical data reuse, and (2) allow RL individuals to inherit from the best individual, avoiding retraining from scratch. However, we find that reward function changes may lead to policy collapse. To stabilize learning. we propose two mechanisms: reward scaling, which aligns the scale of the new reward function with the elite one, and parameter constraint loss, which minimizes the distance between the policy and critic and their elite counterparts in parameter space. Moreover, different reward functions may result in significant performance variations among policies, the experiences from inferior policies can contaminate the replay buffer, leading to suboptimal policies. Thus we propose Thompson sampling [26]-based interaction mechanism, which prioritizes more frequent interactions for superior policies. In experiments on 16 robotic manipulation and 4 MinAtar tasks with human-designed reward initialization, LaRes outperforms strong RL, ERL, and reward design baselines in both sample outperform efficiency and final performance. When trained without initialization, LaRes continues to achieve ffute-of-thh-art results in manipulation and locomotion tasks
We summarize our contributions as follows: (1) We propose novel hybrid framework that focuses on improving sample efficiency from both the sample and policy perspectives. (2) From the sample perspective, we design a shared replay buffer with reward relabeling mechanism to fully utilize historical data. (3) From the policy perspective, we propose reward scaling and parameter constraint mechanisms to ensure training stability, and a Thompson sampling based interaction mechanism to balance exploration and exploitation. (4) Empirical results show that LaRes consistently outperforms strong baselines across wide range of tasks and experimental settings.
2

---

i Generate Improve Reword Functions with LLM
Periodic Summary
Update RL Population
tips
Expert
Expert (optional)
Elite Information
Update non-elltee policies Qt
Mastty-
Reword Function
Q2
Improv
Improve
TE
bef computt_renard (.) # code hane Raharn rewerd.
Reword Function
Seart oper
LLM
cup target paint
X[2
Q2
Enitialization Improve
Reward Population RL Papulation
?
Reinforcement Learning Process
Reward Relabel
Function
Reward
Q:
Experiences
& done
Shared Reploy Buffer
Shored Reploy Buffer
Reward Funchion
Reword
Functian
Fir Po
Reward Function
Reward
Function
Optimize Hi Qi Vi
Function Diputs
s,a.5' done
Relabel experiences
are updated
Figure The optimization flow of LaRes. In the first iteration, ① LaRes generates the reward function population using the LLM, followed by @ RL training. After a certain number of training steps, @ results are summarized, and then @ the non-elite RI. individuals are replaced by the elite ones. Next, @ the reward function population is enhanced based on the best reward function, followed by 3 the reward relabel phase. The subsequent iterations follow the sequence: 2, 5, a, and 3
2 Background
Reinforcement Learning Consider a Markov decision process (MDP) [1], defined by a tuple (S, AS P, R, 7, T) At each step i, the agent uses a policy æ to select an action de T(-ss;) A according to the state 8t E S and the environment transits to the next state 8t41 according to transition function az) and the agent receives a reward r = R(s a.). The return is defined as the discounted cumulative reward, denoted by R, Σr -r=r;; where E [0, 1) is the discount factor and 7 the maximum episode horizon. The goal of RL is learn an optimal policy that maximizes the expected return. SAC [27] is one representative RL algorithm that maximizes expected reward while maintaining high entropy for effective exploration. SAC maintains a policy and double Q-networks, with the policy optimized as follows:
C, = (Eg, (a)ss((())) min Qo. (s, (s) Co((:)= = - (-+
where V'(8) [0a[=E[[mm
B.r[minn miny Qú, (s', - a olloo((a(()))) and a is the temperature parameter.
Evolutionary Algorithm Evolutionary Algorithms (EAs) [7, 28, 9] are a class of black-box optimization methods. EAs typically need to maintain population of individuals. In previous hybrid works, the population individuals are often represented in various forms [13]. e.g., policy networks, value function networks, and the evolution of the population is achieved through crossover and mutation operations on the parameters. In this paper, the population maintained by the EAs exists in the form of reward function code, defined follows: P (f,, 12,,.,, Fx). The reward function fitness fFFt) is evaluated based on the performance of the policy guided by F, We improve the reward function through LLMs.
Related Work
The integration EAs with RL has demonstrated strong capabilities across various tasks [13, 16]. Some works incorporate RI. into EAs to improve population initialization [29], evolutionary operators [30], and other processes [31]. Other works integrate EAs into RI. for hyperparameter tuning [32], action selection [33], and exploration [34]. Another emerging active direction is to fuse

---

the strengths of both approaches. For example, methods such as ERL [12], PDERL [18], CEM- RL [17]. ERL-Re² [19], EvoRainbow [20] leverage the complementary strengths of EAs and RL. improving both sample efficiency and exploration in policy search. Other works, like VFS [35] and VEB-RI. [36], focus on optimizing value functions. In addition, some works have extended ERI. concepts to areas like multi-agent systems [37, 38] and game testing [39]. In contrast, LaRes focuses on improving the reward function to enhance policy learning efficiency in complex tasks.
Early works on reward design primarily focus on inverse RL [40-43], where rewards are constructed based on expert demonstrations. Subsequently, the emergence of RL from human feedback (RLHF) enables learning reward models directly from human feedback [44-48]. In addition, some works improve reward quality through reward shaping [49-53]. With broad domain knowledge and strong coding capabilities [54-59], LLMs have been successfully applied across various fields [60-65]. Among them, recent works explore leveraging L.LMs to generate reward functions. For example, 1,2R [66] employs L.MSs to write reward code based on predefined APIs. Eureka [23] follows an evolutionary approach, maintaining a population of reward functions to guide policy learning and iteratively refining them with an LLM. DrEureka [67] uses LLMs to write reward functions and configure domain randomization parameters to achieve sim-to-real transfer. R* [25] decomposes reward design into structural evolution and parameter optimization. However, these works overlook sample efficiency, a key evaluation metric in RL. In contrast, LaRes focuses on improving sample efficiency from both the sample and policy perspectives.
4
LLM-based Adaptive Reward Search
This section provides an overview of LaRes and elaborates its key mechanisms from three perspectives: high-level reward evolution, low-level policy learning, and inter-level coordination.
4.1 LaRes Optimization Flow
Unlike previous ERL methods that use EAs and RL to co-optimize the policy parameters, LaRes decomposes the task-solving process into reward function search and policy learning. The overall framework shown in Figure LaRes employs LLM construct and evolve the reward function population. We then employ RL to learn the corresponding policies and evaluate the reward function fitness based on the policy performance. Below, we briefly summarize the algorithmic process, from high-level reward evolution, lower-level policy learning, and coordination between the two levels.
Population initialization. In the initial iteration, provide human-designe reward functions (optional), task descriptions, and environment information as the context to the LLM. Using tailored prompts, the LLM generates n reward functions to construct the initial reward function population. For each reward function, an RL agent is initialized for environment interaction and policy learning.
Population Evaluation and Evolution. With T-step learning. we select the best reward function together with its corresponding policy and Q-function based on policy performance, c.g. success rate. Subsequently, provide the best reward function and learning process information (c.g., success rates, cumulative rewards) as feedback to the LLM for reflection. The LLM then generates new improved reward functions that replace the non-elite ones in the population.
The above process introduces the high-level reward search, focusing on reward function generation, evaluation and evolution. Below, we introduce the low-level policy learning.
Policy Learning. We employ the off-policy RL algorithms (e.g., SAC [27]) learn the policies guided by different reward functions. To improve sample efficiency and mitigate exploration chal- lenges in RL, experiences from all policies are stored in shared replay buffer. To reduce replay buffer contamination from inferior policy experiences, we propose the selection-based interaction using Thompson sampling [26]. Further details are provided in Subsection 4.3.
The coordination between the two levels is crucial, with a particular focus on how policy learning adapts to the periodic evolution of the reward function.
Continual Learning. After the reward function is improved, continuing learning with the previous policy and Q-functions may lead to suboptimal results, while learning new RL policy from scratch incurs a significant sample cost. Thus we propose a continual learning approach, where the best policy and its corresponding Q-functions are used initialize those of the non-clite agents. However,

---

we find that the elite policy is prone to performance collapse when guided by new reward functions. To solve the problem, we propose the reward scaling and parameter constraint mechanisms, which will be discussed in Subsection 4.3.
Historical Data Reuse Off-policy RL algorithms like SAC can reuse historical data through replay buffer. However, when the reward function changes, directly training on previous data may lead to policy collapse due reward inconsistency problem, while discarding historical data would result in significant sample waste. To efficiently reuse historical data, we relabel the shared replay buffer based on the new reward function population.
Next, we present a detailed introduction to the key components of LaRes.
4.2 High-level Reward Evolution
Inspired by previous works [22, 23), we provide the raw environment variables as context to the LLM, along with tailored prompts that guide it in generating and refining reward functions. The LLM then iteratively generates n reward functions in code format, which are subsequently optimized from three key perspectives: 1) Components Optimization. Adding new reward components for guidance or removing ineffective reward components. 2) Weight Optimization. Adjusting the weights between different reward components. 3) Calculation Optimization. Modifying the calculation method of reward components, such as using alternative activation functions.
The L.M must consider all three aspects simultaneously. If optimizatiio is not performed, brief explanation should be provided. In the improvement phase, the best reward function is selected according to the policy performance. We then provide the policy performance, episodic reward, and other related statistics to the LLM, which reflects on this feedback and generates new reward functions replace the non-elite ones in the population. Through the above process, we can achieve the iterative improvement of the reward function population. To ensure the stability of policy learning, the elite reward functions will not be replaced and continue to guide their corresponding RL agents. Additionally, under settings with human- -esigned reward initialization, we always maintain the human husanndesigned reward function, resulting in a total of n reward functions.
4.3 Low-level Policy Learning
Given the presence of multiple reward functions, we employ the parallel training approach, learning an RL agent for each reward function. To improve sample efficiency, we maintain shared replay buffer that stores the experiences from all policies. The key difference is that, the conventional replay buffer stores experiences in the format {s, a, ,T, done]. due to the presence of the reward function population, each experience needs to be relabeled by the reward function population, resulting in the reward, which
format {s, a, s', .T......, F', done, info]. r represents the original human exists only when a humann -dennned reward function is provided, {ri....,7y Tn } represent the rewards from the population, and info denotes the variable inputs to the reward function for reward calculation. Each RL agent is trained using its corresponding rewards.
In addition, due to the significant performance differences in policies guided by different reward functions, allocating the same number of interaction sleps to each agent would lead to resource waste. To improve the experience quality in the replay buffer while mitigating the impact of poor experiences, we propose Thompson sampling-bas Bayesian inference-based sampling method that estimates the distribution of the expected reward for each action based on historical data. Actions are selected by sampling from these distributions, and after performing the action, the reward distribution is updated using the reward feedback. In LaRes, we define n actions, where the i-th action corresponds to selecting policy i for interaction. We update the Beta distribution parameters using feedback from task success and failure. Specifically, the reward u/s of each action follows Beta distribution t/z ~ Beta( (as, 3c), and the action with the highest sampled reward is selected for interaction. For action i, B; and a; are updated as follows:
interaction mechanism. Thompson sampling [26] is a
(2) where ni. and f. represent the number of successes and failures of policy i, respectively. Under this mechanism, superior policies have more interaction resources, while inferior policies have fewer opportunities for interaction. Additionally, after the reward function improves, we reset the Thompson sampler to reallocate resources. Note that following previous ERL. works, the RI. agent guided by the human-designed reward function does not participate in sampling and interacts in each iteration.
df = ar + ng 8 3, + n5,45
5

---

The iterative improvement of the reward function population can introduce several problems for low-level policy learning. In this subsection, we provide a detailed explanation of how address these challenges. When the reward function changes, we adopt continual learning approach to directly configure the best policy and Q functions for the new reward function. This approach avoids the high sample cost of training from scratch and mitigates the performance degradation caused by continuing to train an inferior policy. However, we find that when the reward function changes, policy learning is prone to collapse. This is primarily caused by two factors: reward scale difference and reward design difference.
To address the scale difference, we introduce a reward scaling mechanism. Specifically, we calculate the mean Metite and variance elis of the elite reward function based on the replay buffer as a surrogate for its true mean and variance. Meanwhile, we calculate the mean Mnew and variance new of the newly generated reward function. We align the means and variances of the two reward functions using the following formula:
Celite
Fscaled
(Fnew
(new) + Melite-
Onew
By scaling both the replay buffer rewards and the new interaction rewards, we can efficiently address the reward scale difference problem. However, distributional differences still exist. In previous ERL works, evolution typically occurs at the parameter level, under the assumption that better individuals located near the parameters of the optimal individual. Inspired by this, we propose parameter constraint mechanism that constrains the parameter changes of the policy and value functions to further mitigate the collapse problem. Specifically, introduce the following loss:
(4) where Oelite-OO.elite and & ielii are the policy parameters and value function parameters of the elite agent. By adding the above constraint loss during both policy and value function updates, we prevent the policy from drifting away from the elite parameters, thereby improving training stability.
C. = ||0 Oetite 1| Lq. = l - @o. + |1 $1 -1,eleel
4.4 LaRes Algorithm
LaRes is flexible framework that can be combined with any off-policy RL method. We provide the pseu- docode in Algorithm 1. Specifically, we first use the LLM to generate a reward function population based on the human-designed reward function FF (in human reward function initial- ization setting), environment variables V. and other information (line 3). In each iteration, we first initialize the Thompson sampler T's (line 5). Next,
Algorithm LaRes Framework
I: Require:
Task description L, environment variables V, reward function prompt P, coding L.LM LLM,, Human- designed reward function Fn (Optional)
2: Initialize Shared Replay Buffer D, Thompson sampler
T'.. an RL population IRRL = {T Q1, Tay Qn)
3: Initialize reward function population:
I PReward LLM (L, V, P, Fn (optional))
4: for Iterations do
5: Reset the Thompson sampler Ts
6:
for T environment steps do Policy interaction based on sampler Ts Add all rewards for each experience using IP Rewaad Add all experiences to D)
we enter the training phase, which be- gins with policy interactions (line 6- 11), including both the Ts sampled policy and the policy guided by the human-designed reward (line 7). For each experience, rewards are caleu- lated using |Rwwaadd (line 8), and all experiences are added to the shared re- play buffer D (line 9). Subsequently. parallel RL training is conducted (line 10). If the current iteration is not the initial iteration, a parameter constraint loss is added to the non-elite agent training process to ensure learning sta- bility. After T environment steps of training, we evaluate the performance of the learned RL policies and select the best reward function Frest, together with its corresponding policy Moest and value function Qieaeline 12). Using the LLM
7:
8:
9:
10:
RL parallel training, add parameter constraint loss to non-elite agents
11:
end for
12:
Select the best Reward Function Frest- Tbest and Quest
13:
Improve reward population with LLM reflection
PReward Reflection L, V, P. Fn (optional), Fbest)
14:
Relabel replay buffer D using the new PReward Initialize the non-elite RI. agents with the best Thest and Qbest
15:
16:
Reward rescaling for new reward functions
17: end for

---

Button Press Topdown
Buskechall
Coffee Pull
Dial: Tura
1.4
Saaioos Tart
0-2
0.2
0.
03
03
02
85
02
02
05
12
03
02
04
20 Sirpe (K.27)
DeroUrEcC Sepeotie)
De Skeu
Lenrencee Sepelnte?
1.0
Hoads Finorr
1.0
Harmer
Pio--Dut-OF-loll
1.0
Pep-Unnplu--Sihe
Saocen Burn
Bac
0.4
0.6
8.
0.0
044
D.C
0.
0.3
24
0.
02
04
05
00
0.2
0.4
06
D4
Caroxcasnn Stppe (41
Earborner Depe
Pop: Ensert Side
Coffee Puahh
Push Blakk
0.5
Seccer
077
0.5
03
02
0.0
03
02
10.
00
03
Deps
Figure 2: Performance comparison on 12 robot manipulation tasks from MetaWorkd.
reflection mechanism, we improve the reward function population by replacing non-elite individuals (line 13). The experiences in D are then relabeled based on the new reward function (line 14), and the parameters of the non-elite individuals are initialized using Thest and Qhest (line 15). To ensure reward scale consistency, apply reward rescaling mechanism to adjust the reward function (line 16). The process then proceeds to the next iteration. Through the above process, LaRes achieves efficient policy learning through reward search.
Experiments
We first conduct experiments various tasks to compare LaRes with other strong baselines. To gain deeper insights into LaRes, we then perform detailed analyses. Furthermore, an ablation study is conducted to verify the effectiveness of each component.
5.1 Experimental Setups
We evaluate LaRes wide range of benchmarks, including manipulation tasks from the MetaWorld and ManiSkill3 suites [68, 69], MinAtar tasks with image inputs [70], and locomotion tasks from MuloCo [71]. Under the setting with numan-designed reward initialization, we evaluate LaRes on 20 tasks, including 12 Meta Morld tasks. 4 ManiSkill tasks, and 4 MinAtar tasks. In manipulation tasks, we implement LaRes based on SAC. We compare LaRes with the following baselines: 1) RL baselines, i.e., SAC [27]: 2) ERL-related baselines, including ERL [12], PDERL [18), CEM- RL [17], 2 [19], EvoRainbow [20]. 3) Reward-search baselines, i.e., SAC-based Eureka [23] and Texr2Reward [22]. R* [25]. In Min tasks, we implement LaRes based on DQN and compare with DQN. Under the no-initialization setting, we compare LaRes with other reward design methods on 5 MetaWorld tasks and 2 locomotion tasks (Ant & Humanoid).
We the official codes implement the methods new benchmarks following the original papers. For fair comparison, we fine-tune them in each task to provide the best performance. We use GPT-4o-mini as the LLM backbone under the human- designed reward initialization setting, and GPT-4o under the no- no-initializatio setting. All algorithms are trained with 1 million environment steps on MetaWorld and locomotion tasks, million environment steps on ManiSkill3 and MinAtar. All statistics are obtained from 5 independent runs, consistent with previous literature. We report the average with 95% confidence intervals. For LaRes, we set the population size 5. We perform iterations of the reward population evolution. All implementation details are provided in Appendix B.

---

Place Sphere
Pick StugleYCC
Stadk-Cube
Push T
Spuioos Bata
Seacoun Baxc
04
SAC
3 025 120-935
0.
12
28
107 (0:0 1.87 130
0.07 132 032 8.0 1,07 800
20r0sne Sepu (512)
Derbumace Sepeoiie)
tenmensel (x1e)
Figure 3: Performance comparison on 4 robot manipulation tasks from ManiSkill3,
Method
Window- Clooe
Window-Open
Drawer-Open
Button-
Clor-Close
LLM zero-shot
51% 1,388966
6% 1 419084
8% | 762346
6.%% I 5600288
15% 1565900
Eureka
50% 303786
55% 657098
15% 731876
61% | 924163
98% 283206
LaRes
100% 164850
100% 358403
100% 164850
100% 112800
100% 65626
Table 2: Comparison under the no-initialization setting (Success Rate Samples Needed).
5.2
Performance Evaluation with Human Reward Function Initialization
We begin by evaluating LaRes and other baselines across 12 different manipulation tasks in the MetaWorld benchmark. The RL and ERL baselines learn policies guided by human-designed rewards. The experimental results are shown in Figure 2. We observe that LaRes significantly outperforms SAC (guided by human-designed rewards) across all tasks, which demonstrates that LaRes effectively discovers better reward functions and fully leverages the learned multiple policies achieve superior performance. Furthermore, LaRes outperforms other ERL methods in both sample efficiency and final performance across most tasks, especially the harder ones. Notably, unlike other ERL methods. LaRes does not combine EAs and RL for co-optimizing policy and value function parameters. Instead, focuses on reward function search and continuous policy learning. The results highlight both the importance of reward function design and the effectiveness of LaRes. Finally, we observe that LaRes significantly outperforms other reward design methods. Compared to Eureka and R* LaRes makes more efficient use of the generated data and ensures stable policy learning, while Texi2Reward lacks the capacity for continuous improvement, making the policy more prone to suboptimality or collapse.
We further evaluate LaRes and SAC on four tasks from ManiSkill3. Unlike MetaWorld, these tasks leverage GPU parallel sampling, enabling higher sampling efficiency. The results presented in Figure demonstrate that LaRes can also significantly improve SAC. This demonstrates that LaRes consistently improves performance across different benchmarks, further validating the effectiveness of LaRes.
Can LaRes improve other off-policy algorithms and be applied to discrete ac- tion spaces? To verify this, we integrate LaRes with DQN and evaluate on 4 discrete-control tasks from MinAtar, and report the average scores at 0.5 and 2 million environment steps in Table 1. We observe that LaRes significantly improves the performance of DQN trained with human-designed rewards. This result further validates the effectiveness and generality of LaRes.
Method
Breakout
Asterix
Freeway
SpaceInvaders
DQN
13.52 21.86
2.96
11.55
38.89
53.93
23.18
65.38
LaRes
21.87
31.28
14.13
28.18
50.87
57.24
42.27
84.72
Table 1: The scores at 0.5 & 2 million env steps on MinAtar tasks.
5.3 Performance Evaluation without Human Reward Functions
In the previous subsection, we mainly explore the results under the setting where a human- -eeigndd reward function is provided initialization. A natural question arises: Can LaRes still outperform other reward design methods when no human-designed reward function is available? To answer this question, first evaluate LaRes on 5 tasks from Meta World, which similar those used in Rewa2d Compared to the tasks in the previous subsection, these tasks are relatively easier, allowing policies achieve 100% success rate. The experimental results are presented in Table 2. We observe that LaRes outperforms other methods in both sample efficiency and best performance, which indicates that LaRes is also efficient in its approach to learning from scratch.

---

Beyond the above manipulation tasks, we fur- ther evaluate LaRes on two challenging loco- motion tasks, Humanoid and Ant. As shown in Table 3, LaRes again surpasses other reward design methods, demonstrating its effectiveness across different tasks.
Speed 1M (m/s)
Eureka
ROSKA [72] LaRes
Ant
1.49
2.77
4.44
Humanoid
2.02
2,95
3.42
Table 3: Performance on locomotion tasks.
5.4 Analysis & Ablation
In this section, we answer the following questions through experiments: Q1. Does LaRes effectively improve the reward function? Q2. Does the Thompson sampler in LaRes efficiently and dynamically adjust the policy interaction? Q3. Can the reward scaling and parameter constraint mechanisms effectively prevent policy collapse?
To answer QI. we present perfor- mance comparison between the pol- icy guided by the human-designed re- ward and the best one guided by the generated reward function LaRes. The results are shown in Figure We observe that the generated reward function significantly outperforms the numan-designes reward function both terms of sample efficiency and final performance. This indicates that LaRes can discover higher-quality reward function than the human- designed expert reward function. Additionally, we provide the LLM summary of the best reward function generated on the Hammer task. Due to space limitations, the complete reward functions are provided in Appendix C. We observe that the LLM is able make reasonable improvements based on the three designed aspects, enabling more efficient exploration of the reward space.
Hammor
1.00
Hand-Inneet
Firnon-teddgaa Rewar!
0.5
Glocrated Reward
0.8
Sacces Rute
0.5
Stccess Sate
0.6
04
09
0.0
6.2
0.4
0.6
08
0.0
0.2
0.4
0.6
0.3
Enstorment Steps Octo")
Eaviionnccnt Stigs Octe")
Figure 4: Performance comparison of the policy success rate guided by human-designed rewards and generated rewards.
LLM Summary of Improved Reward Function
The reward for lifting the hammer ('a') and for hitting the target nail ("b') was increased significantly to enhance the focus on these critical actions. The increased penalties when nearing the target position (e.g. adjusted bounds to '0.015") will encourage more precise interactions with the target area.
Reward Components:
2. Reward Weights: The weight of the successful lift and placement were modified to provide enhanced emphasis these actions, thereby creating stronger incentive for the agent to accomplish these milestones, which seems crucial considering the policy's 's current low success rate.
3. Reward Calculation: The overall normalization and adjustment of reward components includes changes in thresholds and the temperature parameter, which should provide more stability and control in training. Lowering the temperature value improves the agent's s control over the learning process and minimizes excessive reward scaling.
To answer Q2, first present de- tailed learning curves of policies guided by different functions the Hammer task. We observe that the policies show significant differences, which aligns with the intuition that the modifications guided by LLMs do not always yield positive results. Then we visualize the selected probability of each policy using Thompson sam- pler. We can observe that F3 is se- lected with high probability in each iteration, while Fs, due to inferior performance, has a low probability of being selected. Additionally, conduct an ablation study
2
Hammer
0.3
Success Ruie
02
00
Envionnent Steps Oxlo")
Devkonneee pele?)
Figure 5: (Left) Policy learning performance guided by dif- ferent reward functions, (Right) The variation curve of the probability of policy selection.

---

Dial-Tura
Fick- Out--e--ill
0.5
Luiken
Letks
Lullee wit Thanppoos
Lefcs vin bdt
courpia
00
02
Eat
G3
00
0.2
2.4
05
100
00
02
0.4
08
0.
0.2
0.4
0.
0.5
Eavivanctt Sopetcizsy
Znvinnncnt Sepeocie
(4ach
Enibvracc Skge pxic.)
(a) sampling interaction mechanism
(b) Reward scale & parameter constraint.
Figure 6: Ablation study on LaRes
the Thompson sampling mechanism. The results in Figure 6a show that removing Thompson sam- pling leads to a performance decline. This is consistent with the intuition that uniformly distributing interaction resources leads poor performance.
To answer Q3, we conduct an ablation study on the reward scaling and parameter constraint mecha- nisms. The experimental results in Figure 6 demonstrate that LaRes w/o reward scaling and parameter constraints exhibits performance fluctuations during the learning process. However, adding reward scaling and parameter constraints leads to more stable and efficient policy learning, highlighting effectiveness these mechanisms. More experiments on hyperparameters and different LLM backbones are provided in Appendix D.
Finally, we present an overhead analysis. LaRes employs parallel training architecture simultane- ously train multiple policies, effectively reducing the training time required for multi-policy learning, LaRes incurs an approximately 20% increase in time overhead, primarily due inter-process commu- nication. This overhead could be further reduced [73]. such as adopting asynchronous communication. In addition, one limitation of LaRes is its increased computational resource requirements, which we aim to address in future work.
6 Conclusion
This paper introduces LaRes, a novel evolutionary reinforcement learning hybrid framework focused on reward function search. Driven by LLM, LaRes operates on three levels: reward function search at the high level, policy learning at the low level, and coordination between the two levels. Specifically, the high level employs the LLM to refine human-designed reward functions, generating improved reward functions. The low level employs Thompson sampling to adaptively select policies for interaction based their performance and utilizes shared replay buffer to enhance sample efficiency. The coordination between the two levels focuses on continuous policy learning. To achieve this, propose reward relabeling mechanism to efficiently reuse historical data, along with reward scaling and parameter constraint mechanisms to mitigate the policy-collape problem. Across 16 robotic manipulation tasks, LaRes demonstrates significant improvements in both sample efficiency and final performance compared to other strong baselines.
Acknowledgments
This work is supported by the National Natural Science Foundation of China (Grant Nos. 624B2101, 62422605, 92370132), National Key Research and Development Program of China (Grant No. 2024YFE0210900 and Xiaomi Young Talents Program of Xiaomi Foundation. We would like to thank all the anonymous reviewers for their valuable comments and constructive suggestions, which have greatly improved the quality of this paper.
References
[1] R. Sutton and A. G. Barto. Reinforcement learning an introduction. 1998.
[2] M. I. Jordan and T.M. Mitchell. Machine learning: Trends, perspectives, and prospects. Science, 2015.
[3] M. Andrychowiez, Denil, Gomez, M. W. Hoffman, Pfau, T. Schaul, B. Shillingford, and N. D. Freitas. Learning to learn by gradient descent by gradient descent. NewrIPS, 2016.
10

---

[4] T. Johannink, S. Bahl, A. Nair, J. Luo, A. Kumar, M. Loskyll. J. A. Ojea. E. Solowjow, and Levine. Residual reinforcement learning for robot control. ICRA, 2019.
[5] O. Vinyals, Babuschkin, W. Czarnecki, and Others. Grandmaster level stareraft II using multi-agent reinforcement learning. Nature, 2019.
[6] L. Zou, L. Xia, Z. Ding, J. Song, W. Liu, and D. Yin. Reinforcement learning to optimize long-term user engagement in recommender systems. In KDD, 2019.
[7] Bäck and H. Schwefel. An overview of evolutionary algorithms for parameter optimization. Evol. Comput., 1993.
[8] P. A. Vikhar. Evolutionary algorithms: A critical review and its future prospects. In ICGTSPICC. IEEE, 2016.
[9] Z. Zhou, Y. Yu, and C. Qian. Evolutionary Learning: Advances in Theories and Algorithms. Springer, 2019.
[10] D. Dasgupta and Z. Michalewicz. Evolutionary algorithms in engineering applications. 2013.
[11] K. Gao, Z. Cao, L. Zhang, Z. Chen, Y. Han, and Q. Pan. A review on swarm intelligence and evolutionary algorithms for solving flexible job shop scheduling problems. JAS, 2019.
[12] S. Khadka and K. Tumer. Evolution-guided policy gradient in reinforcement learning. In NeurlPS, 2018.
[13] Li. J. Hao, H. Tang, X. Fu, Y. Zheng, and K. Tang. Bridging evolutionary algorithms and reinforcement learning: A comprehensive survey. IEEE TEVC, 2024.
[14] J. Hao, T. Yang, H. Tang, C. Bai, J. Liu, Z. Meng, P. Liu, and Z. Wang. Exploration in deep reinforcement learning: From single-agent to multiagent domain. TNNLS, 2023.
[15] T. Gangwani and J. Peng. Policy optimization by genetic distillation. In ICLR, 2018.
[16] O. Sigaud. Combining evolution and deep reinforcement learning for policy search: a survey. arXiv preprint, 2022.
[17] A. Pourchot and O. Sigaud. CEM-RL: combining evolutionary and gradient-based methods for policy search. In ICLR, 2019.
[18] C. Bodnar, B. Day, and P. Lió. Proximal distilled evolutionary reinforcement learning. In AAAI, 2020.
[19] J. Hao, P. Li, H. Tang, Y. Zheng, X. Fu, and Z. Meng. Erl-re:: ment learning with shared state representation and individual policy representation. In ICLR, 2023.
Efficient evolutionary reinforce-
[20] P. Li, Y. Zheng, H. Tang, X. Fu, and J. Hao. Evorainbow: Combining improvements in evolutionary reinforcement learning for policy search. ICML, 2024.
[21] S. Nickum, A. G. Barto, and L. Spector. Genetic programming for reward function search. IEEE Trans. Auton. Ment. Dev., 2010.
[22] T. Xie, S. Zhao, C. Henry Wu, Y. Liu, Q. Luo, V. Zhong, Y. Yang, and T. Yu. Text2reward: Reward shaping with language models for reinforcement learning. In ICLR, 2024.
[23] Y. J. Ma, W. Liang, G. Wang, D. Huang, O. Bastani, D. Jayaraman, Y. Zhu, L. Fan, and A. Anandkumar. Eureka: Human-level reward design via coding large language models. In ICLR, 2024.
[24] Y. Zeng, Mu, and Shao. Learning reward for robot skills using large language models via self-alignment. In ICML, 2024.
[25] P. Li, J. Hao, H. Tang, Y. Yuan, J. Qiao, Z. Dong, and Y. Zheng. R*: Efficient reward design via reward structure evolution and parameter alignment optimization with large language models. In ICML, 2025.
11

---

[26] D. Russo, B. V. Roy, A. Kazerouni, I. Osband, and Z. Wen. A tutorial on thompson sampling. Found. Trends Mach. Learn.. 2018.
[27] Haarnoja, A. Zhou, P. Abbeel, and Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In ICML, 2018.
[28] Thomas Bäck. Evolutionary Algorithms in Theory and Practice: Evolution Strategies, Evolu- tionary Programmingg Genetic Algorithms. 1996.
[29] H. Ye, J. Wang. Z. Cao, H. Liang, and Y. Li. Deepaco: Neural-enhanced ant systems for combinatorial optimization. NeurlPS, 2023.
[30] G. Cideron, T. Pierrot, N. Perrin, K. Beguir, and O. Sigaud. QD-RL: efficient mixing of quality and diversity in reinforcement learning. arXiv preprint, 2020.
[31] K. Xue, Xu, L. Yuan, M. Li, C. Qian, Z. Zhang, and Y. Yu. Multi-agent dynamic algorithm configuration. In NeurIPS, 2022.
[32] M. Jaderberg, V. Dalibard, S. Osindero, W. M. Czarnecki, J. Donahue, A. Razavi, O. Vinyals, T. Green, I. Dunning, K. Simonyan, C. Fernando, and K. Kavukcuoglu. Population based training of neural networks. arXiv preprint, 2017.
[33] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakr- ishnan, V. Vanhoucke, and S. Levine. Scalable deep reinforcement learning for vision-based robotic manipulation. In CoRL, 2018.
[34] S. Chang, J. Yang, J. Choi, and N. Kwak. Genetic-gated networks for deep reinforcement learning. In NeurIPS, 2018.
[35] E. Marchesini and C. Amato. Improving deep policy gradients with value function search. In ICLR, 2023.
[36] P. J. Hao, H. Tang, Y. Zheng, and F. Barez. Value-evolutionary--aasdd reinforcement learning. In ICML, 2024.
[37] S. Majumdar, S. Khadka, S. Miret, S. McAleer, and K. Tumer. Evolutionary reinforcement learning for sample-efficient multiagent coordination. In ICML, 2020.
[38] J. Hao, H. Tang, Y. Zheng, and X. Fu. Race: Improve multi-agent reinforcement learning with representation asymmetry and collaborative evolution. In ICMI.,
[39] Y. Zheng, X. Xie, T. Su, L. Ma, J. Hao, Z. Meng. Y. Liu, R. Shen, Y. Chen, and C. Fan. Wuji: Automatic online combat game testing using evolutionary deep reinforcement learning. In ASE. IEEE, 2019.
[40] S. C. Adams, T. Cody, and P. A. Beling. A survey of inverse reinforcement learning. Artif. Intell. Rev, 2022.
[41] A. Y. Ng and S. Russell. Algorithms for inverse reinforcement learning. In ICML, 2000.
[42] B. D. Ziebart, A. L. Maas, J. A. Bagnell, and A. K. Dey. Maximum entooymentropp entropy inverse reinforcement learning. In AAAI, 2008.
[43] J. Ho and S. Ermon. Generative adversarial imitation learning. In NeurIPS, 2016.
[44] T. Kaufmann, P. Weng, V. Bengs, and E. Hüllermeier. A survey of reinforcement learning from human feedback. arXiv preprint. 2023.
[45] Christiano, J. Leike, T. B. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. In NewrIPS, 2017.
[46] K. Lee, L. M. Smith, and P. Abbeel. Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. In ICML, 2021.
[47] Sharma, B. Sundaralingam, Blukis, C. Paxton, T. Hermans, A. Torralba, Andreas, and
D. Fox. Correcting robot plans with natural language feedback. In RSS, 2022.
12

---

[48] L. Guan, K. Valmeekam, and S. Kambhampati. Relative behavioral attributes: Filling the gap between symbolic goal specification and reward learning from human preferences. In ICLR. 2023.
[49] Y. Burda, H. Edwards, A. J. Storkey, and O. Klimov. Exploration by random network distillation. In ICLR, 2019.
[50] A. Y. Ng. D. Harada, and S. Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In ICML, 1999.
[51] R. Devidze, P. Kamalaruban, and A. Singla. Exploration-guided reward shaping for reinforce- ment learning under sparse rewards. NeurIPS, 2022.
[52] Y. Hu, W. Wang, H. Jia, Y. Wang, Y. Chen, Haoand F. Wu, and C. Fan. Learning to utilize shaping rewards: A new approach of reward shaping. NeurIPS, 2020.
[53] Goyal, S. Nickum, and J. Mooney. Using natural language for reward shaping in reinforce- ment learning, arXiv preprint, 2019.
[54] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, and Others. Language models are few-shot learners. In NeurIPS, 2020.
[55] OpenAl. GPT-4 technical report. CoRR, 2023.
[56] H. Touvron, T. Lavril, G. Izacard, X. Martinet, and Others. Llama: Open and efficient foundation language models. CoRR, 2023.
[57] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, and Others. Llama 2: Open foundation and fine-tuned chat models. CoRR, 2023.
[58] A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, and Others. The Ilama 3 herd of models. CoRR, 2024.
[59] Jing Liang, Hongyao Tang, Yi Ma, Jinyi Liu, Yan Zheng, Shuyue Hu, Lei Bai, and Jianye Hao. Squeeze the soaked sponge: Efficient off-polic reinforcement finetuning for large language model. arXiv preprint 2Xiv:2507.06822 2025.
[60] W. Zhao, K. Zhou, L. Li, T. Tang, and Others. A survey of large language models. arXiv preprint arXiv:2303. 18223, 2023.
[61] M. Hadi, R. Qureshi, A. Shahand M. Irfan, A. Zafar, M. Shaikh, and Others. A survey on large language models: Applications, challenges, limitations, and practical usage. Authorea Preprints, 2023.
[62] H. Naveed, A. Khan, S. Qiu, M. Saqib, and Others. A comprehensive overview of large language models. arXiv preprint, 2023.
[63] E. Kasneci, K. Seßler, S. Küchemann, and Others. Chatgpt for good? on opportunities and challenges of large language models for education. Learning and individual differences.
[64] L. Wang, C. Ma, X. Feng, Z. Zhang, H. Yang, and Others. A survey on large language model based autonomous agents. Frontiers of Computer Science, 2024.
[65] Y. Wang, Z. Xian, F. Chen, T. Wang, and Others. Robogen: Towards unleashing infinite data for automated robot learning via generative simulation. In ICML, 2024.
[66] W. Yu, N. Gileadi, C. Fu, S. Kirmani, and Others. Language to rewards for robotic skill synthesis. CoRL, 2023.
[67] Y. Jason Ma, W. Liang, H. Wang, S. Wang, Y. Zhu, L.. Fan, O. Bastani, and D. Jayaraman. Dreureka: Language model guided sim-to-real transfer. CoRR, 2024.
[68] Yu, D. Quillen, 7. He, R. Julian, K. Hausman, C. Finn, and S. Levine. Meta-world: benchmark and evaluation for multi-task and meta reinforcement learning. In CoRL, 2019.
13

---

[69] S Tao, F Xiang. A Shukla, Y Qin, X Hinrichsen, and Others. Maniskill3: GPU parallelized robotics simulation and rendering for generalizable embodied AI. CoRR, 2024.
[70] K. Young and T. Tian. Minatar: An atari-inspired testbed for more efficient reinforcement learning experiments. CoRR, 2019.
[71] E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In IROS, pages 5026-5033, 2012.
[72] C. Huang, Y. Chang, J. Lin, J. Liang, R. Zeng. and Li. Efficient language-instructed skill acquisition via reward-policy co-evolution. In AAAI, 2025.
[73] A. Farkas, G. Kertész, and Lovas. Parallel and distributed training of deep neural networks:
A brief overview. In INES, 2020.
[74] J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. H. Chi, Q. Le, and D. Zhou. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS, 2022.
[75] S. Yao, D. Yu, J. Zhao, I. Shafran, T. Griffiths, Y. Cao, and K. Narasimhan. Tree thoughts: Deliberate problem solving with large language models. In NeurlPS, 2023.
[76] D. Sudholt. The benefits of population diversity in evolutionary algorithms: A survey of rigorous runtime analyses. In Theory of Evolutionary Computation - Recent Developments in Discrete Optimization. 2020.
14

---

A Limitations & Future Work
For limitations and future work, firstly our work is empirical proof of the effectiveness of the LaRes idea and we provide no theory on optimality, convergence, and complexity. Secondly, although LaRes adopts a parallel training architecture, it still incurs additional time overhead, primarily due to inter- pree--poesss data communication. This overhead could be further optimized [73], for instance, by implementing an asynchronous communication mechanism. Furthermore, the computational cost of LaRes scales proportionally with the number of policies being trained. Thirdly, this work represents only an initial exploration of reward function search in the ERL framework. The proposed architecture can be further optimized, for example, by incorporating buman-designed reward-guided policies into the Thompson sampling-base interactions or adopting more efficient sampling mechanism and continuous policy learning mechanism. Fourthly, LaRes does not explore more efficient generation mechanisms. For example, leveraging reasoning techniques within LLMs, such as Chain-of-Thougg (CoT) [74] or Tree-of-Thought (ToT) [75] reasoning. could potentially enhance the quality of the generated reward functions. Finally, diversity is a key aspect of EA [76]. When constructing reward functions, we do not explicitly consider individual diversity and instead rely on the randomness of LLMs. However, this approach often fails to ensure sufficient differentiation among individuals.
Overall, LaRes represents preliminary attempt at leveraging L.LMs for reward function search within the ERL framework. However, it still has several limitations and potential areas for further improvement. We aim to address these challenges and enhance the method in future work.
B Method Implementation Details
The MetaWorld experiments are carried out on Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz. ManiSkill3 leverages GPU acceleration; therefore, we conduct experiments on NVIDIA GTX 2080 Ti GPU with Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz.
B.1 Implementation of Baselines
For all ERL baseline algorithms, we use the official implementation, including ERL', 2 CEM-RL ER EvoRainbow The reward-search baselines primarily include Text2Reward, Eureka and R* For fair comparisons, all these methods are implemented based on the SAC algorithm with consistent hyperparameter settings. In addition, handle discrete action spaces, we implement a DQN-based version of LaRes. For the implementation of Texi2Reward, we employ the LLM generate improved reward functions based on human-designed reward functions and interface definitions. Eureka can be regarded as a variant of LaRes without continual learning data sharing. To achieve this, we modify the LaRes by removing unrelated components and ensuring the same number of evolutionary iterations and population size. R further builds upon Eureka by introducing two mechanisms: parameter optimization and structure evolution.
B.2 Implementation of LaRes
The implementation of LaRes has three versions: MetaWorld, ManiSkill3, and MinAtar. We build LaRes for MetaWorke based on the SAC implementation from EvoRainbow, for ManiSkill3 using the official SAC implementation provided by ManiSkill3, and for MinAtar using the official DQN implementation provided by VEB-RLS.
For the training framework, we construct parallel framework using Python's multiprocessing library. This framework consists a central server and multiple workers. The server is primarily responsible for policy interactions, data distribution, parameter distribution, data labeling, and relabeling. The number of workers corresponds to the number of reward functions, and each worker is tasked with
5 hatr:://iitttlccomm//sseeeeeeeer
15

---

training a specific policy and sending the trained parameters back to the server for interactions and computations.
In all experiments, we maintain a reward function population of size 5, along with the original human-designed reward function, resulting in total of 6 reward functions. The improvement of the reward function population involved a total of evolutions (including the generation of the initial population). The clite size to 3 for all tasks. Thompson sampling parameters a and 3 are to I by default.
For all tasks in MetaWorld, trained for million environment steps, while for all tasks in ManiSkill3, we trained for 2 million environment steps. Consequently, for tasks in Meta World, population evolution is performed every 200,000 environment steps, whereas for tasks in ManiSkill3, is performed every 500,000 steps.
The following describes the policy interaction process.
For MetaWorld, first perform n rounds of population policy interactions, during which individual policies are sampled using the Thompson sampler. If the same individual is sampled again, we will skip the interaction. Subsequently, we conduct policy interactions guided by the human-designed reward function. Each interaction corresponds to one episode. For ManiSkill3, we use 16 environ- ments for parallel sampling by default. rounds of policy interactions are performed as one policy interaction cycle, resulting in 64 environment steps (the default number of interaction steps before each training iteration in the SAC implementation).
After the interaction process, the collected data is annotated with multiple rewards by the reward population, and the corresponding variable information are recorded for subsequent relabeling and other operations.
For training settings, we follow the configurations of SAC. Specifically, we set the UTD ratio to I for MetaWorkd, and to 0.5 for ManiSkill3.
For the process of generating reward functions using the LLM, it is essential to extract necessary information, e.g., the husanndesigned reward function, the input variables, and the success criteria. Subsequently, we use Prompts 1 and 2 as input to the LLM to generate the initial population, with the suggestions from Prompt 3 appended to Prompt Subsequently, during each evolution process, information about the best reward function is provided in Prompt 4, and the L.MM is instructed to construct new individuals through reflection. The following describes the prompt design for the LLM.
Prompt Role Definition and Task Description
You are a reward engineer trying to write reward functions to solve reinforcement learning tasks effective as possible. Your goal is write reward function for the environment that will help the agent learn the task described in text. You can introduce or remove some reward components for better learning. Here the current reward function. your task to refine and enhance it: [task |tagett]
16

---

Prompt 2: Human-Designed Reward Function and User Format
The reward function is defined as compute_reward function [task __s__________ tio__odd___t string
Sappper_caging reward function
[task obs __dd__ode_______________eeee code_string 2]
Based on the return values of the functions mentioned above, some criteria are determined follows:
(criteria ciriiaa_ooeeeeeeeeeeeeeeeeee code_string)
These variables are crucial for constructing your reward function. Note that it only applies to the return values the functions mentioned above. Therefore, you should not modify the calculation methods for these variables in the functions, as it may disrupt the conditions evaluation. Instead, focus on leveraging this information design more efficient reward guidance.
Here are the modification suggestions (suggestion).
Please strictly follow them to rewrite the "compute_rewwe reward" and leriggrrrpagppee ceging_rewwrdd" reward" above separately, using the keys from the list below function inputs. However, do not introduce any keys that are not present the list. (input_dice dietddicccctccee sttisstting]
Repeatedly verify that all input variables in the function definition exist in the list, ensuring in naming or the introduction of new variables.
17

---

Prompt 3: Suggestions
The code output should be formatted as a python code string: """python .." The return variables must be consistent with those provided the given functions. You should neither add nor remove variables, nor modify their names. will specifically search for the "compute reward" and " Please carefully consider the sub-tasks that need to be completed sequentially to achieve the current task, and determine what rewards necessary for guiding each task.
reward" reward functions.
Carefully read the logic of the code above and improve the code in three ways, each of which must include the following:
(1) Reward Components: Add or remove certain components. If there are no modifications, please provide brief reason. for example, add xxx reward component to encourage the agent to do xxx and apply weight for better xxx
(2) Reward Weights: Adjust the weight certain reward components or change reward coefficients. If there are modifications, please provide brief reason. for example, change the reaching reward weight from 5.0 to 10.0 for better xxx
(3) Reward Calculation: Modify the reward calculation methods. If there are modifications, please provide a brief reason. for example, change the reaching or catching reward calculation method add exp Finally, summarize three areas of improvement and provide valid reasons for their effective-
reward component encourage the agent do
Some helpful tips for writing the reward function code:
(I) You may find it helpful to normalize the reward a fixed range by applying transforma- tions like np.exp to the overall reward or its components.
(2) If you choose transform reward component, then you must also introduce a tempera- ture parameter inside the transformation function; this parameter must be a named variable in the reward function and it must not be an input variable. Each transformed reward component should have its own temperature variable.
(3) Please do not simply transform the reward components or adjust the hyperparameters. Some unnecessary reward components can be removed, while some components that may be effective for learning can be added the final reward.
(4) Make sure the type of each input variable is correctly specified; All the necessary information is provided in the function inputs, and "self" neither referenced nor called. (5) Do not modify the conditions for determining success, proximity to the object, or object grasping, as this would compromise the evaluation criteria.
(6) It is necessary adjust some parameters of the existing reward function, such scaling the reward for grasping, scaling the proximity reward, or scaling the success reward. For example "If condition: reward 1.0 to reward +=0.5 for higher importance".
(7) an error occurs while calling function, implement the desired functionality based on your understanding instead of repeatedly calling the function.
Prompt 4: Reflection Tips
Based the above reward function, the current RL policy's win rate {win_rate), the human designed expert reward is (current_scorg and the reward from the currently designed function is (current our ourrooorroee score). Below are the scores of the current policy on different metrics across multiple rounds during the evaluation process: (current __ttput] Please carefully analyze the policy feedback. Some helpful tips for analyzing the policy feedback:
(1) If the success rates are always near zero, then you must rewrite the entire reward function.
(2) If the current policy has already performed well on certain metrics, the focus should shift
to the subsequent tasks.
(3) If the reward is excessively large, it may need to be appropriately scaled to avoid learning issues.
18

---

C Generated Reward Examples by LaRes
Below is an example of the reward function constructed by LaRes, which successfully guided policy learning to achieve nearly 100% success rate on the Hammer task in its final stages. The following three contexts represent the main reward function, the subfunction and the improvement summaries. can be observed that the L.M capable of thoroughly analyzing the current issues and providing constructive improvement ideas.
Generated Reward Function
compute_ revard obs, HAMMER, HAKMER_HANDLE,
.target_pos, NailslideJoint.quue left_ pad, right_pad
center,, obj_init_pos, init_tep): hand obs [:3]
hammer
[4:7]
hammer_head = hammer up. apraaray ([O.16, 0.06, 0.0])
# Hammer position adjustment for prozimity consideration
Sammer_threshen
hammer copy ()
threshold HAMMER_HANDLE_L HANGHE
if abs (hammer [o] hand [o]) threshold:
shamerrthreehedeee [o] = hand [o]
Quaternion error reward calculation
ideal np array ([1.0, 0.0, 0.0, 0.0])
error np. www.linlgg..oo norm [obs[ ideal)
reward_quat max (1.0 error / 0.3, 0.0) Modiffed the
denominator for stricter orientation
Computing caging and gripping rewards
anggingand.grippiinn caging htpparrriggt.ppttppppeeeeeroooo init_tep, init_top, htt.:paaccc..nmmmeetetteeeeeee tod.aucces.ttrreh-..00....
reward_grat (caging_and + caging) / 2.0
# Lifting reward with enhanced importance
lifted = hammer head [2] > 0.02
pos_ Increased smportance of lifting the hammer
_target_pos hammer_heat
a 0.5
b = 1.5 # Increased importance of hitting the target nail reward_ing ieware_in_place = float (lifted) bb* geward_utils tolerance
up. linalg..g..... norm (pos_error),
bounds=(O, 0.015), $ Reduced bounds for improved precision
margin=0.1,
sigmoid="long_tall")
Overall reward normalization
temperature 5.0 # Adjusted for better normalization
reward (2.0 reward_grab + 4.0 reward_in.placo)
reward_quat temperature
Success condition evaluation
success hailslideloint_qpon if auccess and revard > 0.5: # Reducing the threshold for checking success
0.09
reward 10.0
return (reward, reward_grab, reward_quat, reward_in_plaee success)
19

---

Generated Subfunction
def
-gripper_
Siiper_caginggrewarrd
left_pad,
right_pad,
tcp_center,
obj.init_pos,
init_tep,
action.
obj-pos,
obj.radius=0.015,
pad.succes..cresh=00000
objjcc__eeach.raddss00...00
xw.thresh=0.01,
desired_grrpperr.fffotr11110
figi.
fedium,denstt==Falsee,
):
"""Reward for agent grasping the hammer. "**
if high_density and :enssty::
raise ValueError ("Can only be either high_density or medium_density")
Calculate distances for caging rewards
pad_y_lr np. hstack (leett_pad11), right_pad (1)))
pad_to_obj_1r up abs (pad_y_lr obj-pos [1])
pad_to_ uid_to_obl tottooooiit_irrrrrei up. (pad.y__11 obj_initt_pos [1])
Define caging rewards based on y-azis
Baging_lr_marpe =
-pad_suceess_threet
caging_lr =[reward_utils. tolerance too [i] bounds=
(obj.radius sad_success_ thrrsshh_hhehh)
aagin=caging..r_aarre in range(2)]
for i
caging-y reward_utils. hrr.lsshaaccherppouuttcccc (*caginggt)))))
Caging reward for =-2 reference
tcp, xz = tcp_center, [0, 2]
caging_xz_margia = linain norm norm
init_tcp [xz]) thethresh
caging_xz reward_utils tolerance (np. linalg norm (tep[xz)
[x=p))((x2),,
(0nds==0,,x2thhree, mag.imaggg......argg....
sigmoid=="ong__aai""
Determine gripper closure estent
gr gripper_closed min (max (0, action [-1]),
Combining the various caging rewards
caging reward_utils. www.ddariiee.aaaccee.ppooddo gripping gripper_closen
caging_x2)
caging 0.97 else 0.0
bgging_and.gripple reward_utils hamacher_product (caging, gripping)
if high_density:
return anging.aad.grripping caging medium_density:
tep tep_center
tep_to_obj up www.iiian..oooooo norm (obj_pos tcp)
tcp_to_obj.inte np norm (ar_mooojiiiippppp.
init_tcp)
teach_margin tbs(tap.to.obj_.ng
bbjject_reac__radinn
reach = aovard.utile..tllereee (tcp_to_obj bounds=(O,
object_readie abject_reachhrraiiss..r marginn margin=
reach_ reach.marginnsiigooidd" taill
return caging and_gripping, reach
20

---

Improvement Summary
### Summary of Improvements
1. **Reward Components**; The reward for lifting the hammer ('a") and for hitting the target nail ("b") was increased significantly to enhance the focus on these critical actions. The increased penalties when nearing the target position (e.g. adjusted bounds '0.015") will encourage more precise interactions with the target area. 2. **Reward Weights**: The weight of the successful lift and placement were modified to provide enhanced emphasis on these actions, thereby creating stronger incentive for the agent to accomplish these milestones, which seems crucial considering the policy's 's current low success rate.
3. **Reward Calculation The overall normalization and adjustment reward components now includes changes in thresholds and the temperature parameter, which should provide more stability and control in training. Lowering the temperature value improves the agent's s control the learning process and minimizes excessive reward scaling. These adjustments are proposed to directly address the demonstrated need for improvements given the agent's 's historical performance metrics, particularly with near-zero success rate. The enhanced focus on the essential tasks and tighter constraints on the rewards should facilitate more efficient learning trajectory towards successful manipulation tasks.
D
Additional Experiments
Table 4: Performance under different population sizes.
Pop size
2
5
10
soccer
0.48
0.71
0.63
pick-out-of-hole
0.38
0.81
0.65
hammer
0.55
0.80
0.85
Parameter analysis experiment on population size. The results are shown in Table 4. We observe that population size of 5 generally yields the best performance. A larger population may introduce potential out-of-distribution issues, while smaller population can limit the algorithm's 's exploration capacity.
Table 5: Performance under different numbers of interaction steps.
Interaction steps
100,000
200,000
400,000
soccer pick-out-of-hole
0.68
0.71
0.55
0.83
0.81
0.60
hammer
0.53
0.80
0.69
Parameter analysis experiment on interaction steps. The results are shown in Table 5. We observe that frequency of 200k generally performs well. A smaller evolution frequency may lead to insufficient training of the lower-level policy, while larger frequency can result in under-exploration of the reward function search space.
Table 6: Performance under different elite sizes.
Elite size
1
2
3
4
soccer
0.63
0.78
0.71
0.75
ofcf-l
0.73
0.67
0.81
0.57
hammer
0.72
0.75
0.80
0.68
21

---

Parameter analysis experiment on elite size. The results are shown in Table 6. We find that generally yields the best results. An elite size that is too small tends increase the risk falling into suboptimal solutions.
Table 7: Performance comparison using different LLM backbones.
Backhone
basketball
soccer
pick-out-of-hole
hammer
4o-mini 0.87
0.71
0.81
0.80
deepseck-v3 0.88
0,68
0.58
0.63
qwen-plus 0.63
0.88
0.98
0.50
Comparative experiment using different LLMs as backbones. We conducted evaluations with different LLMs as backbones, including Qwen-plus, GPT-4o-mini, and DeepSeek- -V. The results are shown in Table 7. We observe that although different models exhibit some variation in performance across tasks, LaRes consistently achieves high success rate regardless of the underlying LLM framework.
Parameter analysis experiment on the weight of constraint loss. The results are shown in Table LaRes does not tune this hyperparameter: it is set to the default value of 1.0 across all tasks. While further tuning may lead to improved performance, we find that 1.0 is generally sufficient achieve strong results.
Table 8: Parameter analysis of the weight of constraint loss.
Coefficient
10.0
1.0
0.1
0.01
pick-out-of-hole
0.60
0.81
0.85
0.78
soccer
0.64
0.71
0.81
0.74
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: This paper focuses on developing framework that enables efficient coordina- tion between reward function optimization and policy learning, with the aim of achieving higher sample efficiency. The main claims presented in the abstract and introduction accu- rately reflect the contributions and scope of the work.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We provide discussion of the limitations in Appendix A.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
23

---

Answer: [NA]
Justification: Our work focuses on empirical evaluation and does not provide theoretical proofs formal analysis. We leave the development of theoretical guarantees and formal analysis as future work.
Guidelines:
The answer NA means that the paper does not include theoretical results, All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide the detailed pseudocode, experimental hyperparameter settings and the designed prompts.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
24

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
We commit to releasing the code upon the public availability of the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
sellic//uudess htt///ee///eeeuaiiiseseeeeeeeeee
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: // //nips. peips.cc/pubbl ppwplcc/guddesseeeppppppbpb.paee httpg//dddd//eeddubibimiieio SrodeeSumiissionPo..
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the detailed experimental settings in both the main experiment section and the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Following the experimental settings of prior work, conduct each experiment with 5 independent runs and report error bars to reflect statistically significant variability.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
25

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: These details are provided in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: / http://tt....ccppiici/////////// dewudeeeeeeleeeesss
Answer: [Yes]
Justification: Yes, the research conducted in this paper fully conforms with the NcurIPS Code of Ethics in all respects.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction)
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: This paper presents work whose goal is to advance the field of RL. There many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Guidelines:
26

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes, all creators and original owners of the assets used in the paper properly credited and fully respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
27

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer:[NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
28

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: LLMs are employed for reward function generation, and comprehensive details are provided in the main body of the paper well as in the appendix. All components of the proposed methods are independently designed and original, with no dependence on LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: /ttp/reurpp..//// cc/ for what should or should not be described.
/LLM)
29


Reviews:
[{"review_id": "hwuo7DPpK5", "reviewer": "Reviewer_bxvD", "summary": "This paper proposes LaRes, a framework that uses LLMs within an evolutionary algorithm to search for optimal reward functions for RL tasks. By evolving reward functions directly, it aims to automate a critical part of the reinforcement learning workflow. It introduces several techniques like reward relabeling and parameter constraints to improve sample efficiency and stability.", "questions": "In addition to the points raised in the weakness section, I have three further academic questions for the authors:\n1. In the framework, the elite reward functions are not replaced, and the RL agent guided by the reward function designed by humans does not participate in Thompson sampling to ensure the stability of learning. However, does this design also bring the risk of prematurely converging to local optima? If these strategies are not globally optimal, will the mechanism continuously guide the search process, thereby hindering the exploration of new and potentially better reward paradigms? Have you considered other mechanisms to balance stability and the need for long-term exploration?\n2. The paper demonstrates that the final evolved reward functions are effective. However, what is the nature of this evolutionary trajectory? Do the reward functions converge towards a specific structure? Do they tend to become more complex over generations, or do they become simpler as the LLM distills the problem's essence?\n3. Appendix A mentions that the current method relies on the randomness of LLMS to ensure diversity, but does not explicitly consider individual differences. Have you considered adding explicit instructions in the Prompt to guide the LLM to generate more diverse reward functions? Will this further enhance the performance of LaRes?", "limitations": "See the above weakness section.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths:\nIt's always good to see novel approaches to the reward design problem in RL. I particularly appreciate the systematic framework design, which demonstrates sophisticated LLM integration through closed-loop optimization with quantitative feedback. The core idea of using an LLM as an evolutionary operator to search the reward function space is creative and could inspire follow-up work along this direction.\n## Weaknesses: \nThere are a few points that I would like to suggest here to make the paper even stronger.\n* Although paper shows that LLM-generated rewards improve performance significantly, hard-exploration tasks are often difficult because they require non-intuitive intermediate rewards. It would be insightful if the authors can discuss the relationships between the LLM's generative priors and performance on such tasks. Specifically, why might an LLM-guided search still fail on a hard-exploration task even if it can generate dense rewards? Does the LLM's common sense reasoning create a bias that prevents it from discovering the unconventional reward components needed to solve these problems?\n* When the reward function population is updated, all experiences in the shared replay pool need to be relabeled. How much computational overhead is this process for a replay pool of samples of different sizes? I hope authors can conduct supplementary experiments for analysis and examine the trade-offs related to performance.\n* LaRes contains a series of hyperparameters of the genetic algorithm, such as the population size of the reward function, the number of evolutions, the elite size, etc. However, it seems that the article does not discuss the sensitivity of the framework's performance to these hyperparameters. It is hoped that supplementary experiments can be conducted to illustrate this.\n* This paper relies exclusively on GPT-4o mini as the core LLM engine, which raises questions about the generalizability of the findings to other model families, particularly mainstream open-weight models like Llama or Qwen.  Without validation across different LLM architectures, the broader applicability of the LaRes framework remains uncertain. I would strongly recommend that the authors conduct additional experiments with alternative LLM to demonstrate the robustness and generality of their approach.", "was_revised": true, "final_justification": "Thank you to the authors for the detailed rebuttal. I currently have no further questions. The additional experiments on different LLMs and hyperparameter sensitivity have resolved most of my technical concerns.\n\nMy overall suggestion is to better clarify the scope of the paper's contributions. The paper demonstrates that LaRes is highly effective on a wide range of complex manipulation tasks. However, given the authors' insightful discussion on the challenges of reward hacking and LLM bias, it is difficult to fully believe that the same method can readily solve hard-exploration problems where the necessary reward signals are non-intuitive.", "rebuttal": "We appreciate the reviewer's valuable review and constructive comments, and we would like you to know that your questions provide considerably helpful guidance to improve the quality of our paper. \n\nWe will try our best to address each of the concerns and questions raised by the reviewer below:\n\n**1. [Re: It would be insightful if the authors can discuss the relationships between the LLM's generative priors and performance on such tasks. Specifically, why might an LLM-guided search still fail on a hard-exploration task even if it can generate dense rewards? Does the LLM's common sense reasoning create a bias that prevents it from discovering the unconventional reward components needed to solve these problems?]** \n    \nFirst, task decomposition is essential for a given task, as it can significantly reduce the difficulty of reward generation. Directly using LLMs to generate long and complex reward functions is often impractical.\n    \nWe fully agree with the reviewer’s observation that even with dense rewards, learning a successful policy may still fail. This is mainly due to a potential misalignment problem—dense rewards are not necessarily positively correlated with task success, leading to reward hacking. This issue arises from multiple factors. On one hand, individual reward components may be poorly designed—for example, critical reward terms may be underweighted and dominated by less important ones, or inappropriate activation functions may be used.\n    \nMoreover, reward design can be viewed as a black-box optimization problem, where some reward components that seem intuitively useful may not be necessary for successful policy learning. This highlights the need for RL–driven evaluation of reward functions. It also implies that when a reward function fails to guide learning effectively, the capability of the RL algorithm itself is a non-negligible factor. For instance, the performance of SAC and AC can differ significantly under the same reward design.\n    \nTherefore, we argue that reward functions generated by LLMs, like those manually designed by human experts, are inevitably prone to being suboptimal or even ineffective. Thus, iterative refinement and reflective optimization are often necessary to achieve reliable and aligned reward signals.\n    \n**2. [Re: When the reward function population is updated, all experiences in the shared replay pool need to be relabeled. How much computational overhead is this process for a replay pool of samples of different sizes?.]**\n    \nThe computational overhead of this process is minimal. We directly relabel the replay buffer through code, without involving any gradient operations, resulting in very low complexity.\n    \nWe measure the relabeling time and find that relabeling 200,000 samples takes only 5.89 seconds, and the cost scales linearly with the number of samples.\n    \n**3.[Re: LaRes contains a series of hyperparameters of the genetic algorithm, such as the population size of the reward function, the number of evolutions, the elite size, etc. ]**\n    \nWe thank the reviewer for the valuable feedback. Regarding the population size, we followed the common settings used in prior ERL works, setting it to 5, with 5 evolution number( same with Eureka). The elite size is set to 3.\n    \nTo address the reviewer’s concern, we conducted the following additional hyperparameter experiments:\n    \n| Pop size | 2 | 5 | 10 |\n| --- | --- | --- | --- |\n| soccer  | 0.48 | **0.71** | 0.63 |\n| pick-out-of-hole | 0.38 | **0.81** | 0.65 |\n| hammer | 0.55 | 0.80 | **0.85** |\n    \nWe observe that a population size of 5 generally yields the best performance. A larger population may introduce potential out-of-distribution issues, while a smaller population can limit the algorithm’s exploration capacity.\n    \n| **interaction steps** | 100.000 | 200.000 | 400.000 |\n| --- | --- | --- | --- |\n| soccer  | 0.68 | **0.71** | 0.55 |\n| pick-out-of-hole | **0.83** | 0.81 | 0.60 |\n| Hammer | 0.53 | **0.80** | 0.69 |\n    \nFor the evolution number, we define it in the number of environment interaction steps between two evolutionary updates.\n    \nWe observe that a frequency of 200k generally performs well. A smaller evolution frequency may lead to insufficient training of the lower-level policy, while a larger frequency can result in under-exploration of the reward function search space.\n    \n| Elite size | 1 | 2 | 3 | 4 |\n| --- | --- | --- | --- | --- |\n| soccer | 0.63 | **0.78** | 0.71 | 0.75 |\n| pick-out-of-hole | 0.73 | 0.67 | **0.81** | 0.57 |\n| hammer | 0.72 | 0.75 | **0.80** | 0.68 |\n    \nFor the elite size, we found that 3 generally yields the best results. An elite size that is too small tends to increase the risk of falling into suboptimal solutions.\n    \n**4. [Re: I would strongly recommend that the authors conduct additional experiments with alternative LLM to demonstrate the robustness and generality of their approach.]**\n    \nWe thank the reviewer for the valuable suggestion. We conducted evaluations across a diverse set of benchmarks, including Qwen-plus, GPT-4o-mini, and DeepSeek-V3. The experimental results are shown below.\n    \n|  | basketball | soccer | pick-out-of-hole | hammer |\n| --- | --- | --- | --- | --- |\n| 4o-mini | 0.87 | 0.71 | 0.81 | 0.80 |\n| deepseek-v3 | 0.88 | 0.68 | 0.58 | 0.63 |\n| qwen-plus | 0.63 | 0.88 | 0.98 | 0.50 |\n    \nWe observe that although different models exhibit some variation in performance across tasks, LaRes consistently achieves a high success rate regardless of the underlying LLM framework.\n    \n**5. [Re: In the framework, the elite reward functions are not replaced, and the RL agent guided by the reward function designed by humans does not participate in Thompson sampling to ensure the stability of learning. However, does this design also bring the risk of prematurely converging to local optima? If these strategies are not globally optimal, will the mechanism continuously guide the search process, thereby hindering the exploration of new and potentially better reward paradigms? Have you considered other mechanisms to balance stability and the need for long-term exploration?]**\n    \nWe agree with the reviewer's perspective that exploration versus exploitation is a fundamental challenge in any learning or optimization problem. We did consider using UCB to address this issue more effectively; however, UCB introduces several hyperparameters that require careful tuning. While better performance might be achievable through fine-tuning, we chose to adopt Thompson Sampling for the sake of simplicity.\n    \nAs for long-term exploration, one viable approach is to increase the number of environment interaction steps per generation for each population member. However, this presents a trade-off: too many steps may lead to sample inefficiency, while too few may result in suboptimal performance.\n    \nFinally, regarding the decision to exclude the RL reward function from evolutionary optimization, this follows prior ERL settings. This design ensures that when the population fails to discover better solutions, the RL reward function can still provide a performance lower bound.\n    \n**6. [Re: what is the nature of this evolutionary trajectory? Do the reward functions converge towards a specific structure? Do they tend to become more complex over generations, or do they become simpler as the LLM distills the problem's essence?]**\n    \nIn practice, we observe that the structure of the reward function typically remains relatively stable from the early to later stages of training. For example, in a pick-and-place task, the reward function usually consists of components such as approach reward, grasp reward, and reward for moving toward the target. The main changes over time tend to lie in the tuning of hyperparameters—both within and between reward components—as well as in the implementation details of each module, such as different transformation functions. Overall, the reward function does not become increasingly complex; instead, once the general structure is established, subsequent efforts focus on refining and optimizing specific details.\n    \n**7. [Re: Have you considered adding explicit instructions in the Prompt to guide the LLM to generate more diverse reward functions? Will this further enhance the performance of LaRes? ]**\n    \nWe thank the reviewer for the valuable suggestion. In our implementation, we set a relatively high temperature for the LLM (i.e., 1.0) to promote population diversity.\n    \nTo address the reviewer’s concern, we experiment with designing diversity-enhancing prompts. Specifically, in each generation step, we feed the previously generated result back into the LLM as an additional input, encouraging the model to produce more diverse outputs. The results are shown below.\n    \n|  | hammer | soccer | **pick-out-of-hole** |\n| --- | --- | --- | --- |\n| Diversity version | 0.60 | 0.68 | 0.60 |\n| Org version | 0.80 | 0.71 | 0.81 |\n    \nWe observe that this approach leads to a decline in performance. Upon closer inspection of the generated reward functions, we identify a key issue: while the method increases diversity, it often results in reward functions that omit essential modules. For example, after the first reward function is generated with a human-aligned structure, subsequent ones tend to introduce entirely different reward components for the sake of diversity.\n    \nAs previously discussed, good reward functions tend to share a common structure, differing mainly in their finer details. We therefore believe that enhancing diversity requires more principled mechanism design. Simply relying on prompt-based guidance is insufficient to ensure both high diversity and strong performance.\n\n--- \nWe hope our replies have addressed the concerns the reviewer posed and shown the improved quality of the paper. **We are always willing to answer any of the reviewer's concerns about our work and we are looking forward to more inspiring discussions.**"}, {"review_id": "DQ4hjyaUo7", "reviewer": "Reviewer_4fMG", "summary": "This paper introduces LaRes, a novel framework that integrates Large Language Models (LLMs) with Evolutionary Reinforcement Learning (ERL) to iteratively search for and optimize reward functions instead of policy parameters—a departure from traditional ERL methods. The core idea is to evolve reward functions using LLMs and evaluate them via reinforcement learning performance, thus enhancing both sample efficiency and final task success.", "questions": "- Could the authors provide any theoretical insights into convergence properties or bounds for LaRes? How can we trust the stability of LLM-driven reward evolution in the long run?\n- How does LaRes scale with more complex or high-dimensional tasks beyond the 16 robotic benchmarks tested?", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- Shifting the focus from policy parameter search to reward function search using LLMs is both novel and impactful.\n- The framework is presented clearly with diagrams (e.g., Figure 1) and step-by-step algorithmic breakdowns (Algorithm 1).\n\nWeaknesses:\n- The paper acknowledges that LLM-generated functions may lack diversity, which could limit the exploration potential of the evolutionary process.\n- While the authors also explore scratch generation, the method generally relies on human-designed reward functions as a base, which may limit applicability in settings where expert knowledge is unavailable.", "was_revised": true, "final_justification": "The MinAtar, Ant, and Humanoid experimental results submitted by the authors convince me of LaRes’s potential in handling more complex or higher-dimensional tasks. Overall, I consider this an good paper and have therefore decided to maintain my score.", "rebuttal": "We appreciate the reviewer's valuable review and constructive comments, and we would like you to know that your questions provide considerably helpful guidance to improve the quality of our paper. \n\nWe will try our best to address each of the concerns and questions raised by the reviewer below:  \n**1. [Re: The paper acknowledges that LLM-generated functions may lack diversity, which could limit the exploration potential of the evolutionary process]**\n    \nYes, we promote diversity by constructing an LLM-based population and employing a relatively high temperature during generation. While we acknowledge that more effective mechanisms for ensuring diversity may exist, this is not the primary focus of LaRes at the current stage.\n    \n**2. [Re: While the authors also explore scratch generation, the method generally relies on human-designed reward functions as a base, which may limit applicability in settings where expert knowledge is unavailable.]**\n    \nWe thank the reviewer for the helpful suggestion. We conducted experiments on the following two domains where no human-designed reward function is required:\n    \n- **5 Tasks from Metaworld (win rate & sample cost):**\n    \n|  | **window-close** | **window-open** | **drawer-open** | **button-press** | **door-close** |\n| --- | --- | --- | --- | --- | --- |\n| LLM zero shot | 51%   | 388796 | 6%    | 419084 | 8%    | 762346 | 65%  | 560428 | 15%  | 156590 |\n| Eureka | 50%   | 303786 | 55%  |  657098 | 15%   | 731876 | 61%  | 924163 | 98%  | 283206 |\n| LaRes | 100% | 164850 | 100% | 358403 | 100% | 164850 | 100% | 112800 | 100% | 65626 |\n- **Openai Humanoid & Ant tasks:**\nWe have currently completed experiments up to 500K steps, and we will update the results for 1M steps before the discussion deadline.\n\n| 500k | Eureka | ROSKA | LaRes |\n| --- | --- | --- | --- |\n| Ant | 1.49 | 1.57 | **3.61** |\n| Humanoid | 1.91 | 2.53 | **2.71** |\n\nWe observe that LaRes still demonstrates a significant advantage.\n    \n**3. [Re: Could the authors provide any theoretical insights into convergence properties or bounds for LaRes? How can we trust the stability of LLM-driven reward evolution in the long run?]**\n    \nThe results generated by LaRes are often superior to those produced by standard RL algorithms. This is primarily because LaRes follows the ERL framework and always maintains the expert reward function within the reward population. When LLM-generated reward functions perform poorly, the policy guided by the original reward function dominates the process, ensuring a performance lower bound. In addition, LaRes constructs an RL policy population, which further augments sample diversity and enhances overall performance.\n    \nOur work primarily focuses on empirical validation of the proposed method's effectiveness, supported by extensive experiments across 12 MetaWorld tasks, 4 ManiSkill tasks, 5 human-reward-defined tasks, as well as Humanoid and Ant tasks. Regarding theoretical support, we acknowledge in the Limitations section that the current work lacks formal theoretical guarantees, which we consider an important direction for future research.\n    \n**4. [Re: How does LaRes scale with more complex or high-dimensional tasks beyond the 16 robotic benchmarks tested?]**\n    \nTo address the reviewer’s concern, we primarily extended our experiments in two directions:\n    \n- **MinAtar**, a benchmark based on image inputs.\n- **Ant & Humanoid**, two of the most challenging control tasks from OpenAI's suite.\n    \nThe results on **MinAtar**:\n    \n|  | **Breakout** | **Asterix** | **Freeway** | **Space_invaders** |\n| --- | --- | --- | --- | --- |\n| DQN | 13.52 | 21.86 | 2.96 | 11.55 | 38.89 | 53.93 | 23.18 | 65.38 |\n| LaRes | 21.87 | 31.28 | 14.13 | 28.18 | 50.87 | 57.24 | 42.27 | 84.72 |\n    \nThe results on **Humanoid** and **Ant** without human-designed reward functions, we have currently completed experiments up to 500K steps, and we will update the results for 1M steps before the discussion deadline.\n\n| 500k | Eureka | ROSKA | LaRes |\n| --- | --- | --- | --- |\n| Ant | 1.49 | 1.57 | 3.61 |\n| Humanoid | 1.91 | 2.53 | 2.71 |\n\nWe observe that LaRes also demonstrates superior performance on these tasks, further highlighting its effectiveness and generality.  \n\n---\n\nWe hope our replies have addressed the concerns the reviewer posed and shown the improved quality of the paper. **We are always willing to answer any of the reviewer's concerns about our work** and we are looking forward to more inspiring discussions."}, {"review_id": "lt76lazp9V", "reviewer": "Reviewer_XhCm", "summary": "This paper proposes LaRes, an RL framework for LLM, which focuses on the reward function search. By applying reward function updates and Thompson selection mechanism, the proposed method can improve the historical data usage and prevent the policy collapse.", "questions": "1. Provide more experimental analysis on the computation cost.\n2. Why LaRes exhibits larger variance compared to the baseline.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n\nThe proposed method applies LLM to achieve reward functions searching, which is an innovative framework.\n\nWeakness\n\n1. The relation between high-level reward evolution and low-level policy learning is not clear. Furthermore, the relation between the framework and techniques like continuous learning, Thompson sampling, and reward scaling should be clarified.\n\n2. The authors claimed that LaRes can cooperate with other RL methods, which is not demonstrated in the evaluation. Moreover, the current design requires multiple Reward functions to fine-tune the model, some experiments on computation cost should be involved.\n\n3. The main comparison in evaluation adopts SAC, which was proposed in 2018; more comparisons with SOTA methods are recommended. Furthermore, the baselines in ERL-ralated are out-of-date except ERL-Re and EvoRainbo.\n\n4. In Fig2,3,4, several results of LaRes exhibit larger variance than baseline; more discussion on it is required. \n\n5. No appendix, and the code is not open-sourced.", "was_revised": true, "final_justification": "Overall, the proposed idea is interesting, but the efficiency on complex tasks has not been fully explored (although the authors provide some initial experiments in rebuttal). Also, the theoretical proof is not adequate. Thus, it is good for me to accept this work, but I will not champion it. I will keep my original score.", "rebuttal": "We appreciate the reviewer's valuable review and constructive comments, and we would like you to know that your questions provide considerably helpful guidance to improve the quality of our paper.\n\nWe will try our best to address each of the concerns and questions raised by the reviewer below:   \n**1.[Re: The relation between high-level reward evolution and low-level policy learning is not clear. Furthermore, the relation between the framework and techniques like continuous learning, Thompson sampling, and reward scaling should be clarified.]**\n    \nLaRes is primarily focused on sample-efficient task solving. To address this challenge, we decouple the problem into two levels: high-level reward function optimization and low-level policy learning. At the upper level, an LLM is used to construct a population of reward functions, which guide the learning of policies at the lower level. To further improve sample efficiency, we adopt a sample-level sharing strategy, where trajectories collected by different RL agents are shared and relabeled based on their respective reward functions.\n    \nThe introduction of Thompson Sampling also aims to enhance sample efficiency by allowing stronger policies within the RL population to interact more frequently with the environment, thereby reducing waste from weaker policies.\n    \nIt is important to note that our notion of continual learning differs from conventional definitions, which typically focus on network plasticity. In contrast, LaRes is designed to maintain policy stability in the face of evolving reward functions, ensuring that performance does not degrade or collapse when the reward function changes.\n    \nTo achieve this, we introduce reward scaling and a consistency loss. The former ensures distributional alignment of rewards before and after changes, while the latter constrains policy updates to prevent performance drops.\n    \n**2. [Re: The authors claimed that LaRes can cooperate with other RL methods, which is not demonstrated in the evaluation. ]**\n    \nLaRes is compatible with any off-policy RL algorithm. To address the reviewer’s concern, we integrated LaRes with DQN and evaluated it on MinAtar, a set of discrete control tasks based on image inputs. The experimental results are shown below:\n    \n|  | **Breakout** | **Asterix** | **Freeway** | **Space_invaders** |\n| --- | --- | --- | --- | --- |\n| DQN | 13.52 | 21.86 | 2.96 | 11.55 | 38.89 | 53.93 | 23.18 | 65.38 |\n| LaRes | 21.87 | 31.28 | 14.13 | 28.18 | 50.87 | 57.24 | 42.27 | 84.72 |\n    \nWe observe that LaRes consistently leads to performance improvements.\n    \n**3. [Re: the current design requires multiple Reward functions to fine-tune the model, some experiments on computation cost should be involved.]**\n    \nLaRes requires optimizing multiple RL policies in parallel, each guided by a different reward function. Executing this process serially would indeed incur significant computational overhead. To address this, we implemented a distributed framework using a server–client architecture: sampling is performed on the server side, while each client independently optimizes its own policy. Overall, due to scheduling and communication overhead, LaRes incurs approximately 20% additional runtime.\n    \nRegarding computational resources, all experiments on MetaWorld are conducted using CPU only. Each run consumes (pop size + 1) CPU cores—one for the server process and the remaining for individual worker processes corresponding to each policy in the population.\n    \n**4. [Re: The main comparison in evaluation adopts SAC, which was proposed in 2018; more comparisons with SOTA methods are recommended. Furthermore, the baselines in ERL-ralated are out-of-date except ERL-Re and EvoRainbo. ]**\n    \nWe thank the reviewer for the insightful comment. LaRes is inherently flexible and can be built upon different underlying algorithms. In general, the stronger the backbone algorithm, the better the overall performance of LaRes.\n    \nTo address the reviewer’s concern, we conducted a comparison using two advanced RL algorithms: TD7 and CrossQ. The results are shown below.\n    \n|  | Coffee-Pull | Pick-out-of-hole | Soccer | Hammer |\n| --- | --- | --- | --- | --- |\n| TD7 | 0.65 | 0.0 | 0.53 | 0.82 |\n| CrossQ | 0.60 | 0.0 | 0.59 | 0.23 |\n| LaRes | 0.82 | 0.81 | 0.71 | 0.80 |\n    \nWe observe that although LaRes uses the relatively weaker SAC as its backbone, it still outperforms the advanced RL algorithms TD7 and CrossQ. This highlights the critical role of reward function optimization in achieving strong performance.\n    \n**5. [Re: In Fig2,3,4, several results of LaRes exhibit larger variance than baseline; more discussion on it is required. Why LaRes exhibits larger variance compared to the baseline]**\n    \nWe thank the reviewer for the helpful suggestion. The performance of LaRes is indeed influenced by the inherent randomness of the LLM. In some runs, suboptimal reward functions are generated, leading to reduced policy learning efficiency and increased variance in certain tasks. We believe this issue can be mitigated by either increasing the number of reward function optimization iterations or enlarging the reward function population, both of which allow for a more thorough search and make the framework more robust to the stochastic nature of the LLM.\n    \n**6. [Re: No appendix, and the code is not open-sourced.]**\n    \nWe provide a detailed appendix, including Limitations and future work (Appendix A), Implementation details (Appendix B), Reward function examples (Appendix C)\n    \nWe commit to releasing the code as soon as the paper is made public, which we consider an important contribution of this work.\n\n**7. [Re: Further experiments on Locomotion tasks]**\n\nTo further demonstrate the effectiveness of LaRes, we evaluate it on the two most challenging tasks in OpenAI MuJoCo: Ant and Humanoid. Notably, this version does not incorporate any human-designed reward function.\n\nWe have currently completed experiments up to 500K steps, and we will update the results for 1M steps before the discussion deadline.\n\n| 500k | Eureka | ROSKA | LaRes |\n| --- | --- | --- | --- |\n| Ant | 1.49 | 1.57 | 3.61 |\n| Humanoid | 1.91 | 2.53 | 2.71 |\n\nWe observe that LaRes also demonstrates superior performance on these tasks, further highlighting its effectiveness and generality.\n\n---\n\nWe hope our replies have addressed the concerns the reviewer posed and shown the improved quality of the paper. **We are always willing to answer any of the reviewer's concerns about our work and we are looking forward to more inspiring discussions.**"}, {"review_id": "dmzF6IdLuP", "reviewer": "Reviewer_c6GE", "summary": "The paper introduces LaRes, a hybrid framework combining evolutionary algorithms (EAs) and reinforcement learning (RL) with large language models (LLMs) to optimize reward functions for policy learning. LaRes uses LLMs to generate and evolve reward function populations, a shared replay buffer for sample efficiency, and mechanisms like Thompson sampling and parameter constraints to stabilize training. Experiments on 16 robotic manipulation tasks show LaRes outperforms strong baselines (e.g., SAC, ERL variants) in sample efficiency and final performance, though some tasks exhibit marginal gains over EvoRainbow.", "questions": "- Elite Criterion: How is the \"elite\" reward function/policy formally defined (e.g., success rate threshold, moving average performance)? Does the definition vary across tasks or remain consistent? \n- Ablation on Human-Designed Rewards: Have experiments been conducted where the human-designed reward is excluded from the population, and how does this affect performance? Comparison with Eureka and Roska on their benchmarks is suggested. This would validate the necessity of human guidance.\n- Normalization vs. Scaling: Why use reward scaling with elite statistics instead of normalizing all rewards to the Normal distribution? Does the current approach introduce unnecessary complexity? \n- Hyperparameter Sensitivity of Parameter Constraints: Has the weight of the parameter constraint loss (Equation 4) been ablated to assess its impact on stability and performance?", "limitations": "yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n- Novel Reward Function Optimization: By leveraging LLMs to iteratively evolve reward functions, LaRes addresses a critical gap in prior ERL work that focused on policy parameter search. The framework demonstrates tangible improvements over human-designed rewards, highlighting the value of automated reward engineering.\n- Sample Efficiency Mechanisms: The shared replay buffer with reward relabeling and Thompson sampling-based interaction prioritization effectively reuses historical data and focuses on high-performing policies. This design reduces sample waste and enhances learning stability, as validated by ablation studies.\n- Comprehensive Evaluation: The paper conducts rigorous experiments across two benchmarks (MetaWorld, ManiSkill3) and compares against diverse baselines, including RL, ERL, and LLM-based reward search methods. The results consistently show LaRes’s superiority.\n\n**Weaknesses**\n- Elite Definition and Terminology Ambiguity: The paper uses \"elite\" and \"non-elite\" to denote high-performing vs. low-performing reward functions/policies, but the criteria for defining \"elite\" (e.g., success rate, cumulative reward) are not explicitly standardized. This ambiguity could confuse readers, particularly when discussing policy inheritance and parameter constraints.\n- Dependence on Human-Designed Rewards: While LaRes starts from human-designed rewards and retains them in the population, the paper lacks experiments exploring fully LLM-generated rewards from scratch (without human input). Comparisons to Eureka under such conditions—where reward functions are initialized without human guidance—would strengthen the claim of LLM-driven optimization.\n- LLM Generation Dynamics: The process of LLM-generated reward function updates (e.g., how many new functions are created per iteration, which ones are replaced) is not clearly defined. For example, the paper states the population size is 5, but it is unclear whether LLMs generate all 5 functions or replace a subset.\n- Notation and Clarity in Formulation: The description of the Thompson sampling mechanism (e.g., \"i-th action\" referring to policy selection) is ambiguous, as it could be misinterpreted as action-space elements rather than policy indices. Clarifying this notation earlier in the text would improve readability.\n- Continual Learning and Plasticity Concerns: The paper mentions using the best policy’s parameters for initialization but does not address potential plasticity loss (e.g., catastrophic forgetting), a known issue in continual learning [1]. Compared to methods like RoSKA [2], which explicitly handle this, the framework could be enhanced in its theoretical robustness.\n\n[1] Maintaining plasticity in deep continual learning\n\n[2] Efficient Language-instructed Skill Acquisition via Reward-Policy Co-Evolution", "was_revised": true, "final_justification": "The authors have addressed my concerns.", "rebuttal": "We appreciate the reviewer's valuable review and constructive comments, and we would like you to know that your questions provide considerably helpful guidance to improve the quality of our paper. \n\nWe will try our best to address each of the concerns and questions raised by the reviewer below:  \n**1. [Re:  How is the \"elite\" reward function/policy formally defined (e.g., success rate threshold, moving average performance)? Does the definition vary across tasks or remain consistent?)]**\n    \nThe criterion for defining an \"elite\" is the success rate. We evaluate each policy over 20 episodes to assess the quality of the corresponding reward function. This definition is applied consistently across all manipulation tasks.\n    \n**2. [Re: Dependence on Human-Designed Rewards: While LaRes starts from human-designed rewards and retains them in the population, the paper lacks experiments exploring fully LLM-generated rewards from scratch (without human input). Comparisons to Eureka under such conditions—where reward functions are initialized without human guidance—would strengthen the claim of LLM-driven optimization. （Comparison with Eureka and Roska on their benchmarks is suggested. This would validate the necessity of human guidance.）]**\n    \nWe thank the reviewer for valuable suggestions. LaRes can also be applied in settings where no human-designed reward functions are available.\n    \nTo address the reviewer’s concern, we evaluated LaRes on several tasks used in Text2Reward, where no expert rewards are provided. The results—measured in terms of success rate and sample cost—are shown below:\n    \n|  | **window-close** | **window-open** | **drawer-open** | **button-press** | **door-close** |\n| --- | --- | --- | --- | --- | --- |\n| LLM zero shot | 51%   | 388796 | 6%    | 419084 | 8%    | 762346 | 65%  | 560428 | 15%  | 156590 |\n| Eureka | 50%   | 303786 | 55%  |  657098 | 15%   | 731876 | 61%  | 924163 | 98%  | 283206 |\n| LaRes | 100% | 164850 | 100% | 358403 | 100% | 164850 | 100% | 112800 | 100% | 65626 |\n    \nWe observe that our method still significantly outperforms other methods.\n    \n**3. [Re: Comparison with Eureka and Roska on their benchmarks is suggested. This would validate the necessity of human guidance.]**\n\nWe thank the reviewer for the valuable suggestion. LaRes primarily emphasizes sample efficiency, which requires development based on off-policy RL. However, modifying existing algorithms within IsaacGym to support efficient off-policy learning is quite challenging.\n    \nTo address the reviewer’s concern, we instead conducted experiments on OpenAI Humanoid and Ant tasks based on MUJOCO, which are commonly used in both Eureka and Roska. These experiments are performed without any human-designed rewards, and each is limited to 1 million environment steps.\nThe results are shown below, we have currently completed experiments up to 500K steps, and we will update the results for 1M steps before the discussion deadline.\n\n| 500k | Eureka | ROSKA | LaRes |\n| --- | --- | --- | --- |\n| Ant | 1.49 | 1.57 | **3.61** |\n| Humanoid | 1.91 | 2.53 | **2.71** |\n   \n    \nWe observe that LaRes significantly outperforms both Eureka and Roska on the Ant and Humanoid tasks.\n\n**4. [Re: LLM Generation Dynamics: The process of LLM-generated reward function updates (e.g., how many new functions are created per iteration, which ones are replaced) is not clearly defined. For example, the paper states the population size is 5, but it is unclear whether LLMs generate all 5 functions or replace a subset.]**\n\nIn our experiments, we set the population size to 5 and maintained 3 elites. If the RL policy guided by the human-design reward function is selected as an elite, we generate 3 new reward functions; otherwise, 2 new reward functions are generated. The individuals to be replaced are selected based on their success rate, in ascending order.\n        \n**5. [Re: Notation and Clarity in Formulation: The description of the Thompson sampling mechanism (e.g., \"i-th action\" referring to policy selection) is ambiguous, as it could be misinterpreted as action-space elements rather than policy indices. Clarifying this notation earlier in the text would improve readability.]**\n        \n We thank the reviewer for the valuable suggestion. We improve the presentation in the revised version.\n        \n**6. [Re: Continual Learning and Plasticity Concerns: The paper mentions using the best policy’s parameters for initialization but does not address potential plasticity loss (e.g., catastrophic forgetting), a known issue in continual learning [1]. Compared to methods like RoSKA [2], which explicitly handle this, the framework could be enhanced in its theoretical robustness.]**\n        \nWe thank the reviewer for the valuable input. As shown in our previous responses, we provide comparative experiments; here, we would like to offer some clarification.\n        \nROSKA is based on PPO and primarily emphasizes policy reuse, whereas LaRes is built upon off-policy RL and focuses on sample reuse from a finer-grained perspective. Moreover, the notion of continual learning in LaRes emphasizes maintaining learning stability and consistency when the reward function changes, thereby reducing performance fluctuations. This differs from conventional continual learning, which typically focuses on network plasticity.\n        \nWe appreciate the reviewer’s suggestion and will include this discussion in the revised version.\n        \n**7. [Re: Normalization vs. Scaling: Why use reward scaling with elite statistics instead of normalizing all rewards to the Normal distribution? Does the current approach introduce unnecessary complexity?]**\n        \nIn practice, the dataset is constructed using a learn-from-scratch approach. While using a normal distribution is a possible direction, it poses a challenge: during the first generation, the population must train for a long time to collect sufficient data to characterize the current reward distribution. This makes it difficult to directly model the initial reward functions using a normal distribution.\n        \nIf we leave the first-generation reward functions unchanged and only map subsequent rewards to a normal distribution, this would lead to a distribution shift problem.\n\nTherefore, we analyze the reward distribution of elite individuals from the previous generation, and re-map the outputs of newly generated reward functions to match this distribution.\n        \n**8. [Re: Hyperparameter Sensitivity of Parameter Constraints: Has the weight of the parameter constraint loss (Equation 4) been ablated to assess its impact on stability and performance?]**\n\nWe thank the reviewer for the helpful suggestion. LaRes does not tune this hyperparameter; it is set to the default value of 1.0 across all tasks. While further tuning may lead to improved performance, we find that 1.0 is generally sufficient to achieve strong results.\n\n|  | 10.0 | 1.0 | 0.1 | 0.01 |\n| --- | --- | --- | --- | --- |\n| pick-out-of-hole | 0.60 | 0.81 | 0.85 | 0.78 |\n| soccer | 0.64 | 0.71 | 0.81 | 0.74 |\n\n---\n\nWe hope our replies have addressed the concerns the reviewer posed and shown the improved quality of the paper. **We are always willing to answer any of the reviewer's concerns about our work and we are looking forward to more inspiring discussions.**"}]

Author rebuttal:

Meta-review: (a) Summary

This paper investigates how to improve reinforcement learning(RL) via evolutionary algorithms(EA) in the reward function space. It proposes to generate reward function population using LLM and EA for selecting the optimal reward to guide policy learning. It also introduces reward relabeling and parameter constraints to improve sample efficiency and stability. Experiments validate the effectiveness of the proposed method.

(b) Strengths
- Novel reward function optimization: The core idea of using an LLM as an evolutionary operator to search the reward function space is creative and could inspire follow-up work along this direction.
- Sample Efficiency Mechanisms: The shared replay buffer with reward relabeling and Thompson sampling-based interaction prioritization effectively reuses historical data and focuses on high-performing policies. This design reduces sample waste and enhances learning stability, as validated by ablation studies.
- Comprehensive Evaluation: The paper conducts rigorous experiments across two benchmarks (MetaWorld, ManiSkill3) and compares against diverse baselines, including RL, ERL, and LLM-based reward search methods.

(c) Weaknesses
- LLM-generated functions may lack diversity, which could limit the exploration potential of the evolutionary process.
- Dependence on Human-Designed Rewards: the method generally relies on human-designed reward functions as a base, which may limit applicability in settings where expert knowledge is unavailable.
- Computation cost: when the reward function population is updated, all experiences in the shared replay pool need to be relabeled.
 
(d) rebuttal

This is a borderline paper. The reviewers find the proposed approach novel and sample-efficient. The authors addressed the concerns on human-designed rewards, llm-generated functions, and computational overhead with additional experiments and clarification. Most reviewers remain positive after the rebuttal.

(e) Decision

Shifting the focus from policy parameter search to reward function search using LLMs is both novel and impactful; it could inspire follow-up work along this direction. The shared replay buffer reduces sample waste and enhances learning stability. I recommend acceptance.
Decision: Accept (poster)