Title: LaRes: Evolutionary Reinforcement Learning with LLM-based Adaptive Reward Search
Venue: NeurIPS 2025
Accepted: yes
Keywords: Evolutionary Reinforcement Learning
Abstract: The integration of evolutionary algorithms (EAs) with reinforcement learning (RL) has shown superior performance compared to standalone methods. However, previous research focuses on exploration in policy parameter space, while overlooking the reward function search.
To bridge this gap, we propose **LaRes**, a novel hybrid framework that achieves efficient policy learning through reward function search. LaRes leverages large language models (LLMs) to generate the reward function population, guiding RL in policy learning. The reward functions are evaluated by the policy performance and improved through LLMs.
To improve sample efficiency, LaRes employs a shared experience buffer that collects experiences from all policies, with each experience containing rewards from all reward functions.
Upon reward function updates, the rewards of experiences are relabeled, enabling efficient use of historical data. 
Furthermore, we introduce a Thompson sampling-based selection mechanism that enables more efficient elite interaction.
To prevent policy collapse when improving reward functions, we propose the reward scaling and parameter constraint mechanisms to efficiently coordinate reward search with policy learning.
Across both initialized and non-initialized settings, LaRes  consistently achieves state-of-the-art performance, outperforming strong baselines in both sample efficiency and final performance.
The code is available at https://github.com/yeshenpy/LaRes.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces LaRes, a hybrid framework combining evolutionary algorithms (EAs) and reinforcement learning (RL) with large language models (LLMs) to optimize reward functions for policy learning. LaRes uses LLMs to generate and evolve reward function populations, a shared replay buffer for sample efficiency, and mechanisms like Thompson sampling and parameter constraints to stabilize training. Experiments on 16 robotic manipulation tasks show LaRes outperforms strong baselines (e.g., SAC, ERL variants) in sample efficiency and final performance, though some tasks exhibit marginal gains over EvoRainbow.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- Novel Reward Function Optimization: By leveraging LLMs to iteratively evolve reward functions, LaRes addresses a critical gap in prior ERL work that focused on policy parameter search. The framework demonstrates tangible improvements over human-designed rewards, highlighting the value of automated reward engineering.\n- Sample Efficiency Mechanisms: The shared replay buffer with reward relabeling and Thompson sampling-based interaction prioritization effectively reuses historical data and focuses on high-performing policies. This design reduces sample waste and enhances learning stability, as validated by ablation studies.\n- Comprehensive Evaluation: The paper conducts rigorous experiments across two benchmarks (MetaWorld, ManiSkill3) and compares against diverse baselines, including RL, ERL, and LLM-based reward search methods. The results consistently show LaRes’s superiority.\n\n**Weaknesses**\n- Elite Definition and Terminology Ambiguity: The paper uses \"elite\" and \"non-elite\" to denote high-performing vs. low-performing reward functions/policies, but the criteria for defining \"elite\" (e.g., success rate, cumulative reward) are not explicitly standardized. This ambiguity could confuse readers, particularly when discussing policy inheritance and parameter constraints.\n- Dependence on Human-Designed Rewards: While LaRes starts from human-designed rewards and retains them in the population, the paper lacks experiments exploring fully LLM-generated rewards from scratch (without human input). Comparisons to Eureka under such conditions—where reward functions are initialized without human guidance—would strengthen the claim of LLM-driven optimization.\n- LLM Generation Dynamics: The process of LLM-generated reward function updates (e.g., how many new functions are created per iteration, which ones are replaced) is not clearly defined. For example, the paper states the population size is 5, but it is unclear whether LLMs generate all 5 functions or replace a subset.\n- Notation and Clarity in Formulation: The description of the Thompson sampling mechanism (e.g., \"i-th action\" referring to policy selection) is ambiguous, as it could be misinterpreted as action-space elements rather than policy indices. Clarifying this notation earlier in the text would improve readability.\n- Continual Learning and Plasticity Concerns: The paper mentions using the best policy’s parameters for initialization but does not address potential plasticity loss (e.g., catastrophic forgetting), a known issue in continual learning [1]. Compared to methods like RoSKA [2], which explicitly handle this, the framework could be enhanced in its theoretical robustness.\n\n[1] Maintaining plasticity in deep continual learning\n\n[2] Efficient Language-instructed Skill Acquisition via Reward-Policy Co-Evolution\n\nQUESTIONS:\n- Elite Criterion: How is the \"elite\" reward function/policy formally defined (e.g., success rate threshold, moving average performance)? Does the definition vary across tasks or remain consistent? \n- Ablation on Human-Designed Rewards: Have experiments been conducted where the human-designed reward is excluded from the population, and how does this affect performance? Comparison with Eureka and Roska on their benchmarks is suggested. This would validate the necessity of human guidance.\n- Normalization vs. Scaling: Why use reward scaling with elite statistics instead of normalizing all rewards to the Normal distribution? Does the current approach introduce unnecessary complexity? \n- Hyperparameter Sensitivity of Parameter Constraints: Has the weight of the parameter constraint loss (Equation 4) been ablated to assess its impact on stability and performance?\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-30"}

comments: SUMMARY:
This paper proposes LaRes, an RL framework for LLM, which focuses on the reward function search. By applying reward function updates and Thompson selection mechanism, the proposed method can improve the historical data usage and prevent the policy collapse.

STRENGTHS AND WEAKNESSES:
Strength:

The proposed method applies LLM to achieve reward functions searching, which is an innovative framework.

Weakness

1. The relation between high-level reward evolution and low-level policy learning is not clear. Furthermore, the relation between the framework and techniques like continuous learning, Thompson sampling, and reward scaling should be clarified.

2. The authors claimed that LaRes can cooperate with other RL methods, which is not demonstrated in the evaluation. Moreover, the current design requires multiple Reward functions to fine-tune the model, some experiments on computation cost should be involved.

3. The main comparison in evaluation adopts SAC, which was proposed in 2018; more comparisons with SOTA methods are recommended. Furthermore, the baselines in ERL-ralated are out-of-date except ERL-Re and EvoRainbo.

4. In Fig2,3,4, several results of LaRes exhibit larger variance than baseline; more discussion on it is required. 

5. No appendix, and the code is not open-sourced.

QUESTIONS:
1. Provide more experimental analysis on the computation cost.
2. Why LaRes exhibits larger variance compared to the baseline.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-01

comments: SUMMARY:
This paper introduces LaRes, a novel framework that integrates Large Language Models (LLMs) with Evolutionary Reinforcement Learning (ERL) to iteratively search for and optimize reward functions instead of policy parameters—a departure from traditional ERL methods. The core idea is to evolve reward functions using LLMs and evaluate them via reinforcement learning performance, thus enhancing both sample efficiency and final task success.

STRENGTHS AND WEAKNESSES:
Strengths:
- Shifting the focus from policy parameter search to reward function search using LLMs is both novel and impactful.
- The framework is presented clearly with diagrams (e.g., Figure 1) and step-by-step algorithmic breakdowns (Algorithm 1).

Weaknesses:
- The paper acknowledges that LLM-generated functions may lack diversity, which could limit the exploration potential of the evolutionary process.
- While the authors also explore scratch generation, the method generally relies on human-designed reward functions as a base, which may limit applicability in settings where expert knowledge is unavailable.

QUESTIONS:
- Could the authors provide any theoretical insights into convergence properties or bounds for LaRes? How can we trust the stability of LLM-driven reward evolution in the long run?
- How does LaRes scale with more complex or high-dimensional tasks beyond the 16 robotic benchmarks tested?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes LaRes, a framework that uses LLMs within an evolutionary algorithm to search for optimal reward functions for RL tasks. By evolving reward functions directly, it aims to automate a critical part of the reinforcement learning workflow. It introduces several techniques like reward relabeling and parameter constraints to improve sample efficiency and stability.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths:\nIt's always good to see novel approaches to the reward design problem in RL. I particularly appreciate the systematic framework design, which demonstrates sophisticated LLM integration through closed-loop optimization with quantitative feedback. The core idea of using an LLM as an evolutionary operator to search the reward function space is creative and could inspire follow-up work along this direction.\n## Weaknesses: \nThere are a few points that I would like to suggest here to make the paper even stronger.\n* Although paper shows that LLM-generated rewards improve performance significantly, hard-exploration tasks are often difficult because they require non-intuitive intermediate rewards. It would be insightful if the authors can discuss the relationships between the LLM's generative priors and performance on such tasks. Specifically, why might an LLM-guided search still fail on a hard-exploration task even if it can generate dense rewards? Does the LLM's common sense reasoning create a bias that prevents it from discovering the unconventional reward components needed to solve these problems?\n* When the reward function population is updated, all experiences in the shared replay pool need to be relabeled. How much computational overhead is this process for a replay pool of samples of different sizes? I hope authors can conduct supplementary experiments for analysis and examine the trade-offs related to performance.\n* LaRes contains a series of hyperparameters of the genetic algorithm, such as the population size of the reward function, the number of evolutions, the elite size, etc. However, it seems that the article does not discuss the sensitivity of the framework's performance to these hyperparameters. It is hoped that supplementary experiments can be conducted to illustrate this.\n* This paper relies exclusively on GPT-4o mini as the core LLM engine, which raises questions about the generalizability of the findings to other model families, particularly mainstream open-weight models like Llama or Qwen.  Without validation across different LLM architectures, the broader applicability of the LaRes framework remains uncertain. I would strongly recommend that the authors conduct additional experiments with alternative LLM to demonstrate the robustness and generality of their approach.\n\nQUESTIONS:\nIn addition to the points raised in the weakness section, I have three further academic questions for the authors:\n1. In the framework, the elite reward functions are not replaced, and the RL agent guided by the reward function designed by humans does not participate in Thompson sampling to ensure the stability of learning. However, does this design also bring the risk of prematurely converging to local optima? If these strategies are not globally optimal, will the mechanism continuously guide the search process, thereby hindering the exploration of new and potentially better reward paradigms? Have you considered other mechanisms to balance stability and the need for long-term exploration?\n2. The paper demonstrates that the final evolved reward functions are effective. However, what is the nature of this evolutionary trajectory? Do the reward functions converge towards a specific structure? Do they tend to become more complex over generations, or do they become simpler as the LLM distills the problem's essence?\n3. Appendix A mentions that the current method relies on the randomness of LLMS to ensure diversity, but does not explicitly consider individual differences. Have you considered adding explicit instructions in the Prompt to guide the LLM to generate more diverse reward functions? Will this further enhance the performance of LaRes?\n\nLIMITATIONS:\nSee the above weakness section.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}