Title: RF-Agent: Automated Reward Function Design via Language Agent Tree Search
Venue: NeurIPS 2025
Accepted: yes
Keywords: Reward Design

Reinforcement Learning

Large Language Models

Language Agent

Monte Carlo Tree Search
Abstract: Designing efficient reward functions for low-level control tasks is a challenging problem. Recent research aims to reduce reliance on expert experience by using Large Language Models (LLMs) with task information to generate dense reward functions. These methods typically rely on training results as feedback, iteratively generating new reward functions with greedy or evolutionary algorithms. However, they suffer from poor utilization of historical feedback and inefficient search, resulting in limited improvements in complex control tasks. To address this challenge, we propose RF-Agent, a framework that treats LLMs as language agents and frames reward function design as a sequential decision-making process, enhancing optimization through better contextual reasoning. RF-Agent integrates Monte Carlo Tree Search (MCTS) to manage the reward design and optimization process, leveraging the multi-stage contextual reasoning ability of LLM. This approach better utilizes historical information and improves search efficiency to identify promising reward functions. Outstanding experimental results in 17 diverse low-level control tasks demonstrate the effectiveness of our method.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes RF-Agent, an LLM-based framework for automated reward function design using  Monte Carlo Tree Search.\nIt treats reward function design as a sequence of mutations, crossovers, etc., and uses LLM to modify the Python code for the reward function, allowing for iterative refinement based on training feedback, task descriptions, and generated design ideas.\nExperiments on IsaacGym and Bi-DexHands benchmarks show improved performance over prior LLM-based baselines (Eureka, Revolve).\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n\n- Automated reward function design is an important problem.\n- As far as I know, framing reward design as MCTS over LLM decisions is novel.\n- The proposed method showed strong empirical results.\n- The paper carefully evaluates contributions of different components (search, actions, self-verification).\n\n# Weaknesses\n\n## No test of generalization\n\nAll tasks are from predefined benchmarks with standard objectives (e.g., Ant: run forward). It's highly possible that there already exists manually designed reward functions in the training set of LLMs: Models such as GPT-4o and GPT-4o-mini have been trained on public code repositories and benchmarks, which very likely include environments like IsaacGym and Bi-DexHands, as well as papers, GitHub repos, and documentation that include reward functions for these tasks. There's no evidence that the proposed method works on novel or compositional tasks (e.g., Ant: run in a circle then reverse). Without such tests, we can't claim the method understands task descriptions in a generative or generalizable sense.\n\nIn my opinion, what RF-Agent demonstrates is search efficiency: it can find or tune reward functions more effectively than greedy (Eureka) or evolutionary (Revolve) baselines, but possibly only within the space the LLM already has internalized. It may be an efficient memory probe, not a general reward function generator.\n\n## Limited understanding of feedback\n\nThe system feeds raw numerical logs $l_{feedback}$ into the LLM in string form. It assumes LLMs can reason over these logs to improve reward design, but this ability is unvalidated and possibly fragile. In my understanding, LLMs are not designed to do numerical trend analysis on raw sequences like a statistics model or a signal processor.\n\n## Computational cost\n\nLike other reward optimization methods, RF-Agent requires repeated LLM calls and full RL training loops. However, no analysis is provided on runtime, cost, or scalability, which is especially relevant given the use of GPT-4o.\nI may have missed some information, but it seems that this paper only stated\n\n> Overall, RF-Agent uses slightly more storage space and token consumption, but experimental results show that this slight sacrifice is worth it.\n\nbut gave no quantitative comparison.\n\nQUESTIONS:\n- l. 50--52: Is there evidence or references showing it's because of exploration and exploitation, not other reasons?\n- l. 112: $\\mathcal{R}$ is the set of $[S \\times A, \\mathbb{R}]$ of all real-valued functions over $S \\times A$, right?\n- l. 113: I understand $\\pi: S \\to \\Delta A$ is a measurable function to the measurable space $\\Delta A$ of all probability measures over $A$, but some readers may be unfamiliar with this notation.\n- l. 114: $\\Pi$ is not defined.\n- l. 114: $F: \\Pi \\to \\mathbb{R}$ is not limited to the (discounted) sum of rewards, right?\n- l. 115: Is the discounting factor specified?\n- l. 117--118: finding $R \\in \\mathcal{R}$, s.t. $\\mathscr{A}\\_M(R) = \\arg\\max\\_{\\pi \\in \\Pi}F(\\pi)$?\n- l. 120: $\\mathcal{L}$ is not defined.\n- l. 121: If we never need to train the language model, I think we can safely drop $\\theta$ or use a more informative name.\n- l. 124--125: How are $l_{feedback}$ and $l_{task}$ obtained?\n- l. 125--126: $\\max\\_{l \\in \\mathcal{L}} F(\\mathscr{A}\\_M(p_\\theta(l)))$ is weaker than $\\max\\_{R \\in \\mathcal{R}} F(\\mathscr{A}\\_M(R))$, which is weaker than finding $R \\in \\mathcal{R}$, s.t. $\\mathscr{A}\\_M(R) = \\arg\\max\\_{\\pi \\in \\Pi}F(\\pi)$. There's a gap at each step. More rigorously, I think we should define the sequential decision-making problem formally.\n- Eq. (1): I can guess $R \\sim p\\_\\theta(x, z), z \\sim p\\_\\theta(x)$ means using an LLM twice, but the notation is confusing.\n- l. 205 and l.218: What does $\\sim$ in $s_{node \\sim \\{elites\\}}$ and $s_{node \\sim \\{tree\\}}$ mean?\n- Appendix I: maybe the author should show diff instead of just pasting all steps.\n\nLIMITATIONS:\nThe author discussed the need for multiple interactions with the LLM and multiple RL training iterations.\nHowever, I believe that the lack of evidence regarding generalization ability in novel or compositional tasks is a more significant limitation.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes RF-Agent, a novel framework for automated reward function design in low-level control tasks. The core innovation lies in treating large language models (LLMs) as language agents and framing reward function design as a sequential decision-making process enhanced by Monte Carlo Tree Search (MCTS). This approach aims to address the limitations of existing methods, such as poor utilization of historical feedback and inefficient search, which hinder performance in complex control tasks. Experimental evaluations across 17 tasks demonstrate that RF-Agent outperforms state-of-the-art LLM-based methods, achieving higher success rates and training efficiency.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n### 1. Innovative Framework Combining LLMs and MCTS\n\nThe paper presents a compelling integration of LLMs and MCTS, treating reward function design as a sequential decision problem. This approach effectively leverages LLMs' contextual reasoning abilities and MCTS' search optimization to address the limitations of greedy or evolutionary algorithms used in previous work. The tree-based structure allows for systematic exploration and exploitation of reward function spaces, enhancing the utilization of historical information.\n\n### 2. Comprehensive Experimental Validation\n\nRF-Agent is evaluated across a diverse set of 17 low-level control tasks, covering locomotion and complex manipulation in IsaacGym and Bi-DexHands. The results show consistent superiority over baselines (Eureka, Revolve) in terms of success rates and training efficiency, even with lightweight LLM backbones like GPT-4o-mini. The quantitative evaluations, including average normalized scores and training curves, provide robust support for the method's effectiveness.\n\n### 3. Clear Visualization and Comparison\n\nThe paper includes informative figures (e.g., Fig. 1, Fig. 2) that effectively illustrate the sequential decision-making view, method comparisons, and RF-Agent's tree search process. These visualizations help readers understand the differences between RF-Agent and baselines, as well as the step-by-step optimization path of reward functions. The comparison tables (e.g., Table 1) clearly present performance metrics, making it easy to assess the method's advantages.\n\n### 4. Thoughtful Design of Action Types and Reasoning Mechanisms\n\nRF-Agent introduces a variety of action types (mutation, crossover, path reasoning, different thought) to guide LLM-generated reward functions, each serving a specific purpose in exploring or exploiting the search space. The self-verify and thought-align mechanisms help mitigate LLM hallucinations and ensure alignment between design thoughts and reward function implementations, enhancing the framework's reliability.\n\n## Weaknesses\n\n### 1. Incomplete Ablation Studies\n\nWhile the paper conducts ablation studies on the search method, action types, and reasoning components, the evaluation of some action types and their combinations is insufficient. For example, the specific impact of each action type (e.g., crossover vs. path reasoning) on different task types is not thoroughly analyzed. The ablation on \"reasoning\" only removes self-verify and thought alignment, but does not explore the individual contributions of each reasoning component.\n\n### 2. Lack of Comparison with Relevant Methods\n\nThe paper does not compare RF-Agent with methods like Roska [1], which uses continuous learning and reward function evolution within a policy improvement process, or Progress Function [2], which employs LLMs to model task completion progress as an intrinsic bonus. These comparisons would help situate RF-Agent within the broader landscape of reward function design and highlight its unique contributions.\n\n[1] Efficient Language-instructed Skill Acquisition via Reward-Policy Co-evolution\n\n[2] Automated Rewards via LLM-Generated Progress Functions\n\n### 3. Limited Analysis of Token Consumption and Compute Resources\n\nAlthough the paper mentions that RF-Agent uses slightly more storage and tokens than baselines, it does not provide detailed quantitative comparisons of token consumption or compute resources. Given that MCTS requires generating more reward functions (and thus more LLM calls) than greedy or evolutionary methods, understanding the trade-off between performance and computational cost is crucial for practical adoption.\n\nQUESTIONS:\n### 1. Token Consumption Comparison\n\nGiven that RF-Agent's MCTS process requires the LLM to generate more reward functions than baselines like Eureka and Revolve, have the authors compared the number of tokens generated by the LLM for each method? Understanding the token consumption is essential for evaluating the practicality and cost-effectiveness of RF-Agent, especially when using expensive LLM models.\n\n### 2. Comparison with Roska and Progress Function\n\nWould the authors consider comparing RF-Agent with methods like Roska and Progress Function? Roska's continuous learning and reward evolution during policy improvement, as well as Progress Function's use of LLMs to model task progress, present alternative approaches to reward design. Such comparisons could shed light on the strengths and weaknesses of RF-Agent in different scenarios.\n\n### 3. Component Ablation Details\n\nThe proposed method is comprised of many intricate designs for different purposes; however, they are not comprehensively analyzed. Could the authors elaborate on how each component, ranging from the score design to the action types, influences performance in different task domains? \n\n### 4. Thought-Align Mechanism Effectiveness\n\nThe thought-align mechanism aims to resolve inconsistencies between LLM-generated design thoughts and reward function code. Could the authors provide quantitative results on the effectiveness of this mechanism? For example, what percentage of reward functions show improved alignment after thought-align, and how does this impact training performance?\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents RF-agent a method to generate (dense) reward functions using LLMs via sequential-decision making process based on  MCTS. While the contemporary approaches focus on LLM-based methods that rely on evolution-based methods or greedy-search based methods, this work proposes a tree based method that utilizes as MCTS and UCT to handle the exploration-exploitation process efficiently and generate better performing reward functions compared to the existing methods. \n\nThe LLM acts as a reward function generator that generates the reward function in python. There is another evaluation metric $F(\\cdot)$ that captures how well a policy $\\pi$ performs on a task, along with some feedback $l_{feedback}$ that indicates the feedback from executing the generated reward function on the task. The problem is to integrate this feedback in a sequential manner, so as to find better performing reward functions using the available actions (mutation, crossover, reasoning traces, etc.). This results in better ability to design dense reward functions from the sparse evaluation metric - compared to human designers or other baselines - in the low-level control simulation-based gym environments.\n\nSTRENGTHS AND WEAKNESSES:\n# Strength\n- The paper is well written and easy to follow.\n-  Strong empirical results over the baselines that shows the promise of approaching this problem from the sequential decision-making lens.\n- Rigorous experimental procedure and evaluation . \n\n# Weakness\n- Some of the motivation for the problem setting is not clear. For instance, why can't the evaluation function/metric and the feedback be directly used for optimizing the policy here? The goal is to find the best performing policy so it's not clear why we need an additional step to map $F \\rightarrow R$, and then utilize the feedback for $R$ ? From the experiments, it seems the idea is to transform $F$ (spare reward) to $R$ (dense reward). It seems like both action space design and the feedback function design are the key component for the success of this method. The $l_\\{feedback}$ is the crucial component that makes the search process possible - the space of possible LLM generated reward function is infinite and only this signal guides the algorithm in the right direction. What are the assumptions here? How to design such feedback function is an important aspect which is overlooked in the current draft. \n\n- Another concern I have is that the methods/environments might already be in the training-set of the closed source LLMs like GPT, so there is a risk of data contamination. When you take this into account with the feedback prompt, maybe that can be a factor in the success of this method (as well as the baselines). How does this perform on OOD tasks or novel reward functions? \n\n- There is ablation for $A$, that demonstrates these actions are justified for the domains in this work, but the paper needs a more thorough discussion on how to design this to make the work more generalizable.\n\nQUESTIONS:\nAs I mentioned in the weakness section, I'm mostly concerned about two things regarding the $l_{feedback}$: (1)  How well this method scales on OOD tasks and reward functions; and (2) any ablation or insights into it's design. I believe the work can be improved by addressing these concerns, and would convince me to change my score.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores a novel method of using Large Language Models (LLMs) to generate reward functions in reinforcement learning (RL). Considering that current methods underutilize historical feedback and the optimization capabilities of LLMs, this paper proposes a systematic framework that uses LLM to generate and update reward functions with the help of Monte Carlo Tree Search (MCTS) . The paper employs MCTS to make decisions and update the reward functions, and emphasizes the role of action expansion in reshaping the reward functions. Experimental results on low-level control tasks demonstrate that the proposed approach can generate high-performance reward functions, thereby facilitating policy learning.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n-The paper is well-written, clear, and easy to understand.\n-The motivation for introducing MCTS to optimize reward functions is natural, so that the training effects corresponding to various reward functions can be quickly listed and backtracked, with optimization at the nodes. The paper demonstrates that under this design, it is possible to arouse the potential of LLM for efficiently designing reward functions.\n-The design of action expansion stimulates the LLM to improve existing reward functions from multiple perspectives, making better use of historical information and potential directions for optimizing reward functions.\n-Experimental results on low-level control tasks provide adequate evidence for the effectiveness of the proposed reward optimization framework and demonstrate its generalizability.\n\nWeaknesses:\n\n-LLM's design thoughts for reward generation lack a finer-grained structure. For example, when generating reward functions, LLMs could be guided to decompose the design thought into more granular components at the reward function level. This might include explicitly defining the set of usable variables, identifying which subsets contribute to specific components of the reward, and describing how these components combine into the overall reward. In other words, standardizing the design thought process could be an effective way to enhance LLM's search strategies. By doing so, during action expansion, the LLM could quickly backtrack to identify which variables and reward components were used or effective, enabling faster iteration and improving the explainability of the reward optimization process. A possible implementation is discussed in [1]. \n-Lack of analysis on MCTS iterations and computational cost. Specifically, the paper does not analyze the number of iterations or the time required for MCTS. It remains unclear how many rounds of RL training and reward updates are needed for a given task to achieve stable and successful execution. This raises concerns about the overall training cost and whether it falls within an acceptable range compared to alternative methods.\n[1] Heng Z K, Zhao Z, Wu T, et al. Boosting Universal LLM Reward Design through Heuristic Reward Observation Space Evolution[J]. arXiv preprint arXiv:2504.07596, 2025.\n\nQUESTIONS:\n-Building on the above weaknesses 2, while the motivation for combining MCTS with reward optimization is reasonable, it is unclear how many iterations of reward function updates are required for the tasks in the experiments to achieve stable success. Overall, for training a task from start to successful execution, what is the RL training cost? How does the total training time compare to other methods? Is it within a reasonable and acceptable range? \n-In Figure 2, does the process from s_1^0 to its leaf nodes s_2^0 and s_2^1 represent action expansion? If not, how does it differ from action expansion? \n-Is there a possibility to provide a detailed explanation of the selection phase? When a node is selected, will other nodes at the same level also be visited later during the search? My understanding is that during the backpropagation phase, all nodes on this path are updated, after which other branches are explored until the reward function corresponding to a certain node successfully completes the task. Is there a limit to the action expansion depth along a single path? Has the author experimented with different depth limits? \n-Are there ablation studies on whether to use design thought? Regarding Weakness 1 mentioned above, is there a test that compares the impact of using different design thoughts on reward generation and optimization? Alternatively,  are there experiments with using Chain of Thought (CoT) reasoning to guide LLMs in generating reward functions?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}