RF-Agent: Automated Reward Function Design via Language Agent Tree Search
Authors: Ning Gao
Xiuhui Zhang
Xingyu Jiang
Mukang You
Mohan Zhang
Yue Deng
Venue: NeurIPS 2025 spotlight ()
Keywords: Reward Design
Reinforcement Learning
Large Language Models
Language Agent
Monte Carlo Tree Search
Primary area: reinforcement_learning
TL;DR: We propose RF-Agent, an automated RL reward function design framework via language agent tree search.

Abstract: Designing efficient reward functions for low-level control tasks is a challenging problem. Recent research aims to reduce reliance on expert experience by using Large Language Models (LLMs) with task information to generate dense reward functions. These methods typically rely on training results as feedback, iteratively generating new reward functions with greedy or evolutionary algorithms. However, they suffer from poor utilization of historical feedback and inefficient search, resulting in limited improvements in complex control tasks. To address this challenge, we propose RF-Agent, a framework that treats LLMs as language agents and frames reward function design as a sequential decision-making process, enhancing optimization through better contextual reasoning. RF-Agent integrates Monte Carlo Tree Search (MCTS) to manage the reward design and optimization process, leveraging the multi-stage contextual reasoning ability of LLM. This approach better utilizes historical information and improves search efficiency to identify promising reward functions. Outstanding experimental results in 17 diverse low-level control tasks demonstrate the effectiveness of our method.

RF-Agent: Automated Reward Function Design via Language Agent Tree Search
Ning Gao, Xiuhui Zhang, Xingyu Jiang, Mukang You, Mohan Zhang, Yue Deng"
Beihang University
37 Xueyuan Road, Haidian District, Beijing
(gaoning_ai, zhangxiuhui, jxy332rhd) edu. cn youmukang@gmail. com, {zmh666, ydeng)@buaa, @buaa edu. cn
Abstract
Designing efficient reward functions for low-level control tasks is a challenging problem. Recent research aims to reduce reliance expert experience by using Large Language Models (LLMs) with task information to generate dense reward functions. These methods typically rely on training results as feedback, itera- tively generating new reward functions with greedy or evolutionary algorithms. However, they suffer from poor utilization of historical feedback and inefficient search, resulting in limited improvements in complex control tasks. To address this challenge, propose RF-Agent, framework that treats L.MMs as language agents and frames reward function design as a sequential decision-making process, enhancing optimization through better contextual reasoning. RF-Agent integrates Monte Carlo Tree Search (MCTS) to manage the reward design and optimization process, leveraging the multi-stage contextual reasoning ability of LLMs. This approach better utilizes historical information and improves search efficiency to identify promising reward functions. Outstanding experimental results in 17 diverse low-level control tasks demonstrate the effectiveness of our method. The source code is available at htttts//giihuu..cmm/degg libbab/RF-AAeen
Introduction
Reward design is crucial component in Reinforcement Learning (RL), significantly affecting the performance and training efficiency of the policy, especially low-level control tasks like locomotion 40 and complex munipulationn3]]115 Although task evaluation metrics (e.g., success rate, movement speed) can be directly used as rewards, their sparse one-dimensional nature presents challenges for policy optimization. Therefore, reward shaping functions crucial for guiding policy learning more efficiently [46). The most common approach involves human experts manually crafting dense reward function, which is interpretable but requires extensive expertise and may be suboptimal[4] 4 To address this, Inverse RL|65] 54 [3] and Preference- based RL[10 10, 22 35 attempt to generate dense rewards by learning reward models from expert demonstrations or preferences. However, they still rely on large amounts expert data and lack interpretability.
(34) through dense reward
Recently, large language models (LLMs) have excelled in not only traditional NLP tasks 33 but also logical reasoning and agent-based tasks 50 58). Given their impressive world knowledge [2 and coding capabilities [56] 60), a natural thought is to leverage LLMs to generate interpretable dense reward functions, replacing expert effort. Approaches like L2R 59 and Text2Reward 55
"Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(+) Sequential Decision making View
(c) Max Score improvement comparison
State
LLM
Action
State
Agent
Spore Pe+1
Reward Function
Policy Re
Environment
Feodback
(b) Method comparison
Belathe Law
S
S
Performance
Greedy-based egg.
Evolution-based egg.
Search-based (Ours RF-Agent)
Figure 1: Quick view. (a) Reshaping ALM-based Reward Function Design Problem from sequential
decision-making view. (b) Abstract pipeline comparison between different approaches. (c) Compari- son between different methods on max success rates with the numbers of complete training times.
made strides in this direction, achieving remarkable results. In addition, more recent methods 28 have further advanced the field by combining more refined feedback design and evolutionary concepts. We view these methods as part of a sequential decision process, as shown in Fi The interaction between the policy and environment forms an internal system, where the LLM, as a language agent, continuously interacts with this system. The action of the LLM represents the process of generating a reward function, and the internal system provides results after training as feedback based on the action. The action and feedback in the current decision-making process are finally integrated into the history state and utilized by LLM for future actions.
17 61 62 [8]
From the sequential generation modes, shown in J. The first mode, exemplified by Eurcka(28), follows greedy algorithm (88 approach, where the best reward function is retained after a batch generation of reward functions and serves the starting point for the next iteration. The second mode, represented by [17) adopts an evolutionary algorithm 11] approach, maintaining a population and evolving new reward functions by integrating individual information from the population. However, when utilizing LLMs to design reward functions for complex control tasks (91, the performance improvement of these methods remains limited, illustrated in Te. This limitation hinders the generation of high-quality and training-efficien reward functions through LLMs, constraining the performance of this pipeline. We identify two key factors leading limited performance improvement. First, from the search perspective, greedy or evolutionary algorithms struggle in complex decision- making problems, as they fail to effectively balance exploitation and exploration, often resulting in premature convergence to local optima or excessive exploration of possibilities (37). Second, regarding information usage, existing methods only retain local historical information, overlooking the potential decision paths that could transition from low-performing to high- high-performing reward functions b ours).
perspective, current research can be categorized into two reward
Based on the analysis above, we introduce a novel reward function design framework, RF-Agent, which treats the LLM as an agent address the challenges outlined by enhancing their reasoning and decision-making capabilities. Specifically, we employ a tree structure to represent the entire reward design process [FigT bb ours) and its history states. Each node in this tree corresponds to distinct decision strategy and its associated reward function, effectively leveraging the multi-stage contextual reasoning capabilities of LLMs. To further improve the framework, we incorporate the Monte Carlo Tree Search (MCTS) algorithm to balance exploration and exploitation within the decision space. Additionally, during the action expansion phase, we integrate set of heuristic action types to guide the LLM in generating reward functions with varying inclinations. This enables RF-Agent to appropriately incorporate global decision information from other paths, ensuring the full utilization and refinement of promising reward functions. Finally, environmental feedback and self-verify metrics contribute to an overall evaluation, which forms value prior for node selection, thereby enhancing the search optimization capabilities of the entire RF-Agent framework.

---

We evaluate the proposed RF-Agent across 17 tasks in IsaacGym [1] and Bi- Bi-DexHands(9), covering task types such as locomotion and robot arm/hand manipulation. The results indicate that our approach outperforms the current nate-of-the-art L.LM reward function design methods, producing high- performance and training-efficient reward functions. Such achievement demonstrates the potential of leveraging language agents in combination with reasoning and search frameworks for reward function design. Ablation studies further validate the effectiveness of our architecture.
Related Works
Reward Design with LLMs. Reward engineering has long been challenge in reinforcement
learning 226.. with the common approach being manual creation of reward functions by experts [23]
4. Inverse R I9 65
54 [ and Preference-based [L 22 35 generate rewards through
learned models, but still require expert demonstrations and lack interpretability. Recent advancements
leverage LLMs, which exhibit strong world knowledge and logical reasoning [58] 14), in tasks like
code generation 8 39] and robotic control planning 27 44 49]. Some works have extended this
to generating rewards for RL. For instance, 24 uses LLMs to generate binary rewards based on user intent in negotiation games, and [12] calculates rewards based on the similarity between state transitions and goals generated by L.MMs. However, these methods suffer from excessive L.M calls and inconsistencies. In contrast, L.2R[59) and T2R(55] generate dense, interpretable reward functions, while T2R directly offers plug-and-pla reward function Python code, introducing a promising paradigm for reward engineering. Recent works like Eureka use greedy iterative approaches to optimize reward functions, while Revolve[[7] incorporates evolutionary algorithms to evolve new rewards. However, both methods still face challenges with inefficient use of historical feedback and low search efficiency. Our RF-Agent addresses these issues by introducing Monte Carlo Tree Search to manage reward function design and optimization process from a decision-making making perspective. It fully utilizes LLMs' contextual reasoning and historical feedback, thereby significantly enhancing the performance of the designed reward function.
LLM-based Agent for Decision Making. The world knowledge and logical reasoning abilities of LLMs enable them to serve as agents for solving decision- decision-making problems 49 6). For
example, LLMs have been used as high-level controllers in robotic planning 22, interacted with web browsers to fulfill user needs 33] and tackled multi-step decisions in text-based adventure games 43 As tasks become more complex, methods such as Twosome (47] attempt to fine-tune LLM--based agents to interact directly with the environment, but this often incurs high training costs. In contrast, training-fte free approaches that utilize multi-step reasoning to leverage LLMs' contextual learning have gained popularity [53] 5I 58 63 This "Test-time Compute" paradigm (25 enhances reasoning to influence decision performance, with Chain-of- Thhought (CoT) [3] being the most prominent method. ReAct(58) incorporates CoT into decision-
while self-refine 29] and Reflexion 42 add self- feedback mechanisms. However, these linear methods neglect alternative decision paths. ToT[57] breaks the decision process down using tree search, and methods like LATS LATS(64]] and RAP[16) further introduce Monte Carlo Tree Search (MCTS) to enhance reasoning during decision-making, Inspired by these approaches, we frame reward function design decision-making process, enhancing the logical reasoning of LLM to better analyze historical feedback and generate high-quality reward functions.
3 Problem Setting
In standard reinforcement learning (RL) 46 tasks, the problem is commonly modeled as a Markov Decision Process (MDP), formalized by the tuple M = (S, A, A, R,y, In this formulation, S denotes the state space and A represents the action space. Y is the discount factor, and po is the initial state distribution. The environment's dynamics are described by the transition function T : x A S + [0, 1]. which specifies the probability of transitioning
new state given the current state and action. The reward function R. S A R defines the immediate reward received after taking action in given state.
We then introduce the Reward Design Problem (RDP) 45), given by the tuple p= (M, R, ग. F}. where M (S, A. T) is the environment world model including state space S, action space A, and transition function T. indicates the space of all reward functions where each E R is reward function that maps state-action pair to scalar R. S A A. n: S A indicates
3

---

Abstrac Env Observation
Task Info
Instruction
Optimiration
RFF KF-Agent
BeseeelaeiSiingaappll dil ceespen, aburttiinntuti object, poae
Tha iaveher daal hunde cap. bood and reing the cap oepther
Bos, dowtthe the deigg sloe fanetio...
Parent Node
max
Child Node
i) Selection
Trace
Value Score
Vadit Dalance
Self-Verify
sil caleupp wig.t haw.le_adeeeeeoe pox.
Score
Sg
Reward Func Ri Score 0.0014
Anlgn thnghtz
Sn
Leaf Node
Specific action
Child Node
Expansion
dg
Inittalization
dileact to grile sermiddien gev gro, chawe))
add ressoee sone bask soaard companents
soved recurd
soaal semens gpo srossst grd....
Mutation
Repeat N timus or till early
Foodback
Key lever 2001, 0.0014)
cpecit
Modiy the ghos tunciion free paranetells
eg. slensty pararstern and gov settego.
CrowOver
conaiiz perfermzg campunera
Reward Punc R Score
foon Ailermr reword foxctiees haod fedbick
Coderleeeee hand perstealy remords
baids dinacs
Path Ressoning
Comaee new fracciio by susenuiinng la an Pg optireluttio juth Iupietiins
wock ceatteyy covent tanllwryy (kel,
Local Paith
Global Path Messue
O
Different Thonght
Chore foeetie diftrure from giren aade
Score
G
Kepp hostrical resurd fooe aoda (acall
Imag pruteiity Change eorraes
IlI) Simulation
Calc- die publiing sceeal loadd on angale veledty
Sopal:
Brnort
Policy
Feedlback Score
Reward R Score 0,9890
Brance
O Node O Node in simulation
Child Node
Update
Parent Node
Iv) Bockpeopagitien
odd Bloos
inaiiity baadd licoe rolldty
Reward Punction [ Hisstory State A Tree 30; Policy ua Language Model
Action) dis. Set
Node Valuo dmm
Child Score
Node Visit dim
Child Viait
Evaluate
Self Vectfy
(a), Stngle Path Optimisation Trace
(b) Tree Growth Process
(c) Mote Cado Tree Search Cycle
Figure Illustration of our RF-Agent with (a) an exemplary reward function optimization path from the tree, (b) total tree growth process, and (c) iteration based on MCTS. Please refer to the Appendix for prompts used in RF-Agent, mainly including initialization, expansion, and other process.
policy sampled form the policy space II and F() : + R is the evaluation metric that measures the real performance of on given task. Given reward function R. the tuple (M, R) forms Markov Decision Process M. We additionally denote (.) : + IT as learning algorithm (e.g., PPO[4]] that outputs the policy under the given MDP (M, R). The goal of the RDP is to output a reward function It E R such that the policy π == AMM (t) under the MDP (M, R) achieves the highest evaluation score F(T).
In our Reward Function Design Problem (RFDP) setting, each R is specified as Python code and given by reward designer G L + R according to the input string L as instruction. Here we choose a pre-trained language model po as the reward function designer G. After the policy æ completes the training of the fix number of steps under the given reward function R and learning algorithm AMs we will receive feedback from the environment about this learning process. In addition to the evaluation score F, string feeaback as language feedback will indicate the execution situation. Finally, given a string bask that specified the task, the objective of the RFDP is to output a reward function code Rt ~ (l(() such that (( (R)) is maximized.
4 Method
In this section, we first explore the challenges associated with solving the LLM--based RFDP problem. with particular emphasis on the in-context learning capabilities of LLMs. We then introduce RF- Agent, a framework that enhances the multi-stage contextual reasoning LLMS by integrating Monte Carlo Tree Search (MCTS), thereby improving the efficiency of reward function search through better utilization of historical feedback and evaluation metrics. Finally, we show an exemplary reward function optimization process to illustrate the effectiveness of RF-Agent.

---

4.1 Rethinking LLM-Based Reward Function Design Problem
From a decision decision-making perspective, each reward function R generated by an LLM pe can be viewed as action a, with the corresponding input instruction I as the state This presents key challenge in RFDP: the state and action space defined on language is vast, while the evaluation metrics F and feedback I feedback provided by the environment after training are sparse and text-based. As result, using feedback to directly optimize pe(-) via gradients is impractical and costly.
The in-context learning ability of LLMs offers promising alternative to solve the above problem. In many text-based tasks, LLMs act as language agents, leveraging varying state information as context to iteratively make decisions [58, 64] Based on the analysis, identify that previous LLM- M-based RFDP methods simply treat LLM contextual learning as a basic usage, failing to integrate it effectively within the RFDP framework. These methods suffer from limited historical information utilization and low search efficiency to generate high-performance reward functions. To address these issues, we propose the RF-Atent framework, which fully exploits multi-stage contextual reasoning of LLMM to better utilize evaluation metrics and textual feedback in RFDP and improve the efficiency of reward function search.
4.2 RF-Agent for Reward Function Design
Overview. Our RF-Agent is illustrated in Zig We structure the optimization to solve the LLM-based RFDP Monte Carlo Tree Search process, with the root node Throot serving as a virtual node. The remaining nodes each represent a reward function R with its training feedback. Based on classical MCTS, first generate an initial set of child nodes, after which the RFDP solution process consists of four stages: selection, expansion, simulation, and backpropagation. These stages iterate until the maximum iteration limit N reached or early convergence. In the selection phase, we select the most promising node based on an improved Upper Confidence Bound for Trees (UCT), which incorporates evaluation metries, self-verify metrics, and exploration count. During the expansion phase, RF-Agent utilizes specific actions to guide the LLM in generating reward functions for child nodes, ensuring effective exploration of the reward function space. Finally, in the simulation and backpropagation phase, RF-Agent trains policies with generated reward functions and receives feedback to update the node states for next selection. See Appendix for the algorithm of Rg-ntttt.
Initialization. During the initialization phase, in addition to the brief task information lask, RF- Agent is only granted access the observational part of the environment code, Lober consistent with previous approaches[2] [28] 17 We then provide both inputs to the LLM, leveraging its world knowledge and zero-shot instruction-following capabilities to generate the reward function:
R ~ pa (z (2) 2), ><pe(z)
pe(z), where r [kacks Lober laction]
Here, laction refers to the instructions, and z represents the design thought in the reasoning process. As shown in Fig a, we prompt the LLM to first generate the design thought and then create the specific reward function based on it, leveraging the advantages of logical reasoning (53).
The generated reward functions are then integrated with the environment to train the policy under the learning algorithm AM. The process returns evaluation metric F (e.g., task success rate) and feedback feedback. which tracks the changes of key components in R during training as shown in App. B.2 Then, the construction from the root node nyyott to the first layer of child nodes is complete, with each node state represented as = [z, F,F,F,,,ebbarrr F.oobbaaack]. The tree structure stores all of the node states throughout the development, enabling their full utilization in the selection and expansion stages.
Selection to the Potential Node. In the selection stage, RF-Agent balances exploration and exploita- tion to identify promising nodes as shown in Fig2 c.i and blue arrow in b. improving search efficiency for promising reward functions. We complete this stage by sequentially selecting child nodes with the highest UCT values until a leaf node. In the RFDP setting, each node maintains value Q(s) reflecting the score of its reward function and those of its descendants, along with a visit count N (s)..
Additionally, RF-Agent introduces a self-verify score to improve selection. In complex control tasks, reward functions often have low scores F, even nearly zero, resulting the value Q constructed based on F not being able to effectively reflect the potential of the node. Inspired by the inherent evaluation capabilities of L.LMs 42 we prompt the LLM to first analyze how an expert completes the task, then output score representing the likelihood of obtaining expert-level policy
5

---

by training under the current R as shown in App.
B.4 The final UCT calculation is as follows:
(CT(se ((echale) Qmán
2 log (Sparenn)) + 1) ((shhad)
+ +..(1 +
(Vsett (Schaa)))
Qmax
Q Qmnn
Here, we normalize the Q-value to enhance the robustness of the UCT calculation in different tasks. The parameter À controls the weight of the exploration term and linearly decays during the search to promote early exploration while ensuring later convergence. Uself represents the self-verify score scaling with softmax a and also decays with X.
Expansion with LLM-based actions. In the expansion stage, RF-Agent incorporates historical information guide the LLM in generating new reward functions similar Eq but with different laction- Previous methods (28) typically regenerate reward functions based on a single prompt and limited historical data, thus cannot fully leverage the LLM in-context learning abilities and limit effective exploration of the reward function space. When humans tackle problems, complete decision-makinn process usually involves various types of action, with different actions employed under different states. Inspired by this, RF-Agent enhances exploration of the reward function space by defining different action types generate the reward functions with utilization of historical information from the whole tree, as shown in Figge.) ii and orange arrow in b. See App. B.3 and [.] for detailed prompts and effects. The action types including [4m11 0m2; 4cs, 0r4; 0.45 are as follows:
o Mutation am1 & am2 Laction = Imstation pprenn)) We introduce the mutation to represent local modifications the existing reward function. Given parent node, the mutation action guides the LLM to analyze the state of the parent node to optimize its reward function. Specifically, am guides the LLM to modify the structure of the reward function, such adding or removing a component, while am2 focuses on adjusting the parameter weights within the reward function.
Crossover 4.3 laction Lerossover (5 porent, (cliees ). To enhance the utilization of global historical information and accelcrate the search process, we introduce a crossover operation to extract information from tigh-peeformance nodes. RF-Agent maintains high-performance nodes from the tree based on evaluation scores F. We sample & nodes from the elite set, weighted by their scores, and input them along with the parent node state to the LLM. Then RF-Agent guides the LLM to analyze the feedback and extract useful reward components from these nodes, combining them form new reward function.
elite set, dynamically storing
Path Reasoning 0,1 laction = Lreason (8 parentt Soncestor). ). Each path from the root node to a leaf node represents unique reward function optimization trace. Similar to how humans recall past experiences when solving new problems, RF-Agent leverages the knowledge in the reward function optimization trace by reasoning. By truncating k-length tree path from the current node to the root, RF-Agent guides the LLM to reason the reward function optimization process, identifying the design strengths and further generating a new reward function.
Different Thought aas laction = laif (Sparent, (toder To prevent premature conver- gence during the search and enhance exploration of the reward function space, we introduce 0,5 to generate thoughts distinct from existing reward functions in the tree. RF-Agent randomly selects nodes from different paths, combines them with the parent node, and guide the LLM to generate reward function structurally different from those in the selected nodes.
Simulation with Reward Function. In the simulation stage, we train the policy using the reward function R of expanded nodes, as shown in Fig. c.iii. Generally, can be executed at once, then complete node s = [z. R, F, I feedback] will be established. In complex environments, however, early -eneereedd reward functions may encounter execution errors. In such cases, we extract the traceback error and prompt the LLM to adjust R. This adjustment may cause the reward function f to diverge from the original design thought z, and even without adjustments, LLM hallucinations can misalign R and x 20) Since expansion actions are partly based on 2, its accuracy is crucial. To address this, RF-Agent introduces simple yet effective thought alignment process: during expansion, we generate 2 before R: in simulation, once the reward function compiles correctly, we regenerate a more complete design idea Enew based on R or R. See Appendix Bnd 1.2 for details.
Backup with Evaluation. The purpose of the backpropagation stage is to update the value ((s) and visit count (s)), with generating a self-verify score Uself as shown in Fig and green arrow in b. The process and motivation for generating the self-verify score are detailed in the expansion part,

---

while the update process for (ss) and (s) with update rate is as follows:
(s)) = (1- (s)) + +
max o CChildd(ee
Q(s'): ((s)) = E N(s')
(3)
#' ('Child(s)
4.3 Optimization of Reward Function in RF-Agent
As shown in 2ig2 a, we illustrate an exemplary reward function optimization path for Swing Cup task. During early exploration, RF-Agent tends to expand extensively try different reward functions such as using mutation actions to add the handle distance reward in R3-- Later, RF-Agent focuses on exploiting a few nigh-performanc paths, such as continuing with R3 by crossover of elite nodes to generate R4. which modifies parameters and incorporates angular velocity stabilization rewards. Further optimization through path reasoning led to R which adds a linear velocity stabilization reward and achieves high performance. See Appendix F.3 for more optimization examples.
Experiments
5.1 Experiment Environments
We test RF-Agent in two low-level control environments: IsaacGym 31 and Bi-DexHands (), encompassing 8 control agents and 17 diverse tasks. We first evaluates on 7 representative tasks across locomotion and robot arm/hand manipulation from IsaacGym. To further more complex tasks, we selected 10 tasks from the BinDexHands, involving dual-arm manipulation with more intricate actions such as door closing, cup rotation, and bottle cap twisting. We divided these 10 tasks into two groups, expert-easy and expert-hard. based on human reward function success rates, enabling comprehensive comparison of different LLM-based RFDP methods and human performance, Task details are provided in the Appendix
our method on
5.2 Baselines
Human. These use dense reward functions from the benchmark, written by reinforcement learning researchers who designed the tasks, representing expert-level reward engineering techniques.
Sparse. These are equivalent to using the evaluation score F, which directly reflects task completion quality, as the reward. This reward is typically sparse or amnle-dimensional
Eureka 28 automatically generates dense reward functions by combining task information and environmental observations. More importantly, it generates rewards in batches and retains the best one based on feedback, optimizing the reward function through greedy iterative approach.
Revolve [7] maintains a population and applies evolutionary operations to modify the reward functions with human-in-the-logg feedback. In our fully automated reward function design scenario, we implemented the Revolve auto version, where feedback is also provided by the environment.
5.3 Training Setup
Policy Training. Following Eureka, all tasks implemented with well-tuned PPO[0] and default hyperparameters provided by the benchmark. We tested the final reward functions through individual training with 5 different seeds and measured the quality of the reward functions by reporting the average maximum evaluation score achieved at each policy checkpoint.
Evaluation metrics. In the Bidex tasks, the success of task is determined by binary 0-1 signal, thus the evaluation score directly based on the success rate. In the Isaac tasks, the metric depends on its goal. For example, the goal of the Ant task is to move as quickly as possible, so the metric is set as the forward speed. See appendix for detailed task descriptions and evaluation scores.
Method Implementation. Since Eureka, Revolve, and RF-Agent optimize reward functions through sampling, we ensure fair comparison by setting the same total sampling number of reward functions. In IsaacGym, we set the total limit to 80 and use the GPT--4-mini-077 18 and GPT-40-0806 models for L.L implementation 2T). For the Bidex tasks, to evaluate the search performance of the method itself under complex control tasks, we increase the upper limit to 512 and use only the GPT-4o-mini
7

---

Table 1: Quantitative evaluations on IsaacGym. Bold indicates the best results within the same group. Avg norm score represents the average performance after calculating human normalized scores (Method - Sparse) /(H uman - Sparse) for each task.
locomotion
manipulation
task
Avg norm score
Ant
Anymal
Humanoid
Quadcopter
AllegroHand FrankaCabinet ShadowHand
Sparse
0.14
2.05=a.24
3.01
1.35=0.04
0.03+4.00
0.04
0.04
Human
6.75+0.30
0.03+0.00
5.89+n.51
-0.08-0.08
1141+112
0.11+0.07
8.56+1.23
LLM-based Reward Function Design with GPT-4o-mini
Eureka
4.87+1.34
0.67=0.04
3.29
-0.07=0.c2
13.87=3.04
0.01+0.00
10.81+1.06
0.63
Revolve
5.1611.37
0.8020.09
3.1210.33
-0.08
18.4020.07
0.19
10.6141.38
Ours
9.22+0.90
0.01+0.00
5.91+1.21
0.03+0.00
1.88
0.35
13.13
1.70
LLM-based Reward Function Design with GPT-4o
Bureka
3.77.0.85
0.01=0.00
5.44
0.03=a.02
22 02.
0.52#0.21
12.2740.91
2.00
Revolve
5.90+0.82
0.01
4.87
-0.04=0.22
23.11#1.10
0.57+0.50
9.02+1100
Ours
1.1020.11
0.01
6.3710.32
-0.0310.01
24.04:1.41
1.
14.0940.40
2.68
model. For Eureka and Revolve deployment details, please refer the Appendix configuration of our RF-Agent, please refer to the Appendix
D As for the
5.4 Experiment Results
Advantages of RF-Agent in multi-class control tasks. Tab reports the absolute scores with standard deviation and average normalized performance of different methods each IsaacGym task. We observe the following advantages:
(1) Our method outperforms other LLM-based reward function design methods across various tasks and control types, demonstrating RF-Agent's ability to leverage LLMs for reward function generation, with this advantage unaffected by changes in the LLM backbone model.
(2) Even with lightweight LLM backbone model 4o-mini, RF-Agentt-ennerated reward functions maintain superiority over human expert methods in nearly all tasks, while Eureka and Revolve do not, particularly in locomotion tasks.
(3) With more powerful LLM backbone 4o, almost all methods achieve better reward functions within a limited number of training iterations. However, it is worth noting that Eureka experiences oscillations on FrankaCabinet, indicating inability to generate effective reward functions with lightweight model. Revolve fails to optimize performance on ShadowHand, while RF-Agent's performance improvement remains stable across all tasks.
These advantages highlight RF-Agent's effective use of contextual reasoning and historical informa- tion, enabling superior search efficiency for high-performance reward functions. Additionally, further modily two control tasks to validate the generalization capabilities of different methods in out-of-distribution scenarios. Detailed results can found in the Appendix
(a) Bides-Expert-Eory
Success R.8t
Spare
GrsspandHaaee
BustaCap
Catchunderam
(b) Bides- Experi-Hladd
1+4
Success Rate
08
SwingCup
Ultindorom
Ketble
Catchebrooo
Figure 3: Success rates comparison with standard deviation upper bar BiDex Expert-Easy/Hard.

---

High Performance of RF-Agent under complex manipulation. Fig presents the results of different methods on BidexHands. with tasks divided into Expert-Easy and Expert-Hard groups based on human reward function outcomes for easier comparison. As shown, RF-Agent maintains clear performance advantage over both human experts and other LL.M-based reward function design methods, particularly more complex tasks. Specifically, in the Expert- -asy tasks, where human reward functions have high success rates and smaller objects (e.g., cubes, bottle caps) are manipulated. Eureka and Revolve fail to reach half of human performance on tasks like GraspAndPlace,, BottleCap, and BlockStack. In contrast, RF-Agent matches or slightly exceeds human performance. In the Expert- Hard tasks, where human success rates lower and larger objects (e.g., kettles, doors) manipulated, LLM-based methods perform all well, with RF-Agent showing an obvious advantage in most tasks. These results demonstrate that RF-Agent can design effective reward functions even in more complex task environments.
Figure 4: Success rates with exploration steps under reward functions by different methods.
RF-Agent designs training-efffcient reward functions. Fig shows the training curves for different reward functions on BiDexHands. Compared to Eureka and Revolve. Reward functions generated by RF-Agent can efficiently train the policy to converge to higher success rate range, illustrating the high-quality reward function generation ability of RF-Agent, See Appendix F. for curves of others.
The powerful search improvement capabilities of RF-Agent. Fig shows the average maximum scores achieved across tasks as reward functions are iteratively generated, reflecting the actual improvement of reward functions. In both Expert-Easy/Hart tasks, RF-Agent demonstrates high optimization efficiency, significantly outperforming other methods. These results indicate that RF- Agent effectively addresses the issue of limited reward function improvement in complex control. We also provide the maximum reward achieved on each single task Appendix F.2
(a) Bidex-Expertie Expert-Eas
(b) Biles-Expert-Hardd
Figure 5:
Reward function optimization performance with sampling counts.
5.5
Ablation Studies.
(a) Efflett search methed
Effeet diffcrent
(d) Elfect remenins
Percert Differnce
-0.2
-02
-66
-6.
CarbaCabent Maggltaad
-08
Crandy
K0 Putation
Faatcring
Offfee
seteerrtty thoihht
Figure 6: Ablations on the search method, action types, and reasoning component. We report the percent difference in performance compared to the complete RF-Agent. The experiment is built on Ant, FrankaCabinet, and AllegroHand cover three task types with 4o-mini as the designer.
Search Method. To evaluate the impact of balancing exploration and exploitation on RF-Agent, we replace the improved UCT-based selection process with common search methods such DFS, BFS,

---

and Greedy. As shown in Fig each method experienced significant performance loss on at least one task, highlighting the importance of balance for search efficiency.
Action types. To evaluate the impact of the actions designed in the expansion stage, we relatively replaced the different action types with basic action, providing only the parent node state generate new reward functions. In Fig b,,, each action type holds its value, and removing all actions leads to significant degradation. This demonstrates that the action design of RF-Agent effectively leverages historical information and LLM in-context learning ability to generate high-quality reward functions. Additionally, we further investigate the impact of different action combinations on the experimental results. Detailed results and explanations can be found in the AppendixF.5
Reasoning. To evaluate the impact of LLM reasoning paradigm RF-Agent, remove self- verify and thought-align in Fig G The results show that the reasoning paradigm helps mitigate L.MM hallucinations and provides effective evaluations, benefiting RF-Agent in generation and search. Furthermore, we also conduct an ablation study on the thought alignment component itself. Please refer the AppendixF.6
6
Conclusion
This paper proposes RF-Agent, which automates the design of high-performance reward functions for low-level control tasks by combining language agents and tree search. By treating the reward function design process as decision problem, RF-Agent integrates MCTS into the reward function design process, utilizing the multi-stage contextual reasoning of LLMs to improve search efficiency and leveraging diverse action design with historical information to make effective improvements to the reward function. The results demonstrate that RE-Agent can effectively generate nigh-performance reward functions for various and complex low-level control tasks, validating its effectiveness.
Limitation. Our RF-Agent, similar to Eureka and Revolve, belongs to the category of reward function design methods that utilize feedback and LLM. These methods are rather computationally expensive and tone-ccnsu due requiring multiple interactions with the LLM and repeated policy training. While RF-Agent achieves better generation results with relatively smaller LLM models, it does not reduce the need for multiple RL training iterations. Our future work will focus on reducing the number of RL training cycles while maintaining the effective iterative improvement.
Acknowledgments and Disclosure of Funding
This work was supported by the National Natural Science Foundation of China (Grant No. 62031001 Grant No.62325101.
10

---

References
[1] Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-fine international conference on Machine learning, page 1, 2004.
[2] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do as i can, not as say: Grounding language in robotic affordances. arXiv preprint urXiv:204.00691,, 2022.
[3] OpenAl: Marcin Andrychowiez, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob Mc- Grew, Jakub Pachocki, Arthur Petron, Matthias Plappent, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-20, 2020.
[4] Serena Booth, W Bradley Knox, Julie Shah, Scott Niekum, Peter Stone, and Alessandro Allievi. The perils trial-and-error reward design: misdesign through overfitting and invalid task specifications. In Proceedings of the AAAI Conference on Arlificial Intelligence, volume 37, 2023.
pages
[5] Cameron B Browne, Edward Powley, Daniel Whitchouse, Simon M Lucas, Peter Cowling, Philipp Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samothrakis, and Simon Colton. survey of monte carlo tree search methods. IEEE Transactions on Computational Intelligence and Al in games, 4(1):1 43, 2012.
[6] Thomas Carta, Clément Romac, Thomas Wolf, Sylvain Lamprier, Olivier Sigaud, and Pierre- Yves Oudeyer. Grounding large language models in interactive environments with online reinforcement learning. In International Conference Machine Learning, pages 3676-3713. PMLR, 2023.
[7] Guillaume MJ Chaslot, Mark HM Winands, and H Jaap van Den Herik. Parallel monte-carlo tree search. In Computers and Games: 6th International Conference, CG 2008, Beijing, China, September 29-October 1, 2008. Proceedings 6, pages 60-71. Springer, 2008.
[8] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 2021.
[9] Yuanpei Chen, Tianhao Wu, Shengjie Wang, Xidong Feng, Jiechuan Jiang, Zongqing Lu, Stephen McAleer, Hao Dong, Song-Chun Zhu, and Yaodong Yang. Towards human-level bimanual dexterous manipulation with reinforcement learning. Advances in Neural Information Processing Systems, 35:5150-5163, 2022.
[10] Paul Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
[11] Kenneth De Jong. Evolutionary computation: unified approach. In Proceedings of the Genetic and Evolutionary Computation Conference Companion, pages 373-388, 2017.
[12] Yuqing Du, Olivia Watkins, Zihan Wang. Cédric Colas, Trevor Darrell, Pieter Abbeel, Abhishek Gupta, and Jacob Andreas. Guiding pretraining in reinforcement learning with large language models. In International Conference on Machine Learning, pages 8657-8677. PMLR, 2023.
[13] Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In International conference on machine learning, pages 58- PMLR, 2016.
[14] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, arXiv preprint arXiv:2501.122448 2025.

---

[15] Ankur Handa, Arthur Allshire, Viktor Makoviychuk, Aleksei Petrenko, Ritvik Singh, Jingzhou Liu. Denys Makoviichuk, Karl Van Wyk, Alexander Zhurkevich, Balakumar Sundaralingam. et al. Dextreme: Transfer of agile in-hand manipulation from simulation to reality. In 2023 IEFE International Conference on Robotics and Automation (ICRA), pages 5977-5984. IEEE, 2023.
[16] Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhit- ing Hu. Reasoning with language model is planning with world model. arXiv preprint Skki::2305.199999 2023.
[17] RISHI HAZRA, Alkis Sygkounas, Andreas Persson, Amy Loutfi, and Pedro Zuidberg Dos Martires. REvolve: Reward evolution with large language models using human feedback. In The Thirteenth International Conference on Learning Representations, 2025.
[18] Zen Kit Heng, Zimeng Zhao, Tianhao Wu, Yuanfei Wang, Mingdong Wu, Yangang Wang, and Hao Dong. Boosting universal I1m reward design through heuristic reward observation space evolution. arXiv preprint wrXiv:2504.075%, 2025.
[19] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
[20] Lei Huang, Weijiang Yu. Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang. Qiang- long Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions Information Systems, 43(2):1-55, 2025.
[21] Aaron Hurst, Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.221266 2024.
[22] Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving. Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. Advances in neural information processing systems, 31, 2018.
[23] W Bradley Knox, Alessandro Allievi, Holger Banzhaf, Felix Schmitt, and Peter Stone. Reward (mis) design for autonomous driving. Artificial Intelligence, 103829,, 2023.
[24] Minae Kwon, Sang Michael Xie, Kalesha Bullard, and Dorsa Sadigh. Reward design with language models. arXiv preprint arXiv:2303.00001, 2023.
[25] Xinzhe Li. A survey on Ilm test-time compute via search: Tasks, Ilm profiling, search algorithms, and relevant frameworks. arXiv preprint arXiv:2501.1006) 2025.
[26] Yuxi Li. Deep reinforcement learning: An overview. arXiv preprint arXiv:1001.07244 2017.
[27] Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 9493-9500. IEEE, 2023.
[28] Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eureka: Human-level reward design via coding large language models. arXiv preprint arXiv:2310. 12931, 2023.
[29] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36:46534-46594, 2023.
[30] Denys Makoviichuk and Viktor Makoviychuk. rl-games: perf-performmace framework for reinforcement learning. https://github. attpp:///gtthue..co///eesss//// -aammeeaeaaerrrreeaaaaeeee games May 2021.
12

---

[31] Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo. Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, al. Isaac gym: High performance gpu-hased physics simulation for robot learning. arXiv preprint arXiv:2108, 10470, 2021.
[32] Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2
2021.
[33] Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, Saeed Anwar, Muhammad Usman, Naveed Akhtar, Nick Barnes, and Ajmal Mian. A comprehensive overview of large language models. arXiv preprint vrXiv:2307.06433 2023.
[34] Andrew Y Ng. Daishi Harada, and Stuart Russell. Policy invariance under reward transforma- tions: Theory and application to reward shaping. In Icml, volume 99, pages 278-287. Citeseer, 1999.
[35] Jongjin Park, Younggyo Seo, Jinwoo Shin, Honglak Lee, Picter Abbeel, and Kimin Lec. Surf: Semi-supervised reward learning with data augmentation for beedback-efficiing preference-based reinforcement learning. arXiv preprint arXiv:2203. 2022.
[36] Joon Sung Park, Joseph O' Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceed- ings of the 36th annual acm symposium on user interface software and technology, pages 1-22, 2023.
[37] Diego Perez-Liebana, Spyridon Samothrakis, Julian Togelius, Tom Schaul, Simon M Lucas, Adrien Couêtoux, Jerry Lee, Chong-U Lim, and Tommy Thompson. The 2014 general video game playing competition. IEEE Transactions on Computational Intelligence and Al in Games, 8(3):229-244 243, 2015.
[38] Zhenting Qi, Mingyuan Ma. Jiahang Xu, Li Lyna Zhang, Fan Yang, and Mao Yang. Mutual reasoning makes smaller ilms stronger problem-solvers. arXiv preprint arXiv:240..06195 2024.
[39] Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, al. Code Ilama: Open foundation models for code. arXiv preprint arXiv:2308. /2950, 2023.
[40] Nikita Rudin, David Hoeller, Marko Bjelonic, and Marco Hutter. Advanced skills by learning locomotion and local navigation end-to-end. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages
IEEE, 2022.
[41] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.0633: 2017.
[42] Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning, 2023,
[43] Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. arXiv preprint arXiv:2010.037768 2020.
[44] Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. Progprompt: Generating situated robot task plans using large language models. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 1523-11530. IEEE, 2023.
[45] Satinder Singh, Richard I. Lewis, and Andrew G Barto. Where do rewards come from. In Proceedings of the annual conference of the cognitive science society, pages 2601-2606. Cognitive Science Society, 2009.
[46] Richard Sutton, Andrew Barto, et al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.
13

---

[47] Weihao Tan, Wentao Zhang, Shanqi Liu, Longtao Zheng, Xinrun Wang, and Bo An. True knowledge comes from practice: Aligning 11ms with embodied environments via reinforcement learning. arXiv preprint arXiv:2401. 100::2411.44111, 2024.
[48] Andrew Vince. A framework for the greedy algorithm. Discrete Applied Mathematics, 121(1- 3):247-260, 2002.
[49] Guanzhi Wang, Yuqi Xie. Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305. 16291,
[50] Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6):186345, 2024.
[51] Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprinl arXiv:2203. 1171, 2022.
[52] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, al. Emergent abilities of large language models. arXiv preprint arXiv:2206. 07682, 2022.
[53] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 01.24824-24881 2022.
[54] Markus Wulfmeier, Peter Ondruska, and Ingmar Posner. Maximum entropy deep inverse reinforcement learning. arXiv preprint arXiv: 1xX.::1577.04888 2015.
[55] Tianbao Xie, Siheng Zhao, Chen Henry Wu, Yitao Liu, Qian Luo,, Victor Zhong, Yanchao Yang. and Tao Yu. Text2reward: Automated dense reward function generation for reinforcement learning. In International Conference on Learning Representations (ICLR), 2024 (07/05/2024- 1 11/05/2024, Vienna, Austria), 2024.
[56] An Yang. Baosong Yang. Beichen Zhang, Binyuan Hui. Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412. 15115, 2024.
[57] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Ad- vances in neural information processing systems, 36: 18s91189911888 2023.
[58] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023.
[59] Wenhao Yu, Nimrod Gileadi, Chuyuan Fu, Sean Kirmani, Kuang-Huel Lee, Montse Gonzalez Arenas, Hao-Tien Lewis Chiang. Tom Erez, Leonard Hasenclever, Jan Humplik, et al. Language to rewards for robotic skill synthesis. arXiv preprint arXiv:2306.08664 2023.
[60] Daoguang Zan, Bei Chen, Fengji Zhang, Dianjie Lu, Bingchao Wu, Bei Guan, Yongji Wang, and Jian-Guang Lou. Large language models meet nl2code: A survey. arXiv preprint arXiv:2212.094200, 2022.
[61] Yuwei Zeng, Yao Mu, and Lin Shao. Learning reward for robot skills using large language models via self-alignment. arXiv preprint 4rXiv:2405.07162, 2024.
[62] Chen Bo Calvin Zhang, Zhang- Wei Hong, Aldo Pacchiano, and Pulkit Agrawal. Orso: Ac- celerating reward design via online reward selection and policy optimization. arXiv preprint 10X 2024.
[63] Zirui Zhao, Wee Sun Lee, and David Hsu. Large language models as commonsense knowledge for large-scale task planning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
14

---

[64] Andy Zhou, Kai Yan, Michal Shlapentokh-Rotten Rothman, Haohan Wang, and Yu- Yu-Xiong Wang. Language agent tree search unifies reasoning acting and planning in language models. arXiv preprini arXiv:2310.04400, 2023.
[65] Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pages 1433-1436 Chicago, IL. USA, 2008.
15

---

A Environments
In this section,we we provide environment details. For each environment, list its observation and action dimensions,the verbatim task dnseription,and the task evaluation score function F.
IsaacGym. The general objective of the IsaacGym 31 environments is to simulate various robotic tasks in controlled virtual setting. These tasks range from balancing and controlling movements to interacting with objects. Each environment has specific goals such making an possible, having a humanoid perform tasks with agility, or controlling a hand to spin an object to target orientation. The environments involve challenges in movement, manipulation, and stability, with each task requiring precise control of the robot or agent, measured by corresponding evaluation score function that encourages the agent optimize its performance.
as fast as
Figure 7: Examples of the IsaacGym environment.
IsaacGym Environments
Environment (obs dim, action dim)
Task description
Task evaluation score function F'
Ant (60,8)
To make the run forward fast possible cur cur_dist prev_dist
Anymal (48, 12)
To make the quadruped follow randomly chosen x, y, and yaw target velocities ssinvel_error + angvel_error)
Humanoid (108, 21)
To make the humanoid run as fast as possible cur_dist prev_dist
Quadcopter (21, 12)
To make the quadcopter reach and hover near fixed position
-cur_de dist
AllegroHand (88, 16)
To make the hand spin the object a target orientation
16

---

number of consecutive successes where current success is 1 [rrot_dist <
0. 1]
FrankaCabinet (23, 9)
To open the cabinet door [[cabinet_pos 0.39]
ShadowHand (211, 20)
To make the shadow hand spin the object to target orientation number of consecutive successes where current success is 1 [root_dist 0.1]
<
Bi-DexHands. The Bi-DexHands()) environment is designed to simulate dexterous manipulation tasks involving two hands. The overall goal the Bi-DexHand environment is to enable the agent to perform various tasks that require coordinated manipulation using both hands. These tasks are modeled simulate real-world dexterous activities, such passing objects between hands, interacting with everyday tools, and performing precise actions requiring dual-hand cooperation. Tasks Bi- De-Deaaadd environment are designed to push the boundaries of agent control, testing their ability to handle complex multi-step processes, balance, and precision, all of which are crucial in dexterous manipulation.
SairgCap
(b) BottCCa
(0
(c)
00 DostCane
(O Lift mederam
(K) BleckStack
@ GragundPPace
@i Ketlk
Figure 8: Examples of the Bi-DexHands environment.
Bi-DexHands Environments
Environment (obs dim, action dim)
Task description
Task evaluation score function F
Over (398, 40)
This class corresponds to the HandOver task. This environment consists of two shadow hands with palms facing up, opposite each other, and an object that needs to be passed. the beginning, the object will fall randomly in the area of the shadow hand on the right side. Then the hand holds the object and passes the object to the other hand. Note that the base of the hand is fixed. More importantly, the hand which holds the object initially can not directly touch the target, nor can it directly roll object to the other hand, so the object must be thrown up and stays in the air in the process [[iis 0.0
GraspAndPlace (425, 52)
This class corresponds to the GraspAndPlace task. This environment consists of dual-hands, an object and bucket that requires us to pick up the object and put it into the bucket 1 1[]block- bucketl 0.2]
17

---

BottleCap (420, 52)
This class corresponds to the Bottle Cap task. This environment involves two hands and bottle, we need to hold the bottle with one hand and open the bottle cap with the other hand. This skill requires the cooperation of two hands to ensure that the cap does not fall 1 [dist 0.0
CatchUnderarm (422, 52)
This class corresponds to the Catch Underarm task. In this task, two shadow hands with palms facing upwards are controlled pass object from one palm the other. What makes it more difficult than the Hand over task is that the hands' translation and rotation degrees of freedom longer frozen but are added into the action space 1 [dist < 0.0
BlockStack (428, 52)
This class corresponds to the Block Stack task. This environment involves dual hands and two blocks, and we need to stack the block tower
0.07 and poal_dist_2 0.07 and 50 (0.05- z_dist_1)
1]
SwingCup (417, 52)
This class corresponds to the SwingCup task. This environment involves two hands and a dual handle cup. need use two hands hold and swing the cup together 1 [rot_dist 087]
LiftUnderarm (417, 52)
This class corresponds to the LiftUnderarm task. This environment requires grasping the pot handle with two hands and lifting the pot to the designated position. This environment is designed to simulate the scene of lift in daily life and is a practical skill
[[ist 0.05]
Kettle (417, 52)
This class corresponds to the PourWater task. This environment involves two hands, a kettle, and a bucket, we need to hold the kettle with one hand and the bucket with the other hand, and pour the water from the kettle into the bucket. In the practice task in Isaac Gym, we use many small balls to simulate the water
[[lbucket- kettle_spout| < 0.05]
CatchAbreast (422,
This class corresponds to the Catch Abreast task. This environment consists of two shadow hands placed side by side in the same direction and an object that needs passed. Compared with the previous environment which is more like passing objects between the hands of two people, this environment designed to simulate the hands of the same person passing objects, so different catch techniques are also required and require more hand translation and rotation techniques 1 [dist] 0.03
DoorCloseDutwary (417, 52)
This class corresponds to the DoorCloseQutward task. This environment also require a closed door to be opened and the door can only be pushed inward initially open outward, but because they can't complete the task by simply pushing, which need to catch the handle by hand and then open close it, it is relatively difficult hanoo__aandle_ee dist < 0.5]
B RF-Agent Details
B.1
Basic Prompts.
Here, we present the fundamental prompts used across all LLM-based reward function design methods, including the system prompt, environment information prompt, code format tip prompt.
The system prompt as follows:
18

---

System Prompt
You are a reward engineer trying to write reward functions to solve reinforcement learning tasks effective possible.
Your goal is to write reward function for the environment that will help the agent learn the task described in text.
Your reward function should use useful variables from the environment inputs. example, the reward function signature can be:
Storch. Jit. script
def rewerd (objeccc... torch. Tensor, goal_pos: Temsor) <> Tuple[ Tensor, , Dict(str: torch. Tensor]::)
revard,
Since the reward function will be decorated with @torch.jit.scrjjj please make sure that the code is compatible with TorchScript (e.g., torch tensor instead of numpy array).
Make sure any tensor variable you introduce the same device as the input tensors.
The environment information prompt with task IsaacGym-Ant is as follows:
Environment information Prompt with Task Ant
The task is: Ant. You need make the run forward as fast as possible.
The Python environment is:
class Ant (VecTask):
*. ""*Rest the environment definition enitted. e*e
def compute_ onw.uttteoaaaeaiinonnaaaaao (self):
sin) self.gyn. www..//nrreee.ttreccaooeoeeeeeee
self gyn. refreeh_dof__ste self gyn. satpehhhtttttorrotttttttteeeeeee (self.sim)
(self simm)
self.obe obs_buf self potentiale(:] self celf.pre pre. [:], up_vec
[:], self self.heading-ve[[]]
aamputo.att_obaerattioo
self. obe_buf, root.root_stae stotestates, sel., targets, self polentoten,,sl,, self inv_start. self dolffdof_limitt__ee
dof_poo, self.dof_vel.
dolf.dof_limits._.
self.dot ded__vl__scale,
helf.vee_.ensor.tenee
selfoactions, self.dt, self.
Cantact_force_ace
self. basis_seeco self baris_ self.up.axis_r
def compute, ant aat__obeervatinee obf_buf, roet_states, targets, potentials, inv_start_rot, dof_pos, dof_vel, dof aff_limits_lover dof_linitssuuper. doclvel_scale, sensor noror_torreetorquee. actiona, dt, contact_force_ecat
basis_vecO, baris_veci, up_axit_idx):
type:
Tensor, Tensor, Tensor, Tensor, Tensor, fleat, Tensor, Tensor, float, float, Tensor, Teasor, int) +> Tuple [Tensor, Tensor, Tensor, Tensor, Tensor]
(Tenaor, Tensor,
poriiton root_ 0:3]
perto_ortation root_states 3:7]
velocity root_statesl;, 7:10]
ang_velocity root_states [;, 10:13]
to_target targets torso_pasition
to_target 2] 0.0
prev_potentiale..n potentials. clone
potentials norm (oratto p-2, dim--1) dt
torso quat, up-proj, beading.proj, up_ hoading_vec
saapute_heading, and_up( torso_rotation, inv_start_rot, tortarget, vecO, basie_vecl, 2)
19

---

vel, 1o,,o,, angvel.loc, roll, pitch, yav, tugle_to.targe conpute_rot toaeo_quat, velocity, ang. velociiy targets, toreo_position)
dof pof_poa_acaled unscale (dot___dd lovta, dof_Linata_upper)
abs torch. (ttortoo.pozttion ap_axia__dx] (iiriaa. 1), vel_loc, angvel_1oc,
yau unsqueeze (-1), roll. unsqueeze sngle_to, target..
unsqueeze (-1),
pproo.unqqueeze_C heading.proj wtt.gupeeooouuuu unequeeze dof-poa_scaled.
dof_vel dof_vel_scale,
torque vier 24)
actions), din--1)
return obs, potentials, pror.potentiall..or up_rec, heading_vec
The environment information prompt with task BidexHHadss-Over is as follows:
Environment information Prompt with Task Bidex-Over
The task is: This class corresponds to the HandOver task. This environment consists of two shadow hands with palms facing up, opposite each other, and object that needs to be passed. In the beginning, the object will fall randomly in the area of the shadow hand on the right side. Then the hand holds the object and passes the object to the other hand. Note that the base of the hand is fixed. More importantly, the hand which holds the object initially can not directly touch the target, nor can directly roll the object to the other hand, so the object must be thrown up and stays in the air in the process.
The Python environment is:
class ShadouHandDver (VacTask):
*"*t of the environment. definition onitted.
compute_ (self):
self gelf.gya. stw..rrreehdddd.sssteeeeeeerr (snl...inn
self-gyn. www.:erreeeei...ttoo..tettttetee
(self.aim)
self www...rrreeen..r..........att... tensor ain)
wtt.::ge..eeeet......e.seeeeee (self, sin)
salf.gyn.ree
(self.
obs_ttpe ["point_cloud"] self.gym. self att.ttt..
(sel.....) sim) (self. aim)
self oelf.tbjecttpoe self rolffroot_state_teeee [self: caaillfoojjetiiteeeee
0:7]
self.object.ppo relf.root_state_ [self.abjectt.nddi self.object self rolf.roet_sttteeeee tensor obllfoobject_idieeee 3:7]
0:3]
oolfeobject_liave oblft
roos__tate_teentt tensor collf.ebject_indee
7:10]
self wtt.rrrooottttttttteeee [self. wsweet.beectttneeeeeee object_indices, 10: 13]
self self.gosl_poee self gosl_states 0:7]
self self.goal_pos self colf.goal_stater(:: 0:3]
rolf.goal.rot self soll.goal
self. aaf..fingrrtip.ttat self rolf. self. hut.e..tngrrtipp.hae..
1[:, 0:13]
self fingertprgts
self, [lggerttp
0:3]
self ais..iggrrtipp.ssthrrrr
zelf zelf.rrigid.body_ttaeerr self. ;, self wtd.rriidddbbdd.areeee..e self.
cingertip.anotter_Laaat
self annggrrtip_aothee...o
i, 0:3]
if self self.obe_ obp_ "full_state";
self self conpute_ cowpute_fuul_eeeaeeeee
elif self. obl..oosstypp "point_cloud":
self. wtt.pppwww..............eeeraaaa
suff.aspe asyanetrie_oba
self conpute, full state (True)
20

---

The code format tip prompt is as follows:
Code format tip Prompt
Some helpful tips for writing the reward function code:
(1) You may find it helpful to normalize to a fixed reward range tike-1,1 or 0,1 by applying transformations like torch.exp( to the overall reward its reward components in order to have reasonable values for the optimization procedure. This does not mean that you should bound the rewards wrong bounds might hinder the learning algorithm from achieving the optimal policy.
(2) If you choose transform a reward component, then you must also introduce temperature parameter inside the transformation function; this parameter must be a named variable in the reward function and must not be an input variable. Each transformed reward component should have its own temperature variable.
(3) If the task has a goal state, it might helpful to add a bonus to the reward function if the agent achieves the goal (up to some tolerance).
(4) Make sure the type of each input variable correctly specified; a float input variable
should not be specified as torch.Tensor
(5) Make sure you don't give contradictory reward components.
(6) Most importantly, the reward code's input variables must contain only attributes of the provided environment class definition (namely, variables that have prefix self.). Under no circumstance can you introduce new input variables.
B.2 Feedback Example and Analysis Prompts.
Here we use the Ant task as an example to show the specific content of feedback. This feedback is inherited from Eureka 23 and tracks the changes in set of key variables during the training process, including the key components of the designed reward function, task scores and episode length.
Feedback example on task Ant
feward_forward_weer
'- 0.02', 1.07', 1.47'/ 1 1.90',
3.54', 3.67'
Max: 3.73, Mean: 2.48, Min: -0.02
reward tt__tttttasae target:
'0.00', 0.00', 0.00', 0.00', 0.00', 0.000, 0.00'', 0.00', 0.00', 0.00'
Max: 0.00, Mean: 0.00, Min: 0.00
task score:
0.02', 1.08' 1.48'/ 1.90', 2.29' 2.60' /..45/ 3.622
Max: 3.67, Mean: 2.46, Min: -0.02
episode__engths: '59.19', 180.39' 285.69'' 425.37', 519.28', 578.28', 634.42',' 644.16', 633.70' 649.94'
Max: 717.00. Mean: 494.84, Min: 59.19
Based on given feedback, we provide some general tips for analyzing feedback.
Trained results analysis tip
(1) reflects the agent's 's actual task score or success rate after training under the current reward function.
21

---

If the values for certain reward component are near identical throughout, then this means RL is not able to optimize this component as it is written.
(3) some reward components' magnitude is significantly larger smaller, its value may not conducive to policy learning.
The above feedback mechanism will used in the implementation of Revolve to achieve automatic feedback of revolve in the experimental environment.
B.3 Prompts for RF-Agent Actions.
In RF-Agent, five different action types are set in the expansion stage, as well as the initial initialization action, as shown below:
The initialization prompt requires LLM to form a design idea based on the given environmental information and task description, and generate reward function based on this idea.
Initialization prompt
First, describe the design idea and main steps of your reward function one sentence. The description must be inside a brace outside the code implementation.
Next, write reward function based on this idea. The output of the reward function should consist of two items: (1) the total reward,
(2) dictionary of each individual reward component.
The code output should be formatted as a python code string: """python... w..m Do not give additional explanations.
The mutation 0mm1 prompt requires the LLM to locally optimize its reward function based on the feedback information of the parent node. In particular, 4m1 guides the LLM to modify the reward function composition mechanism.
Mutation 0m1 prompt
have one reward function with its design idea and code follows.
Design Idea: (design_iden) Code: creward_function
We trained a RL policy using the provided reward function code and tracked the values of the individual components in the reward function well as global policy metrics such success rates and episode lengths after every (epoch_fred) epochs and the maximum, mean, minimum values encountered:
(trained_resulee
Analysis tips for trained results:
Please create a new reward function that has different form but can be modified version of the provided reward function. The new reward function should have higher task score. Try to introduce more novel idea and add or remove some basic reward components from the environment.
The mutation 6m2 prompt requires the LLM to locally optimize its reward function based on the feedback information of the parent node. In particular, am2 guides the I.LM adjust the parameter weights within the reward function.
22

---

Mutation 0m22 prompt
have one reward function with its design idea and code as follows.
{design_idee ] Code: freward function)
Design Idea:
We trained RL policy using the provided reward function code and tracked the values of the individual components in the reward function well as global policy metrics such as success rates and episode lengths after every (epoch_freg) epochs and the maximum, mean, minimum values encountered:
[raiied_resuily
Analysis tips for trained results: (trained result analysis_tin tip)
Please identify the reward function parameters and create reward function that has different parameter settings compared to the provided version. The new reward function should have higher task score.
The crossover 4.3 prompt uses parent node information and nodes information in the elite set to generate new reward function by combining the reward component. After sorting the scores corresponding to nodes in the set, the sampling of elite sets is randomly selected using the reciprocal of sorting as the weight.
Crossover 4.3 prompt
have [nums] existing reward functions with their design ideas and codes as follows.
creward_funcggat
We trained RL policy using the provided reward function code and tracked the values of the individual components the reward function well as global policy metrics such as success rates and episode lengths after every (epoch freg) epochs and the maximum, mean, minimum values encountered:
Brained_resulte
Analysis tips for trained results: (trained_ secineddresult_anne resull__nallsii__u
Please create a new reward function inspired by those reward functions. Try to list the common ideas in those high score reward functions and combine high-performing reward components from different reward functions based on the data tracked. The new reward function should have higher task score.
The path reasoning 0,4 prompt uses parent node information and its ancestor information to generate a new reward function by reasoning this optimization path.
Path Reasoning a,4 prompt
have (nums) existing reward functions related to optimization sequence with their design ideas and codes follows.
Dowward_func_grrg
We trained RL policy using the provided reward function code and tracked the values of the individual components in the reward function as well as global policy metrics such as
23

---

success rates and episode lengths after every (epoch_freee freq) epochs and the maximum, mean, minimum values encountered:
Analysis tips for trained results:
Please create new reward function inspired by all the above reward functions. Try to reason the optimization path and list some ideas those reward functions that are clearly helpful to better improvement. The new reward function should have higher task score than any of them.
The different thought 0,5 prompt uses parent node information and random nodes information from the tree generate a new reward function by requiring to design new reward function with different structures from these nodes.
Different Thought aus prompt
have [nums) existing reward functions with their design ideas and codes as follows.
[reward_func_] func_group)
We trained RL policy using the provided reward function code and tracked the values of the individual components the reward function well global policy metrics such success rates and episode lengths after every (epoch_freq) epochs and the maximum, mean, minimum values encountered:
(trained results]
Analysis tips for trained results: (trained_result, result_analysis_tipp
Please create new reward function that has a totally different form from the given algorithms. Try generating codes with different structures, flows algorithms. Remember the new reward function should have a higher task score.
B.4 Prompts for Thought-align and Self-verify.
Thought-align. After the reward function is successfully executed, the RF-Agent resummaries the reward function to eliminate the inconsistency between the original design idea and the reward function expression caused by hallucinations or modifications.
shought-align prompt
Following is the Design Idea of reward function for the problem and the code for implementing the reward function.
Design Idea: (design_ide idea]
Code: (reward function)
The content of the Design Idea cannot fully represent what the reward function has done informative. So, now you should re-describe the reward function using less than 4 sentences. Hint: You should reference the given Design Idea and highlight the most critical design ideas of the code. You can analyze the code to describe which reward components it contains, which observations they are calculated based on, what are the parameters and structure of the reward coupling process, and what special ideas are used.

---

Self-verify. In order to identify potential functions faster in the early selection stage, RF-Agent appropriately added self-verify score from the LLM as part of the selection criteria, and the impact of this score will gradually weaken in the later stage.
Self-verify prompt
Be sure to remember the definition of the task. Additionally, have an existing reward function with its design idea and code as follows.
Design Idea: /design_ideee idea] Code:
Now, based on the task definition, imagine how expert-level strategy completes the task, describe the execution planning and motion process of the expert strategy for the task, and then evaluate the given reward functions according to your imagined description process, and judge the similarity of the given reward function with the imagined process, with the numerical value limited to [-1,1]. Finally, return the value enclosed square brackets, such as [0.5].
C Training Details
shown in the paper, we deploy our RF-Agent with leaf node parallel scheme 7 in the expansion stage, we set this parallel number to 8. Thus, we configure the actions as [2, 2, 2, 1, 1] per expansion to make the ratio nodes that utilize local and global information at 1 The initial value of À is set to 0.4 across all tasks, Uself is constrained within the range of [-1, 11 in order to have a more obvious distinction after softmax, is randomly sampled from [2, 4] to control the number of nodes sampled in a,3, 0,4 and das and is set to 0.7.
Hyper-parameters.
Deployment resources. We deployed RF-Agent on a Nvidia Geforce RTX3090 cards with 128 core CPUs and 256GiB memory server. On issacgym, the average time consumed by each different task was 9 hours; on Bi-DexHands, the average time consumed by each task was 40 hours.
Other evaluation details. During the evaluation, we tested on seeds different from those during the search. The number of training steps in the environment consistent with the default training steps in the IssacGym and Bi-DexHands environments. For example, the uniform number of steps on Bi-DexHands is le8.
D Baseline Details
Eureka. Eureka 28 has been tested on IsaacGym and Bi-Dexhands using the GPT-4 model as the reward function generator. Given the high cost of GPT-4 and its reduced relevance as a mainstream choice, we used 4o and 4o-mini models for our experiments. Therefore, we also re-conducted experiments with Fureka using these two models. The parameter configuration for Eureka remains consistent with its default settings, maintaining 16 samples per iteration.
Revolve. Since Revolve[17 has not been previously tested IsaacGym and Bi-Dexhands, we re-implemented the experiments. We retained its algorithm and evolutionary operation prompts, with the main changes being in the system prompt for the environment and replacing feedback information with automated environment feedback. These two types of prompts are consistent with those used in Eureka and RF-Agent. The parameter configuration for Revolve remains consistent with its default settings, maintaining 16 samples per iteration during population evolution.
E
Cost Comparison
Training resource consumption. Fureka, Revolve and Ours RF-Agent are using the same total number of reward functions to training the policy, thus the training cost is almost the same.
25

---

Storage Usage. RF-Agent needs to store historical information of the entire tree, Revolve needs to store historical information of the population, and Eureka hardly needs store historical information, but these additional historical information is composed of strings, whose order of magnitude is at the MB level.
Token consumption. The main consumption of each method is concentrated on the process of generating reward functions in combination with environmental information and instructions. RF- Agent also has thought-align and self-verify processes compared to Eureka and Revolve, but the token used in these two processes much lower than the reward function generation process.
We tallied the input token (prompt), output token (completion), and the actual monetary cost for RF- Agent, Eureka, and Revolve on the Ant and Humanoid tasks. It is important note that imput tokens can be processed in a single, efficient forward pass (a 'prefill); whereas output output_tokens tokens are generated autoregressively (a "decode' process), which is computationally intensive. This efficiency difference is reflected API pricing (e.g., Input: $0.6 per 1M tokens vs. Output: $2.4 per 1M tokens).
Table 4: Cost on Ant Environment
Ant
Prefill Tokens
Decode Tokens
Total Cost (S)
Eureka
82458
54505
0.1802868
Revolve
162348
43973
0.202944
Ours-total
257564
69704
0.321828
Ours-action
203282
53462
Ours-others
54282
16242
Table 5: Cost Humanoid Environment
Humanoid
Prefill Tokens
Decode Tokens
Total Cost (S)
Eurcka
78940
60021
0.191414
Revolve
169545
44118
0.20761
Ours-total
254555
66557
0.31247
Ours-action
204283
50270
Ours-others
50272
16287
The data in the tables leads to the following conclusions: RF-Agent utilizes 2-3 times more prefill tokens than the baselines, while the number of decode tokens only moderately higher. Furthermore. we find that this asymmetric token consumption aligns perfectly with the design philosophy of RF-Agent. The higher usage of prefill tokens is a direct result of leveraging historical information (i.e., previously generated thoughts and reward functions). As the MCTS tree grows, these artifacts are flexibly combined and included in the prompt, forming multi-step reasoning process for the LLM. This process enables the LLM to generate more sophisticated and effective reward functions, which in turn leads to the significant performance growth observed our experiments.
F Additional Results
F1
Training Curves in Bi-DexHands.
In Fig 9 selected Bi-DexHands. RF-Agent can maintain its dominant performance in most tasks.
we show all the training curve results of the Expert-Easy and Expert-Hard groups on the
F.2 Reward Function Optimization Performance.
In Fig 10 we show the maximum score obtained on each task the number of reward function samples increases. The results show that our RF-Agent has better iterative optimization capabilities on most tasks.
26

---

Figure Success rates with exploration steps under reward functions by different methods on Bidex-hands.
Figure 10: Reward function optimization performance with sampling counts in Bi- BinDexHands.
Cask
Poru
Pevarc
Autien
Anter
Artins
Artien
Sin Time
Figure 11: Examples of the best node growth path.
F.3
More Optimization Path Examples of RF-Agent
We show the optimization paths of the nodes that end up with the best performance two Bi- HexHands tasks, as shown in Fig[TT We found that the optimization paths on both tasks contain certain global actions, which shows that it is important to optimize reward functions using global historical information. In addition, the optimization paths on CatchAbreast further illustrate that those weak reward functions should be easily abandoned, and mixing them with reward functions on other paths can also obtain performance advantage nodes.
E.4 Out-of-distributtin tasks performance
Considering that the training data for advanced large language model baselines might include certain common reward functions, we design and conduct two entirely new experiments using the Isaac Gym Ant environment:
Ant Lie Down Task description: "To make the ant lie down on its back (belly-up) as quickly and stably possible. Its four feet should point towards the sky." The evaluation score for this task based on success condition, defined by an indicator function |/orientation, reward 0.9), corresponding to stable belly-up pose.
27

---

Ant Patrol and Return Task description: "To make the ant first run forward to the checkpoint position, then turn around and return its starting origin position." For this task, success is < 0.5), signifying the completion of the entire two-stage
1 fttask_phase == 1 and trip.
Using the same experimental configuration as in our main Isaac Gym experiments and employing the GPT-44-mini model, we obtained the following results:
Table 6: Comparison of methods on OOD-Ant tasks.
Task
Sparse
Eureka
Revolve
Ours
Lie Down 0.12:10.33
0.64:10.08 0.52.10.07
0.5910.11
0.73:10.08
Patrol and Return
0.05
0.63
0.82100.06
In these experiments, the 'Sparse' condition refers to using only the evaluation score (i.e., the [[..] indicator function) the reward signal. While it is not feasible to provide a handcrafted expert baseline for such novel tasks, the experimental results sufficiently demonstrate two key points: (1) The paradigm of LLM-based reward generation capable of generalizing these specialized, novel tasks. (2) Our RF-Agent continues to outperform the existing baseline methods on these new challenges.
F.5 Detailed ablation studies on action combination
We further conducted combinatorial study of action Ant and Allegro Hand tasks:
Table 7: Detailed ablation study on action combinations.
Action Combination
Ant
Allegro Hand
a w/ mutation
5.13(-18%)
18.76(116%)
b w/ mutation + crossover
5.77(-7%)
21.90(-3%)
C w/ mutation + reasoning
5.94(-4%)
21.26(-5%)
d w/ crossover + reasoning
4.85(-22%)
19.21(115%)
e w/ mutation + crossover + reasoning
6.08(-2%)
22.19(-25%
Full Version (w/ actions)
6.22
22.52
By analyzing the ablation results, we can categorize our actions from the perspective of information utilization. The mutation action can be seen as local operator, as it modifies single parent node. In contrast, crossover and reasoning act as global operators, as they synthesize information from multiple nodes entire ancestral paths.
The results clearly indicate that relying exclusively on one type of operator leads to a significant performance drop. For instance, using only local action (w/ mutation) or only global actions (w/ crossover reasoning) both result considerable performance degradation. Therefore, a balanced combination of both local and global operators is crucial for achieving the best performance.
F.6 Detailed ablation studies on thought align
We present ablation study on the thought align mechanism. In current design, this process consists of two steps:
(1) First Design Thought: Following Shain-of-Thhouhh like process, the LLM first generates high-level 'design thought' before composing the actual reward function code.
(2) Second Thought Align:
To mitigate potential hallucinations from a single generation step, after the reward function generated, we prompt the LLM re-examine the code's content and then revise the original 'design thought' to ensure it accurately reflects the code's 's logic.
To quantitatively validate the effectiveness of this mechanism, conducted further ablations these intermediate components:
The results indicate that the design thought mechanism is crucial across all tasks. Notably, the second thought align step has particularly significant impact on the more complex manipulation task (Allegro Hand).
28

---

Table 8: Ablation study on thought alignment components.
Task
full thought align
w/o first design thought
second thought align
w/o all
anl
6.22
5.08(-18%) 18.07(-19%)
5.94(-4%) 18.55(-17%)
4.96(-20% 14.64(-35%)
allegro hand
22.52
G
Algorithm of RF-Agent
Algorithm provides pseudo-code for the proposed RF-Agent method, which can be combined with Fig to further familiarize yourself with the entire RF-Agent process.
Algorithm 1: Reward Function Design via Language Agent Tree Search (RF-Agent)
Input:
score Function F, reward designer G, the number of initial nodes N,, max evaluation times N, action set [aml1 0m2; 4c3, ay4, adsl, counts of action [Amd Rmm2 fee3, T4d4, na5]. UCT initial balance parameter Ào
Initialization Stage
Initialize virtual root node yooot
3 Set t
Initialization N, nodes with designer G and link all the N, nodes to the n
// Eq
5 Evaluating after training policy x with from N, nodes
// MCTS Search
7 for 1 S N do - λo N-t N
// MCTS Selection Stage
10
Noyoot while is not leaf node do argmax, UCT(s');
11
// Eq 2 with A
end while
14
// MCTS Expansion Stage
for a in action set do expand node with action a and designer G in na times; end
16
// Eq with laction
18
// MCTS Simulation Stage
19
Evaluating after training policy with from newly generated nodes // MCTSS Backpropagation Stage
20
21
Evaluate the self-verify score uself newly generated nodes.
while is not the root node Nroot do
23
Update (s)) and N(8);
// Eq
end while
25
+ EmEactionsse set 7h
26 end
Return: The node s* with the best reward function R²
H Licenses
The licenses and URL of baselines. benchmarks and deployment algorithm are listed in
Table 9: The licenses and URL of baselines. benchmarks and deployment algorithm.
Resources
Тype
License
URL
Eurekaf 28
Codes for Baseline
MIT
https: h/tp://ggttttttt// con/eureka research/ Eareka
Revolve 17
Codes for Baseline
Available Online
nttps 2it000.
KancGym 31
Codes for Benchmark
BSD 3 C-lausse License
https: gitaub com/ 15aac 51m/ Lesacuynaave
Bi-DexHanda
Codes for Benehmark
Apache 2.0
Attpe:/ /ttp////1/ kituub Coa/PKU PKU www.e//eeeeeeeeeeeeeeeoeo
RL-games 30
Codes for Algorithm
MIT
Sttps://gatimnn http://hwww..ccaaaaaaa/////e..
29

---

RF-Agent Reward Examples
L1 Reward after Diverse Actions Examples
We randomly selected a task and corresponding node to show what the tendency of the new reward function generated by RF-Agent under different actions. Here select the node with RF-Agent developed on CatchAbreast to the path 'root-Oi-e3-ml-e. as the current node. First, the reward function of the node is displayed, and then the reward function generated by different actions under the node is displayed turn.
Current reward function on CatchAbreast.
atorch. strrip
def compute_ (objer
torch goal_poa:
left_hand_ppt Tensorr right_hand_pos: Tecchreensr -> Tuple (torch. Tensor, [str, torch. Tensor]]:
distance_to_gom torch. corch..o
goal_pas, dim=-1)
tossance_to_lef_.hee dbstance_to_riitt_bbm
(1rf hand_pp_a ho object_pos, dim--1)
object_pos, dim--1)
Revard being
goal
distance__eward torch. torch.clanp (1.0 tiistance_to_goall / 0.5), mia=0, max-1)
Ravard for successfully passing the objact, bonus for being within
proxinity threshold
buccess_banus torch. shere (drch.
< 0.1,, torch.tensor (1.0, devica= (ojeece)os.device), torch.tenaor (0.0, device-object.ppa.ddee
a Rand balance reward penalizing uneven novement
hand_ rovad -torch. (distancee,
tistanca.to.right.haa
Proxinity revard encourage handa efficiently reach object combined_haad handdd_handdddttttaae prexinity.rexarz clamp (1.0 (combined bietanie 0.8), min-0, max=1)
distance distance_to..efttttt to_ distance_to_ri1e roghtith__han hand
Smoothness penalty to encourage fluid hand novements
aeaddddlaaeee.etttee..ttt anootb a.othh_ovveeat.paaall - 0.01 hand lelesaance_bevvee__ee
torch. porm haad right_ dim
# Total reward calculation
total notal_revard reward distance_ravard succeas_bonus hand read
proxlnity_reward anooth_novement.peaally
Revard components dictionary
revard, crrardd_omponents
"alstance_roxard"" dlstance_revard
success_bonua"" success_benus,
baand_balance_rexarde hand_balance_reuar
"proxinity_.revedd"" proxinity. proxinity_rovare serry_reeerd..
snnoth__oveenntt.onne anoth
return revard covpae_co
Mutation 0m1-
Reward function after am CatchAbreass
jitrrc..jt..aee
computa_revard (object__o
Tensor, goal_pos:
left_hand_pos torch. Tensor, right_hand torch. Tensor) Tuple [torch.
distance_to_gee torch.Teenoor]]
Tensor, Diet [str, torch. Teneor]]:
goal_pos, dim--1)
norch.norm (left object_pos, dim--1) Lierance_to_rigttbman norda.oomttighh..aadd.. object_pos, dim--1)
slstance_to_lett_he
baing very to goal
distance
distance_revard torch. clamp (1.0 (diatance_to_goo 0.6), min-0, max-1)
30

---

Success bonus increased for being within a tighter proxinity threshold success_
(dietance_ to- 0.05, torch, (1.0, object_pos.derico/ torch. tentor (0.0, devicorobject_pos.co (ttttooooeet.po..dorceee)) device)) torch.tenror
Balance revard penslizing
but normalizing based hand
distance
madd_balanc..pnallz
aba (otsstaace..o__eet..aa..
distance_to torreett_rrrhhhteeee
(igriigt__aad)
normalized, bandaai
clamp ((distance_ left_bandd (0.6 0.6), min=0, max=1)
trrranco_tt_rigt_haeee hand_ revare hand_balance, pennaattcepppltt normalizad_
Proxinity revard ancouraging efficient hand placement
combined_hand distance tettacce.ooleet_baand proximity reward clamp (1.0 (combined_ baaeeedd_aaddddieeeteeeee distance / 0.75), min-0, max-1)
to.
Smoothness bonus for transitions encouraging fluid movementa hand_ cearaace_hetzeen_baaeeee hands torch. forch.norm(iee corm(let__aadd_ right_hand_pos, 2-1)
bonus
claap (1.0
0.1)
min=O, max-1)
a Total revard calculation
reward (dietance__rearl succese_bonn
hand, revard
proximity.reward anooth,novemant_bonk /
components dictionary
revard_ component
setsstanc__rexard" [tatance_reeard,
* : success_bonus,
baadd_balsnce_reee hand_ baanddbaaaneereeeeree
"proxinity_ravarde" proxinity.revare
bouseett mnooth_ bonus
return total_revard, revard, comprn_cc
where the key change is adding the normalizing mechanism "hand balance ppcnatty"e
Mutation 0m2-
Reward function after am2 CatchAbreast.
@torch. (abjecttpop torch. Tenaor, goal_poa: torch.
compute_ravard
beft_hand_pos- torch. Tensor, right__hass sight__aand_po: torch. Tensor) Tuple [torch. Tensor, Dict [str, torch. Teneor]]
diatance_to__ooal corf..no goal_pos, dim--1) hand torch, (ow.tooet_aaddd bastaace_to.right.ban
object_pos. din--1) object_pos, dim-.1)
Revard for being clese to the goal with larger range dietance_revare torch. clamp (1.0 (distance_to_goal 0.3), min-0, max-1)
Larger succese success_bonue torch, xhere ddiistane__t__gee < 0.05, torch tensor (2.0, device -object_pos... -object_poo.dericc)) deviice., torch. tenzor (0.0, aaricorooject_ppssddrttt))
passing object
Hand balance reward maintaining a noderate penalty -0.5 torch, (distance_to._.. sistanco.to_riiht__annd
Eahanced preximity revard encouraging efficient hand placement combined, hand distance daatttanne
to_left_hand distance_ to_rightt.haa reaad clampp1 max-1)
hatpsaeed_handddiittaaa / 0.6), min=0,
Smoothness penalty slightly adjusted to mininize harsh panalties hand_ bellstance.bexeen_baeee torch. bordh.ooraleett_aadd..eo right_hand_poo
-) amooth moweeentt_panall,.
dand_diatance,
31

---

Total revard calculation
reward distance_revard success, bonus hand, balance_
prozimity. snoot__ooeeeaa_ppaaaay
components dictionary
reward_
covard_components woopooeensses.....
#istance_revarde distance_revard
"success_ success_benua,
"bsnd_balance paalaace_rexardd
"proxinity_ voxinity_rovarc
"cmooth_novement. ponaallyy.nt..paly....... anotth_.ovamntt.pannl.
return total revard, revard_ conponents
where the key change is change the parameters in some reward components.
Crossover 0c3-
Reward function after a.3 CarahAbreast.
computa_revard (objectttp
Tensor, goal_pos: torch.
left_hand_pos torch. Tensor, right_hand_pon torch. Tensor) Tuple [torch.
Tensor, Dict [str, torch. torch. Teneor]]:
distance_to_gon torch. norch.norm(ebb)) (or_(ebjecttpppo
goal_pos, dim=1)
torch norch.noroeee norn (left, hand object.pos, din-1) dietance_to_rrggt..ba.
torch.nor((righh.baad...
object_pos, dim-1)
goal_revard torch. clamp (1.0 (diatance_ 0.5), min=0, max=1)
Proxinity goal
success_ torch. where (dietance. too_tttogooo < 0.1, terch, tenser (2.0, device- object_por.deviie device), torch tensor 00.0, device device)) Success benus
hand_ sadd_balnnce_reee revard -torch. abs (distance_to__aaee lertllefthaaa
Penalizea uneven novenent between hands hand_ cooeddiattioo..eearrd torch. exp(-torch. beedt_
right_ dim-1)) Encourages closer hand positions
proxinity_reaart -0.5 (distanca_to_loft.bbel
distance_to_right_hnn
Maintain proximity to the object
total reward = goal_ reuad success, anccass_bonus bonus + hand_balance_reenn
hand cow.doiatiitoonnoorrr. proxinity.rexard
rovard_ componenss
geal_revard" goal_reward,
Sescess_boonu"" success_boaus,
haad_ballaneerexxrre hand_balanca_revard
condddaordii reaarrdd" mond_corddmationn.eee reward,
"proxinity_revadd" reord
total revard, reuard_ compoeetnn
where the key change to delete "combined hand distance" and "smooth movement_penaltty from the initial reward function while keeping other nigh-performing components.
Path Reasoning ar4-
Reward function after a,4 Catch Abreast.
storch. fitrch..jt... acript
def (oppuue__reerddbbbeeee.... torch. Tensor, goal_pos torch. Tensor,
Soft_hand_pos: torch Tensor, haght__and_pos: torch. Tensor) Tuple [torch. Fansor, Dict [str, torch. Fensor]]:
sistance_throsbae 0.1
auccesa_bonus anooth_noveeett_pealll 0.01
32

---

Distance metrica
distance to_sccee_o_oooo goo_ cora(ebeett. goal_pos, dim-1)
fistance_to__eft_. wt_.eoo__eaaaaeeeeeeeeeeeeeeeeee hand torch.norm (orm hot object_pos, dim-1)
bistance_to.right.haar
hor..noorrrgggthhadd.... abjact_pas, dim=1)
Revard components
reeaad torch. clamp (1.0 (oiatance_ go. 0.5), Reward for being clase to the goal
Revard for successfully passing the objact puccesaful_paas (distance_to_gon dtype)
Ertrance_threhhhte see typpe
vecard_anccesatiu
auccesstul_pasa auccess_bonus
# Rand balance reward to encourage syachronized novements
hand_ rovard storch. abs (distancce_ to_ftt__and
Histance.to_rsght.hhnn
Proxinity revard for effectively reaching the object
diatance_ to_ to_ distance_to.rinn torghtttttrtttaaaao (eonbined_bannddiettae 0.8), min-0,
proxinity.reward clamp maz=1)
Smoothness panalty for fluid hand novements hand www.diattneeeetteeeeooo
torch. berm herm(left.eend.po
dim
=1)
anooth, www.oooeeettt.ommmmmmm pew.eeenntteeealll..cct. pww.eety bead_ddieanceebeeeeeeeccc
distance_rquat
roward_auccesssul.ppr hand_ reard
rroxinity_reward soothh
Revard components dictionary
aavard_comporennt
"distance_retee revarrd"; distance_revard.
aexxard_ucceesttu__po rauard_
baldn_balanccereeaarrd band_balance_rrvardd)
"proxinity_reverdd" prozinity_rovare
aaaoth_avvenott_paallee revard"
aaorhhooreeer__nnlly.ree..
}
return total revard, sevard_component.
where the key change is that the coefficients in the original reward function have been redefined and some nested methods have been changed.
Different Thought 0.5-
Reward function after a,, on CatchA abreatt..
jit.
def compute_ ceteeeerreeedrdeeeeeeeee
torch Tensor, goal_pos: torch. Tensor, left_ left_hand_poss torch, Tensor, right_hand_pos: torch. Tensor, object_linvel: Tensor]]:
torch. Tensor) Tuple [torch. Tensor, Dict Design Idea: Utilize nested revard structure focusing on progressive subtasks grasping passing object, revarding transitions and smooth trajactories.
Define parameters goal_tolerance - 0.1
grasp_radiue snoothnoss_veight 0.2
Calculate distances for nested revards dietance_to_gee torch. (objectct goal.pes, dim--1) Sistance_to_left_tr distance_to_right_haat
torch norch.norm (objecbeesr_pee left_hand_pos, dim--1)
torch. norchh.or right_ right_hand_pos, dim--1)
Nested revard for successful graaping (vithin grasp radius) grasp_ revard torch. (ttaaadtttannetttlelll
grasp_radius,
(0.5, device=obfect.ppo device), torch geros. like (distance left tllatance.to..ott.bannd))
33

---

grasp_ torch. (tt.thrrrettttttotoooooooo righittt__aad grasp_radiua, torch. device), tensor (0.5, device-object.po ) torch. 2ercs, _ike! to.tance_oo.rihtt..ooo
goal_ revard torch. where (distance_ go_ goal_ toleraleeeee torch,
(1.0, tevicarobject_pon davice), torch zeros_ sarch..aro...kacdhttaaee
Smoothness revard calculated as the negative exponential of object linear relocity
snoothness_rovarc ttrrc (xwooopottoeeeeeaaeaaa dim=-1))
torch perch.nnroobjett.liaee
Total revard combining all companents
reward anoothness_ grasp_reward goal_revard
components
rexard_componant
graap_ grasp_revard,
"goal_revard:: goal_resard,
smmoohhess_ amoothness_ravard
return total_revard compenetete
where the change is to think about new reward forms, introducing a grasping angle and nesting it into the distance-related rewards.
1.2 Design Thought after Alignment
Similarly, the current node frot-0i-e3-ml-e-3 in the previous section is an example. The design thought during the generation process
Before thought-align
The reward function is designed optimize agent performance in the Catch Abreast task by rewarding careful proximity of the hands to the object, effective synchronization of hand movements, and incentivizing reaching the goal with success bonus, while penalizing excessive movement to ensure fluid hand coordination.
After thought alignment:
After thought-align
The reward function designed to enhance agent performance in the Catch Abreast task by incorporating multiple components that prioritize hand coordination and goal achievement. It evaluates proximity to the goal and the object, providing rewards for closeness and success bonus for passing the object within a defined threshold. Additionally, it incorporates hand balance reward to penalize uneven movements between the hands, proximity reward to encourage efficient hand placement, and smoothness penalty to promote fluid motion. The total reward calculated by summing these components while ensuring that fluid hand coordination is incentivized.
Obviously, the design ideas after thinking alignment can more comprehensively summarize the corresponding reward function.
I.3 RF-Agent and Baselines Reward Comparison
Here we list some best reward function from different methods.
IsaacGym-Ant.
34

---

Our RF-Agent.
jit.
def coapuee_eeard terch. Tensor, heading_vee: torch, Tenser,
Lorch.Tenaor -> Tuple[torch) Tensor, Dict [stz, vorchrTenaor]]: forward velocity velocity root_states 7:10] forward_valoctty velocity[:,
x-direction
Neight for different reward componente
hoading_veight - 1.5
weright_weight - 0.5
y=2.pemalty.
Cosine similarity revard maintaining desired heading direction exis)
desired torch. teaser ([1.0, 0.0, 0.0), device-heading.vee.. deviic). (hgadd_az
heading. com torch,
funcctioin itttiitittnntn cosine www.ieooeeniiiieeeeeee sinilarity (desired beading.vec, dim=-1)
reaard heigtng_Reight (heading.cos_sinn forvard
Stability penalty for velocity y and directions y- ves. torch. noreeoooivllllii 1:), p-2, din--1)
Revard being upright, using z-compenent up_vec uprightness
upright_ bonue_ 2.0 upright_bons bonus weiight 1.0))
(up_vec
Total reward is combination of forvard novement, heading alignment, and uprightness
total reward forvard_velocity revadd upright._onus y_2.velociuy_ponalyy
Construct revard components dictionary
"Iorvard_velocit forvard
"heading_rovard" heading_ravard,
" sppright_bonus" upright_bonus.
"ys_.eelocity.penalt"" -y_2.velocity.peall.,
total_revard, covere_components
Eureka.
Storch. script
def rrooeedrooododooeeoooooooooooo torch Tensor, potentials: torch Tenser, prer_poteatiale:
actions: torch. Tensor,
dof_vel: torch. tonsor dof dof_limits_loser Terch.Tensor. dof_iimita_upper: torch. Tensor) Tuple Tuple [torch Tenaor, Dict(etr, Tensor])
Tensor,
Extract relevant state information.
velocity root_states(;; 7:10]
velocity
torso position root_states:;, 0:3] # Position of the torso
Ravard component to encourage forvard velocity
roleityde velocity [t, Aesuming forsard along the X- axis
Modify target closeness www.://ooeeesrrarrretoo.oeeeeeee - 0.2
ceneitivity
prev.potentials potentials
target_closensseeeeeaa torch. (orrhhappprrrett..cmmmssseerrer
Re-examine dof velocity penalty for more sensitivity
dof wof_vel-penalty_cappee.
0.02
dof_vel pof_vel_penalty torch. sun (unrcthch. relf dim--1)
axp_dof_van spp.dot_vel.eenatty torch. txpch.exp( waw.aaatllaaalttttte...o..
dof__vel.pnnatty)
35

---

Actiona panalty adjust temperature greater diatinction
actione_penee penalty.tenperee actions_ torch. eun(torch sunttorca.abussctteeeeee (ba(actions) dim--1)
0.005
exp.actions.ponalty = torch. exp pactions.poallly.oooooooo.
actiont_penally))
Total nornslized reward total total_revard reward forvard voreart
0.6 cap.targee.lloeeessrrraaa axp.actions.penalty
exp_dot..el.penalty
Compile individual componente
dictionary
reward conpon_comonnnnts '{
"orraard.aalocctyyreeeeee rexardd" forsard. forsard_velectty. velocity.reoard,
* orpptargget.cooeeees we_.of_vel..eaatty.
prr_dof_vee__enaa11 expp.cctione.poualy"" spp.actione.eemalyy.
return total_ revard, revard conponetss
Revolve.
Storch. script
def compute_ roote states: torch. torch.Tensor, targeta: torch. torch.Teemorr potentiale:
corch.Teneer
pteypotenttals:: torch, Tenser, torch Tensor) +> Tuple[ torch, Tenser, Dict (str, torch. Tenso]]]:
Extract necessary velocity root_stateal:; 7:10]
Forward speed revard component forward_ velocity 0]
speed along
Forward potential difference rexard component
torward_potential_dttt potentials prov_petentials
Upright conponent
ap_vec_goal torch. tensor ((0.0, 0.0, 1.0), revicemup_rec rec.viiceeeieeeiee unsqueeze (0) aup upright_reeard = torch. aun vup_vec * up_vec_goal, dim=-1)
temperatures for exponential acaling forvard temp 0.35
torwardd.oteattiel_oomgg temp
upright_tenp - 1.0 a Keeping original scale
Exponentially scale the revards
servard_speed_ter torch.
foruard_ ravard forzard_apeed_2 paap) fowildd_oovvaddpotteto forvard_ forvard.potential. temp) sealledapright_reee
forverd. torch.exp(
torch. (upright__enp upright-roeard)
Total revard calculation
total revard scaled_ worddroorrrddddddroooeo + scaled folded.oroad +
notal_revard
scaiee_.prright.reeurdd
revard_ Saxard_coponrannnt componecccnnnnn
Revard componente dictionary
Toroard_epead_reaaa " tit "upright_ravard'.
foaled_
return total_revard, componttn
Bidex-CatchAbreant
36

---

Our RF-Agent.
otorch. seriit
def conpute_revard
torch. Tensor, gosl_pos: torch. Tensor,
Left_hand_pos Tensor, right,hand.ppo Tensor) >> Tuple [torch.
Tensor, Dict [str, torch. [emsor]]:
distance_to.goal torch.norn(ebjeec
goal_pos, dim=1)
distance_.oo.lftt.hmc torch. (leftt dietance.to.rrggt..bnn torch. cer.dnoorrriii.baad.....
object_pos, dim=1)
object_pos, dim-1)
Revard for proximity the with increased tolerance deeaance_reward torch. torch. (1.0 toistance_to_goll / 0.5), mia=0, max-1)
Success bonus for accurate object passing with tighter threshold auccesa_bonua torch. (oottandtttnnc.to...e
0.03, torch. tensor (1.0, -object.pos. device), torch. tensor device-object.pos device))
hand, hand_bslance reward - -0.5 * torch. distribation effort between hands lectraaceett__et_eeeee torch. clamp (1.0 (distance_to_left.t. / 1.5, nined, max=1)
land balance reward incentivizing
distanco_ tartottt_rrghhhaaataeee
tistance_to_righttt.
Proxinity reward to encourage both hands be the object conbined_hand_dae distance distance_to_ prexinity_resard torch. crah.claap(1.0 maz=1)
tostanc hand beabbined_aan_disttaed
0.75), min=0,
Smoothness benus to promote fluid hand novements hand, wtw.daiteeeeeteeeen.. betzeen, torch. nerm (left, hend right, bight__enndpps dim 0.1)
-1)
smooth merrb__oveeeet__eee
torch. torch.clanp(e. (1.0
max=1)
calculation
total reward = rdratance_revard success_ bonus hand_ballane_rrvarr
reerd soott__ / 2.0
Revard components dictionary
aavard_componenta=
"diatance_rexet" distance_rewaze reward,
"success_ success_benus
bband_balence_rekardd
"proxinity_revard"" proximity_revare
"smooth_avoeent__bom smooth_nevenent_6
retura total_ reward, tewerr__caponentt
Eureka.
@torch jit.ipippt def conpute_revard (ebject..p Tensor, goal_pos:
abjact_linvell torch. Tensor) -> Tuple Tansrr, Dict [atr, torch. Tenaor]]:
Constants for temperature adjustnents tentance_tenp
tenperature seneitivity distance velocity_temp 1.0 | Reduced scaling for velocity
Compute distance to goal
dietance_ toagal torch. pobjecbjctt goal_pos, dim-1)
Fegative distance cxp(-cisttncc_ttag distance_to_goal) Exponential distance_ravard torch. decay
mininization
Inproved velacity
with penalty
speeds
apeed_three threshold 0.1 threahold for penalty
din-1)
Penalty for lox speeds encourage proper novement
velacity_panatl
spaed_thresheil -1.0
threh_tt velocity velocity.maguitude) zerch.zozos_like
velocity.magnirnde
Positive resard for aligned velocity toxards the direction
37

---

diraction_vector gonl_pos objact_pos Direction from abject goal
direction nora norn vector, dim-1, keepdin-True)
velired_velocaty direction, / direction_nort # Normalized direction
aligned_veloctty torch.sum(okjett.
dastred_velocie dim=1)
product for slignnent
torch.clane (aligned min=0.0) # Only
positive velocities
velocity. weggcid.ellcityy..... reeeard pelacity.penslty Combine revards penalties
Kornalize velocity
rolaeity.rovard torch. clanpvvllocittyyrrrrr nin-0.0)
Conbine
total total_rekare reward dievance_revard relocity.reward
# Companents for dabugging
reward reward_componee componenta =
"distance_ distance_revard,
reeaoctty. relocity.reward,
return totar_ compee_co
Revolve.
dtorch.jit. script
def reaputo_rvvardccbjee... torch. Tensor, goal_pos: torch.Tensor,
left_hand_pps: torch. Tensor, right_hand_pot: torch. Tensor, left_hand_ron
torch. Forch.Tensor, rigtt_hand_rot: torch, torch. Tenaor) >> Tuple [tnorh Dict [atr, torch. Tensor]]:
Define temperature variables for normalization tomp-pos temp_rot 0.05
temp_catch 0.2 Adjusted tempersture for catch reward
# Compute the distance from the object to the goal nerbeeeccpp goal_pos, dim--1) dietance_to_pe -distance_to_goal Claser is better, so we take negative rovard_position
Compute the distance from the object to the left and right hands (orcheetr
tietaace_to_left_heee epailen prevent log(0)
left__aadd_ps, dim--1) 1e-6
tlatane__t_rrittt_bau hand torch. norn(object_ppo right, hos,,,_andd_os, dim--1) + 1e
Encourage the object to revard_ ceft lest_base -torch. (xp(deettttteeeeoooo....e
with stronger feadback
temp_catch)
Base feedback
revard_catch_leftt
(dirtadie to_left_hand
distance_to _llet_haand / 0.1, torch. tensor (0.00 device-object.pos device) proxinity
baaaace_to_llft_bane
Linear revard_catch_leff. coxard_catch_lett..un
ravard
Combined revarda
corard_catch_right -1.5 crrecaeppcdictnncetoorrrg
/ temp_catch)
seight right penalty reward_catch revard_catch_ corard Combined rexards for guidance
Conpute alignent//roation revards based on the orientation of the hands desired_left_.re rott_rot torch stan2(1eft_n) bo 1), left beft_hand_postee 0])
rightre torch. atan2 (righh
il, hag, 0))
rovard_left_rot -torch. aborch. (befttt hatd_honddd 0] desired
Compare sone angle derived
quaternion.
rigard_right_rot -torch. abs (odd
0] destred.
Combine all revards rovard torch. exp (xp(revard_ temp_catchl expch. exporoorrd temp_rot) torch. exp reward_right_ree temp_rot)
temp_pos) torch. torch. exp(revard_cated
Prepare individual revards retura
38

---

revard_componanta
'reward_position's revard.position,
'rexard_catch': reward_catch
"rerard__atch_ revard_catch.eett
"resard_catch.righh': revard.catch.righ;
'rexard_left_rot'; cowar__left_rot.
"retard_rightrrott: right.
retura total_ conpeettss
39

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We accurately claim the main contributions and scope of our paper in the Abstract and Introduction section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of work in Sec
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
40

---

Justification:
Our work does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We include the needed information for reproducibility See and Appendix
C
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
41

---

Answer:
[Yes]
Justification: Included with the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips.com wttpcccppppccccccec guwdes/dde/eeee Cooeeeoodeesssseeeeeee http://dd///uuudddd/d/oe/e/ea/s/
ssinislicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We include these details Sec Appendix Cld Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We report the mean and standard deviation of the experiments in Tab Tabian and Fig. 3
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
42

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [No]
Justification: Partially. We report the average running time and devices of experiments
description in Appendix C and
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:///wuurr....oppoop/cc/cctaa
Answer: [Yes]
Justification:
We follow the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact in our work.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
43

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
There are no such risks in our paper.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Please check in Appendix
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
44

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Included with the supplementary material.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
45

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: Please refer Sec 4 and Appendix B for detailed usage.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
46


Reviews:
[{"review_id": "QfI6bKrmPP", "reviewer": "Reviewer_87TL", "summary": "This paper explores a novel method of using Large Language Models (LLMs) to generate reward functions in reinforcement learning (RL). Considering that current methods underutilize historical feedback and the optimization capabilities of LLMs, this paper proposes a systematic framework that uses LLM to generate and update reward functions with the help of Monte Carlo Tree Search (MCTS) . The paper employs MCTS to make decisions and update the reward functions, and emphasizes the role of action expansion in reshaping the reward functions. Experimental results on low-level control tasks demonstrate that the proposed approach can generate high-performance reward functions, thereby facilitating policy learning.", "questions": "-Building on the above weaknesses 2, while the motivation for combining MCTS with reward optimization is reasonable, it is unclear how many iterations of reward function updates are required for the tasks in the experiments to achieve stable success. Overall, for training a task from start to successful execution, what is the RL training cost? How does the total training time compare to other methods? Is it within a reasonable and acceptable range? \n-In Figure 2, does the process from s_1^0 to its leaf nodes s_2^0 and s_2^1 represent action expansion? If not, how does it differ from action expansion? \n-Is there a possibility to provide a detailed explanation of the selection phase? When a node is selected, will other nodes at the same level also be visited later during the search? My understanding is that during the backpropagation phase, all nodes on this path are updated, after which other branches are explored until the reward function corresponding to a certain node successfully completes the task. Is there a limit to the action expansion depth along a single path? Has the author experimented with different depth limits? \n-Are there ablation studies on whether to use design thought? Regarding Weakness 1 mentioned above, is there a test that compares the impact of using different design thoughts on reward generation and optimization? Alternatively,  are there experiments with using Chain of Thought (CoT) reasoning to guide LLMs in generating reward functions?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n-The paper is well-written, clear, and easy to understand.\n-The motivation for introducing MCTS to optimize reward functions is natural, so that the training effects corresponding to various reward functions can be quickly listed and backtracked, with optimization at the nodes. The paper demonstrates that under this design, it is possible to arouse the potential of LLM for efficiently designing reward functions.\n-The design of action expansion stimulates the LLM to improve existing reward functions from multiple perspectives, making better use of historical information and potential directions for optimizing reward functions.\n-Experimental results on low-level control tasks provide adequate evidence for the effectiveness of the proposed reward optimization framework and demonstrate its generalizability.\n\nWeaknesses:\n\n-LLM's design thoughts for reward generation lack a finer-grained structure. For example, when generating reward functions, LLMs could be guided to decompose the design thought into more granular components at the reward function level. This might include explicitly defining the set of usable variables, identifying which subsets contribute to specific components of the reward, and describing how these components combine into the overall reward. In other words, standardizing the design thought process could be an effective way to enhance LLM's search strategies. By doing so, during action expansion, the LLM could quickly backtrack to identify which variables and reward components were used or effective, enabling faster iteration and improving the explainability of the reward optimization process. A possible implementation is discussed in [1]. \n-Lack of analysis on MCTS iterations and computational cost. Specifically, the paper does not analyze the number of iterations or the time required for MCTS. It remains unclear how many rounds of RL training and reward updates are needed for a given task to achieve stable and successful execution. This raises concerns about the overall training cost and whether it falls within an acceptable range compared to alternative methods.\n[1] Heng Z K, Zhao Z, Wu T, et al. Boosting Universal LLM Reward Design through Heuristic Reward Observation Space Evolution[J]. arXiv preprint arXiv:2504.07596, 2025.", "was_revised": true, "final_justification": "The authors took the feedback seriously and improved their work accordingly. The remaining issues are relatively minor and shouldn't prevent acceptance, though addressing them would further strengthen the paper.", "rebuttal": "# Response to Reviewer 87TL\nWe sincerely thank **Reviewer 87TL** for the thorough and constructive review. We particularly appreciate the recognition that **the motivation for introducing MCTS is natural and novel** and that **the experimental section sufficiently demonstrates the effectiveness** of our approach. In response to your mentioned weaknesses and questions, we consolidate our detailed replies into the following sections:\n___\n\n## Question 1: About MCTS Iterations\nWe thank the reviewer for acknowledging the novelty of our MCTS-based approach to reward function optimization.\n\nTo ensure a fair comparison, as stated on *line 278*, all methods were allocated an **identical number of total samples** per experiment. Each **sample** constitutes a full cycle of LLM-based reward generation and a subsequent RL training run. Consequently, since the number of samples is identical for all methods in each experiment, the upper bound on total training time is also consistent across all approaches.\n\nTo further analyze efficiency, we can compare the training cost (i.e., number of samples) required to achieve stable performance on the more challenging Bi-DexHands tasks. As shown in *Figure 5 (after line 318)*, which plots success rate versus the number of samples, we can establish performance thresholds. Defining \"stable success\" at 0.6 and 0.8 success rates, the number of samples required by each method to reach these levels is as follows:\n\n| Metric  | Ours (Easy) | Eureka (Easy) | Revolve (Easy) | | Ours (Hard) | Eureka (Hard) | Revolve (Hard) |\n|:--------|:-----------:|:-------------:|:--------------:|-|:-----------:|:-------------:|:--------------:|\n| avg 0.6 |     108     |     ~512      |     ~512       | |     100     |      304      |      144       |\n| avg 0.8 |     264     |     512+      |     512+       | |     296     |     512+      |     512+       |\n\nIn this table, `Easy/Hard` denote the `BiDex-Expert-Easy/Hard groups`. `~512` means a baseline was approaching the target success rate by the 512-sample limit, whereas `512+` means it failed to do so.\n\nThe results clearly show that our method is more sample-efficient. On the Easy tasks, for example, it reaches a 0.6 success rate nearly five times faster than the baselines, which fail to reach the 0.8 success rate within the entire budget. Regarding the precise wall-clock duration, we refer the reviewer to *Appendix C (line 620)*. While the full training budget for each Bi-DexHands task is approximately 40 hours, our method consistently converges to a high success rate in under 20 hours.\n\nWe hope these details resolve any concerns regarding training efficiency and time, and we conclude that RF-Agent is a more efficient method that achieves superior performance in less total training time.\n\n## Question 2: About Aciton Expansion\nYour understanding is comprehensive. The transition from `s_1^0` to `s_2^0` and `s_2^1` is also an action expansion. The reason it is not marked with an \"orange\" arrow in *Figure 2(after line 126)* is to differentiate the stages of the search process.\n\nThe non-orange arrows indicate expansions that were previously completed and whose resulting nodes (like `s_2^0` and `s_2^1`) are already stored in the tree. The orange arrow, however, is used to specifically denote the expansion (to node `s_5^1`) that is currently active. In this way, the figure illustrates a dynamic process where arrow colors distinguish past expansions from the present one.\n\n## Question 3: About Selection Stage and Depth\nYour description about the selection phase is basically accurate. We would like to add an implementation detail for clarification. For ease of deployment, we perform action expansion in parallel from the selected leaf node.\n\nFor instance, when the selection phase reaches a leaf node `s_1^0` (i.e., a node with no children), we simultaneously expand n new child nodes (`s_2^0`, `s_2^1`, ...), each corresponding to a different action. Once this batch of new nodes completes their simulations, their values are backpropagated up the tree (e.g., `s_2^0 -> s_1^0 -> root` and `s_2^1 -> s_1^0 -> root`), just as the reviewer described. The backpropagations are processed sequentially in the order that the simulations finish.\n\nIn the subsequent selection phase, the search might proceed to a different branch (e.g., to sibling node `s_1^1`) if its UCT score is higher. Alternatively, if the backpropagated values have made the `s_1^0` branch more promising, the search may continue deeper into one of its newly expanded children (`s_2^0`, `s_2^1`, ...). This creates an efficient and flexible iterative optimization process.\n\nRegarding tree depth, our maximum depth setting is correlated with the total simulation budget (e.g., a depth limit of 16 for 512 total nodes). Empirically, we observed that the search tree's effective depth naturally stabilized between 8 and 12, which was sufficient to achieve high success rates. Constraining the depth to be **shallower** (<8) confirmed its importance, as it caused a significant **performance degradation** on some tasks (e.g., the success rate on Bidex-CatchAbreast dropped from 0.7 to 0.5).\n\n## Question 4: About Design Thought\nWe are grateful to the reviewer for this crucial point. In our experiments, the \"design thought\" serves as a high-level plan, generated by the LLM itself, for composing the reward function. We require this thought to explain how different components are combined into a whole, which is an approach **analogous to Chain-of-Thought** (CoT).\n\nWe agree that the \"design thought\" process could be made more effective with a more structured framework. As the reviewer insightfully suggests with reference [1], utilizing a state memory to track which state variables are used for the reward function would indeed be a more effective approach.\n\nInspired by this, we conducted a brief preliminary experiment with three conditions: (1) without \"design thought\" (**w/o thought**), (2) with our original self-generated \"design thought\" (**w/ self thought**), and (3) with a \"state-structured thought\" (**w/ state-structured thought**). The \"state-structured thought\" condition implements the Reward State Execution mechanism mentioned in [1]. In this setup, we track the historical usage of variables and prompt the LLM to generate a \"thought\" based on this auxiliary information, ensuring the thought specifies how the selected variables are to be combined.\n\n| Task          | w/o thought | w/ self thought | w/ state-structured thought |\n|:--------------|:-----------:|:--------------:|:--------------------------:|\n| ant           | 5.08(-18%)  |      6.22      |        6.43(+3%)         |\n| frankacabinet | 0.27(-23%)  |      0.35      |        0.46(+31%)        |\n\nThe experimental results show that the \"design thought\" process is necessary, as its absence degrades reward design performance. Furthermore, a more fine-grained thought process, as implemented in the \"state-structured thought\" condition, yields additional performance improvements. The Reward State Execution mechanism is indeed analogous to an external memory module in LLM agent design. We are grateful for the reviewer's profound insight and will add a discussion of the performance gains from this structured approach to our revised manuscript.\n\nRegarding CoT reasoning, our original \"self thought\" method is, in fact, an implementation of this process. As formalized in *Equation 1(after line 165)*, our method first generates a \"self thought\"—the high-level reasoning plan—before composing the corresponding reward function. Therefore, the performance improvement observed when comparing the w/ self thought and w/o thought conditions already demonstrates the effectiveness of CoT reasoning in our framework."}, {"review_id": "jaVO4kBLwF", "reviewer": "Reviewer_zhzV", "summary": "The paper presents RF-agent a method to generate (dense) reward functions using LLMs via sequential-decision making process based on  MCTS. While the contemporary approaches focus on LLM-based methods that rely on evolution-based methods or greedy-search based methods, this work proposes a tree based method that utilizes as MCTS and UCT to handle the exploration-exploitation process efficiently and generate better performing reward functions compared to the existing methods. \n\nThe LLM acts as a reward function generator that generates the reward function in python. There is another evaluation metric $F(\\cdot)$ that captures how well a policy $\\pi$ performs on a task, along with some feedback $l_{feedback}$ that indicates the feedback from executing the generated reward function on the task. The problem is to integrate this feedback in a sequential manner, so as to find better performing reward functions using the available actions (mutation, crossover, reasoning traces, etc.). This results in better ability to design dense reward functions from the sparse evaluation metric - compared to human designers or other baselines - in the low-level control simulation-based gym environments.", "questions": "As I mentioned in the weakness section, I'm mostly concerned about two things regarding the $l_{feedback}$: (1)  How well this method scales on OOD tasks and reward functions; and (2) any ablation or insights into it's design. I believe the work can be improved by addressing these concerns, and would convince me to change my score.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "# Strength\n- The paper is well written and easy to follow.\n-  Strong empirical results over the baselines that shows the promise of approaching this problem from the sequential decision-making lens.\n- Rigorous experimental procedure and evaluation . \n\n# Weakness\n- Some of the motivation for the problem setting is not clear. For instance, why can't the evaluation function/metric and the feedback be directly used for optimizing the policy here? The goal is to find the best performing policy so it's not clear why we need an additional step to map $F \\rightarrow R$, and then utilize the feedback for $R$ ? From the experiments, it seems the idea is to transform $F$ (spare reward) to $R$ (dense reward). It seems like both action space design and the feedback function design are the key component for the success of this method. The $l_\\{feedback}$ is the crucial component that makes the search process possible - the space of possible LLM generated reward function is infinite and only this signal guides the algorithm in the right direction. What are the assumptions here? How to design such feedback function is an important aspect which is overlooked in the current draft. \n\n- Another concern I have is that the methods/environments might already be in the training-set of the closed source LLMs like GPT, so there is a risk of data contamination. When you take this into account with the feedback prompt, maybe that can be a factor in the success of this method (as well as the baselines). How does this perform on OOD tasks or novel reward functions? \n\n- There is ablation for $A$, that demonstrates these actions are justified for the domains in this work, but the paper needs a more thorough discussion on how to design this to make the work more generalizable.", "was_revised": true, "final_justification": "After discussion with the authors, my concerns related to data contamination and $l_{feedback}$ design and ablations were sufficiently addressed, so I'm updating my score to reflect that.", "rebuttal": "# Response to Reviewer zhzV\nWe sincerely thank **Reviewer zhzV** for the insightful and detailed review. We particularly appreciate the acknowledgment that RF-Agent, through **rigorous experiments, provides promising support for solving this problem from a sequential decision-making perspective.** In response to the mentioned weaknesses and questions, we organize our detailed replies into the following sections:\n___\n\n## Question 1: OOD tasks performance\nThe reviewer's concern regarding generalization to novel tasks is indeed valid. To address this directly, we design and conduct two entirely new experiments using the Isaac Gym Ant environment:\n\n1.  **'Ant Lie Down'**: This task tests for *generalization to a novel objective*. The instruction provided to the LLM is:\n    > \"To make the ant lie down on its back (belly-up) as quickly and stably as possible. Its four feet should point towards the sky.\"\n\n2.  **'Ant Patrol and Return'**: This is a *compositional task* designed to test the LLM's ability to generate reward functions for unforeseen, multi-stage problems. The instruction is:\n    > \"To make the ant first run forward to the checkpoint position, then turn around and return to its starting origin position.\"\n\nThe **evaluation score** for each task is based on a success condition, defined by an indicator function `1[·]`.\n* For **'Ant Lie Down'**, success is `1[orientation_reward > 0.9]`, corresponding to a stable belly-up pose.\n* For **'Ant Patrol and Return'**, success is `1[task_phase == 1 and dist_to_start < 0.5]`, signifying the completion of the entire two-stage trip.\n\nUsing the same experimental configuration as in our main Isaac Gym experiments and employing the `GPT-4o-mini` model, we obtained the following results:\n\n| Task              | Sparse    | Eureka    | Revolve   | Ours      |\n|:------------------|:---------:|:---------:|:---------:|:---------:|\n| Lie Down          | 0.12±0.03 | 0.64±0.08 | 0.59±0.11 | 0.73±0.08 |\n| Patrol and Return | 0.05±0.02 | 0.52±0.07 | 0.63±0.05 | 0.82±0.06 |\n\nIn these experiments, the 'Sparse' condition refers to using only the evaluation score (i.e., the `1[...]` indicator function) as the reward signal. While it is not feasible to provide a handcrafted expert baseline for such novel tasks, the experimental results sufficiently demonstrate two key points:\n\n1. The paradigm of LLM-based reward generation is capable of generalizing to these specialized, novel tasks.\n2. Our RF-Agent continues to outperform the existing baseline methods on these new challenges.\n\nWe will add this section to the manuscript and thank you again for this insightful question.\n\n## Question 2: About Feedback Insight and Ablation\n### Part1：Illustration on Feedback\nRegarding `l_feedback`, in current design, it serves as auxiliary information that summarizes the dynamics of the variables within a given reward function. As detailed in Appendix B.2 (line 567), this feedback is composed of three parts:\n\n1. The change trend of each reward component over the training epochs.\n2. Their corresponding statistics (e.g., max, mean, min).\n3. Potential analytical suggestions.\n\nThis design is intended to provide a moderate, not excessive, amount of information. In our framework, *the ideal optimization signal is the final task success rate, `F`*. The `l_feedback` is treated as a secondary, auxiliary factor to help analyze the reward's behavior. This choice is motivated by *practical deployment considerations, where a full state feedback might not be available*. Consequently, we deliberately limit the auxiliary information to only include the dynamics of the reward components themselves.\n\nTo give a concrete example of how `l_feedback` influences the search process: *if a reward component exhibits very little variance during training, `l_feedback` suggests that this component might be ineffective*. This information then guides the node's action expansion. For instance, during a `crossover` operation between two high-performing reward functions (both with high `F`), the `l_feedback` allows the agent to identify and discard ineffective components from both parents. **By combining only the proven, meaningful components, the crossover is more likely to produce a superior child reward function.**\n\nTherefore, our current framework **places a greater emphasis on the design of the search `actions`** . We believe that well-designed `actions` are the primary mechanism for intelligently utilizing the insights from `l_feedback`. \n\n### Part2: Ablation on Feedback\nFurthermore, the current design of `l_feedback` component has been experimentally validated to demonstrate its effectiveness. We conducted a set of ablation studies on its constituent parts:\n\n* `w/o full feedback`: A baseline condition where `l_feedback` is removed entirely. The MCTS optimization, as well as the information provided to the search actions, relies solely on the final evaluation score, `F`.\n* `w/o nums`: In this ablation, we remove the part of the feedback corresponding to the numerical progression of the reward components over the training epochs.\n* `w/o statistics`: This condition removes the corresponding statistical summary (e.g., max, mean, min) of the reward components.\n* `w/o analysis tip`: This condition removes the analytical \"tip\" which is intended to help the LLM interpret the feedback.\n\nWe use the same ablation setting as in the original text and conducted the experiment, and the results are as follows:\n\n| Task          | Ours (w/ full feedback) | w/o full feedback | w/o nums   | w/o statistics | w/o analysis tip |\n|:--------------|:-----------------------:|:-----------------:|:----------:|:--------------:|:----------------:|\n| ant           |           6.22          |    5.09(-18%)     | 5.28(-15%) |   5.63(-9%)    |    5.47(-12%)    |\n| frankacabinet |           0.35          |    0.25(-28%)     | 0.32(-8%)  |   0.26(-25%)   |    0.31(-11%)    |\n\nThe ablation study demonstrates that removing the `l_feedback` component entirely causes a significant performance degradation. However, it is noteworthy that even without this detailed feedback, the combination of `MCTS` and `LLM` is robust enough to achieve reasonable performance by optimizing solely on the final evaluation score.\n\nFurthermore, the results indicate that **each of the three auxiliary information components** within our `feedback` design contributes positively to the final performance. \n\nWe will include a more in-depth discussion of these findings in our revised manuscript.\n\n## Question 3: About Reward Design Problem Motivation\n\nWe thank the reviewer for this insightful question, as it allows us to further elaborate on the motivation behind our **reward design problem setting**. This approach originates from the principles of reward engineering. The evaluation score, `F`, often serves as a sparse and monolithic signal. Using `F` directly as the reward `R` for policy learning is often inefficient and makes the learning process difficult, especially for complex manipulation and compositional tasks.\n\nThis challenge is particularly evident in our experimental results.\n* As shown by the `Sparse` results for `Bi-DexHands` in *Figure 3(after line 296)*, when a task only provides a binary 0/1 success signal at the end of an episode, a policy trained directly on this sparse reward can barely learn within the given step limit.\n* Similarly, as shown in *Table 1(after line 248)*, even for tasks like `Ant` and `Humanoid` that provide a dense metric `F` at every step, using `F` directly as the reward is still significantly less sample-efficient compared to using a shaped reward function.\n\nTherefore, the significance of our problem setting is to explore **how to better construct dense reward signals to efficiently guide policy learning**. From this perspective, a reward function is an interpretable, easily deployable, and modifiable way to provide this guidance. Furthermore, our method investigates how to leverage `LLMs` to generate these better reward functions. This approach enables automated optimization while **reducing the laborious manual effort** required from human experts in reward engineering.\n\n## Question 4: About Action Design Discussion\nThe design of our current `action` framework is intentionally high-level and conceptual. It does not operate on concrete environmental assets or objects (e.g., it does not propose adding a specific 'velocity penalty'). Instead, our action set can be applied to any reward function that exhibits fundamental characteristics, such as using a selection of environment observations, containing tunable hyperparameters, and being composed of multiple components. **This makes our framework inherently generalizable.**\n\nIn designing this framework, we integrated several key ideas: common heuristic operations (like `mutation` and `crossover`), an action that simulates human-like step-by-step thinking (`reasoning`), and an action that emphasizes exploration (`different thought`). Together, these form a comprehensive set of high-level, conceptual actions designed to effectively navigate the complex space of reward functions."}, {"review_id": "p03wdExhpM", "reviewer": "Reviewer_veGp", "summary": "The paper proposes RF-Agent, a novel framework for automated reward function design in low-level control tasks. The core innovation lies in treating large language models (LLMs) as language agents and framing reward function design as a sequential decision-making process enhanced by Monte Carlo Tree Search (MCTS). This approach aims to address the limitations of existing methods, such as poor utilization of historical feedback and inefficient search, which hinder performance in complex control tasks. Experimental evaluations across 17 tasks demonstrate that RF-Agent outperforms state-of-the-art LLM-based methods, achieving higher success rates and training efficiency.", "questions": "### 1. Token Consumption Comparison\n\nGiven that RF-Agent's MCTS process requires the LLM to generate more reward functions than baselines like Eureka and Revolve, have the authors compared the number of tokens generated by the LLM for each method? Understanding the token consumption is essential for evaluating the practicality and cost-effectiveness of RF-Agent, especially when using expensive LLM models.\n\n### 2. Comparison with Roska and Progress Function\n\nWould the authors consider comparing RF-Agent with methods like Roska and Progress Function? Roska's continuous learning and reward evolution during policy improvement, as well as Progress Function's use of LLMs to model task progress, present alternative approaches to reward design. Such comparisons could shed light on the strengths and weaknesses of RF-Agent in different scenarios.\n\n### 3. Component Ablation Details\n\nThe proposed method is comprised of many intricate designs for different purposes; however, they are not comprehensively analyzed. Could the authors elaborate on how each component, ranging from the score design to the action types, influences performance in different task domains? \n\n### 4. Thought-Align Mechanism Effectiveness\n\nThe thought-align mechanism aims to resolve inconsistencies between LLM-generated design thoughts and reward function code. Could the authors provide quantitative results on the effectiveness of this mechanism? For example, what percentage of reward functions show improved alignment after thought-align, and how does this impact training performance?", "limitations": "yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n### 1. Innovative Framework Combining LLMs and MCTS\n\nThe paper presents a compelling integration of LLMs and MCTS, treating reward function design as a sequential decision problem. This approach effectively leverages LLMs' contextual reasoning abilities and MCTS' search optimization to address the limitations of greedy or evolutionary algorithms used in previous work. The tree-based structure allows for systematic exploration and exploitation of reward function spaces, enhancing the utilization of historical information.\n\n### 2. Comprehensive Experimental Validation\n\nRF-Agent is evaluated across a diverse set of 17 low-level control tasks, covering locomotion and complex manipulation in IsaacGym and Bi-DexHands. The results show consistent superiority over baselines (Eureka, Revolve) in terms of success rates and training efficiency, even with lightweight LLM backbones like GPT-4o-mini. The quantitative evaluations, including average normalized scores and training curves, provide robust support for the method's effectiveness.\n\n### 3. Clear Visualization and Comparison\n\nThe paper includes informative figures (e.g., Fig. 1, Fig. 2) that effectively illustrate the sequential decision-making view, method comparisons, and RF-Agent's tree search process. These visualizations help readers understand the differences between RF-Agent and baselines, as well as the step-by-step optimization path of reward functions. The comparison tables (e.g., Table 1) clearly present performance metrics, making it easy to assess the method's advantages.\n\n### 4. Thoughtful Design of Action Types and Reasoning Mechanisms\n\nRF-Agent introduces a variety of action types (mutation, crossover, path reasoning, different thought) to guide LLM-generated reward functions, each serving a specific purpose in exploring or exploiting the search space. The self-verify and thought-align mechanisms help mitigate LLM hallucinations and ensure alignment between design thoughts and reward function implementations, enhancing the framework's reliability.\n\n## Weaknesses\n\n### 1. Incomplete Ablation Studies\n\nWhile the paper conducts ablation studies on the search method, action types, and reasoning components, the evaluation of some action types and their combinations is insufficient. For example, the specific impact of each action type (e.g., crossover vs. path reasoning) on different task types is not thoroughly analyzed. The ablation on \"reasoning\" only removes self-verify and thought alignment, but does not explore the individual contributions of each reasoning component.\n\n### 2. Lack of Comparison with Relevant Methods\n\nThe paper does not compare RF-Agent with methods like Roska [1], which uses continuous learning and reward function evolution within a policy improvement process, or Progress Function [2], which employs LLMs to model task completion progress as an intrinsic bonus. These comparisons would help situate RF-Agent within the broader landscape of reward function design and highlight its unique contributions.\n\n[1] Efficient Language-instructed Skill Acquisition via Reward-Policy Co-evolution\n\n[2] Automated Rewards via LLM-Generated Progress Functions\n\n### 3. Limited Analysis of Token Consumption and Compute Resources\n\nAlthough the paper mentions that RF-Agent uses slightly more storage and tokens than baselines, it does not provide detailed quantitative comparisons of token consumption or compute resources. Given that MCTS requires generating more reward functions (and thus more LLM calls) than greedy or evolutionary methods, understanding the trade-off between performance and computational cost is crucial for practical adoption.", "was_revised": true, "final_justification": "I believe this work features solid experimental validation, and the manuscript would be more comprehensive if updated with this new content. While it incurs higher LLM interaction overhead and lacks direct numerical comparisons with some related methods (e.g., Roska), I recognize its rightful contributions within its specific research branch. Therefore, I have decided to increase the score.", "rebuttal": "# Response to Reviewer veGp\nWe sincerely thank **Reviewer veGp** for the clear and detailed review. We particularly appreciate the positive feedback on our work, especially regarding the **innovative framework combining MCTS and LLMs** and the **clear figures and thorough experimental comparisons.** In response to the mentioned weaknesses and questions, we have organized our detailed replies into the following sections:\n\n---\n\n## Question 1: Token Consumption Comparison\nWe tallied the `input_token` (prompt), `output_token` (completion), and the actual monetary cost for `RF-Agent`, `Eureka`, and `Revolve` on the Ant and Humanoid tasks. It is important to note that `input_tokens` can be processed in a single, efficient forward pass (a 'prefill'), **whereas `output_tokens` are generated autoregressively (a 'decode' process), which is more computationally intensive**. This efficiency difference is reflected in API pricing (e.g., Input: `$0.6 per 1M tokens` vs. Output: `$2.4 per 1M tokens`).\n\n|  Ant         | Prefill Tokens | Decode Tokens | Total Cost ($) |\n|:----------------|---------------:|--------------:|---------------:|\n| Eureka          | 82458          | 54505         | 0.1802868      |\n| Revolve         | 162348         | 43973         | 0.202944       |\n| **Ours-total** | **257564** | **69704** | **0.321828** |\n| &nbsp; ↳ Ours-action | 203282         | 53462         | -              |\n| &nbsp; ↳ Ours-others | 54282          | 16242         | -              |\n\n\n|  Humanoid         | Prefill Tokens | Decode Tokens | Total Cost ($) |\n|:----------------|---------------:|--------------:|---------------:|\n| Eureka          | 78940          | 60021         | 0.191414       |\n| Revolve         | 169545         | 44118         | 0.20761        |\n| **Ours-total** | **254555** | **66557** | **0.31247** |\n| &nbsp; ↳ Ours-action | 204283         | 50270         | -              |\n| &nbsp; ↳ Ours-others | 50272          | 16287         | -              |\n\nThe data in the tables leads to the following conclusions:\n\n1.  `RF-Agent` utilizes **2-3 times more `prefill` tokens** than the baselines, while the number of `decode` tokens is only moderately higher (approx. **1.2-1.4 times**). Given that the `prefill` stage is computationally cheaper than the `decode` stage, the overall increase in resource consumption, as reflected by the total cost, is moderate and remains within an acceptable range.\n\n2.  Furthermore, we find that this asymmetric token consumption aligns perfectly with the design philosophy of `RF-Agent`. The higher usage of `prefill` tokens is a direct result of leveraging historical information (i.e., previously generated thoughts and reward functions). As the MCTS tree grows, these artifacts are flexibly combined and included in the prompt, forming a **multi-step reasoning process** for the LLM. This process enables the LLM to generate more sophisticated and effective reward functions, which in turn leads to the **significant performance growth** observed in our experiments.\n\n## Question 2: Comparison with Roska and Progress Function(LLMCount)\nWe thank the reviewer for suggesting these relevant works. We will cite and discuss the differences between these methods and our own in the revised manuscript.\n\nDue to the time constraints and the lack of publicly method code for reproduction, our comparison is presented in a **table** based on the differences and the results reported in their respective papers. For `ROSKA`, we noted that the number of training steps used for the Isaac Gym tasks differed from our setup. We retrained the reward function under the same steps for comparison.\n\n| Category   | Metric     | Eureka       | Revolve    | LLMCount[2]      | ROSKA[1]       | (Ours) RF-Agent         |\n|:----------------------|:------------------------|:-------------:|:---------------:|:----------------:|:--------------------------------|:--------------------------------------------------------------|\n| **knowledge requirement** | Environment Observation | Yes     | Yes      | Yes   | Yes         | Yes    |\n|                       | Additional Library   | No            | No     | Yes    | No     | No              |\n| **optimization method** | has optimization?       | Yes     | Yes     | No       | Yes       | Yes     |\n|                       | optimization method     | Greedy based  | Evolution based | -       | Greedy based      | exploration and exploitation tree search      |\n|                       | historical utilization  | Nearly Single | Population    | -       | implicitly with policy fusion | explicitly with diverse action by reasoning to llm |\n| **Performance** | normal locomotion     | Standard      | Medium  | No test in paper | Strong     | Stronger   |\n|            | normal manipulation    | Standard      | Medium     | No test in paper | Stronger    | Strong    |\n|          | complex manipulation   | Standard      | Medium  | Medium  | No test in paper   | Strong  |\n\nBased on the reviewer's valuable suggestions and our further analysis, we believe the following points are promising directions for future exploration:\n\n1.  `LLMCount` focuses on achieving competitive results non-iteratively through reward transformation. This makes it highly efficient under very limited sample budgets, but it relies on the construction of external knowledge. For iterative optimization processes like ours, we believe the approach from `LLMCount` could serve as an **excellent initialization method**.\n\n2.  Regarding the use of historical information, `ROSKA` **implicitly** fuses it at the **policy level** to assist its evolutionary process. In contrast, our method **explicitly** leverages history at the **LLM level**, using a rich set of actions and tree search to construct new reward functions. Each approach has its own advantages and is driven by different motivations. Our work, in particular, has been validated with high performance in more complex manipulation environments. **Future research could potentially combine these two distinct optimization paradigms.**\n\n## Q3 & Q4: About Detailed Ablation\nWe thank the reviewer for the valuable suggestion to conduct a more detailed ablation study. Given the rebuttal time constraints , we  include several key ablations as follows.\n\n### Part1: More Ablation on Feedback\nDue to the rebuttal character limits, please check our reply to R2 (zhzV), located in part2 of Q2 for detailed ablation part of the feedback.\n\n### Part2: More Ablation on Action Combination\n\nWe further conducted a combinatorial study of action, which supplemented the ablation in the paper(line 324):\n\n| Action Combination                      | Ant         | Allegro Hand |\n|:----------------------------------------|:-----------:|:------------:|\n| a w/ `mutation`                           | 5.13(-18%)  | 18.76(-16%)  |\n| b w/ `mutation` + `crossover`             | 5.77(-7%)   | 21.90(-3%)   |\n| c w/ `mutation` + `reasoning`             | 5.94(-4%)   | 21.26(-5%)   |\n| d w/ `crossover` + `reasoning`            | 4.85(-22%)  | 19.21(-15%)  |\n| e w/ `mutation` + `crossover` + `reasoning` | 6.08(-2%)   | 22.19(-2%)   |\n| **Full Version (w/ all actions)** | **6.22** | **22.52** |\n\nBy analyzing the ablation results, we can categorize our actions from the perspective of information utilization. The `mutation` action can be seen as a *local* operator, as it modifies a single parent node. In contrast, `crossover` and `reasoning` act as *global* operators, as they synthesize information from multiple nodes or entire ancestral paths.\n\nThe results clearly indicate that relying exclusively on one type of operator leads to a significant performance drop. For instance, using only a *local* action (w/ `mutation`) or only *global* actions (w/ `crossover` + `reasoning`) both result in considerable performance degradation. Therefore, a balanced combination of both local and global operators is crucial for achieving the best performance.\n\n### Part3: More Ablation on Thought-Align\n\nFinally, we present an ablation study on the `thought align` mechanism. In our current design, this process consists of two steps:\n\n1.  **First Design Thought**: Following a Chain-of-Thought (CoT) like process, the LLM first generates a high-level 'design thought' *before* composing the actual reward function code.\n2.  **Second Thought Align**: To mitigate potential hallucinations from a single generation step, after the reward function is generated, we prompt the LLM to re-examine the code's content and then revise the original 'design thought' to ensure it accurately reflects the code's logic.\n\nTo quantitatively validate the effectiveness of this mechanism, we conducted further ablations on these intermediate components:\n\n| Task         | w/ full thought align | w/o first design thought | w/o second thought align | w/o all    |\n|:-------------|:---------------------:|:------------------------:|:------------------------:|:----------:|\n| ant          |          6.22         |       5.08(-18%)         |        5.94(-4%)         | 4.96(-20%) |\n| allegro hand |         22.52         |       18.07(-19%)        |       18.55(-17%)        | 14.64(-35%)|\n\nThe results indicate that the `design thought` mechanism is crucial across all tasks. Notably, the **second `thought align` step has a particularly significant impact** on the more complex manipulation task (Allegro Hand).\n\nRegarding a direct percentage validation of this alignment's impact, it is challenging to isolate. This is because the second step corrects the thought, not the reward function code itself. The benefit of this corrected thought is realized later, when it is used to generate subsequent reward functions for child nodes. However, we are confident that this alignment process positively influences the quality of future rewards generated during the RF-Agent tree search, **as empirically demonstrated by the overall performance gains shown in the table above.**"}, {"review_id": "aB79a0ZLmz", "reviewer": "Reviewer_qnw7", "summary": "This paper proposes RF-Agent, an LLM-based framework for automated reward function design using  Monte Carlo Tree Search.\nIt treats reward function design as a sequence of mutations, crossovers, etc., and uses LLM to modify the Python code for the reward function, allowing for iterative refinement based on training feedback, task descriptions, and generated design ideas.\nExperiments on IsaacGym and Bi-DexHands benchmarks show improved performance over prior LLM-based baselines (Eureka, Revolve).", "questions": "- l. 50--52: Is there evidence or references showing it's because of exploration and exploitation, not other reasons?\n- l. 112: $\\mathcal{R}$ is the set of $[S \\times A, \\mathbb{R}]$ of all real-valued functions over $S \\times A$, right?\n- l. 113: I understand $\\pi: S \\to \\Delta A$ is a measurable function to the measurable space $\\Delta A$ of all probability measures over $A$, but some readers may be unfamiliar with this notation.\n- l. 114: $\\Pi$ is not defined.\n- l. 114: $F: \\Pi \\to \\mathbb{R}$ is not limited to the (discounted) sum of rewards, right?\n- l. 115: Is the discounting factor specified?\n- l. 117--118: finding $R \\in \\mathcal{R}$, s.t. $\\mathscr{A}\\_M(R) = \\arg\\max\\_{\\pi \\in \\Pi}F(\\pi)$?\n- l. 120: $\\mathcal{L}$ is not defined.\n- l. 121: If we never need to train the language model, I think we can safely drop $\\theta$ or use a more informative name.\n- l. 124--125: How are $l_{feedback}$ and $l_{task}$ obtained?\n- l. 125--126: $\\max\\_{l \\in \\mathcal{L}} F(\\mathscr{A}\\_M(p_\\theta(l)))$ is weaker than $\\max\\_{R \\in \\mathcal{R}} F(\\mathscr{A}\\_M(R))$, which is weaker than finding $R \\in \\mathcal{R}$, s.t. $\\mathscr{A}\\_M(R) = \\arg\\max\\_{\\pi \\in \\Pi}F(\\pi)$. There's a gap at each step. More rigorously, I think we should define the sequential decision-making problem formally.\n- Eq. (1): I can guess $R \\sim p\\_\\theta(x, z), z \\sim p\\_\\theta(x)$ means using an LLM twice, but the notation is confusing.\n- l. 205 and l.218: What does $\\sim$ in $s_{node \\sim \\{elites\\}}$ and $s_{node \\sim \\{tree\\}}$ mean?\n- Appendix I: maybe the author should show diff instead of just pasting all steps.", "limitations": "The author discussed the need for multiple interactions with the LLM and multiple RL training iterations.\nHowever, I believe that the lack of evidence regarding generalization ability in novel or compositional tasks is a more significant limitation.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "# Strengths\n\n- Automated reward function design is an important problem.\n- As far as I know, framing reward design as MCTS over LLM decisions is novel.\n- The proposed method showed strong empirical results.\n- The paper carefully evaluates contributions of different components (search, actions, self-verification).\n\n# Weaknesses\n\n## No test of generalization\n\nAll tasks are from predefined benchmarks with standard objectives (e.g., Ant: run forward). It's highly possible that there already exists manually designed reward functions in the training set of LLMs: Models such as GPT-4o and GPT-4o-mini have been trained on public code repositories and benchmarks, which very likely include environments like IsaacGym and Bi-DexHands, as well as papers, GitHub repos, and documentation that include reward functions for these tasks. There's no evidence that the proposed method works on novel or compositional tasks (e.g., Ant: run in a circle then reverse). Without such tests, we can't claim the method understands task descriptions in a generative or generalizable sense.\n\nIn my opinion, what RF-Agent demonstrates is search efficiency: it can find or tune reward functions more effectively than greedy (Eureka) or evolutionary (Revolve) baselines, but possibly only within the space the LLM already has internalized. It may be an efficient memory probe, not a general reward function generator.\n\n## Limited understanding of feedback\n\nThe system feeds raw numerical logs $l_{feedback}$ into the LLM in string form. It assumes LLMs can reason over these logs to improve reward design, but this ability is unvalidated and possibly fragile. In my understanding, LLMs are not designed to do numerical trend analysis on raw sequences like a statistics model or a signal processor.\n\n## Computational cost\n\nLike other reward optimization methods, RF-Agent requires repeated LLM calls and full RL training loops. However, no analysis is provided on runtime, cost, or scalability, which is especially relevant given the use of GPT-4o.\nI may have missed some information, but it seems that this paper only stated\n\n> Overall, RF-Agent uses slightly more storage space and token consumption, but experimental results show that this slight sacrifice is worth it.\n\nbut gave no quantitative comparison.", "was_revised": true, "final_justification": "In the rebuttal, the author provided some empirical evidence regarding generalization ability.\nIf the results are valid, they demonstrate that the proposed method generalizes better than the baselines.\nThis design may be of interest to some readers in the RL community.", "rebuttal": "# Response to Reviewer qnw7\nWe sincerely thank **Reviewer qnw7** for the detailed and comprehensive review. We particularly appreciate the acknowledgment that **framing reward design as MCTS over LLM decisions is novel** and for recognizing our **strong empirical results.** In response to the mentioned weaknesses and questions, we have organized our detailed replies into the following sections:\n\n---\n\n## Question 1: OOD tasks performance\nThe reviewer's concern regarding generalization to novel tasks is indeed valid. To address this directly, we design and conduct two entirely new experiments using the Isaac Gym Ant environment:\n\n1.  **'Ant Lie Down'**: This task tests for *generalization to a novel objective*. The instruction provided to the LLM is:\n    > \"To make the ant lie down on its back (belly-up) as quickly and stably as possible. Its four feet should point towards the sky.\"\n\n2.  **'Ant Patrol and Return'**: This is a *compositional task* designed to test the LLM's ability to generate reward functions for unforeseen, multi-stage problems. The instruction is:\n    > \"To make the ant first run forward to the checkpoint position, then turn around and return to its starting origin position.\"\n\nThe **evaluation score** for each task is based on a success condition, defined by an indicator function `1[·]`.\n* For **'Ant Lie Down'**, success is `1[orientation_reward > 0.9]`, corresponding to a stable belly-up pose.\n* For **'Ant Patrol and Return'**, success is `1[task_phase == 1 and dist_to_start < 0.5]`, signifying the completion of the entire two-stage trip.\n\nUsing the same experimental configuration as in our main Isaac Gym experiments and employing the `GPT-4o-mini` model, we obtained the following results:\n\n| Task              | Sparse    | Eureka    | Revolve   | Ours      |\n|:------------------|:---------:|:---------:|:---------:|:---------:|\n| Lie Down          | 0.12±0.03 | 0.64±0.08 | 0.59±0.11 | 0.73±0.08 |\n| Patrol and Return | 0.05±0.02 | 0.52±0.07 | 0.63±0.05 | 0.82±0.06 |\n\nIn these experiments, the 'Sparse' condition refers to using only the evaluation score as the reward signal. While it is not feasible to provide a handcrafted expert baseline for such novel tasks, the experimental results sufficiently demonstrate two key points:\n\n1. The paradigm of LLM-based reward generation is capable of generalizing to these specialized, novel tasks.\n2. Our RF-Agent continues to outperform the existing baseline methods on these new challenges.\n\nWe will add this section to the manuscript and thank you again for this insightful question.\n\n## Question 2: About Feedback Insight and Ablation\nWe thank the reviewer for this insightful question, which highlights a critical and valid concern about the potential fragility of an LLM's numerical reasoning capabilities.\n\nWe completely agree that LLMs are not inherently designed as numerical signal processors. \nAs detailed in Appendix B.2 (line 567), this feedback is composed of three elements: nums, correspongding statistics, and analysis tip.\nWe conducted the following ablation studies to verify the feedback effectiveness:\n\n* `w/o full feedback`: The baseline where `l_feedback` is entirely removed, and the search relies solely on the final success score, `F`.\n* `w/o nums`: Removes the numerical progression from the feedback.\n* `w/o statistics`: Removes the pre-computed statistical summary.\n* `w/o analysis tip`: Removes the analytical heuristic/tip.\n\nThe results are as follows:\n\n| Task    | Ours (w/ full feedback) | w/o full feedback | w/o nums   | w/o statistics | w/o analysis tip |\n|:--------------|:-----------------------:|:-----------------:|:----------:|:--------------:|:----------------:|\n| ant   |   6.22   |  5.09(-18%)   | 5.28(-15%) |  5.63(-9%) |  5.47(-12%)  |\n| frankacabinet |  0.35  |  0.25(-28%)     | 0.32(-8%)  |   0.26(-25%)  |  0.31(-11%)  |\n\nRemoving `l_feedback` entirely (`w/o full feedback`) causes a significant performance drop. More importantly, the ablations show that removing either the **pre-computed statistics** (`w/o statistics`) or the **analytical tip** (`w/o analysis tip`) individually degrades performance.\n\nWe speculate that `analysis tip` helps LLM to a certain extent how to analyze sequences and further exert effects in subsequent diver actions. More refined optimizations can still be made regarding feedback, which can be discussed in depth in subsequent research, thank you again for your insights.\n\n## Question 3: Computational Cost\nWe tallied the `input_token` (prompt), `output_token` (completion), and the actual monetary cost for `RF-Agent`, `Eureka`, and `Revolve` on the Ant and Humanoid tasks. It is important to note that `input_tokens` can be processed in a single, efficient forward pass (a 'prefill'), **whereas `output_tokens` are generated autoregressively (a 'decode' process), which is more computationally intensive**. This efficiency difference is reflected in API pricing (e.g., Input: `$0.6 per 1M tokens` vs. Output: `$2.4 per 1M tokens`).\n\n| Ant          | Prefill Tokens | Decode Tokens | Total Cost ($) |\n|:----------------|---------------:|--------------:|---------------:|\n| Eureka   | 82458  | 54505  | 0.1802868|\n| Revolve   | 162348   | 43973   | 0.202944  |\n| **Ours-total** | **257564** | **69704** | **0.321828** |\n| &nbsp; ↳ Ours-action | 203282    | 53462   | -    |\n| &nbsp; ↳ Ours-others | 54282  | 16242  | -    |\n\n\n| Humaniod          | Prefill Tokens | Decode Tokens | Total Cost ($) |\n|:----------------|---------------:|--------------:|---------------:|\n| Eureka   | 78940    | 60021  | 0.191414   |\n| Revolve   | 169545   | 44118   | 0.20761 |\n| **Ours-total** | **254555** | **66557** | **0.31247** |\n| &nbsp; ↳ Ours-action | 204283   | 50270  | -     |\n| &nbsp; ↳ Ours-others | 50272  | 16287  | -    |\n\nThe data in the tables leads to the following conclusions:\n\n1.  `RF-Agent` utilizes **2-3 times more `prefill` tokens** than the baselines, while the number of `decode` tokens is only moderately higher (approx. **1.2-1.4 times**). Given that the `prefill` stage is computationally cheaper than the `decode` stage, the overall increase in resource consumption, as reflected by the total cost, is moderate and remains within an acceptable range.\n\n2.  Furthermore, we find that this asymmetric token consumption aligns perfectly with the design philosophy of `RF-Agent`. The higher usage of `prefill` tokens is a direct result of leveraging historical information (i.e., previously generated thoughts and reward functions). As the MCTS tree grows, these artifacts are flexibly combined and included in the prompt, forming a **multi-step reasoning process** for the LLM. This process enables the LLM to generate more sophisticated and effective reward functions, which in turn leads to the **significant performance growth** observed in our experiments.\n\n\n## Other Questions\n**Re l. 50-52:** On one hand, our ablation studies demonstrate the importance of balancing exploration and exploitation. On the other hand, a key reference is \"The 2014 General Video Game Playing Competition,\" a benchmark for complex decision-making games. In this article, methods based on an exploration-exploitation balance, such as MCTS, consistently achieved top ranks (e.g., 1st, 2nd, 3rd), while approaches like genetic and greedy algorithms ranked significantly lower (e.g., 8th, 9th, 12th).\n\n**Re l. 112:** Yes. To maintain consistency with prior work, we assume the reward `R` is a function of the current state and action (`S x A`).\n\n**Re l. 113:** Thank you for the suggestion. We will change the notation for the action space to **A** for better readability.\n\n**Re l. 114 (π):** We will clarify in the text that `π` is a policy sampled from the policy space `Π`.\n\n**Re l. 114 (F):** Yes. `F` is a general evaluation score. It can represent the total sum of rewards or, for long-horizon tasks, a final binary signal (0/1) indicating task success.\n\n**Re l. 115:** Due to space constraints, we did not detail the full MDP definition in the main text. The discount factor `γ` is indeed part of the standard MDP formulation. We will add a detailed explanation to the appendix in the revised version.\n\n**Re l. 117-118:** Your understanding of the formulation is correct.\n\n**Re l. 120:** Thank you. We will add a definition clarifying that `ℒ` is the space of possible language sequences.\n\n**Re l. 121:** Thank you. We chose to retain the parameter `θ` to explicitly represent that different LLMs could be used in our framework, but we agree that it can be simplified for clarity.\n\n**Re l. 124-125:** `l_feedback` consists of a predefined text template combined with numerical values returned during the training process, as detailed in Appendix B.2 (line 567). `l_task` is determined by the benchmark itself, such as the brief text descriptions provided for each task in the Isaac Gym repository.\n\n**Re l. 125-126:** This is a very thoughtful point. In the revised manuscript, we will more formally define the high-level decision-making problem to ensure it strictly aligns with our proposed solution process.\n\n**Re Eq. (1):** This notation describes a single generation process where the LLM's response contains both the thought `z` and the reward function `R`. The model first decodes `z`, and then subsequently decodes `R`.\n\n**Re l. 205 and l. 218:** These terms refer to sampling nodes from the elite set and from the current MCTS tree, respectively. The information from these sampled nodes is then used for the next generation step.\n\n**Re Appendix I:** Thank you for this suggestion. For readability, we initially provided a one-sentence summary comparing the reward functions. We acknowledge that this may lack sufficient detail. In the revised manuscript, we will highlight the specific differences in the code using color."}]

Author rebuttal:

Meta-review: This works aims to find a reward function that can help RL agents optimize for a desired (sparse) goal. To do this, authors make use of LLMs to perform tree search in the reward space. Specifically, output of each node is a python program that specifies a reward function. The feedback obtained for each node corresponds to performance of training an agent with the respective reward function (and some additional stats). This leverages the long-memory and optimization capabilities of LLMs to sequentially update the reward function.

All the reviewers appreciated the novelty of the approach and the exhaustive empirical analysis. Some questions were raised pertaining to the cost and compute time, and also regarding ensuring a fair evaluation since the RL environments might have been already there in the LLM training data. Authors did a commendable job of providing all the details and running additional OOD experiments to affirm the utility of their approach.

Overall, the proposed method is novel, looks at an important problem, and the empirical analysis is thorough.
Decision: Accept (spotlight)