Uncertainty-aware Reward Design Process
Authors: Yang yang
Xiaolu Zhou
Bosong Ding
Miao Xin
Venue: Accepted by TMLR ()

Abstract: Designing effective reward functions is a cornerstone of reinforcement learning (RL), yet it remains a challenging process due to the inefficiencies and inconsistencies inherent in conventional reward engineering methodologies. Recent advances have explored leveraging large language models (LLMs) to automate the design of reward functions. However, LLMs’ insufficient numerical optimization capabilities often result in suboptimal reward hyperparameter tuning, while non-selective validation of candidate reward functions leads to substantial computational overhead. To address these challenges, we propose the Uncertainty-aware Reward Design Process (URDP), a novel framework that integrates large language models to streamline reward function design and evaluation. URDP quantifies candidate reward function uncertainty based on the self-consistency analysis, enabling simulation-free identification of ineffective reward components while discovering novel ones. Furthermore, we introduce uncertainty-aware Bayesian optimization (UABO), which incorporates uncertainty estimation to improve the hyperparameter configuration. Finally, we construct a bi-level optimization framework by decoupling the reward component optimization and the hyperparameter tuning. URDP promotes the collaboration between the reward logic reasoning of the LLMs and the numerical optimization strengths of the Bayesian optimization. We conduct a comprehensive evaluation of URDP across 35 diverse tasks spanning three benchmark environments: IsaacGym, Bidexterous Manipulation, and ManiSkill2. Our experimental results demonstrate that URDP not only generates higher-quality reward functions but also achieves significant improvements in the efficiency of automated reward design compared to existing approaches. We open-source all code at https://github.com/Yy12136/URDP.

Published in Transactions on Machine Learning Research (10/2025)
Uncertainty-aware Reward Design Process
Yang Yang
yungyangg02500amm National Key Laboratory of Cognition and Decision Intelligence for Complez Systems, Institute of Automation, Chinese Academy of Science
Xiaolu Zhou
20232113301055maill bnu. edu.cn
School of Mathemationl Sciences, Beijing Normal University
Bosong Ding
B.Ding 3@ttiburnunkuerrsty..dd edu
Air-Lab, Tilburg University
Miao Xin
miao. Pia ac. cn
National Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Science
Reviewed on OpenReview: /ptp:://operreveeee net/ forun? id- CId5tW1HaR
Abstract
Designing effective reward functions is a cornerstone of reinforcement learning (RL), yet remains a challenging process due to the inefficiencies and inconsistencies inherent in conventional reward engineering methodologies. Recent advances have explored leveraging large language models (LLMs) to automate the design of reward functions. However, LLMs' insufficient numerical optimization capabilities often result in suboptimal reward hyperparam- eter turing, while non-selective validation of candidate reward functions leads to substantial computational overhead. To address these challenges, we propose the Uncertainty-awarr Reward Design Process (URDP), , a novel framework that integrates large language models to streamline reward function design and evaluation. URDP quantifies candidate reward function uncertainty based the self-consistency analysis, enabling simulation-free identification of ineffective reward components while discovering novel ones. Furthermore, we introduce uncerlainty-awarr Bayesian optimization (UABO), which incorporates uncertainty estimation to improve the hyperparameter configuration. Finally, we construct a bi-level optimization framework by decoupling the reward component optimization and the hyperparamcter tuning. URDP promotes the collaboration between the reward logic reasoning of the LLMs and the numerical optimization strengths of the Bayesian optimization. We conduct a comprehensive evaluation of URDP across 35 diverse tasks spanning three benchmark environments: Isaac- Gym, Bidexterous Manipulation, and ManiSkill2. Our experimental results demonstrate that URDP not only generates higher-quality reward functions but also achieves significant improvements in the efficiency of automated reward design compared to existing approaches. We open-source all code at /rttps://gitain www.hhuu..o.../caaa/ com com/YYY1136UURDP
1 Introduction
In reinforcement learning (RL), the design of reward functions serves as pivotal determinant for successfully training agents in soquential decision-making tasks. These rewards guide the learning process by shaping agent behaviors accomplish complex objectives across diverse environments. While conventional approaches such reward engineering and inverse reinforcement learning (IRL) (Arora & Doshi 2021) established early research paradigms, they remain fundamentally constrained by their reliance on human expertise and the
"Corresponding authors.

---

Published in Transactions on Machine Learning Research (10/2025)
availability of high-quality demonstration data, particularly in domains like robotic skill acquisition (Zitkovich
et al. 2023). Recent advancements in large language models (LLMs) (Radford et al. 2019 Brown et al., Liu et al. 2024) have demonstrated remarkable capabilities in natural language understanding, code generation, and contextual optimization, thereby introducing a novel paradigm for automated reward function design.
2020
However, current automated reward design methodologies based on LLMs present two fundamental chal- lenges (Cao et al.) 2024). First, the efficiency of reward function design remains suboptimal. Existing approaches rely heavily simulation-based training processes (Ma et al. 2024a) to evaluate reward function efficacy, which involves extensive and often redundant evaluations, leading to significant computational overhead. Second, the performance of LLM-generated reward functions frequently falls short of expectations. During the optimization process, LLMs fail to fully leverage their reasoning capabilities, resulting in reward functions that inadequately capture the intended task objectives. Given that the design efficiency and the performance of reward functions are closely tied to the speed and effectiveness of policy learning, a key research question emerges: How can we enhance the efficiency of oblaining performant reward functions?
In this paper, we introduce the Uncertainty-aware Reward Design Process (URDP), ) novel framework for automated reward function generation. First, we propose method to quantify the uncertainty of generated samples, which enables the selective elimination of redundant reward function sampling and simulation. This approach is grounded in a key observation regarding consistesy (Wang et al. 2022): : LLMs exhibit higher output consistency when handling well-defined tasks, allowing for more efficient sampling strategies. Second, we identify a critical limitation in current LLM-only evolutionary search approaches, i.e., their suboptimal performance in numerical optimization. To address this, we decouple reward component formulation from reward intensity optimization, delegating the latter to a dedicated numerical optimization module. Specifically, we propose a novel uncertainty- Bayesian optimization (Snock et al. 2012) approach incorporating uncertainty distribution priors, which significantly accelerates convergence in this black-box optimization task. Experimental results demonstrate that URDP surpasses state-of-tho-art methods in both reward performance and computational efficiency, establishing a new benchmark for automated reward design.
Overall, our contributions are summarized as follows: reward function design in reinforcement learning. The framework employs an alternating bi-level optimization process that decouples reward component design from hyperparameter oplinization, thereby significantly enhancing reward function performance. (2) We introduce a self- caaff-oonisteec-bbeeee based reward uncertainty quantification method for the reward component design process. This approach significantly improves the efficiency of the reward component validation. (3) We present an Uncertainty-awan -warr Bayesian optimization algorithm that substantially increases the efficiency of hyperparameter search. (4) Our comprehensive evaluation across 35 tasks spanning 3 distinct benchmarks demoustrates that URDP consistently outperforms existing methods in both reward function generation efficiency and final policy performance, evidenced by rigorous quantitative analysis.
(1) We propose URDP, a novel framework for automated
2 Related work
Reward code generation. The automation of reward code generation has been a critical area of research, aiming to simplify and improve the process of defining task-specific reward functions for RL. As the dual formulation of RL problems, inverse reinforcement learning (IRL) methods (Arora & Doshi 2021 have been extensively investigated for reward function acquisition. However, these approaches are fundamentally limited by their dependence on demonstration data, which severely constrains their scalability in practical applications. Recently, LLM-based reward design methodologies (Kwon & Michael 2023 Yu et al. 2023
Ma et al. 2024a Xie et al. 2024 have demonstrated promising potential, offering paradigm shift in automated reward function development. L2R YYu et al. 2023) introduced a two-stage LLM-prompting framework to generate templated rewards, bridging high-level language instructions with low-level robot, actions. Eureka (Ma et al.) 2024a) leveraged the zero-shot and in-context learning capabilities of advanced LLMs perform evolutionary optimization over the reward code. This method demonstrated the potential of LLMs to generate rewards without task-specific prompting or predefined templates, enabling agents to acquire complex skills via RT. Rexa2Reward (Xie et al. 2024 extended this line of work by generating shaped,
2

---

Published in Transactions on Machine Learning Research (10/2025)
dense reward functions as executable programs grounded in compact environment representations. Unlike sparse reward codes or constant reward functions, Text2Reward produces interpretable, dense reward codes capable of iterative refinement with human feedback. Despite their success, these methods generally do not separate the tasks of defining reward components and tuning paramcters, which may reduce the efficiency of aulomated reward design. Our proposed framework decouples diverse factors in reward design reduce invalid tuning. It. incorporates uncertainty quantification to guide LLMs toward more focused analysis and refinement of reward logic relevance.
Hybrid optimization, Recent advances in large language models (LLMs) (OpenAl 2023 Liu et al. 2024
have demonstrated remarkable progress in text-based complex reasoning tasks (Li et al. 2025). Through techniques such as self-improvement. (Song et al.) 2023). multi-path reasoning (Wan et al. 2024) and reward modeling (Zhong et al.) 2025), LLMs exhibit substantial potential in contextual comprehension (OpenAl 2023), code generation Yu et al. 2024) and task planning (Hao al.) 2023). However, their capabilities in deep logical reasoning (Cheng et al. 2025), particularly in mathematical and numerical optimization domains (Yan al.] 2025), remain underexplored, with significant performance gaps persisting. Several studies attempt to employ LLMs meta-optimizers for diverse optimization problems (Yang et al.). Yet, due to their inherently discrete representation nature, LLMs' effectiveness in high-dimensional, continuous mumerical reasoning tasks requires further investigation (Assran et al. 2025). In reinforcement learning, the reward design inherently involves multiple types of optimization problems. Unlike completely LLMM-nly evolutionary search approaches (Ma et al. 2024b Xie et al. 2024) that utilize LLM- evolutionary search to optimize all elements in the reward function, URDP only applies to the optimization of reward components, while black-box numerical optimization tool is introduced to compensate for the limitations of LLMs in contimuous numerical optimization.
Uncertainty quantification. Uncertainty quantification (UQ) serves fundamental component for reliable automated decision-nin making and has been extensively studied in domains such as Bayesian inference (Gal & Ghahramani 2015 Foong et al. 2020). Recent advances have investigated UQ in black-box language models (Liu et al. 2025 Geng el. al. 2024 Kuhn el. al. 2023 Lin el al.), yielding various approaches
including token-level entropy methods (Kadavath et al.] 2022), conformal prediction-based techniques (Su ct al. 2024), and consistency-basee
frameworks (Lin et al.. While existing methods primarily leverage uncertainty estimation to enhance LLM interpretability (Ahdritz et al. 2024) and mitigate hallucination risks (Shorinwa et al. 2024 Mohri & Hashimoto 2024), our methodology not only actively quantifies the uncertainty in LLMs but also strategically leverages this uncertainty as the foundational mechanism for both reward function design and efficient numerical optimization. Furthermore, we identify and characterize a significant. correlation between the novel reward component and its uncertainty.
3 Preliminary: problem setup and notations
Reinforcement learning (RL) tasks can be modeled as Markov Decision Processes (MDPs) defined by the tuple P,P, where S is the state space, A is the action space, R. is the space of reward functions, R E R is the reward function, and is the discount factor. P(s++1|s, ax) P is the transition probability of moving to state 8.+1 given the current state and action a,, where P is the transition probability space. The agent can generate a trajectory of state and action T= (So, ao, $1, a.....) by executing a policy (aa)ss) in the policy space II. The goal is to find the optimal policy that maximizos the sum of expected returns:
T
J(x) = ETAN
Σ t=0
7 R(s at)
(1)
Following the previous works (Song et al. represented as a function of the reward components (r) and the reward intensities (0)
2023 Ma et al. 2024a Xie et al. 2024), the reward function R. is
R(s, a) la(r, 0, 0,,,a),
(2)
where the function h and the reward components are represented in code form. Here, the reward components are defined as the composable and interpretable sub-terms that constitute the overall reward
3

---

Published in Transactions on Machine Learning Research (10/2025)
function Escontrela et al. (2022). each measuring a specific behavioral aspect task objective. For instance, distance reward measures the agent's proximity to the target, while balocity_rewarcr quantifies the alignment of its velocity toward the goal direction. These components are implemented as modular units, enabling both structural manipulation and numerical tuning during the optimization process (sec the highlighled red text in the example of Appendix D.1). The reward intensily refers to the weight. of a reward component within the total reward. Designing reward functions represented in code form is critical to aligning agent behavior with task objectives. Here, the reward design problem is defined as follows.
Definition 1: Reward design problem (RDP) (Singh et al. 2009). A reward design problem is represented as RDP- (M, R2 TM, F), where M = (S, A, P) is the environment model. RR is the space of reward functions and II is the space of polices. A-((-) R + II is an algorithm to optimize the reward function R E R. F: II + R is the fitness function that produces a scalar evaluation of the RL policy æ : A using the ground truth reward function. The goal of an RDP is to generate reward function R T R using Ass (R) to achieve the highest fitness score (()))(R)).
Evaluating the performance of designed reward function necessitates computationally expensive simulations (training policy x using the designed Rt.. Notably, the original definition of RDP does not impose constraints on sample efficiency. In this work, our objective is to output a reward function code R for a given task context, such that F(AM(()) (R)) is maximized while maximizing sample efficiency throughout the automatic reward design process.
Bayesian optimization (BO) is sequential design strategy for global optimization of black-box func- tions (Shahriari et al. 2015 Snoek al. 2012). Given black-box function f X -> RR Bayesian optimization aims to find an input x" E argminnexxf (x) that globally minimizes It placos prior p(f) over the objective function f to form a surrogate model (usually Gaussian process (Wang et al. 2024 An acquisilion function (e.g., the Expected Improvement (EI) (Ament et al. (023)) Op(s) X - R strategically determines the direction of the search for sampling points. The algorithm iteration procoeds in the following three steps: (1) find the most promising Xn+1 ~ argmaa (x)x): (2) evaluate the function Dn+1 = f(xx11) and update the set of historical observations Dn = (x.,. Ui))= by adding the point (Xn+1, Ba+1), and (3) update (((1))
and ap(f P.41))
4 Methods
The Uncortainty-aware Reward Design Process (URDP) framework incorporates three fundamental clements: (1) Decoupling of the reward component and reward intensity design processes, (2) Reward component generation based on uncertainty quantification, and (3) Uncertainty-awan the efficiency of reward design by minimizing redundant simulations while improving the performance of generated reward functions through the integration of numerical optimization techniques within the decoupled optimization framework.
Bayesian optimization. URDP enhances
4.1 Decoupled Reward Generation and Hyperparameter Optimization
Large language models possess extensive commonsense knowledge about task rewards, enabling them to surpass human-level performance in designing reward components (Yu et al. 2023: Ma et al.) 2024a| Xie et al. 2024) for certain tasks. However, their capability in black-box numerical optimization remains inferior to specialized numerical optimization algorithms (see Section 5.4 Abl-3 for an ablation study). Consequently, existing methods (Ma et al., (2024a) that conflate the optimization of reward components with reward intensities not only yield suboptimal numerical optimization results but also lead to insufficient optimization of the reward components.
URDP framework. Building upon these observations, we propose a decoupled reward function design process. As illustrated in Figure | URDP implements a bi-level iterative optimization procedure. Given the environment. specifications and task descriptions, the agent first samples multiple reward functions from the LLM in the outer loop. It ranks them using uncertainty quantification metries and filters out redundant or potentially unreliable rewards through a process termed Reward Code Sampling with Uncertainty Screening. Subsequently, the agent invokes the proposed numerical optimization tool (UABO) in the inner loop to determine optimal
4

---

Published in Transactions on Machine Learning Research (10/2025)
URDP
Input
Environment
Reward Components Optimization
Hyperparameter Optimization
Can wenkcctaaaarieeeeee
set ayn sefied ydre cyn onteet col du-
LLM
U
0
fund por salragd bur hundurtt wower one oet lggt
samples
UABO
Refinement
conpuse Int.. trm
R code
Task
Feedback prompt
Simulation
Uncertainty Quantification
to open the cabinet door
RL Training
Output:
Issac Gym
final raward fucntion code
Figure 1: URDP implements an alternating bi-level iterative optimization framework for automated reward design problems (RDP). The outer-loop optimization employs LLMs to refine reward components, where uncertainty quantification significantly enhances sampling efficiency. Concurrently, the inner-loop optimization utilizes Uncertainty-Awan Bayesian Optimization (UABO) to determine optimal hyperparameter configu- rations for the reward components. The decoupled architecture strategically leverages the complementary strengths of LLMs in conceptual reward design and numerical optimization tools in precise parameter tuning, achieving synergistic improvements in both final policy performance and computational efficiency.
reward intensity hyperparameters for the current reward configuration through black-box optimization and simulation- ovaluation. Finally, the agent evaluates the fcasibility of the current reward components and constructs a leedback prompt refine them. The feedback available to the process encapsulated in a structured prompt sent the LLM. This prompt involves task-level indicators such as success rate and episode length, component-level statistics (mean, max, and min values of each reward component), sample-level statistics (uncertainly scores, slandard deviation, and range), information about the best-performing reward functions, and adjustment suggestions for refining the reward design. An example of the feedback information, along with the full templates of both the initial and feedback prompts, is provided in App. A In essence, the outer loop optimizes the reward components while the inner loop tunes reward intensity hyperparameters, with their alternating optimization progressively converging toward optimal reward functions. See Alg. for pscudocodc.
4.2 Reward Code Sampling with Uncertainty Screening
In I.LM-based RDP, automated reward function design can be achieved through iterative sampling and simulation. However, existing approaches indiscriminately conduct simulation-base evaluation on all LLM- generated reward function samples, despite the frequent presence of redundant or infeasible candidates within these samples. Our analysis identifies this as critical factor contributing to computational inefficiency (see results Section 5.4 Abl-1). Consoquently, an essential question emerges: how effectively filter out polentially redundand reward function samples before simulations, thereby avoiding computationally expensive yet unnecessary simulation training.
Uncertainty priority. Our sampling approach is grounded in the principle of self-consistency Wang et al. 2022) in LLMs. When explicitly prompted to generate diverse outputs, LLMs that produce highly consistent responses demonstrate well-internalized, task-specific knowledge. Such outputs exhibit high reliability and typically require minimal refinement. Conversely, the generated results show substantial diversity, this indicates uncertainty in the understanding of the task context and underlying concepts. These divergent results exhibit lower reliability and consequently demand more refinement.
5

---

Published in Transactions on Machine Learning Research (10/2025)
Sampling and filtering. Based on this principle, the agent prompts the LLM to generate diverse reward components for given RT, task. The uncertainty score of each reward component for the reward function is quantified by its occurrence frequency across all sampled candidates. To quantify the uncertainty of the reward component, URDP identifies and resolves ambiguitics in reward components using LLMs. It combines textual similarity and semantic similarity analyses to evaluate the relevance and clarity of reward components, assigning the reward component uncertainty score Ur.... to cach reward component 7i, as
UTIER,EMM =1-
(a (mar((Srrss (Fi,)) (Ta - w))/K, [E[L,K]
Σ
(3)
where u (.) is a step function, K denotes the quantity of the reward function samples, M, is the number of reward components in the reward function R2, w is a decision parameter regarding the maximum similarity (w = 0.95), Stext E (0, 1] and Seemantic E (0, 1] are the textual and semantic similarity scores, respectively. Furthermore, the normalized sample uncertainty score Un, is computed to evaluate the overall uncertainty of each reward function sample R. Implementation details and hyperparameter settings are elaborated in App. Using Un, the agent filters out samples containing identical reward components, thereby climinating redundant inner-loop optimization processes that would otherwise ineur unnecessary computational overhead. Moreover, our analysis reveals that highly uncertain reward components may contain unexplored components capable of facilitating effective reward shaping (see Section 5.5 Disc-2). Consequently, the agent implements an adaptive exploration-cxplottaiio strategy, i.c., using Ninner UR. to adjust the maximum number of iterations in the inner loop, where Ninner is the upper limit. For high-uncertainty samples, allocates additional inner- loop iterations to prioritize exploration of optimal hyperparameter configurations for potentially novel reward components. For low-uncertainty samples, it emphasizes exploitation to minimize unnecossary simulations. This approach balances the trade-off between the exploration and utilization of uncertain reward components.
4.3 Uncertainty-aware Bayesian Optimization
While large language models demonstrate significant potential for reward component design, they exhibit suboptimal performance in numerical optimization tasks (see results in Section 5.4 Abl-3). This limitation leads to non-optimal reward intensity configurations in LLM-generated reward functions, representing a key factor in the poor policy learning performance observed in prior approaches. In contrast to existing methods, our URDP framework does not rely on LLM-based agents for direct numerical optimization. Instead, the agent serves as a controller that coordinates specialized numerical optimization tools. Specifically, URDP delegates the inner-lopp optimization of reward intensity parameters to Bayesian optimization (BO) algorithms. Benefiting from BO's superiority in black-box global optimization, this approach achieves significantly better performance than LLM-based optimization.
Although the classical Bayesian optimization algorithm, i.e., Gaussian Process with Expected Improvement (EI) (Ament et al. 2023; Snoek et al., 2012), , demonstrates theoretical advantages, its practical efficiency remains unsatisfactory, particularly due to the substantial computational overhead incurred by acquisition functions during sampling (simulation training). This inefficiency frequently prevents convergence to globally optimal solutions within an acceptable number of samplings. Notably, the uncertainties of individual reward components imply valuable prior knowledge for enhancing BO's sampling elficiency. Given a reward function comprising m components, we model the m reward intensities as a joint probability distribution. Crucially, higher uncortainty in (ri.,,...:Fimml corresponds to a more uniform marginal distribution along that dimension. This observation suggests that sampling should prioritize exploitation over exploration in high- tigghtaccety dimensions. Building upon this smoothness assumption, propose Uncertainty-aware Bayesian optimization (UABO) to address these limitations.
Un ). into both the kernel function and the acquisition function of the standard Bayesian optimization. The Matern kernel in the Gaussian process has the form
UABO incorporates reward component uncertainty scores, Ur = (Ur......., Ur.,,
k(ppp)) - fu(d) = =0² 21-v
(/2rd l
2vd
Kr
(4)
T(v)
6

---

Published in Transactions on Machine Learning Research (10/2025)
where d is the Euclidean distance between p and p', o² is the variance, is the smoothness parameter, is the length scale parameter and Ky is the modified Bessel function of the second kind. We note that the kernel is isotropic, which means that all dimensions (i.e., the intensity parameters of reward components) share the same length scale parameter. To accommodate heterogencous smoothness (uncertainty score (r:)) across different dimensions, we propose an anisolropic kernel function that incorporates uncertainly values as length scales within the distance metric. The distance formulated as follows,
(
-v
(O 10/ 0111
1 0%
du du(0f-1,00) 0,)
+..
+
Ura
U rimm
where - and f are reward intensity hyperparamoters for the reward components r; two iterations. the new kernel function is defined as
Then
k(00-1 ,0,) - Su(da) o2 21-v
(V2vda
Kr
(6)
r(v) UR.
UR.
Furthermore, we leverage Ur, to enhance the performance of the acquisition function. The standard Expected Improvement (EI) acquisition function (Ament et al. 2023) in Bayesian Optimization is follows,
E (x) = E (((o)) [[((=) v'++] [z)) (1xx) (()))-1)
((x) (1)))))))) (
where = m0x -), y² max; ys is the best observed value, and h(2) = (02) + 0(()), @ is the standard normal distribution density and ④ is the distribution function. In our context, is the agent success rate from the simulation RL training, and x is the reward intensitics 0,, The objective of EI is to decide the next promising reward intensity parameters for the next round of simulation.
To reduce inefficient exploration along directions with potentially insignificant influence on the function value, introduce penalty term that constrains the weighted distance between the candidate point and the current optimum, yielding uncertainty-acccleaatee EI acquisition function (uEI) defined as
ublly- (0,, Ur,) El(0.) w(O)),
(8)
M.
w(6)) exp Σ
Ur,.. (0: - 077²
(9)
where w(O)) is penalty term to constrain the weighted distance between O and the current best 0z Specifically, an uncertainty value approaching zero for a particular dimension indicates no restriction on variations along that dimension. Conversely, a large uncertainty weight in certain direction implies that extensive exploration in that direction is discouraged. UABO demonstrates significantly improved convergence efficiency (see Section 5.4 Alb-3), reaching optimal values within limited hyperparameter search steps, thereby substantially enhancing the performance and effectiveness of reward intensity configuration. A formal proof of its convergence lower bound is provided in App. F
5 Experiments
In this section, we evaluate the proposed URDP framework through extensive experiments on a diverse set of environments and lasks, comparing its performance against. human and baseline approaches. All experiments and comparative analyses presented in this paper utilize veepSeek-v3-241 foundational model unless explicitly stated otherwise (sce App. G.2 for more open-source LLMs' results).
(Liu et al. 2024] as the
5.1 Baselines
Eurcka (a et al. 2024a) (Bascline) provides systematic approach for generating reward functions utilizing LLMs. It incorporates foedback from various evaluation results to refine the reward functions via evolutionary iterations. This iterative process continues until an optimal reward function is achieved.
7

---

Published in Transactions on Machine Learning Research (10/2025)
Algorithm 1: Uncertainty-aware Reward Design Process
Task description T, Environment code E, LLM Output: Optimized reward function R..
Input:
foreach iteration Nouter do Generate K reward component samples r; = (r,,1, Tim Timbie using L(T, E, prompt) Uncertainty quantification: Ur, = -Ur.,.....,, Un.m and UR.
Filter out redundancies and reserve REEK- foreach R. 1 .innerr -UR, do Fit probabilistic model for f(6,) on data Di,-1
Choose of by maximizing the acquisition function uFT(0s, Ur,)
Evaluate by simulation training Yi, = f(0))
Augment the data Di, = Di (i..)
Choose incumbent f + argmar (y...aa................ Vs.c) and of max (B1,1..... vi.e}
Construct a feedback prompt for LLM to propose revised components
Choose optimal {(r', 00)) <- argmaa -Vi..... .K-
Recombine (tr,,0)) into R²
return R
Text2reward (Xie et al. 2024 is reinforcement learning method that automatically generates dense reward functions from natural language task descriptions using LLMs, without relying on expert data or demonstrations, and can express human goals in the form of procedural rewards, given to iterations using human feodback.
Human. maintain fair comparison, we adopted the same Human data as reported Eureka (Ma et al. 2024ah. The original shaped reward functions provided in the benchmark tasks were developed by active reinforcement learning researchers who designed the tasks. These reward functions embody the outcomes of expert human reward engineering.
Sparse.Therr functions correspond to the fitness measures F employed to assess the performance of the generated reward signals. Analogous to human leedback, they are also provided as part of the benchmark suite. The detailed configurations for Dexterity tasks and Isaac tasks are consistent with those used in Eurcka (Ma et al. 2024a). The fitness functions of all tasks in ManiSkil12 are specified in App. B.2
5.2 Experimental Setup
Benchmarks. Our environments consist of three benchmarks: Isaac, Dexterity, and Maniskil12. They comprise 35 different tasks. Nine of these tasks are from the original Isaac Gym environment (Nasir al., 2024 (Issac), twenty are complex bi-manual tasks (Chen al. 2022) (Dexterity). and the remaining six are from the Maniskill2 environment (Gu et a..). See App.
B for more details.
Metrics. We examine fore metrics: i. Success Rate (SR). We report the success rates of different reward functions on Dexterity and ManiSkill2 tasks. To ensure fair comparison with the baseline methods, the success rates for ManiSkill2 tasks are calculated using the last 50% of test results from each evaluation, while full test results are used for Dextcrity tasks. (ii) Human Normalized Score (HNS). For Isaac tasks, following F
EUREKA (Ma et al. 2024a), we use HNS = [FHumaa- -Fspamell where F is the environment's 's fitness functions (e.g., binary success Bidexterous tasks; continuous task score IsaacGym tasks). Scores are normalized that Sparse maps to 0 and Human to 1, enabling cross-tasi comparison. An HNS> 1 indicates the generated reward's training result excceding the human basclinc. iii. The Number of Evaluations (NOE), quantified as the total number of sinulations (RL training using the designed candidate reward functions) conducted across all samples during the optimization process. iiii. The Number of LLM Callings (NLC), representing the cumulative number of LLM calls made during both reward function generation and refinement. These metrics collectively provide a comprehensive assessment, with SR and HNS evaluating the performance of the generated reward functions, and NOE and NL.C quantifying the efficiency of the design process.
8

---

Published in Transactions on Machine Learning Research (10/2025)
Policy Learning. The performance of reward functions generated by URDP and comparative methods rigorously validated through RT. training. For both Isaac and Dexterity environments, employ the same high-efficiency PPO (Schulman et al. 2017) implementation as used in Eureka, using identical task-specific hyperparamcters without modification. In the ManiSkil12 environment, we utilized both SAC (Haarnoja) et al. 2018) and PPO algorithms to ensure fair comparison between URDP, Rexa2e and Eureka, strictly maintaining the original hyperparameter configurations across all methods. See App. C.2 for detailed parameter configurations.
5.3 Results
URDP improves the efficiency of the reward function design. Table [ presents & comprehensive comparison of computational efficiency between URDP and Eureka across three benchmarks. When optimizing for the peak reward performance, URDP requires only 52.4% of the simulation episodes (NOE) and 46.6% of the cvolutionary iterations (NLC) compared to Eureka. This significant accelcration demonstrates the superior optimization efficiency of the URDP in automated reward function design. See App E for a per-task
breakdown.
Table 1: URDP demonstrates superior efficiency across all benchmarks.
Isaac
Dexterity
ManiSkill2
Methods
NOE,
NLC|
NOEJ
NLC|
NOEJ
NLCJ
Reward
72.889
5
84.45
6.05
106.667
6.667
Eureka
68.667
4.556
80.05
5.5
98.667
6.167
URDP
39.501
2.495
57.8
3.4
32.33
1.667
URDP demonstrates superior reward function performance. Table 2 presents systematic comparison of reward functions generated by different approaches across benchmark tasks. Under identical simulation budgets (NOE), reinforcement learning agents trained with URDP-derived reward functions achieve significantly higher success rates than competing methods. Notably, URDP demonstrates substantial performance gains of 132%, 45%, and 76% over Eureka across the three experimental environments, representing significant enhancements, Furthermore, URDP-deste designee reward functions outperform manually engineered counterparts by a considerable margin, providing compelling evidence for the efficacy of automated RI. frameworks.
Table 2: URDP exhibits higher reward performance across all benchmarks.
Methods
Isaac (HNS+)
Dexterity (SRt)
ManiSkil12 (SRt)
Sparse
0
0.054
0.101
Human 1.000
0.459
0.434
Rewa2d
1.553
0.452
0.554
Eureka
1.607
0.466
0.449
URDP
3.424
0.675
0.792
URDP achieves synergistic progress in both reward performance and generation efficiency. In Figure it can be seen that the superiority of URDP over Eureka, text2reward and Human, both in terms of the success rate and the reduction in the number of simulations, has been well improved on these typical lasks. Each data point in the line plots represents the elfectiveness of the reward function obtained after a single iteration. The results show that URDP surpasses human -esigndd rewards on certain tasks after just one LLM refinement cycle and achieves optimal performance with significantly fewer itcrations than both Eurcka and Texa2Re which means that it consumes fewer tokens. These results collectively demonstrate that URDP achieves simultaneous improvements in both the efficiency and performance of the reward generation.
5.4 Ablation Experiments
Furthermore, explore the role of each core content in URDP in achieving the above results.

---

Published in Transactions on Machine Learning Research (10/2025)
URDP
Eureka
Text2Reward
Human
Sparse
1.5
2.0
15
1.0
1.5
HNS
10
1.0
0.5
NOE.
NOE-
0
NOE
30
60
90
120
30
60
90
120
40
100
(a) Ant
(b) AllegroHand
(c)
FrankaCabinet
1.0
0.8
0.6
1.0
0.4
0.4
0.5
0.2
0.2
0.0
NOE
0.0
NOE
30
90 120
20
40
60
80
100
0
30
60
90
120
(d) ShadowHand
(e)
DoorCloseOutward
(f)
PickCube
Figure Comparisons of URDP with other methods in Isaac (a-d), Dexterity (e), and ManiSkil12 (f). All curves are plotted until their convergence platcau. The HNS values no longer change across simulations (the plot points in the curves) after at least two conseculive outer-loop iterations. At this point, the inner loop (UABO) no longer yields better hyperparameters, evaluations, and the outer-loop LLM optimization longer yields improved reward components. The same criterion is applied consistently across all methods and figures.
it produces identical results in three consecutive
Abl-1: Uncertainty quantification improves the efficiency of reward design. To evaluate the role of uncertainty sampling, we conduct ablation studies by removing the uncertainty sampling and filtering module from URDP (denoted as URDP w.o. Uncertainty). Experimental results Figure 3 demonstrate that URDP achieves comparable success rates while requiring significantly fewer optimization episodes (NOE) than URDP w.o. Uncertainty, quantitatively validating the efficiency improvement brought by uncertainly-aware sampling. Interestingly, our analysis also reveals the correlation between the uncertainty and the novel reward functions, with detailed mechanistic explanations to be discussed in Section 5.5 (Disc-2).
URDP
URDP w.o. Uncertainty
Human
Sparse
2.0
15
1.5
1.0
HNS
1.0
0.5
5
0.5
0.0
NOEE
0.0
NOE
NOE:
0
50
100
150
200
250
50
100
150
200
250
0
50
100
150
200
250
(a) Ant
(b) AllegroHand
(c) FrankaCal Cabinet
1.0
0.8
0.8
0.6
0
0.4
0.4
0.2
NOE-
0.0
N6E
100
150
50
100
150
0
100
150
(d) ShadowHand
(e) DoorCloseOutward
(C) PickCube
Figure 3: When generating reward functions of comparable performance, URDP requires significantly fewer simulation training episodes, attributable to its effective uncertainty-basee based filtering mechanism.
10

---

Published in Transactions on Machine Learning Research (10/2025)
Abl-2: Decoupled optimization is the cornerstone for the collaborative improvement of perfor- mance and efficiency. This experimental investigation examines the role of decoupling in URDP, where reward components and their associated intensities are optimized separately. To evaluate this mechanism, we ablate UABO from URDP (denoted as URDP w.o. UABO) while maintaining identical configurations otherwise, resulting a system where both reward components and intensities are jointly conligured by the LLM without alternating optimization. Under unrestricted NLC, we compare the HNS or SR achieved by URDP w.o. UABO using equivalent NOE the standard URDP implementation. As shown in Figure | (dashed lines), consistent reductions in both HNS and SR observed across all three benchmark tasks, demonstrating the substantial impact of decoupled optimization on improving the reward performance. Furthermore, URDP exhibits faster convergence (requiring fewer NLC) to optimal solutions, suggesting that decoupling also enhances the elficiency of evolutionary search. A comprehensive discussion of this phenomenon and the underlying mechanisms presented Section 5.5 (Disc-1).
URDP
URDP W.BO
URDP W.LLMO
URDP w.o.UABO
Human
Sparse
1.5
2.0
15
1.0
1.5
HNS
10
1.0
0.5
0.5
0.0
NLC-
0.0
NLG
0
NLC:
2
6
2
(a)
Ant
(b) AllegroHand
(c) FrankaCabine
1.0
1.5
0.8
0.6
HNS
1.0
%044
0.4
0.5
0.2
0.2
0.0
NLC-
0.0
NLC
0.0
NLC.
3 4
6
3
4
5
(d) ShadowHamm Hand
(e) DoorCloseOutward
(f) PickCube
Figure comparison of the URDP, URDP w. BO, URDP w. LLMO and URDP w.o. UABO when all methods utilize identical simulation budgets (NOE).
Abl-3: The UABO plays key role in performance improvement. This experimental study systematically compares the numerical optimization capabilities between large language models (LLMs) and Bayesian optimization approaches within our framework. In the first ablation, we replaced the UABO module with the LLM reflection (denoted as URDP w. LLMO), employing identical prompting strategies to Eureka, thereby configuring both outer-loop (reward components) and inner-loop (reward intensitics) optimization entirely through L.Mss. Figure demonstrates that even under decoupled optimization conditions, LLM- based numerical optimization underperforms the Bayesian optimization, revealing fundamental limitations mathematical optimization capabilities while confirming the critical role of Bayesian methods in enhancing the reward performance. These results substantiate controllers for numerical optimization tools rather than direct optimizers. Subsequent validation experiments replacing UABO with standard BO (URDP w. BO) reveal UABO's superior efficiency (see Figure 5): URDP (UABO) achieves comparable or better performance than URDP w. BO using only 80% of the sampling budget across all Isaac tasks, with consistently superior final reward function performance, demonstrating that uncertainty-aware priors accolerate optimal scarch in the reward function design.
hypothesis that LLMs serve more effectively
5.5 Extended Discussion
Decoupling and performance degradation in evolutionary scarch. As illustrated in Figure (blue line), our experiments reveal performance degradation during evolutionary search in the baseline (Fureka) approach, with performance regression observed in 23% of Isaac tasks. Analysis of the LLMs'
Disc-1:
11

---

Published in Transactions on Machine Learning Research (10/2025)
80
SR: 0.67
URDP
URDP W. BO
SR: 0.68
HN553.34
SR: 0.79
NOH
HNS: 3.36
40
30
20
10
Isaac
Dexterity
ManiSkil12
Figure 5: URDP achieves a significantly greater improvement in efficiency across all benchmarks compared BO.
decision- decision-making in these cases demonstrates that in 75% of instances, the models modified only the reward intensity hyperparameters while leaving the reward components unchanged, indicating a propensity for erroneous judgments numerical optimization. More critically, we identify oscillatory phenomenon in the baseline optimization process (see Figure Go), where LLMs entered persistent cycles of alternating between limited sets of hyperparameters during evolutionary search. This optimization instability resulted in complete convergence failure, with the baseline system trapped in ineffective, non-progressive iterations.
URDP
Eureka
Human
Sparse
0.8
0.6
2.5
2.0
0.6
0.4
0 0.4
%
2
1.5
1.0
0.2
0.2
0.5
0.0
NLC-
0.0
NLC-
0.0
0
2
3
5
0
2
3
6
10
12
14
(a) UidUrrmm
(b) OpenCabinet
(c) Humanoid
Figure 6: The baseline (Eureka) exhibits undesirable performance degradation during evolutionary search on certain tasks (a-b). Notably, the oscillatory phenomenon is detected in the baseline method for Task (c), indicating substantial computational waste of the baseline method.
In contrast, URDP demonstrates superior optimization efficiency, requiring significantly fewer evolutionary iterations (NLC) while exhibiting markedly reduced instances of performance regression and oscillatory behavior during the optimization process. Our empirical results show performance regression in merely 1% of Isaac tasks, with no observed cases of persistent optimization oscillations. These findings provide a strong empirical explanation for the effectiveness of the decoupled reward design approach implemented in URDP.
Disc-2: The correlation between reward component uncertainty and novel reward discovery. Our investigation reveals noteworthy phenomenon: the high-uncertainty reward components (rwt) identified by URDP frequently correspond to novel reward components not previously utilized in human-designed reward functions, with these components exhibiting significant reward shaping effects during RL training, thereby revealing a dual role of the uncertainty in cnhancing both optimization efficiency and final policy performance. Here, we refer lo reward component as Txt when the uncertainty of the reward is greater than 0.9. Through systematic comparisons between reward components designed by the URDP and conventional human-designed rewards, Figure 7a illustrates a strong correlation between the component uncertainty level and the novelty, suggesting that the highei-uneertainty components may represent more innovative reward formulations. Ablation studies conducted by removing Fut components from the reward function (R w.o. rut) and retraining
12

---

Published in Transactions on Machine Learning Research (10/2025)
1.0
0.8
R
Reward Correlation
0.6
1.5
R O.
Uncertainty
0.4
0.2
HNS
1.
0.0
0.5
-0.2
0.0
-0.6
500
1000
1500
2000
2500
3000
-1. 0.0 0.5 1.0 1.5 2.0 2.5
Step
HNS (Log Scale)
(b) RL training curves of reward functions R R (a) URDP Rewards vs. Human Rewards on IsaacGym. nut on the ShadowHand.
Figure 7: (a) Results on IsaacGym tasks. Each marker represents one reward component. The r-axis shows the HNS on log scale, while the y-axis gives the Pearson correlation between generated reward and human-designed reward. Marker color shows the component-level uncertainty U, (darker indicates higher uncertainty). Marker shapes indicate outcome relative to the human baseline (blue circles: URDP wins; red crosses: human wins; green triangles: tics). Tasks in the lower-right region illustrate reward formulations that deviate from human design yet achieve superior performance, suggesting that high-uncertainly components capture novel, previously unexplored reward. These components are angvel_penalst and distance pytnaty in ShadowHand, dof hof__el___ _elallle in Anymal, and progress reward in FrankaCabinet.
(b) ShadowHand training curves show that adding the higher-uncertainty component Tut Tut (angvel (angvel_pennlty)) to consistently yields higher HNS than the ablation without it.
PPO agents reveal statistically significant performance degradation and poorer convergence characteristics in the resulting policies shown in Figure 7b whereas the complete reward function maintains substantially better optimization stability, collectively providing conclusive evidence that Tut components play essent role in effective reward shaping and policy learning regularization. See more examples in App. G.1
6 Conclusion
In this work, we present novel decoupled architecture for automated reward function design in reinforeement learning. Our framework introduces uncertainly quantification inlo reward component design, significantly improving the sampling efficiency of LLMss. Furthermore, we propose Uncertainty-Awar Bayesian Optimiza- tion to enable efficient hyperparameter search. Extensive experimental results demonstrate that our approach outperforms existing methods in both reward function quality and automated design efficiency.
13

---

Published in Transactions on Machine Learning Research (10/2025)
Acknowledgments
This work has been supported by the program of the National Natural Science Foundation of China (No. 61906195) and the independent deployment project of the National Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institution of Automation, Chinese Academy of Sciences.
References
Gustaf Ahdritz, Tian Qin, Nikhil Vyas, Boaz Barak, and Benjamin L. Edelman. Distinguishing the knowable from the unknowable with language models. In International Conference Machine Learning (ICML), pp. 503-549. PMLR, 2024.
Sebastian Ament, Samuel Daulton, David Friksson, Maximilian Balandat, and Eytan Bakshy. Unexpected improvements to expected improvement for bayesian optimization. Advances in Neural Information Processing Systems (NeurIPS), 36:20577 20612, 2023.
Saurabh Arora and Prashant Doshi. A survey of inverse reinforcement learning: Challenges, methods and progress. Artificial Intelligence, 297:103500, 2021.
Mido Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Matthew Muckley, Ammar Rizvi, Claire Roberts, Konstuv Sinha, Artem Zholus, et al. V-jepa 2: Self-supervised video models enable understanding, prediction and planning. arXiv preprint arXiv:2506. 09985, 2025.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, 1lya Sutskever, and Dario Amodei. Language models are few-shot learners. In Proceedings of Advances In Newral Information Processing Systems (NeurIPS), pp. 1877 1901, 2020.
Adam D Bull. Convergence rates of efficient global optimization algorithms. Journal of Machine Learning Research, 12(10), 2011.
Yuji Cao, Huan Zhao, Yuheng Cheng, Ting Shu, Yue Chen, Guolong Liu, Gaoqi Liang, Junhua Zhao, Jinyue Yan, and Yun Li. Survey on large language model-enhanced reinforcement learning: Concept, taxonomy, and methods. IEEE Transactions on Neural Networks and Learning Systems, 2024.
Yuanpei Chen, Tianhao Wu, Shengjie Wang, Xidong Feng, Jiechuan Jiang, Zongqing Lu, Stephen McAleer, Hao Dong, Song- Chun Zhu, and Yaodong Yang. Towards human-level bimanual dexterous manipulation with reinforcement learning. Advances in Neural Information Processing Systems (NeurIPS), 35:5150-5163, 2022.
Fengxiang Cheng, Haoxuan Li, Fenrong Liu, Robert van Rooij, Kun Zhang, and Zhouchen Lin. Empowering 1mss with logical reasoning: A comprehensive survey. arXiv preprint or 201: 15652, 2025.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama herd of models. arXiv e-prints, pp. arXiv 2407, 2024.
Alejandro Escontrela, Xue Bin Peng, Wenhao Yu, Tingnan Zhang, Atil Iscen, Ken Goldberg, and Pieter Abbeel. Adversarial motion priors make good substitutes for complex reward functions. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 25-32. IEEE, 2022.
Andrew Foong, David Burt, Yingzhen Li, and Richard Turner. On the expressiveness of approximato inference in bayesian neural networks. In Proceedings of Advances In Neural Information Processing Systems (NewrIPS), volume 33, pp. 15897-15908, 2020.
14

---

Published in Transactions on Machine Learning Research (10/2025)
Yarin Gal and Zoubin Ghahramani. Dropout a bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of the International Conference on Machine Learning (TCML), 2015.
Jiahui Geng, Fengyu Cai, Yuxia Wang, Heinz Koeppl, Preslav Nakov, and Iryna Gurevych. survey of confidence estimation and calibration in large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2577-6595, 2024.
Jiayuan Gu, Fanbo Xiang, Xuanlin Li, Zhan Ling, Xiqiang Liu, Tongzhou Mu, Yihe Tang, Stone Tao, Xinyue Wei, Yunchao Yao, et al. Maniskill2: A unified benchmark for generalizable manipulation skills. In The Eleventh International Conference on Learning Representations (ICLR).
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning (ICML), pp. 1861 1870. Pmlr, 2018.
Shibo Hao, Yi Gu, Haodi Ma, Joshua Hong, Zhen Wang, Daisy Wang, and Zhiting Hu. Reasoning with language model planning with world model. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (FMNLP), pp. 8154 8173, 2023.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac HatfieldDoodds, Nova DasSarma, Eli Tran-Johnson, al. Language models (mostly) know what they know. arxiv preprint Niw:2207.00222, 2022.
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation natural language generation. In International Conference on Learning Representations (ICLR), 2023.
Minae Kwon and Sang Michael. Reward design with language models. In International Conference on Learning Representations (ICLR), 2023.
Zhong-Zhi Li, Duzhen Zhang, Ming- Liang Zhang, Jiaxin Zhang, Zengyan Liu, Yuxuan Yao, Haolian Xu, Junhao Zheng, Pei-Jie Wang, Xiuyi Chen, et al. From system 1 to system 2: A survey of reasoning large language models. arXiv preprint Xrx::2200 17419, 2025.
Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. Generating with confidence: Uncertainty quantification for black-box large language models. Transactions on Machine Learning Research.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepscek-v3 technical report. arXiv preprint arX 19437, 2024.
Xiaoou Liu, Tiejin Chen, Longchao Da, Chacha Chen, Zhen Lin, and Hua Wei. Uncertainty quantification and confidence calibration in large language models: A survey. arXiv preprint orXiv:2503.1588. 2025.
Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eureka: Human-level reward design via coding large language models. In International Conference on Learning Representations (ICLR), 2024a.
Yecheng Jason Ma, William Liang, Hungju Wang, Sam Wang. Yuke Zhu, Linxi Fan, Osbert Bastani, and Dinesh Jayaraman. Dreurcka: Language model guided sim-to-roal transfer. In Robotics: Science and Systems (RSS), 2024b.
Christopher Mohri and Tatsunori Hashimoto. Language models with conformal factuality guarantecs. In International Conference on Machine Learning (ICML), pp. 36029 36047. PMLR, 2024.
F. J. Narcowich, J. D. Ward, and H. Wendland. Refined error estimates for radial basis function interpolation. Constructive Approzimalion, 2003.
Muhammad U. Nasir, Sam Earle, Christopher Cleghorn, Steven James, and Julian Togelius. Llmatic: Neural architecture search via large language models and quality diversity optimization. 2024.
15

---

Published in Transactions on Machine Learning Research (10/2025)
OpenAL. Gpt-4 technical report. arXiv preprint arXiv:2303.0877/ 2023.
Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingleng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL https: //ap:s///rxiv.../ wwapxii..og/geeeoo wtt.aa///........e..
Alec Radford, Rewon Child Jeffrey Wu, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 2019.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alee Radford, and Oleg Klimov. Proximal policy optimization algorithms. Xiv preprint or arXiv:1707.063/ 06347, 2017.
Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P Adams, and Nando De Freitas. Taking the human out of the loop: A review of bayesian optimization. Proceedings of the TEEE, 104(1):148-1755 2015.
Ola Shorinwa, Zhiting Mei, Justin Lidard, Allen Z Ren, and Anirudha Majumdar. A survey on uncertainty quantification of large language models: Taxonomy, open research challenges, and future directions. arv preprinl ar Xrv:224 2. 05563, 2024.
Satinder Singh, Richard T. Tewis, and Andrew G Barto. Where do rewards come from. In Proceedings of the annual conference of the cognitive science society, pp. 2601 2606. Cognitive Science Society, 2009.
Jasper Snock, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. Advances in neural information processing systems (NeurIPS), 25, 2012.
Jiayang Song, Zhehua Zhou, Jiawei Liu, Chunrong Fang, Zhan Shu, and Lei Ma. Self-refined large language model as automated reward function designer for deep reinforcement learning in robotics. arXiv preprint wxXi:22399006677 2023.
Jiayuan Su, Jing Luo, Hongwei Wang, and Lu Cheng. Api is enough: Conformal prediction for large language models without logit-access. In Findings of the Association for Computational Linguisties: EMNLP 2024, pp. 979 995, 2024.
Ziyu Wan, Xidong Feng, Muning Wen, Stephen Marcus McAleer, Ying Wen, Weinan Zhang, and Jun Wang. Alphazero-like tree-search can guide large language model decoding and training. In Forty-first. International Conference Machine Learning (ICML), 2024.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Huai hsin Chi, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. Xi preprint or XrX::2203.1 111101..11111....11111111 2022.
Zi Wang, George E Dahl, Kevin Swersky, Chansoo Lee, Zachary Nado, Justin Gilmer, Jasper Snoek, and Zoubin Ghahramani. Pre-trained gaussian processes for bayesian optimization. Journal of Machine Learning Research, 25(212): 18,--3,3 2024.
Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian- Nie. C-pack: Packed resources for general chinese embeddings. In Proceedings of the 47th international ACM SIGIR conference on research and development in information retrieval, pp. 641 649, 2024.
Tianbao Xie, Siheng Zhao, Chen Henry Wu, Yitao Liu, Qian Luo, Victor Zhong, Yanchao Yang, and Tao Yu. Text2reward: Reward shaping with language models for reinforcement learning. In International Conference on Learning Representations (ICLR), 2024.
Yang Yan, Yu Lu, Renjun Xu, and Zhenzhong Lan. Do phd-level Ilms truly grasp elementary addition? probing rule learning vs. memorization in large language models. arXiv preprint arXiv:2504.05266 2025.
16

---

Published in Transactions on Machine Learning Research (10/2025)
Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. Large language models optimizers. In The Twelfth International Conference on Learning Representations (ICLR).
Wenhao Yu, Nimrod Gileadi, Chuyuan Fu, Sean Kirmani, Kuang-Huei Lee, Montse Gonzalez Arenas, Hao- Tien Lewis Chiang, Tom Erez, Leonard Hasenclever, Jan Humplik, Brian Ichter, Ted Xiao, Peng Xu, Andy Zeng, Tingnan Zhang, Nicolas Heess, Dorsa. Sadigh, Jie Tan, Yuval Tassa, and Fei Xia. Language to rewards for robotic skill synthesis. In Proceedings of Conference on Robot Learning (CoRL), 2023.
Zhuohao Yu, Weizheng Gu, Yidong Wang, Zhengran Zeng, Jindong Wang, Wei Ye, and Shikun Zhang. Outcome-refinngg process supervision for code generation. arXiv preprint arXiv:242.. 15118, 2024.
Jialun Zhong, Wei Shen, Yanzeng Li, Songyang Gao, Hua Lu, Yicheng Chen, Yang Zhang, Wei Zhou, Jinjie Gu, and Lei Zou. A comprchensive survey of reward models: Taxonomy, applications, challenges, and future. Xie preprinl or Xiv:22000 12328, 2025.
Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, et al. Rt-2: Vision- lasooo-languagg-acttion models transfer web knowledge to robotic control. In Conference on Robot Learning (CoRL), pp. 2165-216 2183. PMLR, 2023.
17

---

Published in Transactions on Machine Learning Research (10/2025)
A Full Prompts
In this section, we provide all prompts and a concrete example of the feedback used in the URDP framework.
Prompt Initial system prompl
reinforcement learning Your goal is to write reward function for the environment that will help the agent learn the task deacribed in text. Your reward function should use useful varisbles from the enviromeet as lapute. ds exanple, the reward function signature be: @torch.jat.script def compate, compate_reearddobjeer robjrd torch.
engineer trying write reward
effactive as possible.
goal_ _paa torch.Tensor) Tuple[torch. Tensor, Diet [atr, torch. Tensor]]
return resard, Since the reward function will be decorated with Storch. storch.jat. aor.pt please nake sure that the code is conpatible with TorchSeript (eig., use torch teneor instead nunpy array). Hake any tensor variable you introduce is the device the input tensors.
Prompt 2: Reward reflection and feedback
Ve trained policy the provided reward function code and tracked the values of the individual componenta the revard function as sell as global policy netrics such as success rates and apiaode langtha aftar avary {apoch_ freq] apochs maximun, minimum values encountered: <REXARD REFLECTION HERE1>
Ve calculated score for each eample based on the uncertainty of the reward terz. Ne then calculated the standard and cample scores iteration, <REVARD REFLECTION HERE2>
extrone deviations
Please adopt the following recomendations for the next iteration of revard function generation: the standard deviation less than 0.05 and the extrens deviation less than iteration and instead encourage exploration of nev conbinations of revard terns. recoanended to retain revard itens with higher (3) Only the combination of revard itens and the content of the revard runction need to be optimised, not the nunerical optinieation.
recomennde stop the
the sample and those lover
carefully analyze policy
provida nev, improved
Sona helpful tips for analyzing the policy Teadback:
the success rates are slways near zero, then you must resrite the entire revard function
(2) If revard components' nagnitude is significantly larger, then re-scale its value proper range Please analyze each existing rezard compenent in the suggested manner above first, and than write the revard function coda.
Prompt Code formatting tip
The outpat of the reward function should consiat of txo itens:
(1) the total reward,
(2) a dictionary of each individual revard component.
code output should formatted python code string: """pyhhon
helpful tipa uriting revard
code:
(1) You may find it helpful nornelige the revard to fixed range applying tramsfornations Like torch. exp to the overall revard or its componente
(2) If you choose traneforn revard component, then nust also introduce temperature paraneter inelde the transfornatio function; this paranetar mast be a naned variable in the resard function and it must not be an input component should (3) Make sure the type of each input variable is corractly spacified; a float input variable should not be specified as torch.Tensor
varisble. Each transforned
temperature varisble
(4) Nost isportantly, the revard code't 's input variables muet contain only attributes of the provided environment claes definition (nanely, variables have prefix self.).
circunstance can you
new input
Example of feedback
Iteration URDP Sample: pythen
@torch.jit.ecrip: def compate_reent reward(
object_rot: torch. Teneor, goal_rot: forch.Tensor,
object_angvel; torch.
18

---

Published in Transactions on Machine Learning Research (10/2025)
fingertip_pos norch.eensor,
object_pas: torch. Tensor,
actions: Torch-Tenaor Tenaor
Tuple[torch: Tensor, Dictletr, torch. ;
Reward function for spinning object target orientation.
orientation_ temp:
aliganent, _emp: float 0.5
spin_ temp: float 0.3
control.penaie wetpol.eeeett__eee flost -0.01 Eingertip.dist_tage float -0.1
device abject_rot.devics
1. Orientation aligmnent revard (nain objective)
Quaternion dot product (cosine of half the sugle between rotations) juat_diff quat_ml(object__t orientation_ orientation_temp (1.0 torch.abo(qutt.difff.. 0]))
conjugate
2. Angular velocity aligument with the desired rotation axis went object to spinning, not just avg_angvel nean nean aba (objectt_angeel din-1) spin_reward spin_teap avg_angvel
target
3. Control panalty (amall penalty for large actions) action_penalty control_pmmalyy.terg torch.nean (torch. (aaaeccco din-1)
Fingertip
penalty (keep fingertips
object)
fingertip_died diet (orch.
object_pos ....u...sqqueee(()) din--1)
avg. aig_fingertip..is torch mean (fangeffiinddeeeeeeeee.. dim-1)
distance_ponalle Sangertip.dist.ttee wig.tingrrtip.die
Conbine componenta
lotal_reward = orientation_revaar spin_revard action,penalty + distance_penally
Return components for debugging
reward dict {
Dorientationnrewarrd orientation_rewarf
"spin_ reward"; apin_reward,
penalty:; sction,
ddietance_penaltye distance_penalty
return total_retteeeeee revard, reward_dict
Iteration Feedback:
Ve trained an RL policy using the provided revard function code and tracked the values of the individual conponents in the as success rates opisode lengtha after every apocha, naximum, and minimun values encountered:
function,
as global palicy
Best Sample (0) zetrics:
coantation_rrvardd ['0.67', 10.60', '0.61', 10.68", 10.67', 10.89", 10.68', 10.68', 10.67", '0.67"), 1,, Max: 0.61, Hean: 0.58, Min:
rewarde ['2.10', '2.12', 12.28', 12.42', 12.78', 12.84', 13.07", 14.01', 14.74', 15.09"), Max: 5,62, Vean: 3,31, Min: 1.79
action_penalty: ['-0.191, 1-0.21', 1-0.221, 1-0.23', 1-0.241, 1-0.251, 1-0.26', 1-0.27', 1-0.28', 1-0.291) 1, Nax: -0.19, -0.25, Nin: -0.30
lettance_ponalty, ['-0.03', 1-0.03', 1-0.03", 1-0.03', 1-0.03', 1-0.03', 1-0.03', 1-0.02', 1-0.02', 1-0.02"], Max: -0.02,
Hean: -0.03, Nin: -0.03
succese_rate: [10.00', 10.00", 10.05', 10.01", 10.03', 10.01", 10.08', 10.04', 10.02', 10.02'), Max: 0.20, Mean: 0.03, Mins 0.00
spisode_ lesggteele [16.88', 185.14', '177.95', 1218.96', 1263.25', '2'8.06', '308.33' '311-40', 1305.05', 1322. 21 Max: 326.63, Nean: 243.48, Min: 6.88
Ve calculated a score for each sample basad on the uncartainty of the revard tarn. We then calculated the standard and sample
deviations
iteration,
Sample Statistics:
Standard Deviation: 0.080
Range: 0.309
Sample netrice:
orientation_revard: [10.07', 10.06', '0.07', 10.07', 10.07', 10.07", 10.07', 10.07", 10.07", '0.08') ), Nax: 0.07, Mean: 0.07,
Hin: 0.08
segvel_revard: [10.00', 10.01', 10.001, 10.01", 10.00", 10.011, 10.011, , 10.01", 10.011, 10.01 Max: 0.01, Nean: 0.01, Min:
dof pof__ve__pnnaty: [168.31', 174.191, 174.67', 21', 186.641, 193.441, 198.93', 920, 19'. '90,.19", Nax: 109.81, Nin: fingertip Lingertip.distance
Hean: silstance_pennalee
[10.071, 10.09", 10.08", 10.07', 10.08", 10.081, 10.07", 10.071, 10.07', 10.08'), Max: 0.09,
Hean: 0.08,
19

---

Published in Transactions on Machine Learning Research (10/2025)
success_rate: ['0.00', '0.03', 10.01', '0.00', '0.02', 10.01', '0.02', 10.00", 10.00', '0.00'), Max: 0.18, Noan: 0.02, Min: 0.00 apissod__oongher ['6.68', '112.69", '217.70', 1263.16', '267.88', '352.34', '346.89', 1360.60', 134..1",, '355.96') Max: 379.91, Nean: 275.82, Nin: 6.88
Semple 6 netrics:
orientation_revard: [10.66', 10.65', '0.65', 10.65', '0.65', 10.66", 10.66', 06', 10.65', 10.66", 10.66') "), Max: 0.68, Mean: 0.66,
Min: 0.65
D10.30", "0.43", '0.53", 10.53', 10.51', 10.47', 10.49', 10.49", "0.52", 10.50 Max: 0.53, Nean: 0.48, Min:
0.27
babbiiitt_rewaarde ['0.96', '0.97', '0.97', 10.98", '0.98", 10.98', '0.98", '0.98", 10.98', , 10. '0.98'), Max: 0.98, Maan: 0.98,
Min: 0.95
contact_revard: ['0.99', 10.99", 10.99", 10.99', 10.99', '0.99', 10.99", '0.99', '0.99', 10.99 1, Nax: 0.99, Mean: 0.99, Min: 0.99
action_reg: ('-0.05', 1-0.06', 1-0.06', '-0.07', 100.07' 100.771 1-0.08', 1-0.08', 1-0.08', 1-0.09"] ], Max: -0.05, Nean:
-0.07, Min: -0.09
success_ rate: ['0.00', 10.02', 10.00', 10.04', 10.01" 10.00', 10.01', 10.01", 10.04', 10.0011, Max: 0.13, Mean: 0.02, Min: 0.00
apiiodd_lenngher [16.88', '104.14', '264.72', '294.7'' '321.16', '351.02', '347.58', '327.18 1343.47', (341.25"], Nax:
395.64, Noan: 272.77, Min: 6.88
Sample 8 netrics:
rrtentation_ravare [10.35', '0.35', 10.35', , 10.35', 10.35', '0.35', 10.35', 10.35', 10.35', 10.35 '3, Max: 0.36, Mean: 0.36,
Mim: 0.34
sngvel_revard: [10.36', '0.62', '0.68', '0.75', '0.74', 10.71', 10.79', 10.78', 10.84', '0.85'), Max: 0.88, Nean: 0.74, Min: 0.35
positionn revard: ['1.00', '1.00', '1.00', '1.00', '1.00', 10.99', , '0.99", 10.99", '0.99', 10.99' '1 Max: 1.00, Maan: 1.00,
Min: 0.99
contact_ rentact ['1,00', '1.00', 11.00", '1.00', '1.00', '1.00', , '1.00', '1.00', '1.00', 11.00'1, Nax: 1.00, Mean: 1.00,
Min: 1.00
petionnpenalty: ['-0.10', 1-0.12', '-0.14', '40.14', '-0.15', '-0.15', 1-..15', '-0.15', '-0.15', 1-0,66), Nax: -0,10,
Hean: -0.15, Nin: -0.17
success_rate: ['0.00', 10.00", 10.01', 10.01', '0.02', 10.00', 10.01', 10.03', 10.02', 10.05'), Max: 0.14, Nean: 0.02, Min: 0.00
tatsodd_langther ('6.88", '107.01', '227.35', '219.53', '249.13', '292.37', 2281.23', '331.81', '298.09', '306.40'] Max:
399.00, Nean: 250.83, Nin: 6.88
Sample netrice:
arientattion_reaare ['0.80', *0.80', 10.80', '0.80', 10.80", 10.80', 10.80', 10.80', 10.80", 10.80'), ,, Max: 0.81, Mean: 0.80,
Min: 0.80
angvel_revaree ['0.56', 10.69", '0.80', 10.88', '0.92', 10.91", 10.93', 10.94', 10.94', 10.94') "), Max: 0.96, Moan: 0.87, Min: 0.54
fingertip_reward ['1.00', '1.00', '1.00', '1.00', '1.00', '1.00', '1.00', '1.00', '1.00', '1.00',, Max: 1.00, Nean: 1.00,
Min: 1.00
action_revard: ['0.99', 10.99', 10.99", '0.98', '0.90', '0.98', 10.98', 10.98', '0.98", '0.98') Max: 0.99, Noan: 0.98, Min: 0.98
success_rate: ['0.00', 10.01", 10.02', 10.03', 10.02', 10.00', 10.01', 10.00", 10.00', 10.00'), Max: 0.17, Noan: 0.01, Min: 0.00
geitaode_lengthe ['6.8'', '104.65', '285.08', 3312.46', '354.28', '356.70', '275.36', 1371.96', '368.43', 1370.56'1 Max:
395.92, Nean: 296.21, Min: 6.88
Sample 3 netrics:
peieetatioo_reware [10.00', 10.00", 10.00", 10.00', '0.00", 10.00", 10.00', 10.00', 10.00", '0.00 Nax: 0.00, Mean: 0.00, Min: 0.00
angvel_revare ['0.36', '0.68', '0.81', 10.87', '0.85', 10.90', 10.89', 10.92', '0.92", '0.92'), Max: 0.93, Moan: 0.84, Min: 0.35
fingertip_reward: [*-0.00', '-0.00', '-0.00", 1-0.00", '-0.00', 100.00', 1-0.001, 1-0.00', 1-0.00', 1-0_00'), Max: -0.00,
Hean: -0.00, Nin: -0.00
action_penalty: ['-0.01', '-0.01', '-0.02', 120.02', 1-0.021, '-0.02', 12..0'' 1-0.02', 1-0.02', 1-0.02"]
Max: -0.01,
Hean: -0.02, Nin: -0.02
object_vel pel_paatl ['-0.00', '00.00', '00.00', '00.00', 100.00', , 000000" 100.00', 100.00" 100.00" , 1-0.00 Nax: -0. -0.00,
Hean: -0.00, Nin: -0.00
success_rate: [10.00', 10.02', 10.01', 10.00' '0.00', 10.0'', 10.01' 10.01' 10.00', 10.00') Max: 0.05, Nean: 0.01, Min: 0.00
spisodd_leengtan [16.88', 183.31', '2'8.44', 1169.25', 828 522 2222.2'', 1236.94', 60'. 1282.27 (), Max: 399.00, Nean: 209 30, Min: 6.88
Please adopt the following recomeedations for the next iteration of revard function generation:
(1) If the standard deviation is less than 0.05 and the extrene devistion is less than 0.1, it is recounended to stop the iteration and instead encourage exploration of new conbinations of revard terms.
(2) For all revard itens with higher acores
recomnendad keep then and for those with lovar scares it is reconnendad
to renove then.
(3) Only the conbination of revard itens and the content of the revard function nead to optinised, not the nunerical optinisation.
Please carefully analyze the policy feedback and provide a nev, improved revard function that can better solve the task. Sona helpful tips analyzing the policy feadback:
(1) If the success rates are always near zero, then you must revrite the entire reward function (2) If sone revard components' nagnitude is significantly larger, then you must re-ecale its value to proper range
20

---

Published in Transactions on Machine Learning Research (10/2025)
B Benchmark Details
B.1 An Introduction to the Benchmarks
Isaac.The Isaac Gym benchmark includes broad set of continuous control tasks, covering locomotion, balancing, aerial control, and dexterous manipulation. Robols range from low-DoF systems (e.g., Cartpole, Ball Balance) to complex agents such as Humanoid, Anymal, AllegroHand, and ShadowHand. Each task presents different control challenges, roquiring procise joint coordination, stable gait generation, or fino-grainod object interaction. All lasks provide observations, including joint positions, velocilies, root orientation, and task-specific data such object pose or goal location. The control mode varies by task task-torque, velocity, end-effector control depending on the robot type. Randomization of initial states and physical parameters (e.g., mass, friction) is applied during training to improve robustness and generalization. The benchmark emphasizes both low-level motor control and high-level strategy in physics-rich environments.
Dexterity. The Dexterous benchmark focuses on dexterous manipulation using the 24-Dof ShadowHand across wide range of object-centric tasks. These include stacking blocks, turning faucets, opening doors, rotating bottles, catching objects, and tool use. The tasks require precise finger control, contact-rich interactions, and adaptability to diverse object geometrics and behaviors. Each environment provides proprioceptive input (joint stales, fingertip positions), well as object-related observations (pose, velocity, goal state). Control is applied via joint position or velocity commands. The tasks involve significant variability in object placement, orientation, and physical properties, encouraging the development of general and robust manipulation policies. The benchmark highlights the challenge of high- high-dimensional motor coordination in worl-word,,, unstructured settings.
Maniskill2. In the ManiSkill2 environment, 7-DoF Franka Panda robotic arm is used by default, For tasks focused on stationary manipulation such Lift Cube, Pick Cube, Turn Faucet, and Stack Cube- a fixed-base arm configuration is employed. In contrast, tasks involving mobility, such as Open Cabinet Door and Open Cabinet Drawer, utilize a single-arm robot mounted on a Sciurus mobile base. The Push Chair task handled by a dual-arm system, also equipped with the Sciurus17 base. The observation space inchudes robot-centric data like joint angles, joint velocities, and the base's pose (position and orientation in the world frame), along with task-specific inputs such as goal coordinates and end-effector locations. Control is performed in end-effector delta pose mode, which directly manages changes in 3D translation and orientation, the latter expressed in axis-angle form relative to the end-effector's frame. Each task features variability in key parameters, including the initial and goal states of the object being manipulated, the robot's starting joint configuration, and physical dynamics like friction and damping.
B.2 ManiSkill2 Environment Details
We provide detailed information about the ManiSkil12 environment in this section. Detailed information about the Isaac and Dexterity enviromments is the same as in the Eureka (see the content in the appendix of the paper (Ma et al.] 2024a)). For each environment, we list its observation and action dimensions, the original description of the task, and the task fitness function F.
21

---

Published in Transactions on Machine Learning Research (10/2025)
ManiSkil12 Environments
Environment (obs dim, action dim)
Task description
Task fitness function F
PickCube-vO (51,7)
This class corresponds the PickCube task in ManiSkill. This environment consists of robot arm and cube placed the table. At the beginning, the cube appears at random location and orientation. The agent must control the gripper to approach, grasp, and lift the cube above threshold height. The challenge lies in object localization, precise control, and stable grasping.
[dist cube goal < 0.05]
LiftCube-vO (42,7)
This environment corresponds to the LiftCube task. The agent is required to grasp cube and lift it vertically above a specific height threshold. The task emphasizes accurate vertical movement and stable grasping without disturbing the cube's pose. [[cube_height > 0.2]
TurnFaucet-vo (40, 7)
This class corresponds to the TurnFaucet task. A faucet handle is mounted on wall, and the agent must rotate it clockwise or counterclockwise to target angle. The challenge lies in establishing proper contact, applying sufficient torque, and maintaining stability during the turning motion. cootation_reeardd < 0.1]
OpenCabinetDoor-vl (75, 11)
This environment corresponds to the OpenCabinetDoor task. A cabinet with a sidc-hinged door is presented. The agent must locate and pull the door handle to open it. The task involves estimating the door's hinge axis, approaching from an appropriate angle, and applying a pulling force that aligns with the door's s rotation.
[gooal diff < 0.1 and is static]
OpenCabinetDrawer-vu (75, 11)
This class corresponds to the DrenCabinetDrawer task. The robot must open drawer embedded in a cabinet by locating the handle and pulling outward. The task requires both accurate handle grasping and force application along linear trajectory, while avoiding excessive torque that could misalign the drawer.
[gooa diff < 0.05 and is static]
PushChair-vi (131, 18)
This environment corresponds to the PushChair task. The robot must push movable chair from initial location to designated target region. The chair is free to rotate and slide. The agent needs to make rategic contact with the chair body and adjust its pushing direction dynamically to avoid misalignment and ensure accurate placement.
1 [chair to target dist < 0.3 and chair tilt < 0.2]
C Implementation Details
C.1 Implementation Details of Sampling and Uncertainty
When explicitly prompted to generate diverse outputs, LLMs inevitably produce varying textual expressions for semantically equivalent content- phenomenon particularly evident in reward function generation where code implementations may differ lexically while encoding identical reward semantics. For instance, as demonstrated in Section D.2. two LLM-generated reward function samples might both incorporate velocity-based rewards while exhibiting completely different textual formulations. Failure to detect and climinate such semantic redundancies leads to computationally expensive duplicate evaluations that cannot be effectively identified through surface-level text matching, necessitating deeper semantic analysis for accurate deduplication.
22

---

Published in Transactions on Machine Learning Research (10/2025)
Therefore, the URDP utilizes the BGE-M3 model (Xiao et al. 2024] for the purpose of semantic similarity assessment, whereas the built-in SequenceMatcher in Python is employed for text. similarity assessment. The uncertainty quantification for both reward components and reward functions is implemented through similarity comparison. Specifically, the component uncertainty score (Ur..) is computed by comparing a given reward component against components generated within the same iteration. The reward function uncertainty (Un,) score derived through comparison with all functionally similar reward functions from the same iteration (referred to as similarity group). From each similarity group, only one reward function is randomly selected for training, while the remaining ones discarded, thereby filtering out redundant reward functions. See Alg. 2 for the pseudocode. We employ a similarity threshold of 0.95, where the final similarity metric is determined as the maximum value between semantic similarity and textual similarity.
Algorithm 2: Uncertainty Quantification in the URDP
Input: K reward component samples (12,1,7221... Timm Jiek, text models Stezt and semantic models Ssemantic.
Output: Reward components uncertainty (Ura, Un reward functions uncertainty Un, and similarity sample group Bi.
foreach R4 E K do
foreach ri,1, T1,2........ Tsm do
forcach rg,,, Jjm with j > do if max (Frear(iim Tj,m), S Sseaantia (ri,,, fj,m)) 0.95 then
L countm + countm +
Urimm +1 tt--ouutm/he Up + - M; Un.m UR. + Ur/((Ur, +.... +Ur)) forcach Rjzi do if max(S Rg, Ry), Ssemontie R2), > 0.95 and R, not in other B then L Add R2 to B;
return (Ur,...... Ur UR. B
C.2 Hyper-paramett Settings
The reinforcement learning algorithms employed for valida- tion maintain the defanlt configurations specified for each respective environment, with all hyperparameters comprehensively documented in Tables and 5
All lyperparameters in URDP are listed Table 3
Table Hyperparameters of URDP.
Hyper-parameter
Value
Quantity of the reward samples K
16
Maximum # of iterations Nouter
10
Baseline # of iterations Nsnner
10
Maximum similarity w Smoothness V Length scale
0.95
2.5
Un
We set the quantity of the reward samples K to be the same as that used in Eureka, ensuring fairness in comparisons.
Determination of outer maximum iterations Nouter and inner baseline iterations Ninner. The settings for these two hyperparameters were determined based on experimentally observed safe upper bounds. For the outer maximum iterations Nouter, Figure Sa compares the average NLC (and its variation range, indicated by black lines) required by URDP and Fureka. across different. benchmarks when achieving comparable
23

---

Published in Transactions on Machine Learning Research (10/2025)
HNS and SR. Since the variation range of NLC is within 10 for all cases, set the maximum iterations Nouter = 10 safe upper bound, ensuring effective completion of all tasks. For the inner baseline iterations Ninner, Figure (b presents the average number of inner-loop iterations required by URDP across the three benchmarks, along with their ranges. The required range is also within 10, while Alg. | the inner-loop iteration count defined as enner Up (with UR E [0, 1]). Therefore, we set Nnner = 10 to meet. the
requirements across tasks.
10
URDP
Eureka
8
Inner Loop
7
5
4
3
2
1
Isaac
Dexterity
ManiSkill2
Isaac
Dexterity
ManiSkil12
(a) Outer-loop iterations benchmarks
(b) Inner-loop iterations URDP benchmarks
Figure 8: Statistics of the outer-loop (equivalent to NLC) and inner-loop itcration counts. (a) The number of outer-loop ilerations required across different benchmarks, with black lines indicating the variation range. (b) The statistics and variation ranges of inner-loop iterations for URDP during the experiments.
Influence of maximum similarity The maximum similarity parameter primarily determines the criterion for assessing sample similarity. Both the reward function and the uncertainty scores Un and U, designed in URDP are influenced by this parameter, thereby affecting the overall task score. To examine its impact, we conducted an ablation study by setting w to 0.9 (denoted as URDP w=0.9) and compared with our default setting of w 0.95. As shown Figure Da reducing w to 0.9 leads to decline both the success rate during the training iterations and the final
rate. The main reason, illustrated in Figure 9b] is that with 0.9, fewer samples evaluated per iteration. This occurs because the more lenient similarity threshold causes URDP to overlook some high-quality samples, ultimately resulting in drop in success scores.
Evaluation Sample Size
2.3
10
URDP URDP
8
2.1
1.9
4
1.7
2
NLC
1.5
2
3
4
2
3
4
NLC
URDP outperforms URDP w=0. in the Allergo- (b) The number of samples to be evaluated in the NLC process on the AllegroHand task.
Hand task.
Figure 9: Results of the AllegroHand task under URDP and URDP w=0.9- (a) HNS variation during the NLC process, where URDP w=0.9 remains consistently lower than URDP. (b) Reducing w in NLC substantially decreases the number of evaluatod samples.
Effect Smoothness Parameter The parameter determines the differentiability of the kernel function, which directly inffuences the trade-off between exploration and exploitation during Bayesian optimization. theory, smaller values correspond to less smooth kernels that encourage more aggressive exploration. To
24

---

Published in Transactions on Machine Learning Research (10/2025)
evaluate the effect of the Matern kernel smoothness parameter 2.5 in UABO URDP, conduct ablation study by setting different commonly used values (0.5 and 1.5, denoted as URDP w. UABO0.5 and URDP w. UABOL.5 respectively). As shown in Figure 10 when achieving comparable or identical HNS and SR, URDP (UABO2.s) requires fewer NOE, indicating that the smoothness parameter v not only enhances the efficiency of URDP but also aligns well with its theoretical molivalion.
HNS: HNS:
URDP
70
HNS: 16.38
HNS: 16.31
URDP w. UABO URDP UABOs
HNS: 1.52 HNS: 1.52
HNS:
1.54
I
50
HNS: 1.75 HNS: 1.71
NO
30
10
Ant
FrankaCabinet
ShadowHand
Figure 10: URDP with v 2.5 outperforms other ablations in terms of efficiency.
The impact of length scale l on efficiency. In the Matern kernel of a Gaussian Process, the length scale & determines the rate at which the kernel value decays with distance, thereby affecting the explo- ration ration-exploon exploitattoonn balance of UABO in the parameter space. A larger & slows the decay, increasing the correlation between distant sample points and encouraging global scarch, but may lead to slower convergence. In contrast, a smaller é accelerates the decay, enhancing the sensitivity to local neighborhoods and favoring fine-grained local search, although at the risk of premature convergence. To balance these effects, URDP dynamically sets l to the uncertainty score of the reward function UR, whose value is in the range [0,1]. To assess the advantage of this adaptive length scale, performed an ablation study using a fixed { (denoted URDP w. UABO,) as baseline, shown in Figure 11 The experimental results show that & = Un (URDP) achieves comparable solution quality with fewer NOE, thus improving task efficiency.
URDP URDP UABO,
HNS: 2.64
60
HNS: 1.19
HNS: 2.65
HNS: 1.20
HNS: 1.82
HNS: 1.82
NOE
40
0
Anymal
Quadcopter
Humanoid
Figure 11: URDP outperforms URDP with é [URDDP w. UABO,) in terms of efficiency.
D Case Studies
D.1
Case Study 1: LLMs in Numerical Optimization
This study employs two comparative examples to visualize the differences between URDP and Eureka in optimization processes. Example 1(a) and 1(b) present the respoctive design trajectories of URDP and Eurcka for the ShadowHand task, where red annotations denote reward components and blue text indicates reward intensity hyperparameters.
25

---

Published in Transactions on Machine Learning Research (10/2025)
Table Hyperparameters of the SAC algorithm applied to Maniskil12.
Hyper-parameter
Value
Discount factor Y
0.95
Target update frequency
1
Learning rate
3 x 10
Train frequency
8
Soft update
5 x 10-3
Gradient steps
Learning starts Hidden units per layer Batch Size # of layers
4000
256
1024
2
Initial temperature
0.2
Rollout steps per episode
200
Table 5: Hyperparameters of the PPO algorithm applied each task.
Hyper-parameter
Value
Discount factor y # of epochs per update
0.99 (Isaac), 0.96 (Dexterity), 0.85 (ManiSkill2)
(Isaac), 5 (Dexterity), 15 (ManiSkill2)
Learning rate
5 x 10 (Isaac), 3 x 10 (Dexterity, ManiSkil12) 32768, 16384, 8192 (Tsaac), 16384 (Dexterity), 400 (ManiSkil12)
Batch size
Target KL divergence
0.008 (Isaac), 0.016 (Dexterity). 0.05 (ManiSkill2)
# of layers
(Isaac, Dexterity). 2 (ManiSkill2) 16 (Tsaac), (Dexterity), 3200 (ManiSkil12)
# of steps per update
Analysis of Example (a) demonstrates that during evolutionary search iterations, Eureka exclusively modifies reward intensity hyperparameters while failing to improve reward components. Despite multiple optimization attempts, this approach yields degraded performance. This finding reveals a critical limitation: when simultancously optimizing both reward components and their strengths, L.LMs cannot effectively utilize their inherent advantages semantic correlation and autoregressive modcling, while their deficiencies in numerical optimization become particularly pronounced.
In contrast, URDP's decoupled alternating optimization demonstrates systematic effectiveness. As shown in Example 1(b), URDP proactively explores novel reward components (c.g., replacing "distance reward" with "velocity reward"), achieving a 70.3% performance improvement (from 0.929 to 1.583) before hyperparameter optimization. Subsequent Bayesian optimization further enhances performance with an average 22.1% gain, culminating a 102.5% total improvement over the initial configuration. Example confirms this consistent advantage, providing compelling visual evidence of URDP's superior optimization characteristics.
Example Ia)): Eureka reward functions on ShadowHand (Iteration 2, HNS: 1.045; Iteration 3, HNS: 0.898; Iteration 4, HNS: 1.019; Iteration 5, HNS: 0.955).
Iteration 2:
def compute, rewaardobjectt_ooo torch. Tensor, gosl_rot: torch. Tenaor, object_angvel: torch, Tensor) >> Tuple (torch. Tensor, [strr torch, Tensor]]:
Temperature parameters for revard conponente
orientation_tent temp: float 5.0 Increased sensitivity for orientation alignnent angular_veiln www.wellocccccooooo tamp: float 0.6 Adjusted sensitivity for angular velacity
Compute
orientation
object geal
orientation_ diff 1.0 torch. torch.aba aba(corch. sun(objcct_rot goal_rot, dins-1)) a Quaternion dot product
Reward for mininizing the orientation difference orientation_ exrchh.xppee (- orientation_diff)
11
26

---

Published in Transactions on Machine Learning Research (10/2025)
12
Rezard for spinning the object (oncouraging non-zero angular velocity) angular, welocity. nagoitude wolloctty.nagoouuuee torch. (ong/eott__nggell... din=-1) vegulaa_.elocity_reee
14
torch. tanh (angular_ wewpularrreloottyeeeeee angular_ velaaeeyooocccyyygggtuueee
Rescaled to (-1,
16
Combina the revards with focus orientation alignnent total, reward orientation, resard * (1.0 angular inguaa_velocityy seelocity_rouarce
18
# Veighted conbination
18
Raturn the total revard and individual components reward_ components "orientation_ revoardd orientation reward, "angular_vee cegilarrvvloctty_rveeeee angular_velocity. velocity_reaarr
21
23
20
return potal_reward, cwward_comooentts
28
Iteration 3:
50
def compate, revatttrrerdddjccc.ooom Tensor, goal_rot: torch. Tensor, object_angvel: torch, Tensor) >> Taple Tenoorr, [str, torch. :
31
Tenperatore paranetera for reward conponente
Increased sensitivity crientation aligunent angular gogular_velociyy.terg float 1.0 Adjusted sensitivity for angular volocity
orientation tenp:
Compute the difference in orientation betveen the object and the geal orientation_ 1.0 terch.abettered sentercceeeeeeeettrroo goal_rot, dim--1)) Quaternion det product
56
38
Revard for mininizing the orientation difference orientation_ torch.expi wwwp:oxiinntatiii..toooo..
orientation,diiff)
Revard for apinning the object (encouraging non-zero angular velocity)
angular wwgliiilyy.gggiiuude torch norn anggel, dis--1)
48
angular velllarrveocity_reeeee torch. tanh veleulrr_eloocit_teege vegular. Rescaled to [-1, 1]
46
Combine the rewards with focus on orientation aligunent orientation
nggular.vlocity..ott
Veighted conbination
47
48
Return total revard and individual components covard_compoeente
50
"orientation_ revard"; rrientation_rvaare "angular_vee cericiarvellctty..oo reward": angular_veloctee venilaa_vvlocttyreeaaa
52
return Notar_reward, componenss
56
57
Iteration 4:
compate_ rabjrd torch. goal_rot: torch. torch.Tensor, pajjct_angvell ('nnoo) Onpla(torch.Tii Dict (str, torch.Tensor) Tensor]]:
Teapersture parsmeters
conponente
orientation temp: float = 10.0 Increased sensitivity for orientation aligenent Adjusted sensitivity for angular velacity
angular, valer_elloccttyccc float
63
Computa the difference in orientation batueen the object and the geal orientation_
aba(torch. sun(object_rot goal_rot, dim=-1)) Quaternion product
66
Revard mininizing orientation orientation rovard = torch. oxp(-oorienaation temp orientation_diff)
67
Revard for spinning the object (encouraging non-zero angular velocity) angular valocity nagnitude torch. parch.nannabbjct_.aae angval, din=-1) angular_velooe velilar__veooctty_eeeeeee
70
pecplar_velocity_aaggeee Reacaled [-1, tank velouuarrr_eooiit_eeeeeeee
72
73
aligunent
total, revard = oriantation_ revard x (1.0 angular, velocity_reward) # Weighted conbination
75
Return the total reward and individual componente
77
revard, components =
"orientation_res revarrdd_eevvd;; orientation
79
"angular_veie wonguaar_wlocciy_revvrrd
cooollrrvelocityrroe
50
return total reward, revard_ components
52
54
Iteratioa def hettt__veaaddbbjccccoom torch. Tensor, goal_rot: torch.Tensor, object_angvol: torch. Tensor) -> Paple(torch. [str, torch.Tansori. Tansor]]:
87
Temperature paraneters for revard conponents
orientation,tom
Increased sensitivity orientation aligenent
angular_velocityy veliarr_eloccttyteee float 2.0 Adjusted sensitivity for angular velocity
27

---

Published in Transactions on Machine Learning Research (10/2025)
90
Compute the difference in orientation betueen the object and the geal orientation_dift 1.0 torch-aba aba(torch. sun(object_rot goal_rot, dim=-1)) Quaternion product
92
94
Revard for mininizing the orientation difference orientation, reward torch. exch..xpp(c oxp(-orientation
orientation,diff)
96
spinning object (encouraging non-zaro angular velocity)
angular velocity.nngeee nagnitude torch. (armooooett_aggeeeee dim=-1)
90
angular_velowe wew___elloctty__eeaeee torch, tanh (angular httggualaaeelltyyy..oo www.ooetylteeeepppepepeee angular. www.:eeooootyynagooouee Reacaled to [-1, 1]
101
Combine the rewards with focus on orientation aligunent
total_resse orientation, resarrd (1.0 angular, angular_veloctee velocity,rouards Veighted combination
103
components
reward_ components =
106
"orientation__eeeetteee orientation reward,
veggllar_vlociityrrvvrrdd angular_veles cactaar_elloottyeeeeee
108
reward, revard componetnt
Example 1(b): URDP reward functions on ShadowHand. Iteration 1 (before UABO), HNS: 0.929; Iteration
(after UABO), HNS: 1.165; Iteration (before UABO), HNS: 1.583; Iteration (after UABO), HNS: 1.882.
Iteration (before UABO):
def soupute_rvaarddbjeee.. torch. Tensor,, gosl_rot: Tensor) Tuple (torch. Tensor, Dict (str, torch. Tensor]]
Tamperature orientation, temp 1.0 Raducad terperature for batter sensitivity
Lietance_tenp
Tenperature for distance-based revard
object geal quaternions dot_ produco torch, norch.sum(abjacc (abjactt_rot goal_rot, rot, dim=1)
Compute
product
Ensure the dot product is within the valid range [-1, 1] dot dot_product torch torch.clanp(der clamp(dot_proddcc pproduct, -1.0, 1.0)
11
18
Compute the angle difforence between the quaternions angle_diff diff (2. tot_productr=2 1.0)
16
Orientation roward: exponential transformation the angle difference
oriantation rewardd torch. wwrch.axp(-orreeee (-arieenttaii, temp angla_ditt)
18
Distance-based revard: encourages reducing angle
distance_ revard -angle_diff llogative because we vant to ninimize the difference
Success bonua: revard for achieving the target orientation
23
success threshold: float 0.05 Easier threshold for success
where
succesa_threabben threshold, 100.0, Larger
26
orientation reward, total_reward reward orientation, revard + distanco_revard success_bomes
revard: veighted
revard,
27
Dictionary individual revard components
30
revard, components cetantation_revrdddd orientation_rewen Siietancerrevadd.. distance_revard,
52
35
return reward revard_ compo_conponents
37
30
Iteration 1(after UABO):
40
compate_ (objact Tensor, goal_rot: Terch.) Tuple[torch. Tensor, [str, Tensor]]
Taaperatura parametera
42
orientation, seap 1.1134
Reduced temperature for better sensitivity Teaperature for distance-basen revard
dietance_tens 1.11
44
object geal quaternions dot pot_uct torch, (objecctt_r goal_rot, dinr1)
Compute
product
46
47
Ensure the dot product is within the valid [-1, 1]
49
dot_product torch. alamp(dot_produu -1.0, 1.0)
51
Computa the angle difference batueen the quaternions angle_diff torch. acosh.acos(2.. dot_product*<22 1.0)
axponential
angla
28

---

Published in Transactions on Machine Learning Research (10/2025)
orientation_ wwwddaaepoorotnatiin.trrrm angle_diff)
57
Distance-based revard: encourages reducing angle difference
dietance_revard -angle_diff llegative because vant ninimise the difference
50
Success bonus: resard for achieving the target orientation
61
auccess, threshold: float 0.05 Easter threshold for auccess
success bonus whore
niccess_throsholee 100.0, 0.0) Larger
64
Total revard: veighted sum orientation reward, distance reward, and success bonus total_reed rotal_reward orientation_revard distance_reward success_bomue
68
Dictionary individual revard componenta
68
reward_come components
reward, "diatance, r distance_ revard
"orientation, resard":
71
73
return Notal_reward, roaard_coponents.
76
Iteration 2(before UABO):
78
def coapute_rewee (earrrrteerrdeec.rrott torch. Tensor, gosl_rot: forch.Tensor, object_angvel: torch, Tensor) Tuple(torch. Tensor, Dict (str, torch. Tensor]]:
79
Teaperature paraneters for reward componente orientation_temp velocity.tenp 0.1
81
Compute the difference in orientation batueen the objact and the geal orientation diff porch.norn (objectt_oot gott_rot, dims-1)
Compute the angular velocity nagnitude of the object angvel_nagnitudl nagnitude torch.norm (objoctttt dim--1)
88
mininizing orientation orientation, revard torch. carch.axp(-orteee exp(-orientationn temp orientation_diff)
91
Reward for maintaining high angular velocity (encourages spinning) velocity_revare revard torch. oopp (1.0 (angvel_nagnittan) 1e-6)))
93
Combine the rewards with appropriate veights total, reward
orientation, revard
relacity,rexard
98
Dictionary individual revard components for logging
revard_dict {
100
lorientation_reaaaaa.aa resard": cicattation_rvaare "velacity_reware velocity_rounrer
102
return lotal_reward, revard_dict
106
107
Iteration 2(after
109
def compute, conpute_rovardcee (tb.rrrrarddojjttrrooooo torch.Tensor Tensor, goal_rot: torch. Tensor, abgvet_angvel: torch. Tensor) -> Taple[torch.1ee Dict(atr, torch. torch.Tenaae Tensor]]:
110
Temporature parameters for revard conponents
111
orientation tanp 1.2642
velocity_temp 0.3145
113
the object and geal orientation_dat diff torch, corch.ee (objecttettrro goal_rot, dim--1)
Compute
116
Compute the angular valocity nagnitude of the object angvel_magoitun tugiitude torch
118
torch.norn norn (objectt_ dins-1)
120
Revard for mintnizing the orientation difference orientation_rewart torch.eas (=p(oerentation teap orientation_diff)
121
122
123
maintaining high angular valocity (encourages spinning)
124
velocity_reeare reward torch. wecch.rr((vellottyyteee (1.0 (angvel_ mag.ittgge 1e-6)))
126
Combine the rewards with appropriste veighte total_reward 0.7 orientation_reward 0.3 relocity_rovard
127
129
Dicticnary of individual revard components for logging raward dict "orientation_ revard:; orientation_ reward, "velocity_revard": velocity_revard
130
182
29

---

Published in Transactions on Machine Learning Research (10/2025)
154
135
return total reward, rovard_dict
Example (a): Eureka reward functions Humanoid (Iteration 1, HNS: 2.273; Iteration 2, score: 0.412; Iteration 3, HNS: 0.032; Tteration 4, HNS: 0.127).
Iteration 1:
def compate, (roat [tatt_ttate::: staten: torch. Tansor, float) Tuple[torch. Tansor, [ict torch. : Extract the volocity of the hunanoid's torso (root_states 7:10] contains the Linear velocity) velocity root_etates(:, 7:10]
Compute forward speed (ve assuze the humanoid is moving along the x-axis) forward_speed velocity(;,
Ravard for forward speed (acaled smaller range)
10
speed_ teep 1.0 Reduced temperature for better acaling
rpeed_reward forvard_epeed speed_tenp
18
Reward for coneistency (encourage zaintaining high speed) consistency_ 1.0
15
consistancy_ resard torch.axp (xp((consiittennyy temp * (frorvard ppped forch.neannforrr s_addd))))))))
17
Total revard combines speed reward and consistency rexard speed_revard ronsistency_revart
20
Dictionary individual revard components
rovard_dict
23
"spead resard": revard, "coneisteney__wwrrd"" consietency_rewarc
24
return reuard, revard dict
27
20
Iteration httpeeerraarrrrraaetaaaee torch. terch.Tensor, dt: float) -> [trrct Teasee def coapute_revare velocity hunanoid's torso (roat_states[] E;, 7:10] velocity reot_ [tt_sstaeess, 7:10]
31
Tensor, [ict(str, torch, Tenser)): velocity)
34
Computa forsard apeed (ve the hunanoid is moving along the x-axis) forward_apeed speed velocityl;, 0]
36
Revard for forward speed (scaled to maaler range) apead_ temp 0.2 apeed_reward reward forward_apen
39
tanperature for scaling tenp
41
consistency (ancourage maintaining high speed tine) comeistency_tent 0.5 # Incressed tenperature for better scaling
48
consistency_revert torch. wxwpecooossstttccy.eeeeeo torch.bbe (b torch, (frrchar. (ppeod))
torch.
46
Total reuard
spead revard consistency rexard
revard apeed_revard comeistency_redt _eeard
48
Dictionary individual revard corponents
50
reward_dict =
"speed_rexate" reward"; speed_revard,
52
"consistency_reet resard"; consistancy_ reward,
53
55
return revard, resard_dic dict
57
Itaration 3:
60
def compate_ rew.rddroot_ssaaeeeeorooo (rott_ torch. Tenaor, dt: float) Tuple[torch, Tensor, [ict(str, torch, Tensor]]: Extract the velocity the hunanoid's torso (root_statee[;, 7:10] contains the linear velocity) velocity = root_statee 7:10]
62
Compute the forward speed (se assuze the hunanoid is moving along the x-axis) forxard apeed velocityl:,
67
Reward for forvard speed (scaled to a smaller range)
speed_teap 0.1 Reduced temperature for better scaling reward forward
69
spaod_tonp
Revard for consistency (encourago maintaining high speed) consistency, tamp Increased tanperature
scaling
30

---

Published in Transactions on Machine Learning Research (10/2025)
73
consistency_revarg torch.expp-coee wwwppcconsstttncc..ooooo torch. epeed mean (forear (peed))
75
Total revard combines speed resard consiatency rexard reward speed_reward comeistency_reet reeaard
77
Dictionary individual revard conponents
70
reward, dict
50
"speed_ reward"; spead_revard,
"consistancy. revard"; consistancy_ reward,
52
s4
return revard, dect
56
Itaration 4:
59
def compute_ devard(root_statee) torch. Tensor, dt: float) Tuple (torch. Tensor, [ict torch, Tensor]) Extract the velocity the humanoid's torso (root_statee:;; [t, 7:10] contains the linear velocity) velocity rost_states 7:10]
91
98
Conpute the forward apeed (ve assuze the hunanoid is noving along the x-axis) forward speed velocity(:,
96
Reward for forvard speed (scaled to a smaller range)
speed_temp 0.1 Further reduced tenperature for better scaling spead_revane reward forward_ speed spae
98
100
Revard for consistency (encourage zaintaining high speed over tine) consistency_temp
Increased temperature consistency_reeert torch exppeeep((eee exp(-coneistence_.eee
scaling
torch.abe (forward_epees torch. (wwwwaa (pprrd))
108
Total revard combines spead reward and consistency rexard revard spaad_revard recard consistancy_ ravard
105
107
Dicticnary of individual revard conponents revard_dise dict
108
repeed__ewwrd"; speed_revard,
110
"consistemey_revard" consistency_roward,
112
revard, reward_dic
Example 2 (b): URDP reward functions on Humanoid. Iteration (before UABO), HNS: 1.428, Tteration (after UABO), HNS: 2.132; Iteration (before UABO), HNS: 2.481, Iteration 2 (after UABO), HNS: 2.646.
Iteration (beeoore UABO):
def compute, reward conpate_revard
states: torch. tonso. dt: float) >> Suple[torch. Tuplettorch.Taneor, [icr,str, torch. Tenser]:: bunanoid's
Extract velocity
states
velocity = root_statee!:, 7:10]
Compute the forward speed (assuning x-axis in the forward direction) forsard
forsard_spead apeed valocity[:,
Scale up the spead resard to provide stranger feadback greeed_ewwar__tee temp reward torch. axp(speed_revard_tes peawardd_eepppppp forvard_epeed * dt)
Tenperature paraneter speed
11
12
Optional: Add revard for sustained running
14
austained speed_ 0.1 Temperature paraneter for austained apeed resard jueeaiied_speed__eeeat torch, pet_eestained_epeedd__ooo forward_speed dt)
16
18
total_rewant reward speed_reward sustainedd_pped__reve
19
Raturn the total revard and the individual resard components
21
deward_diet =
"speed_reward"; speed_revard,
28
seettaired__pee___eee sustained searddd__ppedd_eeerree
return Notal_reward, revard
25
50
Itaration (after UABO):
craputte_evarddroot..co
Tensor, Tuple [torch. Tensor, [ict(etr, Temsor]]
Extract the velecity of the hunanoid's torso from the root states
velocity root, noot_statess[, 7:10]
31

---

Published in Transactions on Machine Learning Research (10/2025)
Compute the forward speed (assuning x-axis is the forward direction) forsard_apede velocity[:, 0]
36
58
Scale up the speed reward to provide stronger feedback gaadd_eward__tee temp 10.4424 # Tamperature paraneter for speed reward spead_reward reward forch.exp(spaeee Beruddreerrdddee texp forvard_apeed * dt))
40
Optional: Add reuard for sustained running
43
sustained stpeeddddd____e__________ senp 0.3606 Temperature paraneter for sustained speed revard suetaiaed_epeed_rwerrr torch. (eppeustained_eeee..tteo forward_speed dt)
46
Combina rewards
47
total_reward reward revard + sustained spaedd
Raturn the total reward and the individual revard componenta
50
reward_dict =
rspeed_revard"; speed_revard,
52
"sustained "aastained_spead__eee rewadd" sustained, mustained_spaod, revar
54
rotal_ revtrd_dict
57
58
Itaration 2(bafore UABO):
compate_ (wwerddrrrddrrrteeeeeoo stattettttttt:::
float) [toreh Tensor, [ict torch. Tensor]] Extract the velocity of the huranoid's ' toreo from the root states velocity root_states toreo_bed beight root_states 2] # Torso height (2-axie)
61
64
Computa the forward speed (assuning x-axis the formad direction) forward_ apeed valocityl:, o]
66
68
Apply exponential transforuse oonationnn to the speed reward with teaperature paraneter speed_tenp Tomperature paraneter speed speed_reward torch. exp(speed_tenp forward_speed) dt
71
Panalty falling (torso height balow threshold) thresbold fall_threahold
78
fall
0.8 Minimun torso beight to avoid falling
fall_penalty
beight fall_ floatt)
76
76
total_reward apeed_reward fall_penalty
78
Raturn total revard individual revard componentsa
s0
reward_dict =
"'peed_reward"" speed_revard
rowar_ revard_dict
85
S7
2(after UABO):
def computo, (trttrrrrrrrrttttttrrereree states: torch. Tensor, dt: float) -> Tuple [torch. Tansor, [ict(str, torch, Tanser]:: Extract velocity
90
hunanoid's torso from root statea velocity reat_states[;, 7:10]
92
torso, beight root_statess[,, 2] # Torso height: (2-axis)
94
Compute the forward speed (aseuning r-axis is the forvard direction) forward_apecd velocityl;,
97
Apply exponential transfornation to the apeed reward with terperature paranater speed_teap 0.7111 Temperature paraneter for epeed reverd forward_apeed) dt.
99
speed_reward revard Barch.exp(speed_cee
101
Penalty for falling (toreo height below threshold) threshold fall hall_ponalty -10.0 (torso height fall, threshold) (owaahoodd....(((( fflat() ()
torse height
falling
104
Combine the rewards total_reward reward speed_reward fall_penalty
106
108
Return the total revard and the individual revard components revard dict =
108
sepeed_rexard"" speed_revard
111
113
return total_reward, divt
32

---

Published in Transactions on Machine Learning Research (10/2025)
D.2 Case Study 2: the Redundancy in Reward Function Samples
This case study analyzes two representative examples from Eureka within a single iteration, visually demon- strating major source of computational redundancy, Our analysis reveals that Eureka generates multiple semantically equivalent but syntactically varied reward functions within single iteration, all sharing identical reward intensities.
For instance, in Example 3, although the two reward functions (Sample #9 and #15) from iteration exhibit different textual expressions serruurrd_cllccttyrreeeeee vs. reeaar""t_rrwaard). their underlying reward objectives and logic are fundamentally identical. This observation suggests that the apparent diversity among Eurcka-generatod samples may be primarily lcxical rather than semantic. Effective filtering of such pseudo-diversity is therefore essential to eliminate redundant and compulationally ineffieient evaluations. Example 4 is similar example.
Example 3: The reward function codes of the lask Ant after Iteration 1. The scores of Sample #9 and Sample 15 are both 2.012.
Iteration (sanple #9)
def canpute_revard(roooeeeaaaa torch. Tensor, actions: torch. Tensor) -> Tuple (torch. Tensor, Dict [str, torch. Tensor]] ]
Dafine weight paransters forsard_velocity. float = 1.0
action_penalty_.ee teup:
Extract forward velocity (x-axis velocity the world frens)
forward_velocity root_states(;, Velocity slong the x-aris
10
Revard for forward valocity
12
forward_ wewaardd_eloctt__eeeeee revard forvard forvard_velocity vorvard_velocity..oo
14
Penalty for large actions to encourage energy officiency
16
action_penalty -torch. sun square(actions), din--1) actioa_penally_ter
17
Total rovard
forward voroard_velocity. velocityyreward action_panalty
10
Individusl revard conponents
dict =
22
"forward_ velirrdd_eoccityrreeeeee " action_penally,
24
26
return revard, reward_dict
28
29
Itaration (sample #15):
31
def (rottttttottttttt torch. Tensor, actions: >> Tuple[torch, Tenaor, Dict [atr, torch. Tensor]]
58
Define weight paraneters selocity_tompi.
54
aettan_penallyy temp: float 0.01
56
Extract forsard valocity (x-axia velocity) forward_velocke
38
7] # x-axia velocity is at index 7
40
Revard for noving forward fast selocityyresaac forsard, valocity velocity,
48
Penalty for large act sone encourage energy efficiency
caston_panaltty -torch. wwrrch. aeerrch.guuraeeeee sacttnn))) dins-1) facph_pnaaal__eetee
46
47
total_reward reward velacity_resen reward action_penat penaltty
48
Individual revard componenta for debogging and analysia
50
reward_ componetee = mollocity_revarde velocity_reune revard, : action_penalty,
52
56
return total_ reward, revard_ components
33

---

Published in Transactions on Machine Learning Research (10/2025)
Example 4: The reward function codes of the task Ant after Iteration 1. The scores of Sample #3 and
Sample #14 are both 0.059.
Iteratica (sample A3):
def cetputte_reear(objeee.... torch. Tenaor, gosl_rot: torch. tonao.Temaor, object_angvel: torch. Tensor) >> Tuple [torch. Tenaor,
Dict[atr, torch. Tensor]]:
Temperature paraneters for reward conponents
orientation_tenn temp: float 1.0
angular_vei_tenp
Compute difference in orientation using quaternion distance
qust_diff quat_aml (objecttto quat_ conjugete(goal_roo orientation_erron error torch corn(quat_ diffl: 1:4], din-1) Ignore the scalar part for distance orientation, torch. (=arieenttatin
orientation,
11
Penalize angular_vel_per pei_eaalyy torch.exp(- wal-aaanggaallaaaeeaaa angular_vel vel raggiiiddd)
angular velocity
18
14
Combine rewards
18
total_reward reward orientation_revare revard angular_ vel__ rel_pety
18
Return the total reward and individual components for debugging conponette
20
"orientation_ : diientation_rowardd "angular, vel, penolty"" angular, pecilaarvel__eaalee
23
return total_ total_reeard revard, greard_ccmponant
26
27
Iteration (sample #14):
compate, (objact rot: Torch.Tensor, qoal_rot: Tenchrr objact_angvell torch. Tanaor) Tuple Tuple[torch. [toocc.Tensorr, Dict [str, torch, Tensor]]:
29
Tenperature paraneters for reward conponente
orientation_tant 1.0
31
angular_vedd www.cttyy.oooococe
0.1
Compute the difference in orientation using quaternion distance quat_diff diff quat, mul (object_ rot, quat, conju rot))
34
orientation_error torch norn (quattuaaiiiff 1:4], din-1) Ignore the scalar for distance orientation_reeaadd
56
wxp(-orientatiion
orientation_error)
38
Panalize excessive angular velecity
angular_ uaguuty
nor.ooojeooobeeeeeeooeoo angeel, din-1)
angular bellocity_paaaatyy earch.erp(-angular wtpp:aanular..volocttt..cccc angular www..eety.ooggiiiiiio. nagnituds)
41
Combine rewarde
43
total_reward reward orientation_revvar revard angular_vett relocityy.emallt
45
Return the total revard and individual componenta for debugging conponenteeee "orientation_ revard": orientation reward,
48
"angular_velen www.vveellcyyeeeeeee penalltyy"p aegular_velocity..om
50
return total, total_reward, evoard_compoenttt
E Detailed Results
E.1 Evaluation on Efficiency
Table 6 presents the comprehensive evaluation results across all tasks in the three benchmarks. The comparative analysis demonstrates that while achieving comparable SR or HNS to Eureka, URDP requires fewer simulation training episodes and LLM invocations 92% of the experimental tasks, indicating superior sample efficiency and computational economy.
E.2
Evaluation on the Performance of the Reward Function
Table presents comprchensive performance comparison of different methods across all tasks in three benchmarks. The results demonstrate that URDP consistently outperforms the baseline approaches while maintaining comparable reduced requirements for both simulation training episodes and LLM invocations.
34

---

Published in Transactions on Machine Learning Research (10/2025)
Table 6: URDP vs. SOTA with efficiency. URDP performed best in 92% of tasks in terms of NOE and NLC (bolded parts in the table).
Benchmark Environment
Reward
Eureka
UROP
HNS
NOEJ
NLCJ
HNS
NOEL
NLCJ
HNS
NOEJ
NLCJ
Ant
1.527
Cartpole
16
16
15
1
RallBalance
16
16
1
Quadcopter
1.678
1.667
70
1,818
Isaac
FrankaCabinet
16.95
95
17
97
17.130
57
4
Humanoid
Anymal
1.095
87
1.113
91
1.2
47
2
Alne
2.176
2.162
2.182
4
ShadowHaad
1.805
111
1.786
105
1.817
33
2
SR
NOEJ
NLCJ
SR
NOEJ
NLCJ
NOEJ
NLCJ
BlockStack
0.67
112
0.67
112
0.68
Kettle
0.89
85
0.89
72
0.89
78
OutteleseOutward
0.96
83
0.9
72
0.97
37
2
DooeCleeerwward
78
SwingCup
0.84
93
0.84
87
0.84
51
3
Switch
62
0.0
62
0.0
62
CatchUnderarm
AatchAbreast
0.66
0.66
83
0.67
54
Dexterity
DoorOpen
0.04
73
0.04
69
0.06
67
PushBlock
0.14
BottleCap
0.88
110
0.88
96
0.89
67
4
ReOrientation
CatchOOver/Unddeearm
0.91
77
0.9
74
0.93
57
3
Lif.Underarm
Over
3
Pen
0.85
95
0.85
97
0.85
78
poutOaen
Scissors
76
1.0
77
44
3
GraspAndPlnn
NOBJ
NLCJ
NOBL
NLCJ
NOEL
NLCJ
LiftCube
0.906
112
0.905
96
0.906
15
1
PickCube
ManiSkil12
TurnFaucet
0.799
96
0.800
96
0.801
34
2
OpeuCabinet
0.865
0.861
96
0.866
31
2
OpenCabinet Drawer
0.633
112
0.632
96
0.638
2
PushChair
0.657
96
0.654
96
0.657
51
2
35

---

Published in Transactions on Machine Learning Research (10/2025)
Notably, URDP achieves superior performance to human-designed reward functions in 89% of the experimental tasks, highlighting the significant potential of automated reward design methodologies.
Table 7: Task-wise comparison of URDP with other methods. URDP outperforms the compared methods 89% of the tasks.
Benchmark Enviromment
Sparse
Human
Reward
Eureka
URDP
HNSt
HNSt
HNSt
NOE
NLC
HNST
NOE
NLC
HNST
NOE
NLC
Ant
0
0.772
48
0.828
48
3
1.556
48
Cartiooee
1
15
15
BallBalance
0
1
16
16
16
Quadcopter
0
1.041
41
1.25
41
3
1.818
41
Isaac
FrankaClbinet
5.4
5T
57
Humanoid
0
2:217
52
2.306
52
4
2.646
52
Anymal
0.317
47
AllegroHand
0
1.196
40
1.594
40
2.182
40
ShadowHand
1.034
33
1.113
1.817
33
SR1
SR1
SR1
NOE
NLC
SR1
NOE
NLC
SR1
NOE
NLC
BlockStack
0
0.69
0.11
53
0.12
53
0.679
53
0.02
0.89
78
72
DuorClos:Outwanl
0.15
0.06
0.57
37
0.64
37
0.968
37
DnorClosetwward
0
0.74
34
0.83
34
34
SwingCup
0.62
0.53
0.84
Switch
0
U
76
0.01
76
0.02
76
UnddracnUnnderarm
CatchUnderarma
0
0.51
0.58
67
0.63
67
0.73
67
CatchAbrease
0.37
0.27
54
0.34
0.67
54
Dexterity
DnorOpenlnward
0.03
67
67
PushBlock
0.01
0.05
49
0.05
49
0.15
49
BottleCap
0.91
6T
ReOrientation
0.01
0.02
0.25
58
0.28
58
0.33
58
CatchhOverrunddeerrm
ST
57
Underamm
0
0.37
0.83
78
0.85
78
0.89
78
Over
0.9
0.54
41
0.61
0.92
41
Pen
0.01
0.74
0.67
78
0.63
0.85
78
PborOpenOutwand
0.02
0.85
0.76
46
0.87
46
46
Sciseors
0.99
GraepAudPlace
0
0.87
0.41
59
0.43
59
0.77
59
SRt
SR1
SRt
NOE
NLC
SRt
NOE
NLC
9R7
NOE
NLC
Libe
0.143
0.543
0.531
15
0.356
0.906
15
PickCube
0.131
0.179
0. 0.197
20
0.434
20
2
0.885
20
TurnFaucet
ManiSkill2
OpenCabinetDoor
0.028
0.651
0.713
31
0.575
31
0.866
31
Open CrenCaabittDrawer
0
0.37
0.519
43
0.478
0.84
43
PushChair
0.334
0.432
51
0.336
51
0.657
51
F
Proofs
F.1
Determination of the Kernel Function
Theorem 1. The kernel function equation 0 satisfied the properties of symmetry and positive semiidefiiniteness.
Proof. The property of symmetry is obvious. Now we prove the positive semi cemi-definiteness based on the properties of the Matern kernel equation For any given finite set of sample points ((1), ((2), ( ((n) we denote the corresponding kernel matrix as
- p()))
(10)
By performing a coordinate scaling transformation on the sample points, we obtain new sample points
p (019/1, /h.... .p /la,, i = 1,2, ..
(11)
And the Matern kernel matrix is
Kij (0) (00)
(12)
36

---

Published in Transactions on Machine Learning Research (10/2025)
Since Matern kernel equation 4 is positive semi-definite, the kernel matrix equation 12 constructed from the transformed points with Matern kernel is positive semi-definite. Furthermore, the kernel matrix of equation P is essentially equivalent to the Martern kernel matrix equation 12 computed on the transformed sample points, i.c.
nnew (p"").p0)) -r(()),)) =r()))p())
(13)
Ki k( .(01) = fr(wee = fr(r) = K
(14)
Therefore, K is positive semi-definite.
In the newly defined weighted distance equation larger 4 in directions with more rapid variations increase the possibility of exploration, while a smaller l in directions with smoother varialions will reduce exploration and emphasize the exploitation of information from previously sampled points.
F.2 Convergence analysis of the Uncertainty-acellaaate Expected Improvement (uEI)
The basic definitions and theorems have been defined to analyze the convergence rate of Bayesian optimiza- tion (Bull 2011). Here, we briefly restate some of the key definitions required. Let X C Rdd be a compact set with non-empty interior. For a function f X + RR to be minimized, Ka is the correlation kernel for the function If prior distribution л with length-scales
Hx(X) is the reproducing- kernel Hilbert space of Kg on X. Let Py and my denote the probability and expectation operators when minimizing the fixed function f using strategy u. The loss suffered over the ball BR in Ho(X) after n steps by a strategy defined as,
Ln (u. Ho(X), =
sup (EHa(X) II I
EJ [f((t) min
(15)
where ry is the estimated minimum of It is proved that the strategy expected improvement converges at least at rate (wAll/d up to logarithmic factors, where is the parameter in the Matern kernel (Bull 2011).
Theorem 2. Assume that the function depends only on m input variables, m < d, and remains constant along the other d-m directions. Under such on assumption, with an appropriate choice of weighted parameters, the Uncertainty-acceleniii factors, where is the porameler in the Malern kernel.
Expected Improvement converges at least at the rate n- nu(uA))/m up to logarithmic
Proof. From the proof of expected improvement. convergence rate (Bull 2011 d in convergence rate estimation is actually the dimensionality of the sampling space. And the conclusion holds based on the condition that {.a} } is a quasi-uniform sequence in a region of interest (Narcowich et al. 2003 Without loss of generality, let us assume that the function f depends on dimensions in to im, and is invariant with respect to dimensions im+1 to ip. For the uEI strategy, let
we observe that the parameter
0, j = 1.... o0, j = m + d
A
(16)
Consequently, any exploration in the directions of dimensions fm+1 to ia will be discouraged. The effective dimensionality of the sampling space decreases from to m, which leads an improved convergence rate of least n (vAl)/m
Although our theorem has focused on the limiting case in which f entirely independent of certain directions, it illustrates how applying weighted constraints allows for dimension-specifee treatment within the sampling space, thus enhancing the efficiency of the algorithm.
37

---

Published in Transactions on Machine Learning Research (10/2025)
G Additional Analysis
G.1 Uncertainty and Reward Shaping
To validate the role of high-uncertainty reward components, conducted ablation studies by removing these components from the reward function. Figure 12 presents comparative cases between the original reward functions (Rt and its ablated counterparts (Rt w.o. Tur). Our analysis reveals two key findings: (1) the removal of high-uncertainty components lcads to significant performance degradation, with respective decreases of 19%, 83%, and 51% in HNS/SR metrics; and (2) reward functions retaining these components demonst accelerated discovery of critical states curing early RL training phases, effectively reducing inefficient exploration. These results collectively demonstrate the crucial function of high-uncertainly components in both final performance and training efficiency. These results suggest that high-uncertainty reward components contribute positively to reward shaping and play an essential role in guiding effective policy learning.
0.6
0.6
146
0.4
INS
05
5-
0.2
0.2
0.0
600
1900
1000
2000
2500
3000
500
1000
1500
2000
2500
3000
500
1000
1000
2000
2500
3000
Step
Step
Step
(a)
Anymal
(b) Franka Cabine
(c) LifUnrdrarm
Figure 12: The comparison between and W.O.fat suggests that the high-uncertainty reward components Faf (dof vel penally in Anymal, progress. reward FrankaCabinet, drop penally Lifi Underrr)))) contribute to reward shaping during the policy learning.
G.2 LLM Alternatives
URDP with Qwen2.5 and Llama3. In Figure 13 we compare the performance of URDP with DeepSeek-v3- 241226 (the results reported in the paper), URDP with Qwen2.5 (qwen-max-0911) (Qwen et al. (025), and Llama3 (llama-v--70b-instruu)
(Dubey et al. 2024). These results demonstrate the consistency of the effect. of URDP different LLMs and eliminate concerns that the differences in the capabilities of LLMs themselves may affect the results.
URDP vs. URDP(Qwen2.5) vs. URDP(Llama3) HNS
2.5
17.13 17.65 17.05
2.65 2.66 2.62 2.18 2.14 2.11
1,79
1.84 1.79
1,56
1.5
1.00
1.00 1.00
1.00 1,00
0.5
0.0
Cartpole
BallBalance
Quadcopter
FrankaCabinet
Ant
Humanoid
AllegroHand
ShadowHand
URDP
URDP(Qwen2.5)
URDP(LIama3)
Human
Sparse
Figure 13: URDP demonstrates consistent performance across different LLMs.
Limitation and Discussion
In this work, we investigate efficient automated reward design methodologies based on large language models (LLMs). However, constrained by the inherent limitations LLMs in spatial reasoning capabilities, our approach, like other comparable methods, faces challenges in addressing scenario-spocific constraints during reward formulation. representative case emerges in "grasping" tasks where environmental obstacles may restrict robotic manipulation paths, constraints that should ideally be reflected in reward design. While
38

---

Published in Transactions on Machine Learning Research (10/2025)
providing detailed environmental descriptions in prompts may partially mitigate this issue, more fundamental solution would involve inlegrating video-language models (VL.ss) into the reward design framework. VL.Ms demonstrate superior spatial perception capabilities that could enrich the understanding of RL task objectives, environmental constraints, and reward composition. Nevertheless, incorporating VLMs introduces new challenges regarding computational scalability during reward design and tuning processes. We therefore identify this as critical yet underexplored rescarch direction worthy of systematic investigation.
To maintain simplicity in presenting work, employ the base capabilities of large language models without sophisticated inference- time enhancement techniques (e.g., thain-of-thougee test-time training). However, advanced reasoning techniques have demonstrated significant improvements in handling complex logical tasks, as evidenced in code generation and mathematical reasoning domains. We posit these methods would similarly enhance reward function code design. Ultimately, substantial exploration potential remains for large language model techniques automated reward design.
39


Reviews:
[{"review_id": "JYThHpBmMr", "reviewer": "Reviewer_gtm5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper propose a extension to the LLM generated reward design field by proposing a separation scheme between reward function generation and reward weight tuning. The authors use LLM to design specific reward functions and use Bayesian Optimization to find the best reward weights for each reward. Uncertainty qualification is applied for enhancing the reward function generation as well as improve the BO optimization step with a uncertainty aware optimization methods. The method demonstrates superior performance and efficiency against baselines across multiple benchmarks. Extensive case study and ablation study signifies the effectiveness of the proposed method, bundled with adequate convergence analysis. \n\n## Strength\n1. Interesting topic and solid approach. The approach is adequately justified and implemented. The separate of reward design and weight optimization makes sense. \n\n2. Solid benchmark performance against baselines across multiple benchmarks. \n\n\n## Weakness\n1. The writing of this paper can be greatly improved. I find the explanation in the method section a bit unclear. For example, what is feedback prompt and what's the approach used here? I.e., how to judge from the simulation and give feedback to LLM for calling again?\n2. The authors claim current methods has \"(1) insufficient reasoning capability for reward logic derivation, and (2) inadequate exploration of novel reward components\". But it is not very clear how the proposed method address these problems effectively. What experiments is demonstrating reward logic derivation and novel reward components? I am aware of Fig. 7, how does this directly imply novel reward components? For starter, what are these reward terms? I think big claims need more experiments and analysis to demonstrate.\n3. The authors claim in the abstract that \"evolutionary search paradigm demonstrates inefficient utilization of simulation resources,\nresulting in prohibitively lengthy design cycles with disproportionate computational overhead\" This gave me a falsehood of expectation that the authors have tricks to address this more directly. But I couldn't see it from the method section or am I missing something? Except from directly doing BO over the parameters, I don't see special treatment of the simulation. If so, the claim sounds a bit overreaching. \n4. For the method itself, I find it a bit hard to follow. What is \\\\( y\\\\) in this context? What is the objective trying to achieve? Suppose you have a reward function generated by LLM, and you have an optimization trying to find the best reward parameter, then what is EI trying to achieve here? Please write the method section more rigorously. \n5. Since this is related to LLM, what is the exact LLM tool you use for all the experiments including the baselines? As LLM keeps evolving, the ability to generate evolves too. Only through a fair comparison can we draw meaningful conclusions.\n\n\n### Minor Issues\n1. In fig 1, in middle box, is this supposed to say Uncertainty Quantification? The fig itself is of low quality.\n\n2. \\\\( U_{R_i} == U(r_i)?\\\\)", "claims_and_evidence_yn": "Yes", "claims_explanation": "The experiments shown clearly demonstrate the effectiveness against baselines.", "audience_yn": "Yes", "audience_explanation": "Designing reward function and finetuning reward has been a long standing obstacle in the robotics learning community. This work automates the reward designing process, which looks quite interesting and promising.", "requested_changes": "1. The citation format is very distracting. Please change to the correct format with parenthesis and only use NAME, (YEAR) when it is used as a noun.\n\n2. The overall texts looks overwhelmingly generated by language models. I suggest the authors render the texts more carefully. Some sentences I think are overly decorated, such as\ni.  (section 2) Our proposed framework decouples confounding factors in reward design to reduce cognitive load and incorporates uncertainty quantification to guide LLMs toward more focused analysis and refinement of reward logic relevance.\nii.  (4.1) insufficient attention being paid to the optimization of reward components in agent learning.\n\n3. Although I recognize the baseline method (Eureka) have a similar comparison scheme, but it seems that both this work and Eureka claim that LLM can design better rewards than humans. However, given that the current robot community is yet to embrace such generation scheme, I find that claim a bit overreaching. For example, what if we compare LLM generated reward to train an imitation agent and compute joint tracking performance, versus reward designed by humans? Such comparison might be more meaningful here since the metric used, Human Normalized Score, does not make sense to me, or I need further explanation from the authors how this is a valid metric to compare to. Success rate is a good metric, so I still think this work presents a meaningful improvement over reward function genereation.", "broader_impact_concerns": "No broader impact concern as far as I can tell.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YO7U5EBPvo", "reviewer": "Reviewer_iGv2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**Summary:**\n\nThis work proposes URDP, a novel uncertainty-aware reward design paradigm for RL that introduces a bi-level optimization architecture including reward component generation and hyperparameter optimization. The method leverages LLMs' generative capabilities to produce diverse reward components in the outer loop, while employing Uncertainty-Aware Bayesian Optimization (UABO) to learn optimal hyperparameters in the inner loop, with a novel uncertainty quantification mechanism strategically selecting the most promising components to reduce computational costs. Through comprehensive experiments across three benchmarks, the authors demonstrate URDP's effectiveness, particularly highlighting its hybrid approach that combines LLMs' creative reward formulation with Bayesian optimization's numerical precision.\n\n\n**Strengths:**\n\n1. This work thoughtfully addresses the critical challenge of reward function design in reinforcement learning.\n2. The choice of Bayesian optimization proves particularly valuable for handling numerical optimization tasks where LLMs typically struggle. \n3. Through comprehensive benchmarking experiments and carefully designed ablation studies, the manuscript demonstrates both the method's effectiveness and the necessity of its individual components.\n\n**Weaknesses:**\n\n1. Some methodological descriptions could benefit from further clarification:\n    - the difference between $R_i$ and $r_{i,1}, \\dots, r_{i,m}$.\n    - the descriptions of the $(x,y)$ data used in Bayesian Optimization.\n2. The approach also involves several hyperparameters (e.g., smoothness $\\nu$ and length scale $\\ell$), and discussing UABO's sensitivity to these parameters would strengthen the technical discussion.\n\n\n**Question:**\n1. Would you clarify which specific LLM performed the textual/semantic similarity calculations for uncertainty quantification?  \n2. Could you elaborate on the technique used to balance exploration-exploitation in Section 4.2, and how you determined the iteration counts for the inner versus outer optimization loops?  \n3. To what extent does the choice of generative LLM (e.g., Qwen, LLaMA, or GPT-4 Omni) influence final performance?  \n4. Might you explain the meaning of \"NOE\" and the specific simulation context where this was applied?", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims are convincingly supported through comprehensive benchmarking across three domains, rigorous ablation studies validating each component's necessity, explicit efficiency analysis showing computational gains, and systematic comparisons against established baselines demonstrating consistent performance advantages.", "audience_yn": "Yes", "audience_explanation": "This paper's novel integration of LLMs with Bayesian optimization for automated reward design addresses a fundamental RL challenge, directly appealing to researchers in reinforcement learning, Bayesian methods, and generative modeling through its demonstrable performance gains across established benchmarks.", "requested_changes": "I have one suggestion regarding the algorithm's positioning: It would enhance readability to relocate Algorithm 1 to a later section. Currently, it appears before readers encounter essential definitions like the distinction between $R_i$ and $r_{i,1}, \\dots, r_{i,m}$, the composition of dataset $D$, and the specific prompting methodology. Presenting these foundations first would better prepare readers to understand the algorithm's mechanics and significance.", "broader_impact_concerns": "No ethical concerns are noted.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "94wHJNKZ7h", "reviewer": "Reviewer_vsaA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes URDP, a novel framework for automating reward function design in reinforcement learning (RL) using large language models (LLMs) and uncertainty-aware Bayesian optimization. The authors argue that conventional reward engineering and inverse RL approaches are either too reliant on human expertise or costly in terms of demonstration data, and that recent LLM-based approaches still suffer from inefficiencies in reward quality and optimization. The proposed URDP decouples reward logic construction and numerical hyperparameter optimization, introducing a bi-level architecture supported by uncertainty quantification and hybrid optimization. The method is evaluated across 35 tasks in three benchmark environments.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The authors provide comprehensive experimental results.", "audience_yn": "Yes", "audience_explanation": "Given the novelty of combining LLM reasoning with reward function generation and optimization, and the growing interest in automating RL pipelines, this paper would appeal to subsets of the TMLR community, particularly those working on: Reinforcement learning automation, LLM-based control or code generation, Program synthesis and reward modeling, Optimization under uncertainty.", "requested_changes": "1. The paper fails to clearly define the optimization problem being solved. It is unclear what the exact objective functions are and how the components of reward generation and tuning are jointly or separately optimized. A formal problem definition would be helpful to ground the framework in established RL terminology.\n\n2. The manuscript mentions prior work involving evolutionary search and introduces uncertainty-aware Bayesian optimization but does not explicitly compare or connect these paradigms. It is crucial to explain how URDP differs from, improves upon, or integrates with these approaches, especially in how LLMs are used within this hybrid framework.\n\n3. Currently, it is unclear what constitutes a \"high-quality\" reward function. Is it measured by policy performance, training sample efficiency, convergence speed, or interpretability? Without this clarity, the claims in the abstract and results lack a solid basis.\n\n4. The explanation of both RL fundamentals and the specifics of the reward design problem is insufficient in the preliminaries. For a reader unfamiliar with recent LLM-RL research, the paper may be difficult to follow.\n\n5. Figure 1 is visually appealing but does not make clear how URDP fits into the overall RL training loop.\n\n6. Several parts of the paper suffer from vague phrasing and a lack of precision.\n\n7. The abstract does not effectively communicate the problem, proposed method, or key results.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)