Discovery of the reward function for embodied reinforcement learning agents

Domain: physics
Published: 2025-12-11
Editorial outcome: accept

Abstract
Reward maximization is a fundamental principle in both the survival and evolution of biological organisms. In particular, in the contexts of cognitive science and embodied agents, reward-driven behavior has been widely regarded as important in terms of governing complex cognitive abilities such as perception, imitation, and learning. Among the frameworks that are aimed at establishing such abilities, reinforcement learning (RL), which leverages reward maximization to facilitate intelligent decision-making in embodied agents, has been proven to be particularly promising. Importantly, the inherent complexity and uncertainty of real-world tasks pose significant challenges when designing effective reward functions for embodied RL agents. Conventional methods typically rely on manually engineered or externally tuned reward signals, and therefore require significant domain expertise, associated with considerable human efforts and a long convergence time; these issues may even trigger mission failure. This work introduces a bilevel optimization framework that discovers optimal reward functions for embodied reinforcement learning agents through a mechanism called regret minimization. The approach accelerates policy optimization and enhances adaptability across diverse tasks. These findings can support the broader adoption of embodied RL agents in the behavioral and computational sciences and neurosciences, thereby paving the way for artificial general intelligence.

Introduction
Embodied artificial intelligence (EAI) aims to enable embodied agents with specific morphologies (e.g., joints, limbs, and motion capabilities) to learn effective control policies1. A long-standing debate in this field concerns whether these policies should be general2 or task-specific3. Recent advances in large-scale data technology and cloud computing4 have fueled interest in artificial general intelligence (AGI), which aims to develop EAI systems exhibiting general adaptability and embodied intelligence5,6. Naturally, this motivates the pursuit of universal control policies that allow embodied agents to adapt seamlessly to various scenarios7. However, existing machine learning frameworks seldom provide robust solutions for such generality. Reinforcement learning (RL), a versatile decision-making paradigm based on reward maximization8, has achieved notable breakthroughs across various domains over the past few years. For example, RL has surpassed human performance on Atari games9 and StarCraft II10. Another breakthrough in RL is that of robotic control. Robots acquire policies in a similar way to human learning11 and even engage in lifelong learning12. Recently, RL has demonstrated immense potential in biomedical applications, ranging from uncovering cellular movement mechanisms13 to facilitating rehabilitation14. Most recently, DeepSeek has incentivized the reasoning of large language models (LLMs) via RL without supervised fine-tuning as a preliminary step and has achieved remarkable reasoning capabilities15. These advances suggest that RL could serve as a foundational framework for general EAI.

EAI and associated abilities can be understood as efforts that are aimed at maximizing rewards, which is a core principle of both biological16 and artificial systems17,18. Accordingly, reward is sufficient to drive behavior that involves abilities intrinsic to both natural and embodied intelligence, including perception, imitation, learning, and generalization19–21. This may be surprising, as the diversity of such abilities seems to be at odds with a general objective. However, the natural world, in which animals and humans live, and the environments that EAI systems encounter are inherently complex, and sophisticated attributes are required to succeed in such settings. For example, as shown in Fig. 1, the monkey brain can be conceptualized as exhibiting a form of natural intelligence, and the behaviors thereof can be interpreted as following a policy that maximizes cumulative rewards (bananas). Natural organisms tend to embrace various types of cognitive intelligence that improve life22. Likewise, in robotic EAI systems, the reward maximization principle lead the robots to develop diverse intelligence-driven abilities17. The physical structure of the robot functions as the execution system that receives orders from an embodied intelligent brain that optimizes task execution. During this process, the robotic agent seeks to perform optimally and, at times, even demonstrates capabilities in certain areas that closely resemble those of natural intelligence12,23,24. Therefore, the general objective of reward maximization includes many or all of the goals of intelligence25–27. Furthermore, embodied RL agents that learn how to maximize rewards via trial-and-error can acquire behaviors that exhibit most of the capabilities described above. Some researchers have suggested that powerful embodied RL agents could open a path toward AGI28,29. Specifically, during RL, rewards serve as immediate feedback signals to evaluate the embodied agent’s ability, shaping the outcomes of RL in terms of imparting intelligence to the agent30. The quality of the reward function plays a pivotal role in fostering human-like intelligence, as it directly influences the performance and generalizability of RL intelligence, particularly in open-world EAI systems7. On the one hand, different reward signals shape distinct learning dynamics, giving rise to diverse forms of RL intelligence31,32. On the other hand, a high-quality reward function may offer deeper insights into the underlying mechanisms, driving RL intelligence development and the emergent capabilities. Traditionally, reward function design relies on an iterative, manual trial-and-error process33, as illustrated in Fig. 2a. This manual process is not only time consuming and resource intensive34 but also susceptible to overfitting35, creating learned agents that display unintended behaviors, such as reward hacking36, or even inducing the cobra effect paradox. Unsurprisingly, researchers agree that automatic design of the reward function would both eliminate the significant manual demand and improve agent performance37, in turn broadening the applicability of RL across a wide variety of tasks. The design of reward functions for embodied RL agents can be automated via two primary approaches: acquiring external information38–42 or leveraging self-learning processes43. When considering access to external information, reverse engineering the reward function offline using expert demonstrations38 facilitates rapid application of RL to complex tasks40,44. Meanwhile, active learning of reward functions via human feedback45 enables real-time and interactive reward optimization39,42. The two approaches are shown in Fig. 2b. Inverse RL (IRL) and active reward learning (ARL) have significantly improved RL performance in a variety of simulated environments44, including robotics46 and LLMs47. However, IRL often requires optimal demonstrations41, and ARL needs dense and substantial human feedback, which can be challenging in real-world EAI tasks48. Thus, reliance on external information remains a barrier to the swift deployment of embodied RL in real-world scenarios49. A promising alternative approach is automatic learning of the reward function on the basis of the past performance of the agent50–52. Among the various methods, intrinsic motivation-based reward learning is the most widely adopted43. Here, the reward function is generated via heuristics53 or task-independent metrics54. However, despite progress in constructing and exploring reward function structures from various perspectives, a widely applicable method of reward function design that is fully independent of external knowledge has yet to be found. Furthermore, clarifying the formalization of the optimal reward function remains a challenging task.

This work overcomes the existing research bottleneck and takes a significant step toward discovering the optimal reward functions for embodied RL agents. The core insight is that the optimal reward function can be automatically discovered on the basis of the past performance of embodied RL agents with minimal reliance on task-specific prior knowledge. This enables the embodied RL agent to determine the optimal policy rapidly while ensuring convergence. Building on this insight, an approach for discovering the optimal reward function for embodied RL agents is proposed in this work. In this context, the optimal reward function is first defined, and a framework is then developed that seamlessly integrates the search for the optimal reward function with the RL process. While previous works have explored various definitions of the optimal reward function55,56, this work formulates an intuitive and straightforward definition based on regret minimization, as shown in Fig. 2c. This yields a practical form of the optimal reward function that can be easily applied in most RL settings. The discovery of the optimal reward function within the embodied RL agent is conceptualized as a meta learning problem, enabling the integration of optimal reward function discovery with the pursuit of the optimal policy. Although meta learning problems have been explored in prior works50,57,58, we here propose a metagradient algorithm specifically tailored to this setting, creating a bilevel optimization framework that incorporates the RL process, in which RL policy optimization is the lower-level process and the discovery of the optimal reward function is the upper-level process. The overall bilevel optimization framework is shown in Fig. 2d. Regret minimization is the optimization objective of the upper-level. At the upper level, a reward function is established to provide reward signals for the agent. At the lower level, the embodied RL agent interacts with a world model and receives reward signals generated by the upper-level reward function. The RL agent seeks to maximize the received rewards. During practical implementation of the proposed approach, the RL agent continuously optimizes its policy, maximizing the cumulative reward signals until a terminal state is reached, at which point the interaction trajectory is stored in a trajectory buffer. During upper-level optimization, a mini-batch of trajectories is randomly sampled from the trajectory buffer and decomposed into multiple interaction steps. These steps are subsequently used to estimate the policy distribution and the advantage function, which serve as the basis of approximation of the reward metagradient via Theorem 2. After the reward function is updated in the upper-level, the embodied RL agent resumes its interaction with the world model, receiving new reward signals. This alternating optimization is iteratively performed to improve both the policy and the reward function. Furthermore, this method is extended to accommodate both value-based and policy-based agents, and its generalizability is rigorously explored through theoretical analysis. The reward function discovered by the proposed framework is shown to effectively identify the optimal action for the embodied RL agent in a given state, as shown in Fig. 2e, f. Compared to manually designed reward function, the discovered reward function assigns more appropriate rewards to each action, thereby mitigating the risk of incorrect decisions that may arise from poorly designed reward functions (right side of Fig. 2e). For additional implementation details, including the collaborative optimization of the reward function and RL policy, refer to the pseudocode provided in Supplementary Methods. Empirically, the effectiveness of the proposed approach is evaluated in highly realistic simulators and real-world scenarios in a progressive manner, including various tasks such as sparse-reward tasks, high-dimensional control tasks, and real-world tasks. In contrast to the sparse rewards offered in the classic control environments of OpenAI, the reward functions discovered using the proposed approach are dense and, therefore, effectively guide agents that learn policies. In high-dimensional tasks with MuJoCo, a comparison of manually designed reward functions and those autonomously discovered by the proposed approach reveals that the latter effectively focus on critical states, such as success or failure. In real-world evaluation scenarios involving automated energy management of data centers and unmanned systems control of unmanned aerial vehicles (UAVs), the optimal reward function discovery approach not only equips diverse embodied RL agents to perform real-world tasks more effectively than before but also improves their adaptability and accelerates learning. In addition, the proposed approach enables the identification of critical states (e.g., UAV crashes) and reveals the intrinsic latent relationships between rewards and states, despite the absence of an explicit mathematical model. These findings highlight the potential of the proposed framework as a robust means of autonomous discovery of optimal reward functions for embodied RL agents. Consequently, this framework contributes to the scalability of embodied RL agents and highlights its potential for achieving general EAI of real-world scenarios. Furthermore, this work marks a significant milestone in machine learning paradigms, potentially supporting the deployment of embodied RL agents across diverse domains, including the computational and behavioral sciences and neuroscience.

Results
In this section, the proposed approach toward the discovery of optimal reward functions for embodied RL agents is evaluated. Specifically, a series of challenging tasks is first selected, including sparse-reward tasks and high-dimensional control tasks, wherein embodied RL agents may exhibit slow convergence and difficulty converging. Additionally, the approach is assessed in terms of its ability to assist embodied RL agents to address complex real-world tasks in two widely used scenarios. Furthermore, the reward functions discovered by the proposed approach are compared to the baseline rewards, including sparse reward signals, manually designed reward functions, and rewards from three state-of-the-art reward learning methods, including LIRBO50, Tomax31, and SASR54. The comparison serves to illustrate further the relationship between the reward functions and the tasks, as well as how these functions help embodied RL agents to achieve better performance. The experimental setups for both the numerical experiments and the real-world tasks, and the details on how the comparative methods are implemented, are described in the Supplementary Experimental Details.

Case study of sparse-reward tasks
To examine the effectiveness of the proposed approach when external reward signals are sparse, four sparse external reward tasks are selected from the classical OpenAI control benchmarks: CartPole-v1, AcroBot-v1, FourRoom-v0, and LunarLander-v2, as shown in Fig. 3a. The task setup adheres to the guidelines of OpenAI Gym59, with the original sparse rewards serving as the baseline for comparison. The sizes of the observation spaces for the four tasks and the action spaces of the agent are shown in Fig. 3b. Deep Q-network (DQN)60 and proximal policy optimization (PPO)61 serve as the benchmark RL algorithms. To demonstrate the generalizability of the proposed optimal reward function discovery approach, neither the architecture nor the parameters of the PPO and DQN agents are specifically modified for the tasks. All hyperparameters and network architectures remain consistent across the experiments, as detailed in Supplementary Experiment Details. Each experiment is repeated at least five times with different random seeds, and the average rewards and standard deviations during learning serve as metrics to ensure fairness.

Figure 3c, d show the average episodic rewards of embodied RL agents when using our proposed approach compared to those of agents that receive sparse reward signals and reward signals from baseline methods. Agents under the proposed reward function outperform the baseline agents in terms of both convergence speed and learning stability. When receiving sparse reward signals, the agent finds it challenging to learn the optimal policy and achieve fast convergence because of the limited number of samples that can be effectively utilized, resulting in low sample efficiency. This low efficiency requires the agent to engage in extensive trial-and-error, thereby slowing the convergence rate. Reward function design methods have relieved this challenge. LIRBO replaces sparse rewards with intrinsic reward signals, and SASR uses a shaping auxiliary function to the same end. Tomax directly learns historical maximum rewards but sometimes does not perform well, as the initial random exploration in sparse-reward tasks does not deliver an adequate reward to the agent. In contrast, the proposed approach delivers dense reward signals directly, thereby improving the sample efficiency. As a result, the agent learns the optimal policy over fewer trials, as demonstrated by the rapid increases in the rewards during the iterations shown in Fig. 3c, d. Moreover, guided by the proposed approach, agents (especially DQN) quickly discover and maintain optimal policies. The superiority of the proposed method is directly attributable to the enhanced sample efficiency afforded by the dense reward signals. The reward function discovered by the proposed approach supports stable convergence, which is guided by the upper-level optimization objective of performance maximization. When the reward function maximizes the upper-level objective, policy learning during lower-level optimization continues to converge toward the optimal policy rather than arbitrarily diverging. Furthermore, in the more complex LunarLander-v2 task, simple sparse rewards fail to guide the creation of a feasible policy. In contrast, using the proposed approach, the embodied RL agents overcome initial exploration errors and ultimately learn policies that maximize the cumulative rewards. Figure 3e shows the average episodic returns and standard errors throughout the entire learning process. The PPO and DQN agents that employ our proposed approach comprehensively outperform the agents under baseline methods. Additionally, as shown in Fig. 3c–e, our approach enhances learning stability and the convergence speed.

Figure 4a–d illustrates the reward signals received by this embodied RL agent during a single interaction episode across four tasks, wherein signals are delivered by both the discovered and the sparse-reward function. In the sparse-reward context, the agent receives reward signals only when it attains the final goal or fails to do so, significantly hindering its ability to learn policy from experience samples, thereby leading to slow learning. In contrast, the proposed approach takes a significant step toward overcoming this challenge. In simple terms, it delivers dense reward signals, which very effectively accelerate learning. Additionally, given the upper-level objective maximization, the reward function learns to assign high-value rewards to goal states, encouraging the agent to focus quickly on the goal state. Figure 4e–h illustrates the reward surfaces of the sparse reward function and the discovered reward function for each discrete action in Acrobot-v1 task. The reward surfaces are visualized as functions of the two rotational joint angles, θ1 and θ2, when their angular velocities θ̇1 and θ̇2 are 0. Figure 4e shows that the sparse reward function assigns a reward of -1 for all state and action. Figure 4f–h show the rewards assigned by our reward function for specific actions: application of -1 torque (action 0), 0 torque (action 1), or 1 torque (action 2) to the actuated joint. The results show that the reward function of this work assigns higher values to actions involving -1 and 1 torque when the joint angles deviate from the downward resting position. This encourages the agent to apply torque in situations when momentum generation can be effectively created. Conversely, the action corresponding to 0 torque receives higher rewards when the agent is near the neutral region, indicating a learned preference for stabilization or energy conservation when the system is aligned. This structured reward landscape is consistent with the task objective of swinging the outer link above a target height, demonstrating that the reward function of this work successfully captures meaningful control behaviors aligned with the dynamics of the Acrobot-v1 task. Visualizations of the reward functions at different angular velocities can be found in the Supplementary Figs. 2–4.

In summary, attainment of an optimal policy using sparse rewards is challenging because feedback is limited. However, the proposed approach substantially improves learning efficiency by continuously adjusting the reward on the basis of past interactions. Consequently, the approach not only accelerates convergence of the embodied RL agent but also stabilizes the optimized policy because the reward and policy adjustments are synchronous. The results consistently indicate that the proposed approach discovers the optimal reward function and enables agents to overcome the challenges posed by sparse reward signals.

Case study of high-dimensional control tasks
To evaluate the performance of the proposed approach for embodied agents with complex joint structures, four additional complex high-dimensional continuous control tasks are utilized, in which the embodied RL agents have a complex set of joints, rendering it challenging to achieve the goals. The four tasks, Reacher-v2, Hopper-v2, HalfCheetah-v2 and Humanoid-v2, as shown in Fig. 5a, are designed by OpenAI using the MuJoCo physics simulator, which is a free and open-source physics engine that is aimed at facilitating research and development in robotics and other areas requiring fast and accurate simulations. The number of joints controlled in the four tasks is shown in Fig. 5b, ranging from a simple two-link robotic arm with only two joints (Reacher-v2) to a humanoid intelligence with 17 joints (Humanoid-v2). For the four tasks, the original reward functions designed by OpenAI and rewards from other methods serve as the baselines. Two agents, including the continuous versions of the soft actor-critic (SAC)62 and PPO61, serve as the base RL agents. The agent parameters are not fine-tuned for specific tasks but are instead maintained as general-purpose parameters. Throughout the experiments, all hyperparameters remain consistent. For each task, five trials are executed in parallel with different random seeds. During each trial, the agent interacts with the environment for 106 steps, and the reward is upper-level optimized after every 2048 interactions. Additional experimental details can be found in Supplementary Experiment Details.

Figure 5c, d show the cumulative reward curves of embodied RL agents after each episode across the four tasks, with the error bars indicating the standard deviation across the five parallel experiments. The proposed approach exhibits superior performance, even outperforming the reward functions of OpenAI that are carefully designed by humans. Compared to other baseline methods, our approach is more effective when PPO and SAC agents are employed. The convergence speed is faster and the episodic returns are higher. SASR and LIRBO perform similarly, but SASR achieves better when a PPO agent is applied because SASR samples the success rate of historical data. Figure 5c, d also reveal that the proposed approach accelerates agent convergence and improves the cumulative expected returns, particularly when the number of agent joints increases. Figure 5e presents the means and variance of the cumulative rewards given during learning and compares the rewards of the proposed approach with those afforded by the baseline rewards. It is clear that, for both the SAC and PPO agents, the proposed approach not only outperforms the baseline but also reduces risk induced by randomness.

Figure 6 shows the distributions of the manual reward functions (Fig. 6a) and those discovered via the proposed approach (Fig. 6b) in a state space mapped into two dimensions using the t-SNE method63, that aids intuitive visualization of the reward distributions. Visualizations of the reward distributions for other reward function design methods are provided in the Supplementary Fig. 5. Details regarding the implementation of t-SNE are also included in Supplementary Experiment Details. The discovered reward function produces rewards that are very close to those of the carefully designed reward function, particularly in terms of the overall distributions of rewards and states. Such similarity can be attributed to extensive sampling of the action space during optimization. Figure 6a shows that, using the carefully designed reward function, only a few specific states are assigned exceptionally low or high rewards, whereas most states receive less extreme rewards. The specific states are those in which the embodied agents experience collisions, exceed boundaries, or fail to complete the desired action within the maximum number of steps. In contrast, the reward function of the proposed method (Fig. 6a) identifies specific states and assigns them lower or higher reward signals. For example, both reward functions assign relatively low rewards to the states in the lower-right corner of the leftmost figure (Reacher-v2) of Fig. 6. In these states, the robotic arm fails to complete the task. This phenomenon occurs because the objective function is set before the reward function is optimized, as the reward optimization objective is explicitly tied to the agent’s ultimate goal in this work. Furthermore, the discovered reward function allocates more reasonable reward signals across different states, reflected in Fig. 6b as a more concentrated distribution of similar reward values. In contrast, the manual reward function in Fig. 6a gives anomalous reward distributions because of inevitable (human) imperfections. The findings suggest that the discovered reward function not only captures the intrinsic mechanism underlying the reward relationships but also learns the optimal mapping between states and rewards. This implies that the design of reward functions for embodied RL agents with complex joints will no longer constitute a major barrier when training such agents. Using the proposed approach, even when the intricate operational mechanisms of an embodied agent are not clearly understood, a well-optimized reward function can help the agent to exhibit intelligent behavior.

Case study of real-world tasks
To demonstrate the real-world applicability of the proposed approach, real-world assessment scenarios in which RL has been widely applied but still faces significant challenges are evaluated, including autonomous energy management of data centers and unmanned systems control of UAVs.

Case study of energy management task. Reducing the energy consumption of data centers has garnered increasing attention. Many researchers seek to leverage RL methods to automate energy management and reduce the energy and economic losses. However, it is difficult to balance the dynamic environment of modern data centers because of the complex nonlinear interactions among the equipment and the way in which humans operate the equipment. Furthermore, each data center has a unique architecture and environment, meaning that any RL agent designed specifically for that center may not function well in other centers. Traditional reward functions based on the difference between the data center temperature and a standard temperature often fail to capture the interactions. A universal framework for automatic design of reward functions is needed. In this section, the performance of the proposed approach in terms of assisting RL to reduce energy consumption in data centers is evaluated. An embodied RL agent is trained using real-world data to adjust the cooling tower activity by reference to the data center temperature. The trained agent is then tested over a complete operational cycle (typically 1 year). During training, the embodied RL agent is trained over 200 rounds. In each round, the agent learns for 30 days, and makes a decision every minute. The cooling tower uses these decisions to move the data center temperature to the appropriate range, thereby consuming energy, as shown in Fig. 7a. Certain variables affecting the temperature, including the ambient temperature, user count, and data rate, are randomly processed in each round. If the data center temperature exceeds the acceptable range, learning is terminated, and the agent is considered to have failed. The trained agent is then tested over a one-year period. The agent adjusts the data center settings at one-minute intervals. To fully demonstrate the performance of the proposed approach in assisting various types of RL, three different RL agents (DQN60, SAC62, and PPO61) are tested and the energy consumption levels compared to that of a data center without RL-based regulation.

Figure 7b shows the rate of reduction in energy consumption when applying baseline RL and RL guided by the reward design approaches in data centers over a 1-year period. The numerical values are the percentage reduction rates. Figure 7c illustrates the total energy consumptions over 3, 6, 9, and 12 months for each RL agent. Although all RL methods reduce energy consumption, the reward function discovery approach further reduces consumption, particularly over longer times. Notably, as RL algorithms become more advanced, from the DQN to the more sophisticated SAC, the energy reduction rate increases (from 21% for DQN to 52% for SAC). The superiority of the proposed approach is more obvious when compared with the baseline models. With our method, all RL agents (including DQN, PPO, and SAC) achieve energy reductions that exceed 60%. These results highlight the generality and effectiveness of our approach for RL agents involved in energy management and other tasks. Therefore, the proposed optimal reward function discovery approach can be applied to RL agents operating in data centers and elsewhere. Potential applications include improvements in the conversion efficiencies of power plants (extraction of more energy from the same), reductions in energy and water consumption during semiconductor manufacturing, and improved throughput of manufacturing facilities.

Case Study of Unmanned Systems Control Task. In recent years, UAVs have been increasingly deployed in applications such as tracking, capturing, and disaster relief. However, flexible deployment requires precise environmental modeling, which is difficult in most real-world scenarios. RL has emerged as an effective solution to this problem, as in other areas of robotics. However, when controlling UAVs, the reward function requires iterative adjustments, although a reasonably well-defined reward function can be derived via dynamic modeling. In this section, the effectiveness of the discovered reward function is evaluated by applying it to an autonomous UAV quadrotor engaged in a tracking task. The experiments are based on Pybullet, and the UAV model is implemented via ArduPilot and PX4, as shown in Fig. 8a. The UAV must autonomously track a series of trajectory target points specified by the environment, and UAV observations are limited only to the two nearest targets at any given time. Three state-of-the-art RL algorithms, PPO61, SAC62, and twin delayed deep deterministic policy gradient (TD3)64, are chosen as baseline agents for comparisons. No task-specific adjustments are made. Each RL agent interacts with the environment over 106 time steps. Multiple randomized seeds are used to parallelize the experiments. Additional experimental details are provided in Supplementary Experiment Details.

Figure 8b shows the reward curves when agent learning is guided by the reward function of the present work, reward function constructed on the basis of dynamic principles, and other reward design methods. Error bars represent standard deviations. Across the three different RL agents, the proposed approach significantly accelerates learning. For the PPO agent, our approach significantly improves both learning speed and the final performance, enabling the agent to successfully accomplish the task, whereas the hand-designed and other comparison methods fail to do so. For the SAC and TD3 agents, the improvement in learning speed is less pronounced, but our method performs comparably and ensures stable learning. These results demonstrate that the proposed reward function discovery approach is particularly useful when traditional rewards are insufficient, consistently ensuring robust performance across different RL algorithms.

Figure 8c shows the numbers of target trajectory waypoints successfully attained by the UAV during the interactions, supporting the same conclusion.

The differences between the discovered reward function and the carefully designed reward function based on dynamic principles are illustrated in Fig. 8d, e. For the manually designed reward, the UAV receives a reward of 100 upon reaching the target position and -100 upon crashing, while rewards for all other states remain relatively small. Such a design leads to a very sparse reward distribution, as shown on the left side of Fig. 8d. When further analyzing the reward distributions, we exclude those of special states; the resulting filtered state space is that of Fig. 8e. These visualizations clearly demonstrate the advantages of our approach. First, our method fully captures the critical states when a UAV attains a target trajectory point or crashes. Specifically, in Fig. 8d, the lower red box refers to states wherein the UAV successfully attains a target point, associated with a strong positive reward, and the upper blue box highlights crash states associated with negative rewards. These critical states are appropriately highlighted in the reward space. More importantly, our approach significantly reduces reward sparsity by delivering denser and more informative reward signals throughout the state space, rather than concentrating the rewards on only a few special states. Furthermore, our discovered reward function effectively encodes the underlying theoretical relationship between states and rewards. This is evidenced by the fact that states with similar dynamics are assigned reward values that closely match those calculated when using the physical principles of the system. Our method not only preserves the essential characteristics of the task but also facilitates more efficient learning by providing richer feedback to the agent. Visualizations of the reward distributions for other reward function design methods are provided in Supplementary Fig. 6 for further comparison.

The experimental results of data center energy management and unmanned systems autonomous control indicate that the proposed framework effectively addresses existing RL challenges, such as adaptation to dynamically changing environments. Moreover, the distributions of the discovered reward function align with those of reward functions derived via dynamical modeling, particularly in terms of the emphasis on critical states and the overall distribution of rewards in the state space. The experimental findings support the suggestion that the proposed framework can be applied to real-world applications in embodied RL systems.

Methods
Preliminaries
Notations. Throughout the article, the following symbols are used. S: state space. A: action space. π: policy. θ: parameters of π. R: reward function. ω: parameters of R. Π: policy space. R: reward function space. π*: optimal policy. R*: optimal reward function. π*R: learned optimal policy under reward function R. π*R*: learned optimal policy under optimal reward function R*, and π*R* = π*.
VπR: state value of policy π under reward function R. QπR: state-action value of policy π under reward function R. AπR: state-action advantage of policy π under reward function R. GπR: discounted expected return of policy π under reward function R. ξ: trajectory, contains (s(0), a(0), r(0), …, s(t), a(t), r(t), …), where r(t) = R(s(t), a(t)). δ: a small positive constant. |*|: absolute value of a real number.
The general RL problem without a designed reward function is the world model denoted as M = (S, A, ρ0, T, γ). The initial state distribution is ρ0, which is the probability distribution over the initial states. The state transfer matrix is T : S × A × S → [0, 1], and T(s, a, s') specifies the transition probability from state s to s' when the embodied RL agent chooses action a. The discount factor γ ∈ [0, 1) determines the relative importance of immediate reward signals compared to future reward signals. We denote the world model M with a designed reward function R as M[R], which is a Markov decision process (MDP). In MDP, an agent observes states, chooses action a based on policy π, receives rewards signal r from reward function R, and optimizes its policy π to π*R by maximizing GπR:
π*R ∈ argmax GπR = E ξ∼π [∑t=0∞ γt R(s(t), a(t))], (1)
where ξ∼π indicates that the trajectory ξ is derived by an agent obey policy π and interact with world model. Here, we assume that when the optimal reward function R* is used to guide the agent’s policy learning, we have π*R* = π*.

The optimal reward function problem
Before formally providing the definition of the optimal reward function, we first define the proximal between two reward functions and policies.
Definition 1. (Proximal). For two policies π1 and π2, if the absolute difference |Gπ1R1 - Gπ2R2| ≤ δ, then the two policies are proximal with each other. For two reward functions R1 and R2, if the absolute difference |Gπ1R1 - Gπ2R2| ≤ δ, then the two reward functions are proximal with each other. For any policy π ∈ Π, if the absolute difference |GπR1 - GπR2| ≤ δ, then the reward function R1 is proximal to R2, i.e., R1 ∼ R2.
Proposition 1. Given a reward function R, if R is proximal to the optimal reward function R*, then policy π*R will be proximal to the optimal policy π* with R and R*.
Proof. See Supplementary Theoretical Results. □
Definition 1 shows that when the absolute difference between GπR and GπR* is bounded by a small constant for any policy π, the reward function R qualifies as an approximately optimal reward function. However, in many practical applications, we are only interested in the performance of the learned optimal policy, i.e., π*R, rather than the entire policy space. Therefore, we relax the optimal condition based on Proposition 1 and only require the expected returns of the policies π*R and π* are approximately equal under R*. That means, the optimal reward function R* can be obtained through minimizing the regret in this work:
L(π*R, π*|R*) = |Gπ*R*R* - Gπ*RR*|, (2)
where
Definition 2. A reward function R is defined as the optimal reward function if it minimizes the expected regret L(π*R, π*|R*) over the possible reward space R, where the policy π*R is learned by an embodied RL agent interacting with the world model M[R]. Formally, it is expressed as:
R* = argmin R∈R E π*R∼M[R] L(π*R, π*|R*), (3)
where R denotes the space of all possible reward functions and π*R ∼ M[R] indicates that the policy π* is derived under the dynamics influenced by the reward function R.
The optimal reward function problem (ORFP) is defined in Definition 2. However, given an MDP M[R], a direct search for the optimal reward function via equation (3) remains challenging for the following two reasons:
(1) In practice, the target optimal reward function R* is unknown, which makes it not possible to directly evaluate and compare regrets of different candidate reward functions.
(2) Determination of the optimal policy π* for a given reward function R is not straightforward, rendering it challenging to compute the regret L(π*R, π*|R*) via equation (2).
To address the first challenge, an additional reward function R is introduced to provide a unified evaluation of regret. The reward function R can be simply defined as both fixed and sparse. In this work, the function is associated with the ultimate goal of the embodied RL agent and gives a reward when the task is completed. We then measure the regret of different reward functions under the reward function R:
L(π*R, π*|R) = |Gπ*R*R - Gπ*RR|: (4)
Although many researchers have sought to approximate regret65, direct regret computation and minimization using equation (2) remain challenging. However, noting that the optimization objective of the optimal policy π* under the reward function R is to maximize the objective value GπR, we can define the optimal reward function R* as:
R* = argmin R∈R E π*R∼M[R] L(π*R, π*|R) = argmax R∈R E π*R∼M[R] Gπ*RR: (5)
Intuitively, the definition of the optimal reward function in equation (5) holds for most RL problems.

The bilevel optimization framework
We next naturally integrate the process used to discover the optimal reward function of equation (5) into the RL process and define that process within a bilevel optimization framework. As ω and θ represent the parameters of reward function and policy, the bilevel optimization framework is formulated as follows:
ω* = argmax ω∈Ω Jupper(θ, ω) ≜ E πθ*∼M[Rω] GπRθ*, (6)
θ* = argmax θ∈Θ Jlower(θ, ω) ≜ E s∼ρπθ, a∼πθ(⋅|s) [∑t=0∞ γt Rω(s(t), a(t))], (7)
where Ω and Θ denote the spaces of the possible reward function parameters and the policy parameters, respectively. s∼ρπθ denotes that the states is sampled from the distribution ρπθ and a∼πθ(⋅|s) indicates that action a is sampled from policy πθ(⋅|s) given state s. To avoid notation bloat, we use π below to denote the optimal policy π* of reward function R even though R does not appear.
The solutions of the equations ((6)–(7)) can be sought using a straightforward, alternating optimization method that iteratively fixes the reward function Rω and then optimizing the policy from πθ to πθ' according to equation (7). This step is a standard process whereby RL algorithms can be applied to solve lower-level problem. Subsequently, we fix the policy and optimize the reward function Rω to Rω' using equation (6). This alternating process continues until the optimal policy and optimal reward are discovered.

The lower-level problem. Given that the reward function Rω is fixed, the lower-level problem is a standard RL policy optimization problem. In this problem, the embodied RL agent receives reward signals from the upper-level reward function Rω and then interacts with the world model M. Thus, the expected cumulative return Jlower(θ, ω) of the lower-level problem is defined by equation (7), and we rewrite it here:
Jlower(θ, ω) ≜ E s∼ρπθ, a∼πθ(⋅|s) [∑t=0∞ γt Rω(s(t), a(t))]: (8)
Lemma 1. (ref. 67). Given a standard RL problem denoted by a world model M[R] = (S, A, ρ0, T, γ, Rω) with the objective function defined in equation (8), the policy gradient can be defined as follows:
∇θ Jlower(θ, ω) = E μs,a [∇θ log πθ(s, a) · QπRωθ(s, a)], (9)
where QπRωθ is the state–action value function of πθ in the world model M[R] and μs,a is shorthand for s∼ρπθ, a∼πθ(⋅|s).
We directly derive the gradient update form for the policy parameter θ of the lower-level optimization step based on Lemma 1:
θ' = θ + α · [E μs,a [∇θ log πθ(s, a) · QπRωθ(s, a)]], (10)
where α denotes the learning rate during lower-level policy optimization.

The upper-level problem. The upper-level objective of equation (6) is defined by reference to equation (5), which we here restate:
Jupper(θ, ω) ≜ E πθ*∼M[Rω] [GπRθ*]: (11)
As for lower-level optimization, we optimize the reward parameter ω via gradient updates. For a fixed θ' and ω, we update the reward parameter as follows:
ω' = ω + β · [∇ω Jupper(θ, ω) |ω], (12)
where β is the learning rate of the upper-level problem. However, we cannot obtain the distribution μ's,a when optimizing the reward parameters from ω to ω', even if the policy has been optimized to πθ'. The reason is that during RL, we simulate the distribution by allowing the agent to interact with the world model and then store the trajectories. To address this limitation, we approximate the expectations of πθ' in equation (17) using samples from πθ as suggested by Lemma 2.
Assumption 2. Assume that the support of policy π covers the support of policy π' (s, a) > 0, we also have π(s, a) > 0 for any state-action pair (s, a) for which π'(s, a) > 0. Assumption 2 guarantees that the importance sampling ratio of Lemma 2 is always well defined, finite, and computationally stable during optimization.
Lemma 2. (ref. 68). Given two policies π and π', the expectation of a function f(s, a) under π' can be estimated using samples from π and the importance sampling ratio. Specifically,
E π' [f(s, a)] = E π [π'(s, a) / π(s, a) · f(s, a)], (19)
where π'(s, a) / π(s, a) is the importance sampling weight, assuming that π(s, a) > 0 for all actions a taken under π'.
Use of the importance sampling weight ensures consistent reweighting of samples from π to π', enabling unbiased gradient estimation even with different policy distributions. Therefore, on the basis of Lemma 2, we approximate the gradient term in equation (17) as:
ω' = ω + β · ∇ω Jupper(ω) = ω + β · E μs,a [∇ω E μs,a [πθ'(s, a) / πθ(s, a) · AπRθ'(s, a)] · AπRωθ(s, a)] |ω: (20)
Assumption 3. Assume that the policy and reward lying between two consecutive optimization steps are stationary.
In practical implementations, policy updates can be reasonably stable, implying that the policy distributions before and after the update remain near-identical. Consequently, Assumption 3 further simplifies equation (20). On the basis of Assumptions 2–3 and Lemma 2, in the upper-level of the optimization framework, the reward function parameter update method of equation (17) can be approximated as follows:
ω' = ω + β · ∇ω Jupper(ω) ≈ ω + β · E μs,a [∇ω [E μs,a [AπRθ'(s, a) · AπRωθ(s, a)]]] |ω ≈ ω + β · E μs,a [∇ω AπRθ'(s, a) · AπRωθ(s, a)] |ω: (21)
In contrast to equation (17), the proposed gradient update term in equation (21) does not require the policy gradient parameters. Rather, it relies only on the value function, enabling the update method for the reward parameter ω to be applied to both policy-based and value-based RL agents.

Peer review rounds
[{"round": 1, "reviewer_comments": {"reviewer_1": "Discovering reward functions, whether that be ones which encode a task given observed behaviors (as is typical in IRL),\nwhich is more efficiently optimized than a given (perhaps sparse) target reward function, or which encodes an unknown\npreference relation (as is often the case for RLHF) is a crucial and active area of research underlying reinforcement learning.\nThis paper approaches a version of this problem by alternately optimizing the estimated reward function with respect to how\nit affects policy learning and optimizing the policy through RL with respect to the estimated reward. This general pattern is\nfairly common throughout the field of IRL, with specific forms of each optimization problem varying greatly. The overall\npresentation of the work is excellent. However, I do have concerns about some of the language used which edges into\nmaking unsubstantiated claims as well as about the rigor and precision of the theoretical components of the work.\n# General Comments / Questions\nIn what way do you view this work as being specific to, or even especially suited to “embodied” RL versus the general RL\nproblem? The narrative of the work heavily emphasizes the embodied component, but the method itself does not appear to\nbe influenced by embodiment.\nThere are a lot of statements made throughout the paper that I believe make unsubstantiated claims. Generally, I think these\nare simply instances of using overly enthusiastic language and could be easily edited to fix:\n* Second sentence in abstract: Although I agree with the underlying thesis expressed here, “universal” is a significant\noverstatement especially when paired with “necessary and sufficient”. A proper framing would be that this is a broadly\nshared or at least investigated hypothesis, but certainly not “universally recognized.”\n* Page 3, first line: “Consequently, the primate brain must embody diverse types of cognitive intelligence that facilitate\noptimal reward acquisition.” Why the primate brain specifically? Why “must” and “optimal”, as biological life may well have\nfound heuristic or suboptimal approaches.\n* Page 3, near top: “This process, gives the robot agent capabilities that approximate and surpass those of natural\nintelligence.” Ideally soften the statement, but ideally would provide references for this claim.\n* “Powerful embodied RL agents could provide a solution to AGI.” Sure, maybe, but this is just speculation without any\nprovided support.\n* “Furthermore, the fundamental principles underlying optimal reward functions and the tasks remain largely unexplored,\noften requiring a case-by-case approach.” There is a rich literature on reward design and related topics, far from an\nunexplored topic. However, I find this line more pernicious in that this paper does not compare with or even discuss\nconnections to even a single other method for reward design/discovery. Thus, this line seems to attempt to justify this\nomission by claiming the non-existence of related work.\n* Page 5, Framework Overview: “This alternating optimization process is iteratively performed until convergence, at which\npoint the optimal reward function is discovered.” Care is needed here as you have neither proven that the process will\nconverge nor that if it does so it must converge on the optimal reward function.\n# Empirical Results\nFigure 4 does not seem to add much to understanding. It is referenced once, as showing the learned policies are more\nefficient / effective, but we already know that e.g. in Lunar Lander the agent does better due to looking at the learning curve.\nA much better use of this figure space would be investigating more detail on the discovered reward function itself.\nAs mentioned previously, I believe these results need to include comparisons with other reward discovery methods, ideally 2\nor more with one more closely related to the approach taken here and one more different.\nDespite this issue, the empirical results are the strongest part of the paper. There is a solid set of environments, many of\nthem well studied, exhibiting a variety of properties and levels of difficulty. As well the authors do well at varying the baseline\nRL algorithm itself, which shows generality of the method. And finally of course, the empirical results do generally support\nthis method as being effective at improving learning performance.\n# Methods and Theory\nDefinition 1 (proximal) is not defined precisely enough as hidden inside of ‘nearly equal’ there is both (1) a choice of\ntolerance epsilon > 0, and (2) a choice of distance metric (e.g. L_inf). This definition is essentially saying that when the term\n‘proximal’ is used it is used to refer to the identified equivalence classes on policies or reward functions in terms of expected\nreturns. Noticing this, it is worth pointing out that the ~ is a more common choice for notation on equivalence classes than\n\\approx. Similarly, we might naturally consider V^{\\pi}_R or v^{\\pi}_R instead of U(\\pi, M[R]).\nI’ll use this more standard notation for ease of typing comments out.\nProposition 1. Given a reward function R and the optimal reward function R^∗, if R is proximal to R^∗, then policy π under\nR will be proximal to the optimal policy π^∗.\nThis proposition also has immediate issues with precision. Breaking it down:\n* R^* is “optimal” with respect to what notion of optimality? It seems instead the term optimal is simply being used to identify\na reward as the ‘target’ for reward discovery.\n* R is proximal to R^*, but under what policy? Given there are two policies mentioned, I’ll consider if the theory works for\neither one, but obviously this needs to be specified for this proposition to be complete.\n* “policy π under R will be proximal to the optimal policy π^∗”, this is not clearly phrased and can have many meanings. Are\nwe saying the two policies are proximal wrt R? Or are we saying that π is optimal wrt R and the two policies are proximal wrt\nR^*?\nJust after the statement of Proposition 1 there is a fairly strong claim given without proof: “if the policy π optimized under a\nspecific reward function R is proximal to the optimal policy π^*, then the reward function R can be considered the optimal\nreward function R^*.”\nWhich reward function are the two policies proximal wrt? I think the one that makes most sense here is R, so going forward\nwith that the claim is essentially:\nπ ~ π^* wrt R => R \\equiv R^*\nFirst, “can be considered” needs to be specified formally. I think a natural choice would be for this to mean R ~ R^* wrt π^*,\nas this would be fairly nice. However, none of this is true which can be seen with a simple counter example: R = R_0 = 0 (the\nconstant zero reward function).\nSo, we choose R = R_0, and for *any* policy π we have π ~ π^* wrt R because all policies have value zero under this\nreward and as a result all policies are optimal wrt R, thus π being optimized under R is satisfied as is the proximal property.\nHowever, under any reasonable notion of equivalence R_0 should not be considered the optimal reward function R^*.\nNow turning to regret and Definition 2: “A reward function R is defined as the optimal reward function R^∗ if it minimizes the\nexpected discounted regret L(π, R) over the space of possible policies π, where the policy π is learned by an embodied RL\nagent interacting with the world model M[R].” With L defined above with:\nL(π, R) = v^{π^*}_R - v^{π}_R\nRecall the issue raised previously, R_0 perfectly achieves zero regret here. However, this isn’t the whole of the problem.\nNotice that π^* is some policy unrelated to a particular choice of R. So, there will be arbitrarily many reward functions R, with\ncorresponding ‘optimized’ policies π such that L(π, R) << 0, whereas for the desired reward R^* we have L(π, R^*) = 0 (for π\noptimized for R = R^* as the definition calls for). This objective, or definition for the optimal reward problem, appears to be\nmisspecified at a foundational level.\nA small notational note: it might make sense to use L_{π^*} to indicate the dependence on the given optimal policy.\nThe formal statement of Lemma 1 contains much less information, far less than needed for a precise statement, than the\ninformal description just below it. Additionally, this lemma is entirely unnecessary. \\bar{R} is a fixed reward function, π^* is a\nfixed policy, therefore the value of π^* under \\bar{R} will be fixed, which is all this lemma seems to be saying.\nMoving on to the proof of Proposition 1.\nThe first line of the proof indicates that the intended meaning is R ~ R^* wrt π.\nIt then goes on to claim that this implies\n| v^{π}_R - v^{π}_{R^*} | \\leq \\delta\nThis should be in Definition 1. Equations 22 and 23 both assume a unique optimal policy which is not in general true, so the\nequals should be replaced with “\\in” as the argmax may return a set of optimal policies.\nThe inequalities in 24 are very strange. First, it seems to assume R ~ R^* wrt π_R and R ~ R^* wrt π^*.\nSecond, if a suitably formal definition was used then the corresponding thresholds delta > 0, would\npotentially be different for the two equivalence classes. Ignoring that for the moment, as of (24) what has been\nassumed without being clear about it is that R ~ R^* wrt both π_R and π^*.\nThe rest of the proof follows correctly, with one partial exception. Recall my concerns about precision of the definition of\nproximal. Here this comes up again because the result of the proof depends on this lack of precision. Taking epsilon =\n4*delta and saying this means that π ~ π^* wrt R, means that proximal has its meaning further eroded.\nSince the definition of proximal you give, and actively rely on for your proof, allows for the distance between value functions\nto be arbitrarily large (as large as is needed for the narrative) it makes the result much less meaningful.\nWhat do you *actually* care about saying here? For instance, why do we care if the discovered reward function is close\n(proximal) to the optimal one? Does this result support that goal?\nTo be clear, I think that the theory components can be fixed, but I am unsure if they will give interesting results. I strongly\nspect that they can, but not without some work.\n(Remarks on code availability)\nI have looked at the code fairly briefly. It is built on top of CleanRL to handle the agent-environment interaction and RL\noptimization components. The reward function learning code is undocumented but it is well formatted and easy to read and\nunderstand even without further documentation or comments. Overall, I would say that the code is well done because it is\nvery easy to contrast with the baseline CleanRL implementation to see differences (which are almost entirely modular).\nYes, there is a README that should be sufficient for testing out the code. There is an odd pattern of 'typo' like misspellings\nof words, but done consistently (e.g. reward_funciton, optimaze_reward), which is worth fixing.", "reviewer_2": "> Summary\nFor embodied agents in Embodied AI (EAI), reinforcement learning (RL) is a promising method for realizing intelligent\nbehavior to solve complex problems. However, RL suffers from the so-called reward-engineering problem, in which reward\nquality heavily depends on human expertise and is critical for improving agent performance. In this paper, the authors\npropose an optimal reward-function discovery method. Using a meta-gradient approach, they implement the reward\ndiscovery process as bilevel optimization: the upper level refines the reward function, and the lower level performs the\ntraditional RL step. With the proposed method, agents discovered the optimal policy in fewer episodes, indicating high\nsample efficiency. The method worked well on both energy-management and UAV-control tasks. The authors also provide\ntheoretical results for the proposed approach.\n> What are the noteworthy results?\nThe authors propose a meta-gradient–based bilevel optimization method to optimize the reward function. While traditional\napproaches rely on inverse RL or active learning, the proposed method operates without them. It generates a more\ninformative dense reward signal than the inefficient sparse rewards of manually designed functions, thereby accelerating the\nagents’ learning speed and improving their performance.\n> Will the work be of significance to the field and related fields? How does it compare to the established literature? If the\nwork is not original, please provide relevant references.\nTheir proposed method is novel, but it closely resembles [14] or [39] (in the appendix). Although the implementation and\ndesign differ, the core idea remains very similar.\n> Does the work support the conclusions and claims, or is additional evidence needed?\nConsidering the previous comment, comparisons with the methods in [14] or [39] would clarify that the authors’ proposed\napproach is more effective than theirs; the work currently only compares against naive RL baselines. It would also be\npreferable to include visualizations of the learned reward to assess what reward was actually learned. While this may be\ndifficult for high-dimensional control tasks, it could at least be done for classical control tasks.\n> Are there any flaws in the data analysis, interpretation and conclusions? Do these prohibit publication or require revision?\nIn the energy-management task, the consumption reduction achieved by SAC and PPO appears smaller than that of DQN\n(Fig. 7(c)). In the UAV control task, Fig. 8(b) does not clearly demonstrate an acceleration in learning: while PPO shows\nnoticeable improvement, the gains for TD3 and SAC are comparatively modest. Moreover, the advantage illustrated in Fig.\n8(e) over Fig. 8(d) is not immediately apparent. It would be helpful to clarify the form of the reward function that was learned,\nas this would shed light on how the proposed method drives performance improvements. Finally, the t-SNE visualizations\nare somewhat challenging to interpret; additional explanation or annotation could improve their clarity.\n> Is the methodology sound? Does the work meet the expected standards in your field?\nThe methodology is fundamentally sound; however, a more detailed comparison with state-of-the-art approaches would\nbetter highlight the advantages of the proposed method. I therefore recommend conducting additional, comprehensive\nexperiments to this end.\n> Is there enough detail provided in the methods for the work to be reproduced?\nThe methods as described are sufficient for reproduction; however, we will further strengthen the manuscript by including\nadditional implementation details to support a more thorough evaluation."}, "author_response": "Response to Reviewer #1 (Remarks to the Author):\nThe author response and changes made address my two primary concerns with the work (limited\nempirical comparison with other methods and precision of, especially theoretical, claims). The\nupdated language in the article, improved theory statements and inclusion of additional\nexperimental work satisfies these and significantly strengthen the work.\nResponse: We sincerely thank the reviewer for the careful reading of our manuscript and for the\nthoughtful feedback. We are very pleased that the revised version addresses your two primary\nconcerns. Thank you again for your constructive feedback and support.\nMy only remaining, perhaps pedantic, suggestion has to do with the framing of this work and its\ndependence on knowledge of the task relative to existing methods. The authors repeatedly\ndistinguish their approach from others in terms of dependency on \"external knowledge\", e.g. \"In\ncontrast, our approach aims to discover reward functions in a fully autonomous manner, guided\nsolely by environmental interactions and generalization signals, without relying on predefined\nexpert trajectories, heuristics, or task-specific priors.\"\nThis seems to claim that the proposed method does not depend on any prior information about\nthe \"task\", which obviously is not correct. There is indeed information about the task, just as in\nother methods, but this method and others certainly differ in the type and form of that \"external\nknowledge\". The type of dependency in this article is most closely related to those such as LIRBO\nand SASR, making them great choices for comparison in the experiments.\nI think it is important to be more clear that this method does indeed require some such knowledge,\nin what form, and then to contrast with other in these terms. The argument cannot be that this\napproach requires no knowledge of the task, instead of must be that the type of dependency and\ntype of knowledge here is preferable (e.g. sparse reward or fitness vs. expert demonstrations).\nThis detail is decipherable from the methods section, but in the main the claims do not match this\ndetail, and clarity would be much improved on this point by simply stating the dependency and\ncomparison upfront.\nResponse: We sincerely thank the Reviewer #1 for this insightful and constructive suggestion.\nWe agree with your comment that in our method, like other approaches, inherently depends on\ncertain forms of task-related knowledge. The key distinction, as you correctly pointed out, lies not\nin the absence of such knowledge, but rather in the type and form of dependency.\nFollowing your suggestion, we have revised the manuscript to make this argument more explicit\nand precise, as shown below:\nIntroduction\nThe core insight is that the optimal reward function can be automatically discovered on the basis\nof the past performance of embodied RL agents with minimal reliance on task-specific prior\nknowledge.\nWe sincerely thank you for the time, effort, and thoughtful evaluation that greatly contributed to\nimproving the quality of our work.\nResponse to Reviewer #2 (Remarks to the Author):\nI have confirmed the authors' reply and have no further questions.\nThe authors have added comparative experiments with existing methods, and it is evident that\nthe performance of the proposed method is comparable to or better than the existing ones.\nThe visualizations have also been improved, making it easier for readers to understand\nResponse: We sincerely thank Reviewer 2 for the positive and encouraging evaluation. We are\npleased that the revisions have met your expectations.\nThank you again for your time, effort, and supportive comments, which helped us further improve\nthe quality of our work."}]

Decision letter