Title: Going Beyond Heuristics by Imposing Policy Improvement as a Constraint
Venue: NeurIPS 2024
Accepted: yes
Keywords: Deep reinforcement learning
Abstract: In many reinforcement learning (RL) applications, incorporating heuristic rewards alongside the task reward is crucial for achieving desirable performance. Heuristics encode prior human knowledge about how a task should be done, providing valuable hints for RL algorithms. However, such hints may not be optimal, limiting the performance of learned policies. 
The currently established way of using heuristics is to modify the heuristic reward in a manner that ensures that the optimal policy learned with it remains the same as the optimal policy for the task reward (i.e., optimal policy invariance). 
However, these methods often fail in practical scenarios with limited training data. We found that while optimal policy invariance ensures convergence to the best policy based on task rewards, it doesn't guarantee better performance than policies trained with biased heuristics under a finite data regime, which is impractical. In this paper, we introduce a new principle tailored for finite data settings. Instead of enforcing optimal policy invariance, we train a policy that combines task and heuristic rewards and ensures it outperforms the heuristic-trained policy. As such, we prevent policies from merely exploiting heuristic rewards without improving the task reward. Our experiments on robotic locomotion, helicopter control, and manipulation tasks demonstrate that our method consistently outperforms the heuristic policy, regardless of the heuristic rewards' quality.
Code is available at https://github.com/Improbable-AI/hepo.

Going Beyond Heuristics by Imposing Policy Improvement as a Constraint

Abstract

In many reinforcement learning (RL) applications, incorporating heuristic rewards alongside the task reward is crucial for achieving desirable performance. Heuristics encode prior human knowledge about how a task should be done, providing valuable hints for RL algorithms. However, such hints may not be optimal, limiting the performance of learned policies. The currently established way of using heuristics is to modify the heuristic reward in a manner that ensures that the optimal policy learned with it remains the same as the optimal policy for the task reward (i.e., optimal policy invariance). However, these methods often fail in practical scenarios with limited training data. We found that while optimal policy invariance ensures convergence to the best policy based on task rewards, it doesn't guarantee better performance than policies trained with biased heuristics under a finite data regime, which is impractical. In this paper, we introduce a new principle tailored for finite data settings. Instead of enforcing optimal policy invariance, we train a policy that combines task and heuristic rewards and ensures it outperforms the heuristic-trained policy. As such, we prevent policies from merely exploiting heuristic rewards without improving the task reward. Our experiments on robotic locomotion, helicopter control, and manipulation tasks demonstrate that our method consistently outperforms the heuristic policy, regardless of the heuristic rewards' quality. Code is available at https://github.com/Improbable-AI/hepo .

1 Introduction

Reinforcement learning (RL) [1] is a powerful framework for learning policies that can surpass human performance in complex tasks. However, training RL policies with sparse or delayed rewards is often ineffective. Instead of relying solely on sparse task rewards that indicate an agent's success or failure, it is common to augment the sparse reward with heuristic reward terms that provide denser reward supervision to speed up and improve the performance of RL policies [2, 3]. Shining examples of the success and necessity of heuristic reward terms are complex robotic object manipulation [4] and locomotion [5-8] tasks. However, heuristics impose human assumptions that may limit the RL algorithm. For example, a heuristic reward function may encourage a robot to walk like a human, yet there could be faster walking policies that don't resemble human gait.

The key question is how to learn a policy π that outperforms one trained solely on heuristics (i.e., heuristic policy π H ). Practitioners tackle this problem by tuning the balance between the task objective J ( π ) and the heuristic objective H ( π ) in the augmented training objective J ( π ) + λH ( π ) , where λ controls the balance between the two objectives for a policy π . However, it requires careful tuning of λ to make the policy π outperform the heuristic policy; otherwise, the algorithm might prioritize heuristic rewards while neglecting the task objective.

Tuning λ to balance both objectives is time-consuming. We desire an algorithm that finds a policy that outperforms the heuristic policy without requiring such tuning for any given heuristic reward function. Classic methods [2, 9-14] modify heuristic rewards to align the augmented objective's optimal policy with the one for the task objective (i.e., optimal policy invariance), theoretically ensuring that with infinite data the policy outperforms the heuristic. However, in practice, these modified heuristics often fall short on complex robotic tasks compared to policies trained solely on heuristic objectives, as demonstrated in our study (Section 4.1) and prior work [10].

In this paper, we challenge the prevailing paradigm by questioning whether optimal policy invariance is the appropriate objective to prevent heuristics from limiting RL agent's performance in the finite data regime. As optimal policy invariance ensures convergence to the optimal policy with infinite data it may not be practical in many real-world settings. We propose an alternative paradigm that, in every step of training, imposes the constraint of improving task performance beyond a policy trained solely on heuristic rewards (i.e., J ( π ) ≥ J ( π H ) ). This condition J ( π ) ≥ J ( π H ) guarantees that the learned policy π outperforms the heuristic policy π H , effectively surpassing human-designed heuristics. Additionally, such policy improvements can be verified and achieved using many existing deep RL algorithms [15-18] in finite data settings.

Therefore, we enforce the policy improvement condition J ( π ) ≥ J ( π H ) as a constraint, preventing the policy from exploiting heuristic rewards during training. We propose the following constrained optimization objective:

Optimizing this objective at each iteration allows learning a policy performing better than or equal to policies trained only on heuristic rewards. It prevents capitalizing on heuristic rewards at the expense of task rewards. Moreover, it enables adaptively balancing both rewards over time instead of using a fixed coefficient λ . Our contribution is an add-on to existing deep RL algorithms to improve RL algorithms trained with heuristic rewards. We evaluated our method on robotic locomotion, helicopter, and manipulation tasks using the IsaacGym simulator [19]. The results show that our method led to superior task rewards and higher task-completion success rates compared to the policies solely trained with heuristic rewards, even when heuristic rewards are ill-designed.

2 Preliminaries: Reinforcement Learning with Heuristic

Reinforcement Learning (RL): RL is a popular paradigm for solving sequential decision-making problems [1] where the problems are modeled as an interaction between an agent and an unknown environment [1]. The agent aims to improve its performance through repeated interactions with the environment. At each round of interaction, the agent starts from the environment's initial state s 0 and samples the corresponding trajectory. At each timestep t within that trajectory, the agent perceives the state s t , takes an action a t ∼ π ( . | s t ) according to its policy π , receives a task reward r t = r ( s t , a t ) , and transitions to a next state s t +1 until reaching terminal states, after which a new trajectory is initialized from s 0 , and the cycle repeats. The agent's goal is to learn a policy π that maximizes the expected return J ( π ) in a trajectory as below:

where γ denotes a discount factor [1] and E π [ . ] denotes taking expectation over the trajectories sampled by π . In the following, we term J as the true task objective, as it indicates the performance of a policy on the task.

RLwith Heuristic: In many tasks, learning a policy to maximize the true objective J is challenging because rewards may be sparse or delayed. This lack of feedback makes policy optimization difficult for RL algorithms. To address this, practitioners often use a heuristic reward function h with denser reward signals to facilitate optimization, aiming to learn a policy that performs better in J . The policy trained to maximize the expected return of heuristic rewards is called the heuristic policy π H . The expected return of heuristic rewards, termed the heuristic objective H , is defined as:

where h ( s t , a t ) is the heuristic reward at timestep t for state s t and action a t .

3 Method: Improving Heuristic Policy via Constrained Optimization

Problem statement: Optimizing both task J and heuristic H objectives jointly could lead to better task performance than training solely with J or H , but needs careful tuning on the weight coefficient λ among both objectives in max π J ( π ) + λH ( π ) . Without careful tuning, the policy π may learn to exploit heuristic rewards H and compromise performance of J . The goal of this paper is to mitigate the requirement of tuning this coefficient to balance them in order to improve task performance.

Key insight - Leveraging Heuristic with Constraint: We aim to use the heuristic objective H for training only when it improves task performance J and ignore it otherwise. Rather than manually tuning the weight coefficient λ to balance both rewards, we introduce a key insight: impose a policy improvement constraint (i.e., J ( π ) ≥ J ( π H ) ) during training. This prevents RL algorithms from exploiting heuristic rewards H at the expense of task rewards J . To achieve this goal, we introduce the following constrained optimization objective:

This constrained objective (Equation 3) results in an improved policy π over the heuristic policy π H , leading us to call this framework Heuristic-Enhanced Policy Optimization (HEPO) . A practical algorithm to optimize this objective is presented in Section 3.1, and its implementation on a widelyused RL algorithm [15] in robotics is detailed in Section 3.2.

3.1 Algorithm: Heuristic-Enhanced Policy Optimization (HEPO)

Finding feasible solutions for the constrained optimization problem in Equation 3 is challenging due to the nonlinearity of the objective function J with respect to π . One practical approach is to convert it into the following unconstrained min-max optimization problem using Lagrangian duality:

where the Lagrangian multiplier is α ∈ R + . We can optimize the policy π and the multiplier α for this min-max problem by a gradient descent-ascent strategy, alternating between optimizing π and α .

Enhanced policy π : The optimization objective for the policy π can be obtained by rearranging Equation 4 as follows:

This represents an unconstrained regular RL objective with the modified reward at each step as (1 + α ) r ( s t , a t ) + h ( s t , a t ) , which can be optimized using any off-the-shelf deep RL algorithm. In this modified reward, the task reward r ( s t , a t ) is weighted by the Lagrangian multiplier α , reflecting the potential variation in the task reward's importance during training as α evolves. The interaction between the update of the Lagrangian multiplier and the policy will be elaborated upon next.

Lagrangian Multiplier α : The Lagrangian multiplier α is optimized for Equation 4 by stochastic gradient descent, with the gradient defined as:

Notably, ∇ α L ( π, α ) is exactly the performance gain of the policy π over the heuristic policy π H on the task objective J . By applying gradient descent with ∇ α L ( π, α ) , when J ( π ) > J ( π H ) and thus ∇ α L ( π, α ) > 0 , the Lagrangian multiplier α decreases. As α represents the weight of the task reward in Equation 5, it indicates that when π outperforms π H , the importance of the task reward diminishes because π already achieves superior performance compared to the heuristic policy π H regarding the task objective J . Conversely, when J ( π ) < J ( π H ) , α increases, thereby emphasizing the importance of task rewards in optimization. The update procedure for the Lagrangian multiplier α offers an adaptive reconciliation between the heuristic reward h and the task reward r .

3.2 Implementation

We present a practical approach to optimize the min-max problem in Equation 4 using Proximal Policy Optimization (PPO) [15]. We selected PPO because it is widely used in robotic applications involving heuristic rewards, although our HEPO framework is not restricted to PPO. The standard PPO implementation involves iterative stochastic gradient descent updates over numerous iterations, alternating between collecting trajectories with policies and updating those policies. We outline the optimization process for each iteration and provide a summary of our implementation in Algorithm 1.

Training policies π and π H : Instead of pre-training the heuristic policy π H , which requires additional data and reduces data efficiency, we concurrently train both the enhanced policy π and the heuristic policy π H , allowing them to share data. For each iteration i , we gather trajectories τ and τ H using the enhanced policy π i and the heuristic policy π i H , respectively. Following PPO's implementation, we compute the advantages A π i r ( s t , a t ) , A π i H r ( s t , a t ) , A π i h ( s t , a t ) , and A π i H h ( s t , a t ) for the task reward r and heuristic reward h with respect to π i and π i H . We then weight the advantage with the action probability ratio between the new policies being optimized (i.e., π i +1 and π i +1 H ) and the policies collecting the trajectories (i.e., π i or π i H ). Finally, we optimize the policies at the next iteration i +1 for the objectives in Equations 7 and 8:

Maximizing the advantages will result in a policy that maximizes the expected return for a chosen reward function, as demonstrated in PPO [15]. This enables us to maximize the objective J and H . We estimate the advantages A π i r ( s t , a t ) and A π i h ( s t , a t ) (or A π i H r ( s t , a t ) and A π i H h ( s t , a t ) ) using the standard PPO implementation with different reward functions. Therefore, we omit the details of the advantage's clipped surrogate objective in PPO, and leave them in Appendix A.1.

Although PPO is an on-policy algorithm, the use of off-policy importance ratio correction (i.e., the action probability ratios between two policies) allows us to use states and actions generated by another policy. This enables us to train π using data from π H and vice versa. Both policies π and π H are trained using the same data but with different reward functions. Note that collecting trajectories from both policies does not require more data than the standard PPO implementation. We collect half the trajectories with each policy, π and π H , for a total of B trajectories (see Algorithm 1). Then, we update both π and π H using all B trajectories.

Optimizing the Lagrangian multiplier α : To update the Lagrangian multiplier α , we need to compute the gradient in Equation 6, which corresponds to the performance gain of the enhanced policy π over the heuristic policy π H on the task objective J . Utilizing the performance difference lemma [20, 16], we relate this improvement to the expected advantages over trajectories sampled by the enhanced policy π as J ( π ) -J ( π H ) = E π [ A π H r ( s t , a t )] . However, this approach only utilizes half of the trajectories at each iteration since it exclusively relies on trajectories from the enhanced policy π . To leverage trajectories from both policies, we also consider the performance gain in the reverse direction as -( J ( π H ) -J ( π )) = -E π H [ A π r ( s t , a t )] . Consequently, we can estimate the gradient of α using trajectories from both policies, as illustrated below:

At each iteration i , we estimate the gradient of α using the advantage A π H r ( s t , a t ) and A π r ( s t , a t ) on the trajectories sampled from both π i and π i H , and update α with stochastic gradient descent as follows:

where η ∈ R + is the step size. The expected advantage in Equation 11 are estimated using the generalized advantage estimator (GAE) [21].

Algorithm 1 Heuristic-Enhanced Policy Optimization (HEPO) 1: Input: Number of trajectories per iteration B 2: Initialize the enhanced policy π 0 , the heuristic policy π 0 H , and the Lagrangian multiplier α 3: for i = 0 · · · do ▷ i denotes iteration index 4: Rollout B/ 2 trajectories τ by π i 5: Rollout B/ 2 trajectories τ H by π i H 6: π i +1 ←-Train the policy π i for optimizing Equation 7 using both τ and τ H 7: π i +1 H ←-Train the policy π i H for optimizing Equation 8 using both τ and τ H 8: α ←-Update the Lagrangian multiplier α by gradient descent (Equation 9) using τ and τ H 9: end for

3.3 Connection to Extrinsic-Intrinsic Policy Optimization [22]

Closely related to our HEPO framework, Chen et al. [22] proposes Extrinsic-Intrinsic Policy Optimization (EIPO), which trains a policy to maximize both task rewards and exploration bonuses [23] subject to the constraint that the learned policy π must outperform the task policy π J trained solely on task rewards. HEPO and EIPO differ in their objective functions and implementation of the constrained optimization problem. Additional information can be found in the Appendix, covering the objective formulation (Appendix A.1), implementation tricks (Appendix A.2), and detailed pseudocode (Appendix A.3).

Exploration bonuses [23] can be viewed as heuristic rewards. The main difference between HEPO and EIPO's optimization objectives lies in constraint design. Both frameworks require the learned policy π to outperform a reference policy π ref (i.e., J ( π ) ≥ J ( π ref ) ) but use a different reference policy. EIPO uses the task policy π J as the reference policy π ref because they aim for asymptotic optimality in task rewards. If the constraint is satisfied with π J being the optimal policy for task rewards, the learned policy π will also be optimal for task rewards. In contrast, HEPO uses the heuristic policy π H trained solely on heuristic rewards since HEPO aims to improve upon it.

HEPO simplifies the implementation. Both HEPO and EIPO train two policies with shared data, but EIPO alternates the policy used for trajectory collection each iteration and has a complex switching rule, which introduces more hyperparameters. HEPO collects trajectories using both policies together at each iteration, simplifying implementation and avoiding extra hyperparameters.

4 Experiments

We evaluate whether HEPO enhances the performance of RL algorithms in maximizing task rewards while training with heuristic rewards. We conduct experiments on 9 tasks from IsaacGym (ISAAC) [19] and 20 tasks from the Bidexterous Manipulation (BI-DEX) benchmark [24]. These tasks rely on heavily engineered reward functions for training RL algorithms. Each task has a task reward function r that defines the task objective J to be maximized, and a heuristic reward function h that defines the heuristic objective H , provided in the benchmarks to facilitate the optimization of task objectives J . We implement HEPO based on PPO [15] and compare it with the following baselines:

H-only (heuristic only) : This is the standard PPO baseline provided in ISAAC. The policy is trained solely using the heuristic reward: max π H ( π ) . The heuristic reward functions in ISAAC and BI-DEX are designed to help RL algorithms maximize the task objective J . This baseline is crucial to determine if an algorithm can surpass a policy trained with highly engineered heuristic rewards.

J-only (task only) : The policy is trained using only the task reward: max π J ( π ) . This baseline demonstrates the performance achievable without heuristics. Ideally, algorithms that incorporate heuristics should outperform this baseline.

J+H (mixture of task and heuristic) : The policy is trained using a mixture of task and heuristic rewards: max π J ( π ) + λH ( π ) , with λ balancing the two rewards. As [22] shows, proper tuning of λ can enhance task performance by balancing both training objectives.

Potential-based Reward Shaping (PBRS) [2] : The policy is trained to maximize E π [ ∑ ∞ t =0 γ t r t + γh t +1 -h t ] , where r t and h t are the task and heuristic rewards at timestep t . PBRS guarantees that the optimal policy is invariant to the task reward function. We include it as a baseline to examine if these theoretical guarantees hold in practice.

HuRL [10] : The policy is trained to maximize E π [ ∑ ∞ t =0 γ t r t +(1 -β i ) γh t +1 ] , where β i is a coefficient updated at each iteration to balance heuristic rewards during different training stages. The scheduling mechanism is detailed in [10] and our source code provided in the Supplementary Material.

EIPO [22] : The policy is trained using the constrained objective: max π J ( π ) + H ( π ) s.t. J ( π ) ≥ J ( π J ) , where π J is the policy trained with task rewards only. EIPO is similar to HEPO but differs in formulation and implementation, as detailed in Section 4.3.

Each method is trained for 5 random seeds and implemented based on the open-sourced implementation [25], where the detailed training hyperparameters can be found in Appendix A.4.

Metrics: Based on the task success criteria in ISAAC and BI-DEX, we consider two types of task reward functions r : (i) Progressing (for locomotion or helicopter robots) and (ii) Goal-reaching (for manipulation). In progressing tasks, robots aim to maximize their traveling distance or velocity from an initial point to a destination. Thus, movement progress is defined as the task reward. In goal-reaching tasks, robots aim to complete assigned goals by reaching specific goal states. Here, task rewards are binary, with a value of 1 indicating successful attainment of the goal and 0 otherwise. Detailed descriptions of our task objectives and total reward definitions are provided in Appendix C.

4.1 Benchmark results

Setup: We aim to determine if HEPO achieves higher task returns and improves upon the policy trained with only heuristic rewards (H-only) in the majority of tasks. In this experiment, we use the heuristic reward functions from the ISAAC and BI-DEX benchmarks. To measure performance improvement over the heuristic policy, we normalize the return of each algorithm X using the formula ( J X -J random ) / ( J H-only -J random ) , where J X , J H-only, and J random denote the task returns of algorithm X , the heuristic policy, and the random policy, respectively. In Figure 1, we present the interquartile mean (IQM) of the normalized return and the probability of improvement for each method across 29 tasks, following [26]. IQM, also known as the 25% trimmed mean, is a robust estimate against outliers. It discards the bottom and top 25% of runs and calculates the mean score of the remaining 50%. The probability of improvement measures whether an algorithm performs better than another, regardless of the margin of improvement. Both approaches prevent outliers from dominating the performance estimate.

Results: The results in Figure 1 indicate that policies trained with task rewards only (J-only) generally perform worse than those trained with heuristics, both in terms of IQM of normalized return and probability of improvement. PBRS does not improve upon J-only, demonstrating that the optimal policy invariance guarantee rarely holds in practice. Both EIPO and HuRL outperform J-only but do not surpass H-only, demonstrating that neither approach can improve upon the heuristic policy. Policies trained with both task and heuristic rewards (J+H) perform slightly worse than those trained with heuristics only (H-only), possibly because the weight coefficient balancing both rewards is too task-sensitive to work across all tasks. HEPO, however, outperforms all other methods in both IQM of normalized returns and shows a probability of improvement over the heuristic policy greater than 50%, indicating statistically significant improvements as suggested by Agarwal et al. [26]. Complete learning curves are presented in the Appendix B.1. Additional results on various benchmarks and RL algorithms are provided in Appendix B.3, demonstrating that HEPO is effective in hard-exploration tasks using exploration bonuses [23] and with RL algorithms beyond PPO.

4.2 Can HEPO be robust to reward functions designed in the wild?

Setup: We envision to develop an RL algorithm that can effectively utilize heuristic reward functions, thereby reducing the time costs associated with reward design. We simulate reward design scenarios

0.0

0.1

0.2

0.3

0.4

0.5

0.6

P(X > H-only)

0.17

0.50

0.37

0.26

0.12

0.29

0.62

J-only

H-only

HuRL

PBRS

EIPO

HEPO (ours)

Algorithm

0.0

0.2

0.4

0.6

0.8

1.0

1.2

Normalized return

0.17

0.98

0.84

0.70

0.02

0.27

1.13

(a) IQM over ISAAC + BI-DEX

0.0

0.1

0.2

0.3

0.4

0.5

0.6

P(X > H-only)

0.17

0.50

0.42

0.30

0.13

0.29

0.62

(b) Probability of Improvement over ISAAC + BI-DEX

and evaluate the algorithm's performance when trained with heuristic reward functions created under real-world conditions. Unlike the highly engineered reward functions in ISAAC, participants were asked to design their heuristic reward functions within a short time frame. This approach assesses the algorithm's effectiveness when trained with less refined heuristic reward functions. Participants were asked to iterate on their reward design by writing the reward function, training a policy with a given RL algorithm, reviewing the videos and learning curves, refining the reward, and repeating the process. We selected the FrankaCabinet task for this study because its original heuristic reward function is heavily engineered and consisting of many terms. The task of FrankaCabinet is training a robot arm to open a cabinet. We recruited twelve graduate students with varying levels of proficiency in machine learning and robotics and divided them into two groups. One group used HEPO to iterate on heuristic rewards, while the other group used PPO. This approach ensures that the designed heuristic reward functions are not specialized for one algorithm and ineffective for another. Each participant was instructed to edit the heuristic reward function to help RL algorithms maximize the task return. We used the same task reward metric as in Section 4.1. We then used the final versions of their heuristic reward functions to train HEPO and PPO, and reported the normalized return of the learned policies. Note that we adhered to the normalization scheme outlined in Section 4.1, based on the performance of the policy trained with the original heuristic reward function. This approach allows us to observe any performance drop when training policies with less engineered heuristic reward functions.

Quantitative results: Figure 2 shows that HEPO achieves significantly higher (lower confidence bound above the baselines' upper confidence bound) average normalized returns than PPO trained only on heuristic rewards (PPO (Honly)) in 9 out of 12 heuristic reward functions. Additionally, Table 1 indicates that across all heuristic functions, HEPO achieves a higher interquartile mean (IQM) of normalized returns and has a statistically significant probability of outperforming PPO (H-only) with lower confidence bound greater than 0 . 5 . This suggests that even when trained with poorly designed heuristic reward functions, HEPO performs better than PPO (H-only). Notably, PPO (H-only) that is trained with H 2 , H 5 , and H 6 achieves normal-

0

1

2

3

Normalized return

1.60

0.97

0.01

2.12

H1

0.36

0.09

0.00

0.96

H2

1.49

0.53

0.00

2.13

H3

0.000.000.00 0.07

H4

0.30

0.000.00

1.68

H5

0.66

0.15

0.00

2.01

H6

1

2

3

Normalized return

1.60

0.41

0.00

2.11

H7

1.59

0.78

0.00

2.16

H8

0.000.000.000.00

H9

0.000.000.000.00

H10

1.62

0.45

0.00

2.09

H11

0.000.000.000.00

H12

H-only

HuRL

PBRS

HEPO

0

ized returns below 1 , while HEPO achieves returns greater than or close to 1 . Since returns are normalized using the performance of the PPO policy trained with the well-designed heuristic reward function in ISAAC, a return below 1 . 0 indicates a performance drop for PPO (H-only) when using potentially ill-designed heuristic rewards. In contrast, HEPO can improve upon policies trained with carefully engineered heuristic reward functions, even when trained with possibly ill-designed heuristic reward functions.

Qualitative observation: We aim to understand why PPO's performance declines when trained with heuristic reward functions H 2 , H 5 , and H 6 . These functions are similar to the original heuristic reward in FrankaCabinet , but with different weights for each term. For example, in H 5 , the weight of action penalty is 1 , whereas in the original heuristic reward function it is 7 . 5 . This suggests that HEPO might handle poorly scaled heuristic reward terms better than PPO, which is sensitive to these weights. The heuristic reward functions H 12 and H 9 had an incorrect sign for the distance component, which caused the policy to be rewarded for moving away from the cabinet instead of toward it, making the learning task more challenging.

4.3 Ablation Studies

Expanding on the discussion of relation to relevant work EIPO [22] in Section 3.3, our goal is to examine the implementation choices of HEPO and illustrate the efficacy of each modification in this section. HEPO differs from EIPO primarily in two aspects: (1) the selection of a reference policy π ref in the constraint J ( π ) ≥ J ( π ref ) , and (2) the strategy for utilizing policies to gather trajectories. Both studies are conducted on standard locomotion and manipulation tasks, such as Ant , FrankaCabinet , and AllegroHand . In addition, we provide further studies on the sensitivity to hyperparameters in Appendix B.2.

Selection of reference policy in constraint: HEPO and EIPO both enforce a performance improvement constraint J ( π ) ≥ J ( π ref ) during training. HEPO uses a heuristic policy π H as the reference ( π ref = H-only), while EIPO uses a task-only policy ( π ref = J-only). However, relying solely on policies trained with task rewards as references may not suffice for complex robotic tasks, as they often perform much worse than those trained with heuristic rewards. We compared the performance of HEPO with different reference policies in Figure 3a. The result shows that setting π ref = J-only (EIPO) improves the performance over the task-only policy J-only while notably degrading performance, sometimes even worse than H-only , suggesting it's insufficient for surpassing the heuristic policy.

Strategy of collecting trajectories: We use both the enhanced policy π and the heuristic policy π H simultaneously to sample half of the environment's trajectories (referred to as Joint ). Conversely, EIPO switches between π and π H using a specified mechanism, where only one selected policy samples trajectories for updating both π and π H within the same episode (referred to as Alternating ). This study compares the performance of these two trajectory rollout methods. We modify HEPO to

0

1000

2000

Iteration

0.0

0.5

1.0

Normalized return

Ant

0

500

1000

1500

Iteration

0

1

2

FrankaCabinet

0

5000

10000

15000

Iteration

0.0

0.5

1.0

1.5

2.0

AllegroHand

J-only

H-only

HEPO(

ref

= J-only)

HEPO(

ref

= H-only)

(a) Comparison of reference policy

π

ref

choice in HEPO's constraint

J

(

π

)

J

(

π

ref

)

0

1000

2000

Iteration

0.00

0.25

0.50

0.75

1.00

Normalized return

Ant

0

500

1000

1500

Iteration

0.0

0.5

1.0

1.5

2.0

FrankaCabinet

0

5000

10000

15000

Iteration

0.0

0.5

1.0

1.5

2.0

AllegroHand

J-only

H-only

HEPO(Alternating)

HEPO(Joint)

(b) Comparison of trajectory collecting strategies

gather trajectories using the Alternating strategy and present the results in Figure 3b. The findings indicate that Alternating results in a performance drop during mid-training and fails to match the performance of HEPO(Joint) . We hypothesize that this occurs because the batch of trajectories collected solely by one policy deviates significantly from those that another policy can generate (i.e., high off-policy error), leading to less effective PPO policy updates. In contrast, Joint samples trajectories using both policies, preventing the collected trajectories from deviating too much from each other.

5 Related Works

Reward shaping: Reward shaping has been a significant area, including potential-based reward shaping (PBRS) [2, 27], bilevel optimization approaches [28-30] on reward model learning, and heuristic-guided methods (HuRL) [10] that schedule heuristic rewards. Our method differs as it is a policy optimization method agnostic to the heuristic reward function and can be applied to those shaped or learned rewards.

Constrained policy optimization: Recent work like Extrinsic-Intrinsic Policy Optimization (EIPO) [22] proposes constrained optimization by tuning exploration bonuses to prevent exploiting them at the cost of task rewards. Extensions [31] balance imitating a teacher model and reward maximization. Our work differs in balancing human-designed heuristic rewards and task rewards, improving upon policies trained with engineered heuristic rewards. We also propose implementation enhancements over EIPO [22] (Section 4.3).

6 Discussion & Limitation

HEPO for RL practitioners: In this paper, we showed that HEPO is robust to the possibly illdesigned heuristic reward function in Section 4.2 and also exhibit high-probability improvement over PPO when training with heavily engineered heuristic rewards in robotic tasks in Section 4.1. Moving forward, when users need to integrate heuristic reward functions into RL algorithms, HEPO can potentially be a useful tool to reduce users' time on designing rewards since it can improve performance even with under-engineered heuristic rewards.

Limitations: While HEPO shows high-probability performance improvement over heuristic policies trained with well-designed heuristic reward, one limitation is that HEPO does not have a guarantee

to converge to the optimal policy theoretically. One future work can be incorporating the insight in recent theoretical advances on reward engineering [3] to make a convergence guarantee.

Author Contributions

Chi-Chang Lee: Co-led the project, led the implementation of the proposed algorithms and baselines, and conducted the experiments.

Zhang-Wei Hong: Co-led the project, led the writing of the paper, scaled up the experiment infrastructure, and conducted the experiments.

Pulkit Agrawal: Played a key role in overseeing the project, editing the manuscript, and the presentation of the work.

References

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . 2018.

Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Icml , volume 99, pages 278-287, 1999.

Abhishek Gupta, Aldo Pacchiano, Yuexiang Zhai, Sham Kakade, and Sergey Levine. Unpacking reward shaping: Understanding the benefits of reward engineering on sample complexity. Advances in Neural Information Processing Systems , 35:15281-15295, 2022.

Tao Chen, Jie Xu, and Pulkit Agrawal. A system for general in-hand object re-orientation. Conference on Robot Learning , 2021.

Gabriel B Margolis, Ge Yang, Kartik Paigwar, Tao Chen, and Pulkit Agrawal. Rapid locomotion via reinforcement learning. Robotics: Science and Systems , 2022.

Gabriel B Margolis and Pulkit Agrawal. Walk these ways: Tuning robot control for generalization with multiplicity of behavior. In Conference on Robot Learning , pages 22-31. PMLR, 2023.

Ashish Kumar, Zipeng Fu, Deepak Pathak, and Jitendra Malik. RMA: rapid motor adaptation for legged robots. In Robotics: Science and Systems XVII, Virtual Event, July 12-16, 2021 , 2021.

Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning quadrupedal locomotion over challenging terrain. Science robotics , 5(47):eabc5986, 2020.

Sam Michael Devlin and Daniel Kudenko. Dynamic potential-based reward shaping. In 11th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2012) , pages 433-440. IFAAMAS, 2012.

Ching-An Cheng, Andrey Kolobov, and Adith Swaminathan. Heuristic-guided reinforcement learning. Advances in Neural Information Processing Systems , 34:13550-13563, 2021.

Sam Devlin, Logan Yliniemi, Daniel Kudenko, and Kagan Tumer. Potential-based difference rewards for multiagent reinforcement learning. In Proceedings of the 2014 international conference on Autonomous agents and multi-agent systems , pages 165-172, 2014.

Adam Eck, Leen-Kiat Soh, Sam Devlin, and Daniel Kudenko. Potential-based reward shaping for finite horizon online pomdp planning. Autonomous Agents and Multi-Agent Systems , 30: 403-445, 2016.

Babak Badnava, Mona Esmaeili, Nasser Mozayani, and Payman Zarkesh-Ha. A new potentialbased reward shaping for reinforcement learning agent. In 2023 IEEE 13th Annual Computing and Communication Workshop and Conference (CCWC) , pages 01-06. IEEE, 2023.

Grant C Forbes and David L Roberts. Potential-based reward shaping for intrinsic motivation (student abstract). In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 23488-23489, 2024.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Proceedings of the 32nd International Conference on Machine Learning (ICML-15) , pages 1889-1897, 2015.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv:1801.01290 , 2018.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature , 2015.

Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, and Gavriel State. Isaac gym: High performance gpu-based physics simulation for robot learning, 2021.

Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In In Proc. 19th International Conference on Machine Learning . Citeseer, 2002.

John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. Highdimensional continuous control using generalized advantage estimation. International Conference of Representation Learning , 2015.

Eric Chen, Zhang-Wei Hong*, Joni Pajarinen, and Pulkit (* equal contribution) Agrawal. Redeeming intrinsic rewards via constrained optimization. Advances in Neural Information Processing Systems , 35:4996-5008, 2022.

Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations , 2019. URL https://openreview.net/forum?id=H1lJJnR5Ym .

Yuanpei Chen, Yaodong Yang, Tianhao Wu, Shengjie Wang, Xidong Feng, Jiechuan Jiang, Zongqing Lu, Stephen Marcus McAleer, Hao Dong, and Song-Chun Zhu. Towards human-level bimanual dexterous manipulation with reinforcement learning. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2022.

Denys Makoviichuk and Viktor Makoviychuk. rl-games: A high-performance framework for reinforcement learning. https://github.com/Denys88/rl_games , May 2021.

Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in Neural Information Processing Systems , 34, 2021.

Sam Devlin and Daniel Kudenko. Dynamic potential-based reward shaping. In Adaptive Agents and Multi-Agent Systems , 2012.

Yujing Hu, Weixun Wang, Hangtian Jia, Yixiang Wang, Yingfeng Chen, Jianye Hao, Feng Wu, and Changjie Fan. Learning to utilize shaping rewards: A new approach of reward shaping. Advances in Neural Information Processing Systems , 33:15931-15941, 2020.

Dhawal Gupta, Yash Chandak, Scott Jordan, Philip S Thomas, and Bruno C da Silva. Behavior alignment via reward function optimization. Advances in Neural Information Processing Systems , 36, 2024.

Zeyu Zheng, Junhyuk Oh, and Satinder Singh. On learning intrinsic rewards for policy gradient methods. Advances in Neural Information Processing Systems , 31, 2018.

Idan Shenfeld, Zhang-Wei Hong, Aviv Tamar, and Pulkit Agrawal. TGRL: An algorithm for teacher guided reinforcement learning. In Proceedings of the 40th International Conference on Machine Learning , 2023.

Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations , 2014.

A Implementation Details

A.1 Full Derivation

We will detailedly describe the update of the enhanced policy ( π in Equation 7) and the heuristic policy ( π H in Equation 8) at each iteration.

A.1.1 Notations

B HEPO: the buffer to store samples collected by π i

B H : the buffer to store samples collected π i H .

A.1.2 Enhanced Policy π Update

Given a α value, π i +1 is derived using the arguments of the maxima in Equation 4, which can be re-written as follows:

where U π i α and U π i H α are defined as follows:

To efficiently achieve the update process in Equation 12, we aim to utilize previously collected trajectories for optimization, outlined in Equation 7. Here, we refer to [15], using those previously collected trajectories to form a lower bound surrogate objectives, ˆ J π i α ( π ) and ˆ J π i H α ( π ) , as alternatives

of E π [ U π i ( s t , a t )] and E π [ U π i H α ( s t , a t )] to derive π i +1

where E π [ U π i α ( s t , a t )] ≥ ˆ J π i HEPO ( π ) and E π [ U π i H α ( s t , a t )] ≥ ˆ J π i H HEPO ( π ) always hold; ϵ ∈ [0 , 1] denotes a threshold. Intuitively, this clipped objective (Eq. 14) penalizes the policy π that behaves differently from π i or π i H because overly large or small the action probability ratios between two policies are clipped.

A.1.3 Heuristic Policy π H Update

π i +1 H is derived using the arguments of the maxima of H ( π ) , which can be re-written as follows:

Similarly, we again rely on the approximation from [15] to derive a lower bound surrogate objective for both E π [ A π i h ( s t , a t )] and E π [ A π i H h ( s t , a t )] as follows:

where E π [ A π i h ( s t , a t )] ≥ ˆ H π i ( π ) and E π [ A π i H h ( s t , a t )] ≥ ˆ H π i H ( π ) always hold. Different from vanilla heuristic training, instead of solely collecting trajectories from π i H , we collect trajectories from both π and π H to enrich sample efficiency.

A.2 Implementation Tricks

A.2.1 Sample Sharing for Value Function Update

In practice, obtaining real value functions for training is not feasible. We estimate the value function using collected trajectories, but this approach tends to fail because the value function becomes biased toward the policy responsible for trajectory collection.

To prevent error information from the estimated value function interfering with the training procedure, we share the trajectory samples within the B HEPO and B H buffers to update our value functions:

A.2.2 Smoothing Lagrangian Multiplier α Update

The Lagrangian multiplier α determines the desired constraint information during training. However, in practice the gradient α tends to become explosive. To stabilize the α update procedure, we accumulate previous gradients and adopt the Adam optimizer [32] as follows:

where K is the number of previous K advantage expectation records that we take into account. To smooth the current α gradient for each update, we calculate the median of the previous K records. In our experiments, we assigned K a value of 8 .

A.3 Overall Workflow

Algorithm 2 Detailed Heuristic-Enhanced Policy Optimization (HEPO)

1: Initialize policies ( π 1 , π 1 H ) and values ( V π 1 r , V π 1 h , V π 1 H r , V π 1 H h )

2: for i = 1 · · · do

3: # ROLLOUT STAGE

4: Collect trajectory buffers ( B HEPO , B H ) using ( π i , π i H )

5: Compute ( A π i r ( s t , a t ) , A π i h ( s t , a t ) ) via GAE with ( V π i r , V π i h ) ∀ ( s t , a t ) ∈ B HEPO

6: Compute ( A π i H r ( s t , a t ) , A π i H h ( s t , a t ) ) via GAE with ( V π i H r , V π i H h ) ∀ ( s t , a t ) ∈ B H

7: Compute ( ˆ J π i HEPO , ˆ J π i H HEPO ) based on Equation 14

8: Compute ( ˆ H π i , ˆ H π i H ) based on Equation 16

9:

10: # UPDATE STAGE

13: Update ( V π i r , V π i h , V π i H r , V π i H h ) based on Equation 18

14: Update α based on Equation 22

15: end for

▷ i denotes iteration index

A.4 Training details

Following the PPO framework [15], our experiments are based on a continuous action actor-critic algorithm implemented in rl_games [25], using Generalized Advantage Estimation (GAE) [21] to compute advantages for policy optimization. For PPO, we employed the same policy network and value network architecture, and the same hyperparameters used in IsaacGymEnvs [19]. We also include our source code in the supplementary material. In HEPO, we use two policies for optimization, with each policy maintaining the same model configurations as those used in PPO. Below we introduce HEPO-specific hyperparameters used in our experiments in Section 4.1. The hyperparameters for updating the Lagrangian multiplier α in HEPO are listed as follows:

For baselines, we search for hyperparamters λ for J + H in Ant , FrankaCabinet , and AllegroHand , as shown in Section B.2. We set λ = 1 for all the experiments because it shows better performance on the three chosen environments. For HuRL [10], we follow the scheduling setting provided in their paper.

B Supplementary Experimental Results

B.1 All Learning Curves on the task objective J

We present all the learning curves in Figure 4.

Return

Return

0

500

1000

1500

2000

2500

Iteration

0

2500

5000

7500

10000

Return

Ant

0

500

1000

1500

Iteration

1000

750

500

250

0

Anymal

0

1000

2000

3000

Iteration

100

200

300

400

Quadcopter

0

200

400

600

800

Iteration

0

500

1000

Ingenuity

0

2000

4000

6000

8000

10000

Iteration

0

2500

5000

7500

Return

Humanoid

0

5000

10000

15000

Iteration

0.00

0.02

0.04

AllegroHand

0

500

1000

1500

Iteration

0.0

0.2

0.4

0.6

FrankaCabinet

0

1000

2000

3000

4000

5000

Iteration

0.0

0.2

0.4

0.6

FrankaCubeStack

0

5000

10000

15000

Iteration

0.00

0.02

0.04

0.06

0.08

Return

ShadowHand

0

2000

4000

6000

8000

10000

Iteration

0.0

0.1

0.2

0.3

ShadowHandSpin

0

2000

4000

6000

8000

10000

Iteration

0.00

0.02

0.04

0.06

ShadowHandUpsideDown

0

2000

4000

6000

8000

10000

Iteration

0.0

0.2

0.4

0.6

ShadowHandBlockStack

0

2000

4000

6000

8000

10000

Iteration

0.0

0.2

0.4

0.6

0.8

Return

ShadowHandBottleCap

0

2000

4000

6000

8000

10000

Iteration

0.0

0.1

0.2

0.3

ShadowHandCatchAbreast

0

2000

4000

6000

8000

10000

Iteration

0.0

0.2

0.4

0.6

ShadowHandCatchOver2Underarm

0

2000

4000

6000

8000

10000

Iteration

0.0

0.1

0.2

0.3

ShadowHandCatchUnderarm

0

2000

4000

6000

8000

10000

Iteration

0.0

0.2

0.4

0.6

0.8

Return

ShadowHandDoorCloseInward

0

2000

4000

6000

8000

10000

Iteration

0.000

0.005

0.010

ShadowHandDoorCloseOutward

0

2000

4000

6000

8000

10000

Iteration

0.0

0.1

0.2

0.3

0.4

ShadowHandDoorOpenInward

0

2000

4000

6000

8000

10000

Iteration

0.0

0.2

0.4

0.6

ShadowHandDoorOpenOutward

0

2000

4000

6000

8000

10000

Iteration

0.0

0.2

0.4

0.6

0.8

Return

ShadowHandGraspAndPlace

0

2000

4000

6000

8000

10000

Iteration

0.0

0.1

0.2

0.3

0.4

ShadowHandKettle

0

2000

4000

6000

8000

10000

Iteration

0.000

0.025

0.050

0.075

0.100

ShadowHandLiftUnderarm

0

2000

4000

6000

8000

10000

Iteration

0.0

0.2

0.4

0.6

ShadowHandOver

0

2000

4000

6000

8000

10000

Iteration

0.00000

0.00025

0.00050

0.00075

0.00100

ShadowHandPushBlock

0

2000

4000

6000

8000

10000

Iteration

0.0000

0.0002

0.0004

ShadowHandReOrientation

0

2000

4000

6000

8000

10000

Iteration

0.0

0.2

0.4

0.6

0.8

ShadowHandScissors

0

2000

4000

6000

8000

10000

Iteration

0.0

0.2

0.4

ShadowHandSwingCup

0

2000

4000

6000

8000

10000

Iteration

0.050

0.025

0.000

0.025

0.050

ShadowHandSwitch

0

2000

4000

6000

8000

10000

Iteration

0.000

0.005

0.010

0.015

0.020

ShadowHandTwoCatchUnderarm

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

J-only

H-only

HuRL

PBRS

EIPO

HEPO

B.2 Sensitivity to hyperparameters

In this section, we aim to verify HEPO's sensitivity to two main types of hyperparameters: (1) the weight of the heuristic reward in optimization (denoted as λ ) and (2) the learning rate for updating α .

0

1000

2000

Iteration

0.00

0.25

0.50

0.75

1.00

Heuristic normalized return

Ant

0

500

1000

1500

Iteration

0.0

0.5

1.0

1.5

2.0

2.5

FrankaCabinet

0

5000

10000

15000

Iteration

0.0

0.5

1.0

1.5

2.0

AllegroHand

0.01

0.1

1

5

HEPO

0

1000

2000

Iteration

0.00

0.25

0.50

0.75

1.00

Heuristic normalized return

Ant

0

500

1000

1500

Iteration

0.0

0.5

1.0

1.5

2.0

FrankaCabinet

0

5000

10000

15000

Iteration

0.0

0.5

1.0

1.5

2.0

AllegroHand

Lagrangian multiplier step size (

)

0.01

0.0001

Similar to Section 4.3, we conducted our experiments on the Ant , FrankaCabinet , and AllegroHand tasks.

B.2.1 Sensitivity to the λ Value

Both HEPO and J+H can set a scaling coefficient to weight the heuristic reward in optimization, such that the objective becomes J ( π ) + λH ( π ) . This scaling coefficient can be used to balance both objectives. In this study, we compare HEPO and J+H on their performance sensitivity to the choice of λ , exhaustively training both HEPO and J+H with varying λ values. Note that though the formulation of HEPO does not depend on λ , one can still set a λ coefficient to scale the heuristic reward in HEPO. In our experiments, we did not optimize λ for HEPO but for the baselines trained with both rewards (J+H). In Figure 5, we found that J+H is sensitive to λ in all selected tasks, while HEPO performs well across a wide range of λ values. This indicates that HEPO is robust to the choice of λ .

B.2.2 Sensitivity to the Learning Rate for Updating α

HEPO's robustness relies on the α update, as it reflects the necessary constraint information at each iteration. Similar to Appendix B.2.1, setting different initial values of α is equivalent to using different λ values for our estimation, since both can be rewritten as the ratio between H ( π ) and J ( π ) . Both of these parameters indicate the necessary constraint information for conducting multi-objective optimization. In this study, we aim to verify whether HEPO can yield comparable improvement gaps under different initial values of α , thus providing a more robust optimization procedure.

As shown in Figure 6, we observe that HEPO is also robust to the choice of α 's initial values, similar to the results in Figure 5.

B.3 Additional results

Generality of HEPO: Wealso demonstrated HEPO can be implemented over the other RL algorithms in addition to PPO. We integrated HEPO into HuRL's SAC codebase. Despite HuRL using tuned hyperparameters as reported in its paper [10], HEPO outperformed SAC and matched HuRL on the

most challenging task in Figure 7b using the same hyperparameters from Section 4 of our manuscript, showing the generality of HEPO on different RL algorithms.

Better than EIPO on the most challenging task: Comparing HEPO and EIPO on the most challenging task, Montezuma's Revenge, reported in EIPO's paper [22] using RND exploration bonuses [23] (as used in the EIPO paper), Figure 7a shows that HEPO performs better than EIPO. Also, HEPO matched PPO trained with RND bonuses at convergence (2 billion frames) reported in [23] using only 20% of the training data, demonstrating drastically improved sample efficiency.

How quality of heuristic reward functions impact HEPO? We believe that Figure 2 reveals the relationship between HEPO's performance and the quality of the heuristic reward. The policy trained with only heuristic rewards (H-only) represents both the asymptotic performance of in HEPO and the quality of the heuristic itself. We found a positive correlation (Pearson coefficient of 0.9) between the average performances of H-only and HEPO in Figure 2 results, suggesting that better heuristics lead to improved HEPO performance. Figure 7c provides more details.

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Frame

1e8

0

2000

4000

6000

8000

Average task return

MontezumaRevenge-v5

PPO

EIPO

HEPO

(a) EIPO v.s. HEPO on PPO

0

50

100

150

200

Epoch

500

400

300

200

100

0

Average task return

Sparse-Reacher-v2

SAC

HuRL

PBRS

HEPO

(b) HuRL v.s. HEPO on SAC

0.0

0.5

1.0

1.5

2.0

Performance of H-only

0

1

2

3

Performance of HEPO

Corr(HEPO, H-only) = 0.9

(c) Correlation between the performance of HEPO and H- only.

C Environment Details

As depicted in Section 4, we conducted our experiments based on the Isaac Gym ( ISAAC ) simulator [19] and the Bi-DexHands ( BI-DEX ) benchmark [24]. The selected task classes in ISAAC can be partitioned into 4 groups - Locomotion Tracking ( Anymal ), Locomotion Progressing ( Ant and Humanoid ), Helicopter Progressing ( Ingenuity and Quadcopter ), and Manipulation Tasks ( FrankaCabinet , FrankaCubeStack , ShadowHand , and AllegroHand ). In addition, BI-DEX provides dual dexterous hand manipulation tasks through ISAAC, reaching human-level sophistication of hand dexterity and bimanual coordination. Their tasks include ShadowHandOver , ShadowHandCatchUnderarm , ShadowHandCatchOver2Underarm , ShadowHandCatchAbreast , ShadowHandTwoCatchUnderarm , ShadowHandLiftUnderarm , ShadowHandDoorOpenInward , ShadowHandDoorOpenOutward , ShadowHandDoorCloseInward , ShadowHandDoorCloseOutward , ShadowHandSpin , ShadowHandUpsideDown , ShadowHandBlockStack , ShadowHandBottleCap , ShadowHandGraspAndPlace , ShadowHandKettle , ShadowHandPen , ShadowHandPushBlock , ShadowHandReOrientation , ShadowHandScissors , ShadowHandSwingCup .

For Locomotion Tracking, our emphasis lies in assessing the precision of velocities in linear and angular motions, ensuring that the robot responds closely to the assigned values. To this end, we define tracking errors as our task rewards. For Locomotion Progressing and Helicopter Progressing, our emphasis lies in evaluating the progress made by the robots in reaching the assigned destination from a given start point. To this end, we define movement progress as our task rewards. For Manipulation tasks and all tasks within the BI-DEX benchmark, our emphasis lies in whether and how quickly the robotic hands can successfully complete the assigned missions, reaching the desired goal states. To

this end, we define task rewards using a binary label, assigning a value of 1 to indicate successful attainment of the goal state and 0 otherwise.

The following are our reward function definitions, which include heuristic and task reward terms in Python style:

Isaac Gym - Locomotion Progressing Task: Ant

Isaac Gym - Locomotion Progressing Task: Humanoid

Isaac Gym - Helicopter Progressing Task: Ingenuity

Isaac Gym - Helicopter Progressing Task: Quadcopter

Isaac Gym - Manipulation Task: FrankaCabinet

Isaac Gym - Manipulation Task: FrankaCubeStack

Isaac Gym - Manipulation Task: ShadowHand

Isaac Gym - Manipulation Task: AllegroHand

Bi-DexHands: ShadowHandOver

Bi-DexHands: ShadowHandCatchUnderarm

Bi-DexHands: ShadowHandCatchOver2Underarm

Bi-DexHands: ShadowHandCatchAbreast

Bi-DexHands: ShadowHandTwoCatchUnderarm

Bi-DexHands: ShadowHandLiftUnderarm

Bi-DexHands: ShadowHandDoorOpenInward

Bi-DexHands: ShadowHandDoorOpenOutward

Bi-DexHands: ShadowHandDoorCloseInward

Bi-DexHands: ShadowHandDoorCloseOutward

Bi-DexHands: ShadowHandSpin

Bi-DexHands: ShadowHandUpsideDown

Bi-DexHands: ShadowHandBlockStack

Bi-DexHands: ShadowHandBottleCap

Bi-DexHands: ShadowHandGraspAndPlace

Bi-DexHands: ShadowHandKettle

Bi-DexHands: ShadowHandPen

Bi-DexHands: ShadowHandPushBlock

Bi-DexHands: ShadowHandReOrientation

Bi-DexHands: ShadowHandScissors

Bi-DexHands: ShadowHandSwingCup

Bi-DexHands: ShadowHandSwitch

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: This paper aims to explore alternatives for improving task performance in finite data settings using heuristic signals. Our experiments on robotic locomotion, helicopter, and manipulation tasks demonstrate that this method consistently improves performance, regardless of the general effectiveness of the heuristic signals. We are confident that our abstract and introduction sections accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations of our approach are illustrated in Section 6.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We have shown our derivation details and limitations in Section A.1 and Section 6.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have provided detailed derivation and implementation descriptions (including the simulation environments, hyperparameters, and reward definitions for training and evaluations) in our Appendix section. Additionally, we have also provided our source code in our Supplementary Materials.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We have provided detailed implementation descriptions (including the simulation environments, hyperparameters, and reward definitions for training and evaluations) in our Appendix section. Additionally, we have also provided our source code in our Supplementary Materials. The adopted simulation environments are all well-known and available online.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We have provided detailed implementation descriptions (including the simulation environments, hyperparameters, and reward definitions for training and evaluations) in the experiment and Appendix sections. Additionally, we have also provided our source code in our Supplementary Materials, including all the training and environment configurations.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: All of our experimental results were obtained by 5 random seeds. We have provided the mean and standard deviation for our experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [No]

Justification: But each training procedure can be performed on a single GeForce RTX 2080 Ti device. The required computational resources for all the simulation benchmarks are listed on their respective websites.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have carefully examined the ethical guidelines and verified that our work fully adheres to all the principles and requirements.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The paper does not discuss both potential positive societal impacts and negative societal impacts of the work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper has no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Throughout this paper, we have provided proper citations and references for all utilized repositories, benchmark simulations, and models/algorithms to uphold transparency and ensure appropriate attribution.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We will furnish comprehensive documentation for our released code, elucidating its usage and providing information about the original source. Additionally, we have ensured that any code modified from external sources is subject to licenses that permit modification and redistribution.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: No, but we engaged 12 participants in devising reward functions as part of the experiments detailed in Section 4.2.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors address the problem of reward design in reinforcement learning. Specifically, it is common practice to add heuristics in the reward to help the training and a lot of manual engineering is needed to balance this heuristic and the main reward. They propose to modify the standard RL objective to a constrained optimization objective in order to incorporate heuristics in the reward function in a principled way. \n\nTheir formulation consists in optimizing both the task reward and the heuristic reward while constraining the policy to have a better performance in the task reward compared to a policy that is solely maximizing the heuristic reward. This constraint ensures that the policy is not only exploiting the heuristic but also solves the task.\n\nThe method is evaluated on a set of robotic control tasks where heuristics are already provided.\n\nI acknowledge reading the authors rebuttal and updated my score.\n\nSTRENGTHS:\nThey propose a simple formalization of the problem of reward design with adding heuristic to a main reward task. This does not address the general problem of reward alignment, but I think it is a very relevant problem in RL for domains with well defined but sparse rewards.\n\nThe derivation of the algorithm is presented clearly. \n\nThe experiments show that the proposed algorithm outperform the baseline in terms of average return. Except from the missing EIPO, the authors choose relevant baseline to compare against. \n\nThe second experiment of reward designed in the wild is interesting. The author attempt to recreate the process of reward design with human participants. \n\nThe statistical analysis of the experimental results is rigorous. \n\nThe ablation study shows the importance of using the policy trained on the heuristic reward as a reference.\n\nWEAKNESSES:\nThe data collection part of the algorithm is not super clear. My understanding is that both policies are used and they collect the same amount of trajectories for each, why would the amount of data be the same and not more? The theoretical implications of this data sharing part should be clarified as well. It is not obvious to me that equation 7 and 8 actually lead to the correct policy improvement step. \n\nNo theoretical analysis of the algorithm is provided. I am not sure that detailed theorem and proofs would be needed but at least some insights or discussion about the convergence. There is a brief mention that it would not converge to the optimal policy in the limitations but no proof is given. \n\nWhy not include EIPO in the first experiment? Overall, there is no direct comparison with EIPO. The authors comment on certain aspects during the ablation study but it would have been even stronger to add a line for EIPO. \n\nIn the second experiment some results appear to be missing which makes me question the validity of table 1 as well. I hope the author can clarify this in the rebuttal.\n\nOne experiment that is missing and could have been interesting is to study the performance of the algorithm depending on the quality of the heuristic. \n\nI suggest merging the related work section and section 3.3.\n\nQUESTIONS:\n-\tWhy doesn’t the algorithm need more data since you need to collect data with both policies?\n-\tEquation (7) and (8) introduce an off-policy aspect to the algorithm, is convergence still guaranteed?\n-\tWhy use J_random in the formula to measure performance? Why not use a percentage of improvement over $J_{H-only}$?\n-\tWhat happened with H4-9-10-12 in figure 2?\n-\tOn the same problem with different heuristic, how would the performance of HEPO look like depending on the asymptotic performance of $\\pi_H$\n\nLIMITATIONS:\nThe current algorithm seems limited to on policy methods. It also applies to problems where we can make a clear distinction between task reward and heuristic rewards.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a constrained optimization approach for reinforcement learning, ensuring that the learned policy outperforms or matches the performance of policies trained solely with heuristic rewards. In simulations, the proposed HEPO method consistently delivers superior task performance across various tasks, demonstrating improved returns in robotic locomotion, helicopter, and manipulation tasks.\n\nSTRENGTHS:\nThis paper is straightforward and easy to follow, with a clearly stated motivation. It introduces an innovative constrained optimization approach that effectively leverages heuristic signals to enhance task performance in reinforcement learning. The effectiveness of the HEPO method is demonstrated through its consistent superiority across diverse tasks such as robotic locomotion, helicopter control, and manipulation, even with limited training data. Additionally, the proposed method simplifies the reward design process by dynamically balancing heuristic and task rewards without the need for manual tuning, showing significant improvements over traditional heuristic-only and mixed reward approaches.\n\nWEAKNESSES:\nThe paper would benefit from a more thorough theoretical analysis. Including a theoretical guarantee on the convergence rate, an examination of how the selection of heuristics impacts the HEPO method, and an analysis of the effect of the alpha parameter would enhance the robustness and depth of the study.\n\nQUESTIONS:\nCould you please explain how the heuristic is obtained in the implementation? Additionally, it would be helpful to see the computational cost of solving the constrained optimization problem. Is it possible to include a comparison of your method with reward shaping and policy cloning methods? For equation 5, when rearranging equation 4, could you clarify where the term J(pi_H) goes?\n\nLIMITATIONS:\nWhile the HEPO method shows empirical success, it lacks a thorough theoretical analysis, including a formal guarantee on the convergence rate and the effects of heuristic selection and the alpha parameter on performance. Additionally, the computational cost of solving the constrained optimization problem is not addressed, which could impact the feasibility of the approach in large-scale or real-time applications. Furthermore, the study does not compare HEPO with other reinforcement learning techniques such as reward shaping and policy cloning, which would provide a more comprehensive evaluation of its effectiveness.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel approach to improve reinforcement learning (RL) by incorporating heuristic signals. The authors propose a constrained optimization method that uses heuristic policies as references to ensure that the learned policies outperform heuristic policies on the exact task objective. The method, named Heuristic-Enhanced Policy Optimization (HEPO), was tested on various robotic tasks, demonstrating consistent performance improvements over standard heuristic methods.\n\nSTRENGTHS:\n- The paper introduces a simple method to improve reinforcement learning (RL) algorithms using heuristic rewards, by shifting focus from ensuring optimal policy invariance to simply outperforming heuristic policies.\n- The method is validated on various environments, showing consistent performance improvements.\n- The paper is clearly written.\n\nWEAKNESSES:\n- The claim that the contribution is an add-on to existing deep RL algorithms is not fully substantiated, as the authors only evaluate HEPO using Proximal Policy Optimization (PPO). Demonstrating HEPO's performance with other deep RL algorithms, such as Soft Actor-Critic (SAC), would help validate this claim.\n- The method is very similar to Extrinsic-Intrinsic Policy Optimization (EIPO), with the main difference being the reference policy used. The authors should provide more details on this difference, including the number of additional hyperparameters introduced in EIPO and a comparison of its complexity in terms of time with HEPO.\n- The benchmarks used in this paper differ from those in previous works like EIPO [6] and HuRL [7]. The authors should address how HEPO would perform on those benchmarks to ensure a fair comparison.\n- The paper does not include challenging tasks such as Montezuma's Revenge.\n- It is unclear whether the reward functions and heuristics used in the experiments are designed by the authors or sourced from ISAAC and BI-DEX benchmarks.\n\nQUESTIONS:\n- Please check weaknesses.\n- How does HEPO perform with other deep RL algorithms, such as Soft Actor-Critic (SAC)?\n- What specific heuristics are used in the progressing tasks, and how are the rewards sparse/delayed?\n- In figure3: Is the T-only ablation equivalent to the EIPO algorithm or a modified version of HEPO?\n- Line 129 is missing a reference to the application of PPO in robotics with heuristic rewards.\n\nLIMITATIONS:\nThe authors address the limitations and potential negative impacts in their paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}