Going Beyond Heuristics by Imposing Policy Improvement as a Constraint

Authors: chi-chang  lee
zhang-wei  hong
pulkit  agrawal
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 21:56:49

Abstract: In many reinforcement learning (RL) applications, incorporating heuristic rewards alongside the task reward is crucial for achieving desirable performance. Heuristics encode prior human knowledge about how a task should be done, providing valuable hints for RL algorithms. However, such hints may not be optimal, limiting the performance of learned policies. 
The currently established way of using heuristics is to modify the heuristic reward in a manner that ensures that the optimal policy learned with it remains the same as the optimal policy for the task reward (i.e., optimal policy invariance). 
However, these methods often fail in practical scenarios with limited training data. We found that while optimal policy invariance ensures convergence to the best policy based on task rewards, it doesn't guarantee better performance than policies trained with biased heuristics under a finite data regime, which is impractical. In this paper, we introduce a new principle tailored for finite data settings. Instead of enforcing optimal policy invariance, we train a policy that combines task and heuristic rewards and ensures it outperforms the heuristic-trained policy. As such, we prevent policies from merely exploiting heuristic rewards without improving the task reward. Our experiments on robotic locomotion, helicopter control, and manipulation tasks demonstrate that our method consistently outperforms the heuristic policy, regardless of the heuristic rewards' quality.
Code is available at https://github.com/Improbable-AI/hepo.

Main paper: 
 introduction: 
 Reinforcement learning (RL) [1] is a powerful framework for learning policies that can surpass human performance in complex tasks. However, training RL policies with sparse or delayed rewards is often ineffective. Instead of relying solely on sparse task rewards that indicate an agent's success or failure, it is common to augment the sparse reward with heuristic reward terms that provide denser reward supervision to speed up and improve the performance of RL policies [2, 3] . Shining examples of the success and necessity of heuristic reward terms are complex robotic object manipulation [4] and locomotion [5] [6] [7] [8] tasks. However, heuristics impose human assumptions that may limit the RL algorithm. For example, a heuristic reward function may encourage a robot to walk like a human, yet there could be faster walking policies that don't resemble human gait.The key question is how to learn a policy π that outperforms one trained solely on heuristics (i.e., heuristic policy π H ). Practitioners tackle this problem by tuning the balance between the task objective J(π) and the heuristic objective H(π) in the augmented training objective J(π) + λH(π), where λ controls the balance between the two objectives for a policy π. However, it requires careful tuning of λ to make the policy π outperform the heuristic policy; otherwise, the algorithm might prioritize heuristic rewards while neglecting the task objective.Tuning λ to balance both objectives is time-consuming. We desire an algorithm that finds a policy that outperforms the heuristic policy without requiring such tuning for any given heuristic reward function. Classic methods [2, [9] [10] [11] [12] [13] [14] modify heuristic rewards to align the augmented objective's optimal policy with the one for the task objective (i.e., optimal policy invariance), theoretically ensuring that with infinite data the policy outperforms the heuristic. However, in practice, these modified heuristics often fall short on complex robotic tasks compared to policies trained solely on heuristic objectives, as demonstrated in our study (Section 4.1) and prior work [10] .In this paper, we challenge the prevailing paradigm by questioning whether optimal policy invariance is the appropriate objective to prevent heuristics from limiting RL agent's performance in the finite data regime. As optimal policy invariance ensures convergence to the optimal policy with infinite data it may not be practical in many real-world settings. We propose an alternative paradigm that, in every step of training, imposes the constraint of improving task performance beyond a policy trained solely on heuristic rewards (i.e., J(π) ≥ J(π H )). This condition J(π) ≥ J(π H ) guarantees that the learned policy π outperforms the heuristic policy π H , effectively surpassing human-designed heuristics. Additionally, such policy improvements can be verified and achieved using many existing deep RL algorithms [15] [16] [17] [18] in finite data settings.Therefore, we enforce the policy improvement condition J(π) ≥ J(π H ) as a constraint, preventing the policy from exploiting heuristic rewards during training. We propose the following constrained optimization objective: max π J(π) + H(π) subject to J(π) ≥ J(π H )Optimizing this objective at each iteration allows learning a policy performing better than or equal to policies trained only on heuristic rewards. It prevents capitalizing on heuristic rewards at the expense of task rewards. Moreover, it enables adaptively balancing both rewards over time instead of using a fixed coefficient λ. Our contribution is an add-on to existing deep RL algorithms to improve RL algorithms trained with heuristic rewards. We evaluated our method on robotic locomotion, helicopter, and manipulation tasks using the IsaacGym simulator [19] . The results show that our method led to superior task rewards and higher task-completion success rates compared to the policies solely trained with heuristic rewards, even when heuristic rewards are ill-designed.2 Preliminaries: Reinforcement Learning with Heuristic Reinforcement Learning (RL): RL is a popular paradigm for solving sequential decision-making problems [1] where the problems are modeled as an interaction between an agent and an unknown environment [1] . The agent aims to improve its performance through repeated interactions with the environment. At each round of interaction, the agent starts from the environment's initial state s 0 and samples the corresponding trajectory. At each timestep t within that trajectory, the agent perceives the state s t , takes an action a t ∼ π(.|s t ) according to its policy π, receives a task reward r t = r(s t , a t ), and transitions to a next state s t+1 until reaching terminal states, after which a new trajectory is initialized from s 0 , and the cycle repeats. The agent's goal is to learn a policy π that maximizes the expected return J(π) in a trajectory as below:EQUATIONwhere γ denotes a discount factor [1] and E π [.] denotes taking expectation over the trajectories sampled by π. In the following, we term J as the true task objective, as it indicates the performance of a policy on the task.RL with Heuristic: In many tasks, learning a policy to maximize the true objective J is challenging because rewards may be sparse or delayed. This lack of feedback makes policy optimization difficult for RL algorithms. To address this, practitioners often use a heuristic reward function h with denser reward signals to facilitate optimization, aiming to learn a policy that performs better in J. The policy trained to maximize the expected return of heuristic rewards is called the heuristic policy π H . The expected return of heuristic rewards, termed the heuristic objective H, is defined as:EQUATIONwhere h(s t , a t ) is the heuristic reward at timestep t for state s t and action a t .3 Method: Improving Heuristic Policy via Constrained Optimization Problem statement: Optimizing both task J and heuristic H objectives jointly could lead to better task performance than training solely with J or H, but needs careful tuning on the weight coefficient λ among both objectives in max π J(π) + λH(π). Without careful tuning, the policy π may learn to exploit heuristic rewards H and compromise performance of J. The goal of this paper is to mitigate the requirement of tuning this coefficient to balance them in order to improve task performance.Key insight -Leveraging Heuristic with Constraint: We aim to use the heuristic objective H for training only when it improves task performance J and ignore it otherwise. Rather than manually tuning the weight coefficient λ to balance both rewards, we introduce a key insight: impose a policy improvement constraint (i.e., J(π) ≥ J(π H )) during training. This prevents RL algorithms from exploiting heuristic rewards H at the expense of task rewards J. To achieve this goal, we introduce the following constrained optimization objective:EQUATIONThis constrained objective (Equation 3) results in an improved policy π over the heuristic policy π H , leading us to call this framework Heuristic-Enhanced Policy Optimization (HEPO). A practical algorithm to optimize this objective is presented in Section 3.1, and its implementation on a widelyused RL algorithm [15] in robotics is detailed in Section 3.2. 
algorithm: heuristic-enhanced policy optimization (hepo): 
 Finding feasible solutions for the constrained optimization problem in Equation 3 is challenging due to the nonlinearity of the objective function J with respect to π. One practical approach is to convert it into the following unconstrained min-max optimization problem using Lagrangian duality:min α≥0 max π L(π, α), where L(π, α) := J(π) + H(π) + α (J(π) -J(π H )) ,where the Lagrangian multiplier is α ∈ R + . We can optimize the policy π and the multiplier α for this min-max problem by a gradient descent-ascent strategy, alternating between optimizing π and α.Enhanced policy π: The optimization objective for the policy π can be obtained by rearranging Equation 4 as follows:max π (1 + α)J(π) + H(π),where (1 + α)J(π) + H(π) = E π ∞ t=0 γ t (1 + α)r(s t , a t )) + h(s t , a t ) .(EQUATIONThis represents an unconstrained regular RL objective with the modified reward at each step as (1 + α)r(s t , a t ) + h(s t , a t ), which can be optimized using any off-the-shelf deep RL algorithm. In this modified reward, the task reward r(s t , a t ) is weighted by the Lagrangian multiplier α, reflecting the potential variation in the task reward's importance during training as α evolves. The interaction between the update of the Lagrangian multiplier and the policy will be elaborated upon next.Lagrangian Multiplier α: The Lagrangian multiplier α is optimized for Equation 4 by stochastic gradient descent, with the gradient defined as:EQUATIONNotably, ∇ α L(π, α) is exactly the performance gain of the policy π over the heuristic policy π H on the task objective J. By applying gradient descent with ∇ α L(π, α), when J(π) > J(π H ) and thus ∇ α L(π, α) > 0, the Lagrangian multiplier α decreases. As α represents the weight of the task reward in Equation 5, it indicates that when π outperforms π H , the importance of the task reward diminishes because π already achieves superior performance compared to the heuristic policy π H regarding the task objective J. Conversely, when J(π) < J(π H ), α increases, thereby emphasizing the importance of task rewards in optimization. The update procedure for the Lagrangian multiplier α offers an adaptive reconciliation between the heuristic reward h and the task reward r.We present a practical approach to optimize the min-max problem in Equation 4 using Proximal Policy Optimization (PPO) [15] . We selected PPO because it is widely used in robotic applications involving heuristic rewards, although our HEPO framework is not restricted to PPO. The standard PPO implementation involves iterative stochastic gradient descent updates over numerous iterations, alternating between collecting trajectories with policies and updating those policies. We outline the optimization process for each iteration and provide a summary of our implementation in Algorithm 1.Training policies π and π H : Instead of pre-training the heuristic policy π H , which requires additional data and reduces data efficiency, we concurrently train both the enhanced policy π and the heuristic policy π H , allowing them to share data. For each iteration i, we gather trajectories τ and τ H using the enhanced policy π i and the heuristic policy π i H , respectively. Following PPO's implementation, we compute the advantages A π i r (s t , a t ), A π i H r (s t , a t ), A π i h (s t , a t ), and A π i H h (s t , a t ) for the task reward r and heuristic reward h with respect to π i and π i H . We then weight the advantage with the action probability ratio between the new policies being optimized (i.e., π i+1 and π i+1 H ) and the policies collecting the trajectories (i.e., π i or π i H ). Finally, we optimize the policies at the next iteration i + 1 for the objectives in Equations 7 and 8:EQUATIONEQUATIONE τ ∼π i π(a t |s t ) π i (a t |s t ) A π i H h (s t , a t ) (Heuristic policy).Maximizing the advantages will result in a policy that maximizes the expected return for a chosen reward function, as demonstrated in PPO [15] . This enables us to maximize the objective J and H. We estimate the advantages A π i r (s t , a t ) and A π i h (s t , a t ) (or A π i H r (s t , a t ) and A π i H h (s t , a t )) using the standard PPO implementation with different reward functions. Therefore, we omit the details of the advantage's clipped surrogate objective in PPO, and leave them in Appendix A.1.Although PPO is an on-policy algorithm, the use of off-policy importance ratio correction (i.e., the action probability ratios between two policies) allows us to use states and actions generated by another policy. This enables us to train π using data from π H and vice versa. Both policies π and π H are trained using the same data but with different reward functions. Note that collecting trajectories from both policies does not require more data than the standard PPO implementation. We collect half the trajectories with each policy, π and π H , for a total of B trajectories (see Algorithm 1) . Then, we update both π and π H using all B trajectories.Optimizing the Lagrangian multiplier α: To update the Lagrangian multiplier α, we need to compute the gradient in Equation 6, which corresponds to the performance gain of the enhanced policy π over the heuristic policy π H on the task objective J. Utilizing the performance difference lemma [20, 16] , we relate this improvement to the expected advantages over trajectories sampled by the enhanced policy π asJ(π) -J(π H ) = E π [A π H r(s t , a t )].However, this approach only utilizes half of the trajectories at each iteration since it exclusively relies on trajectories from the enhanced policy π. To leverage trajectories from both policies, we also consider the performance gain in the reverse direction as-(J(π H ) -J(π)) = -E π H [A π r(s t , a t )]. Consequently, we can estimate the gradient of α using trajectories from both policies, as illustrated below:EQUATIONAt each iteration i, we estimate the gradient of α using the advantage A π H r (s t , a t ) and A π r (s t , a t ) on the trajectories sampled from both π i and π i H , and update α with stochastic gradient descent as follows:EQUATIONwhere η ∈ R + is the step size. The expected advantage in Equation 11 are estimated using the generalized advantage estimator (GAE) [21] .Algorithm 1 Heuristic-Enhanced Policy Optimization (HEPO) π i+1 ←-Train the policy π i for optimizing Equation 7 using both τ and τ Hπ i+1 H ←-Train the policy π i H for optimizing Equation 8 using both τ and τ Hα ←-Update the Lagrangian multiplier α by gradient descent (Equation 9) using τ and τ H 9: end for 3.3 Connection to Extrinsic-Intrinsic Policy Optimization [22] Closely related to our HEPO framework, Chen et al. [22] proposes Extrinsic-Intrinsic Policy Optimization (EIPO), which trains a policy to maximize both task rewards and exploration bonuses [23] subject to the constraint that the learned policy π must outperform the task policy π J trained solely on task rewards. HEPO and EIPO differ in their objective functions and implementation of the constrained optimization problem. Additional information can be found in the Appendix, covering the objective formulation (Appendix A.1), implementation tricks (Appendix A.2), and detailed pseudocode (Appendix A.3).Exploration bonuses [23] can be viewed as heuristic rewards. The main difference between HEPO and EIPO's optimization objectives lies in constraint design. Both frameworks require the learned policy π to outperform a reference policy π ref (i.e., J(π) ≥ J(π ref )) but use a different reference policy. EIPO uses the task policy π J as the reference policy π ref because they aim for asymptotic optimality in task rewards. If the constraint is satisfied with π J being the optimal policy for task rewards, the learned policy π will also be optimal for task rewards. In contrast, HEPO uses the heuristic policy π H trained solely on heuristic rewards since HEPO aims to improve upon it.HEPO simplifies the implementation. Both HEPO and EIPO train two policies with shared data, but EIPO alternates the policy used for trajectory collection each iteration and has a complex switching rule, which introduces more hyperparameters. HEPO collects trajectories using both policies together at each iteration, simplifying implementation and avoiding extra hyperparameters. 
experiments: 
 We evaluate whether HEPO enhances the performance of RL algorithms in maximizing task rewards while training with heuristic rewards. We conduct experiments on 9 tasks from IsaacGym (ISAAC) [19] and 20 tasks from the Bidexterous Manipulation (BI-DEX) benchmark [24] . These tasks rely on heavily engineered reward functions for training RL algorithms. Each task has a task reward function r that defines the task objective J to be maximized, and a heuristic reward function h that defines the heuristic objective H, provided in the benchmarks to facilitate the optimization of task objectives J. We implement HEPO based on PPO [15] and compare it with the following baselines:• H-only (heuristic only): This is the standard PPO baseline provided in ISAAC. The policy is trained solely using the heuristic reward: max π H(π). The heuristic reward functions in ISAAC and BI-DEX are designed to help RL algorithms maximize the task objective J. This baseline is crucial to determine if an algorithm can surpass a policy trained with highly engineered heuristic rewards.• J-only (task only): The policy is trained using only the task reward: max π J(π). This baseline demonstrates the performance achievable without heuristics. Ideally, algorithms that incorporate heuristics should outperform this baseline.• J+H (mixture of task and heuristic): The policy is trained using a mixture of task and heuristic rewards: max π J(π) + λH(π), with λ balancing the two rewards. As [22] shows, proper tuning of λ can enhance task performance by balancing both training objectives.• Potential-based Reward Shaping (PBRS) [2] : The policy is trained to maximize E π [ ∞ t=0 γ t r t + γh t+1 -h t ], where r t and h t are the task and heuristic rewards at timestep t. PBRS guarantees that the optimal policy is invariant to the task reward function. We include it as a baseline to examine if these theoretical guarantees hold in practice.• HuRL [10] : The policy is trained to maximizeE π [ ∞ t=0 γ t r t + (1 -β i )γh t+1 ],where β i is a coefficient updated at each iteration to balance heuristic rewards during different training stages. The scheduling mechanism is detailed in [10] and our source code provided in the Supplementary Material.• EIPO [22] : The policy is trained using the constrained objective:max π J(π) + H(π) s.t. J(π) ≥ J(π J ), where π J is the policy trained with task rewards only. EIPO is similar to HEPO but differs in formulation and implementation, as detailed in Section 4.3.Each method is trained for 5 random seeds and implemented based on the open-sourced implementation [25] , where the detailed training hyperparameters can be found in Appendix A.4.Based on the task success criteria in ISAAC and BI-DEX, we consider two types of task reward functions r: (i) Progressing (for locomotion or helicopter robots) and (ii) Goal-reaching (for manipulation). In progressing tasks, robots aim to maximize their traveling distance or velocity from an initial point to a destination. Thus, movement progress is defined as the task reward. In goal-reaching tasks, robots aim to complete assigned goals by reaching specific goal states. Here, task rewards are binary, with a value of 1 indicating successful attainment of the goal and 0 otherwise. Detailed descriptions of our task objectives and total reward definitions are provided in Appendix C.We aim to determine if HEPO achieves higher task returns and improves upon the policy trained with only heuristic rewards (H-only) in the majority of tasks. In this experiment, we use the heuristic reward functions from the ISAAC and BI-DEX benchmarks. To measure performance improvement over the heuristic policy, we normalize the return of each algorithm X using the formula (J X -J random )/(J H-only -J random ), where J X , J H-only , and J random denote the task returns of algorithm X, the heuristic policy, and the random policy, respectively. In Figure 1 , we present the interquartile mean (IQM) of the normalized return and the probability of improvement for each method across 29 tasks, following [26] . IQM, also known as the 25% trimmed mean, is a robust estimate against outliers. It discards the bottom and top 25% of runs and calculates the mean score of the remaining 50%. The probability of improvement measures whether an algorithm performs better than another, regardless of the margin of improvement. Both approaches prevent outliers from dominating the performance estimate.The results in Figure 1 indicate that policies trained with task rewards only (J-only) generally perform worse than those trained with heuristics, both in terms of IQM of normalized return and probability of improvement. PBRS does not improve upon J-only, demonstrating that the optimal policy invariance guarantee rarely holds in practice. Both EIPO and HuRL outperform J-only but do not surpass H-only, demonstrating that neither approach can improve upon the heuristic policy.Policies trained with both task and heuristic rewards (J+H) perform slightly worse than those trained with heuristics only (H-only), possibly because the weight coefficient balancing both rewards is too task-sensitive to work across all tasks. HEPO, however, outperforms all other methods in both IQM of normalized returns and shows a probability of improvement over the heuristic policy greater than 50%, indicating statistically significant improvements as suggested by Agarwal et al. [26] . The horizontal axis shows the probability that algorithm X outperforms the policy trained solely with heuristic rewards (H-only). HEPO achieves a 62% probability of improvement over the heuristic policy on average, with the lower bound of the confidence interval above 50%, indicating statistically significant improvements over the heuristic policy.and evaluate the algorithm's performance when trained with heuristic reward functions created under real-world conditions. Unlike the highly engineered reward functions in ISAAC, participants were asked to design their heuristic reward functions within a short time frame. This approach assesses the algorithm's effectiveness when trained with less refined heuristic reward functions. Participants were asked to iterate on their reward design by writing the reward function, training a policy with a given RL algorithm, reviewing the videos and learning curves, refining the reward, and repeating the process. We selected the FrankaCabinet task for this study because its original heuristic reward function is heavily engineered and consisting of many terms. The task of FrankaCabinet is training a robot arm to open a cabinet. We recruited twelve graduate students with varying levels of proficiency in machine learning and robotics and divided them into two groups. One group used HEPO to iterate on heuristic rewards, while the other group used PPO. This approach ensures that the designed heuristic reward functions are not specialized for one algorithm and ineffective for another. Each participant was instructed to edit the heuristic reward function to help RL algorithms maximize the task return. We used the same task reward metric as in Section 4.1. We then used the final versions of their heuristic reward functions to train HEPO and PPO, and reported the normalized return of the learned policies. Note that we adhered to the normalization scheme outlined in Section 4.1, based on the performance of the policy trained with the original heuristic reward function. This approach allows us to observe any performance drop when training policies with less engineered heuristic reward functions. ized returns below 1, while HEPO achieves returns greater than or close to 1. Since returns are normalized using the performance of the PPO policy trained with the well-designed heuristic reward function in ISAAC, a return below 1.0 indicates a performance drop for PPO (H-only) when using potentially ill-designed heuristic rewards. In contrast, HEPO can improve upon policies trained with carefully engineered heuristic reward functions, even when trained with possibly ill-designed heuristic reward functions.Qualitative observation: We aim to understand why PPO's performance declines when trained with heuristic reward functions H2, H5, and H6. These functions are similar to the original heuristic reward in FrankaCabinet, but with different weights for each term. For example, in H5, the weight of action penalty is 1, whereas in the original heuristic reward function it is 7.5. This suggests that HEPO might handle poorly scaled heuristic reward terms better than PPO, which is sensitive to these weights. The heuristic reward functions H12 and H9 had an incorrect sign for the distance component, which caused the policy to be rewarded for moving away from the cabinet instead of toward it, making the learning task more challenging.Expanding on the discussion of relation to relevant work EIPO [22] in Section 3.3, our goal is to examine the implementation choices of HEPO and illustrate the efficacy of each modification in this section. HEPO differs from EIPO primarily in two aspects: (1) the selection of a reference policy π ref in the constraint J(π) ≥ J(π ref ), and (2) the strategy for utilizing policies to gather trajectories. Both studies are conducted on standard locomotion and manipulation tasks, such as Ant, FrankaCabinet, and AllegroHand. In addition, we provide further studies on the sensitivity to hyperparameters in Appendix B.2.Selection of reference policy in constraint: HEPO and EIPO both enforce a performance improvement constraint J(π) ≥ J(π ref ) during training. HEPO uses a heuristic policy π H as the reference (π ref = H-only), while EIPO uses a task-only policy (π ref = J-only). However, relying solely on policies trained with task rewards as references may not suffice for complex robotic tasks, as they often perform much worse than those trained with heuristic rewards. We compared the performance of HEPO with different reference policies in Figure 3a . The result shows that setting π ref = J-only (EIPO) improves the performance over the task-only policy J-only while notably degrading performance, sometimes even worse than H-only, suggesting it's insufficient for surpassing the heuristic policy.We use both the enhanced policy π and the heuristic policy π H simultaneously to sample half of the environment's trajectories (referred to as Joint). Conversely, EIPO switches between π and π H using a specified mechanism, where only one selected policy samples trajectories for updating both π and π H within the same episode (referred to as Alternating).This study compares the performance of these two trajectory rollout methods. We modify HEPO to We show that using the policies trained with heuristic rewards (J-only) is better than using the policies trained with task rewards (J-only) when training HEPO. (b) HEPO(Joint) that collects trajectories using both policies leads to better performance than HEPO(Alternating) that alternates between two policies to collect trajectories. See Section 4.3 for details gather trajectories using the Alternating strategy and present the results in Figure 3b . The findings indicate that Alternating results in a performance drop during mid-training and fails to match the performance of HEPO(Joint). We hypothesize that this occurs because the batch of trajectories collected solely by one policy deviates significantly from those that another policy can generate (i.e., high off-policy error), leading to less effective PPO policy updates. In contrast, Joint samples trajectories using both policies, preventing the collected trajectories from deviating too much from each other.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: We have provided detailed derivation and implementation descriptions (including the simulation environments, hyperparameters, and reward definitions for training and evaluations) in our Appendix section. Additionally, we have also provided our source code in our Supplementary Materials.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. Answer: [Yes] Justification: All of our experimental results were obtained by 5 random seeds. We have provided the mean and standard deviation for our experimental results.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
related works: 
 Reward shaping: Reward shaping has been a significant area, including potential-based reward shaping (PBRS) [2, 27] , bilevel optimization approaches [28] [29] [30] on reward model learning, and heuristic-guided methods (HuRL) [10] that schedule heuristic rewards. Our method differs as it is a policy optimization method agnostic to the heuristic reward function and can be applied to those shaped or learned rewards.Constrained policy optimization: Recent work like Extrinsic-Intrinsic Policy Optimization (EIPO) [22] proposes constrained optimization by tuning exploration bonuses to prevent exploiting them at the cost of task rewards. Extensions [31] balance imitating a teacher model and reward maximization. Our work differs in balancing human-designed heuristic rewards and task rewards, improving upon policies trained with engineered heuristic rewards. We also propose implementation enhancements over EIPO [22] (Section 4.3). 
discussion & limitation: 
 HEPO for RL practitioners: In this paper, we showed that HEPO is robust to the possibly illdesigned heuristic reward function in Section 4.2 and also exhibit high-probability improvement over PPO when training with heavily engineered heuristic rewards in robotic tasks in Section 4.1.Moving forward, when users need to integrate heuristic reward functions into RL algorithms, HEPO can potentially be a useful tool to reduce users' time on designing rewards since it can improve performance even with under-engineered heuristic rewards.Limitations: While HEPO shows high-probability performance improvement over heuristic policies trained with well-designed heuristic reward, one limitation is that HEPO does not have a guarantee to converge to the optimal policy theoretically. One future work can be incorporating the insight in recent theoretical advances on reward engineering [3] to make a convergence guarantee.A.1 Full DerivationWe will detailedly describe the update of the enhanced policy (π in Equation 7) and the heuristic policy (π H in Equation 8) at each iteration.A.1.1 Notations• V π r (s t ) := E (st,at)∼π ∞ t=0 γ t r(s t , a t )|s 0 = s t • V π h (s t ) := E (st,at)∼π ∞ t=0 γ t h(s t , a t )|s 0 = s t • A π r (s t , a t ) := r(s t , a t ) + V π r (s t+1 ) -V π r (s t ) • A π h (s t , a t ) := h(s t , a t ) + V π h (s t+1 ) -V π h (s t )• B HEPO : the buffer to store samples collected by π i• B H : the buffer to store samples collected π i H .Given a α value, π i+1 is derived using the arguments of the maxima in Equation 4, which can be re-written as follows:EQUATIONwhere U π i α and U π i H α are defined as follows:EQUATIONTo efficiently achieve the update process in Equation 12, we aim to utilize previously collected trajectories for optimization, outlined in Equation 7. Here, we refer to [15] , using those previously collected trajectories to form a lower bound surrogate objectives, Ĵπ i α (π) andEQUATIONwhereE π [U π i α (s t , a t )] ≥ Ĵπ i HEPO (π) and E π [U π i H α (s t , a t )] ≥ Ĵπ i H HEPO (π) always hold; ϵ ∈ [0, 1] denotes a threshold. Intuitively, this clipped objective (Eq. 14) penalizes the policy π that behaves differently from π i or π i H because overly large or small the action probability ratios between two policies are clipped.π i+1 H is derived using the arguments of the maxima of H(π), which can be re-written as follows:EQUATIONSimilarly, we again rely on the approximation from [15] to derive a lower bound surrogate objective for bothE π [A π i h (s t , a t )] and E π [A π i H h (s t , a t )]as follows:EQUATIONEQUATIONwhereEQUATIONπ) always hold. Different from vanilla heuristic training, instead of solely collecting trajectories from π i H , we collect trajectories from both π and π H to enrich sample efficiency.In practice, obtaining real value functions for training is not feasible. We estimate the value function using collected trajectories, but this approach tends to fail because the value function becomes biased toward the policy responsible for trajectory collection.To prevent error information from the estimated value function interfering with the training procedure, we share the trajectory samples within the B HEPO and B H buffers to update our value functions:EQUATIONVEQUATIONVEQUATIONThe Lagrangian multiplier α determines the desired constraint information during training. However, in practice the gradient α tends to become explosive. To stabilize the α update procedure, we accumulate previous gradients and adopt the Adam optimizer [32] as follows:EQUATIONwhere K is the number of previous K advantage expectation records that we take into account. To smooth the current α gradient for each update, we calculate the median of the previous K records. In our experiments, we assigned K a value of 8.Algorithm 2 Detailed Heuristic-Enhanced Policy Optimization (HEPO)1: Initialize policies (π 1 , π 1 H ) and values (V π 1 r , V π 1 h , V π 1 H r , V π 1 H h ) 2: for i = 1 • • • do ▷ i denotes iteration index 3: # ROLLOUT STAGE 4:Collect trajectory buffers (B HEPO , B H ) using (π i , π i H )ComputeA π i r (s t , a t ), A π i h (s t , a t ) via GAE with V π i r , V π i h ∀(s t , a t ) ∈ B HEPO 6: Compute A π i H r (s t , a t ), A π i H h (s t , a t ) via GAE with V π i H r , V π i H h ∀(s t , a t ) ∈ B H 7:Compute Ĵπ i HEPO , Ĵπ i H HEPO based on Equation 148:Compute Ĥπ i , Ĥπ i H based on Equation 169: 10: # UPDATE STAGE 11: 1814:π i+1 ← arg max π Ĵπ i HEPO (π) + Ĵπ i H HEPO (π) 12: π i+1 H ← arg max π Ĥπ i (π) + Ĥπ i H (π) 13: Update (V π i r , V π i h , V π i H r , V π i H h ) based on EquationUpdate α based on Equation 2215: end forFollowing the PPO framework [15] , our experiments are based on a continuous action actor-critic algorithm implemented in rl_games [25] , using Generalized Advantage Estimation (GAE) [21] to compute advantages for policy optimization. For PPO, we employed the same policy network and value network architecture, and the same hyperparameters used in IsaacGymEnvs [19] . We also include our source code in the supplementary material. In HEPO, we use two policies for optimization, with each policy maintaining the same model configurations as those used in PPO. Below we introduce HEPO-specific hyperparameters used in our experiments in Section 4.1. The hyperparameters for updating the Lagrangian multiplier α in HEPO are listed as follows: (-ϵ α , ϵ α ) 1.0 Range of the α value [0, ∞)For baselines, we search for hyperparamters λ for J + H in Ant, FrankaCabinet, and AllegroHand, as shown in Section B.2. We set λ = 1 for all the experiments because it shows better performance on the three chosen environments. For HuRL [10] , we follow the scheduling setting provided in their paper.We present all the learning curves in Figure 4 . In this section, we aim to verify HEPO's sensitivity to two main types of hyperparameters: (1) the weight of the heuristic reward in optimization (denoted as λ) and ( 2) the learning rate for updating α. Both HEPO and J+H can set a scaling coefficient to weight the heuristic reward in optimization, such that the objective becomes J(π) + λH(π). This scaling coefficient can be used to balance both objectives. In this study, we compare HEPO and J+H on their performance sensitivity to the choice of λ, exhaustively training both HEPO and J+H with varying λ values. Note that though the formulation of HEPO does not depend on λ, one can still set a λ coefficient to scale the heuristic reward in HEPO. In our experiments, we did not optimize λ for HEPO but for the baselines trained with both rewards (J+H). In Figure 5 , we found that J+H is sensitive to λ in all selected tasks, while HEPO performs well across a wide range of λ values. This indicates that HEPO is robust to the choice of λ.HEPO's robustness relies on the α update, as it reflects the necessary constraint information at each iteration. Similar to Appendix B.2.1, setting different initial values of α is equivalent to using different λ values for our estimation, since both can be rewritten as the ratio between H(π) and J(π).Both of these parameters indicate the necessary constraint information for conducting multi-objective optimization. In this study, we aim to verify whether HEPO can yield comparable improvement gaps under different initial values of α, thus providing a more robust optimization procedure.As shown in Figure 6 , we observe that HEPO is also robust to the choice of α's initial values, similar to the results in Figure 5 .We also demonstrated HEPO can be implemented over the other RL algorithms in addition to PPO. We integrated HEPO into HuRL's SAC codebase. Despite HuRL using tuned hyperparameters as reported in its paper [10] , HEPO outperformed SAC and matched HuRL on the most challenging task in Figure 7b using the same hyperparameters from Section 4 of our manuscript, showing the generality of HEPO on different RL algorithms.Better than EIPO on the most challenging task: Comparing HEPO and EIPO on the most challenging task, Montezuma's Revenge, reported in EIPO's paper [22] using RND exploration bonuses [23] (as used in the EIPO paper), Figure 7a shows that HEPO performs better than EIPO. Also, HEPO matched PPO trained with RND bonuses at convergence (2 billion frames) reported in [23] using only 20% of the training data, demonstrating drastically improved sample efficiency.How quality of heuristic reward functions impact HEPO? We believe that Figure 2 reveals the relationship between HEPO's performance and the quality of the heuristic reward. The policy trained with only heuristic rewards (H-only) represents both the asymptotic performance of in HEPO and the quality of the heuristic itself. We found a positive correlation (Pearson coefficient of 0.9) between the average performances of H-only and HEPO in Figure 2 results, suggesting that better heuristics lead to improved HEPO performance. Figure 7c provides more details. Figure 7 : (a) Comparison of HEPO and EIPO [22] on the most challenging Atari task, Montezuma's Revenge, shown in the EIPO paper [22] . Both are implemented on top of EIPO's PPO codebase using RND exploration bonuses [23] as heuristic rewards H, as suggested in [22] . HEPO outperforms EIPO, achieving the performance (denoted as dashed line) similar to PPO trained with RND at convergence (2 billion frames) reported in [23] in five times fewer frames. (b) HEPO matches HuRL's performance on the most challenging Sparse-Reacher task using HuRL's SAC codebase [10] , despite HuRL being tuned for this task and HEPO using the same hyperparameters from our Section 4. This also highlights HEPO's generality in different RL algorithms. (c) HEPO's performance is positively correlated with that of the heuristic policy trained with heuristic rewards only (H-only), suggesting that HEPO's effectiveness will improve as the quality of heuristic rewards increases.As depicted in Section 4, we conducted our experiments based on the Isaac Gym (ISAAC) simulator [19] and the Bi-DexHands (BI-DEX) benchmark [24] . For Locomotion Tracking, our emphasis lies in assessing the precision of velocities in linear and angular motions, ensuring that the robot responds closely to the assigned values. To this end, we define tracking errors as our task rewards. For Locomotion Progressing and Helicopter Progressing, our emphasis lies in evaluating the progress made by the robots in reaching the assigned destination from a given start point. To this end, we define movement progress as our task rewards. For Manipulation tasks and all tasks within the BI-DEX benchmark, our emphasis lies in whether and how quickly the robotic hands can successfully complete the assigned missions, reaching the desired goal states. To this end, we define task rewards using a binary label, assigning a value of Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [No] Justification: But each training procedure can be performed on a single GeForce RTX 2080 Ti device. The required computational resources for all the simulation benchmarks are listed on their respective websites.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes]Justification: We have carefully examined the ethical guidelines and verified that our work fully adheres to all the principles and requirements.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.• The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [NA] Justification: The paper does not discuss both potential positive societal impacts and negative societal impacts of the work performed. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: This paper has no such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes]  
 
 Appendix:  


Decision: The paper proposes a method, HEPO, for ensuring that heuristics help RL by making RL optimize a constraint objective, where the constraint is given by the value function of a policy trained on heuristic rewards. HEPO is closely related to an existing approach, EIPO, but provides valuable improvements over it in implementation simplicity and reduced number of hyperparameters. The reviewers found the paper easy to read and suggested a number of improvements to the experiments, such as using SAC, that the authors have successfully implemented during the rebuttal stage. The reviewers and the meta-reviewers trust that the authors will include them into the manuscript.

After these new results are incorporated into the paper, the work's only notable remaining weakness (which is addressed in the rebuttals only partly) is the scarcity of its theoretical analysis. However, the metareviewer believes that the paper's algorithmic contributions are sufficiently valuable on their own and fully recommends this paper for acceptance.

Reviews: review: paper_summary: paper_summary: The paper presents a novel approach to improve reinforcement learning (RL) by incorporating heuristic signals. The authors propose a constrained optimization method that uses heuristic policies as references to ensure that the learned policies outperform heuristic policies on the exact task objective. The method, named Heuristic-Enhanced Policy Optimization (HEPO), was tested on various robotic tasks, demonstrating consistent performance improvements over standard heuristic methods.
strength_weakness: strength_weakness: - The claim that the contribution is an add-on to existing deep RL algorithms is not fully substantiated, as the authors only evaluate HEPO using Proximal Policy Optimization (PPO). Demonstrating HEPO's performance with other deep RL algorithms, such as Soft Actor-Critic (SAC), would help validate this claim.
- The method is very similar to Extrinsic-Intrinsic Policy Optimization (EIPO), with the main difference being the reference policy used. The authors should provide more details on this difference, including the number of additional hyperparameters introduced in EIPO and a comparison of its complexity in terms of time with HEPO.
- The benchmarks used in this paper differ from those in previous works like EIPO [6] and HuRL [7]. The authors should address how HEPO would perform on those benchmarks to ensure a fair comparison.
- The paper does not include challenging tasks such as Montezuma's Revenge.
- It is unclear whether the reward functions and heuristics used in the experiments are designed by the authors or sourced from ISAAC and BI-DEX benchmarks.
questions: questions: - Please check weaknesses.
- How does HEPO perform with other deep RL algorithms, such as Soft Actor-Critic (SAC)?
- What specific heuristics are used in the progressing tasks, and how are the rewards sparse/delayed?
- In figure3: Is the T-only ablation equivalent to the EIPO algorithm or a modified version of HEPO?
- Line 129 is missing a reference to the application of PPO in robotics with heuristic rewards.
limitations: limitations: The authors address the limitations and potential negative impacts in their paper.
paper_summary: paper_summary: This paper introduces a constrained optimization approach for reinforcement learning, ensuring that the learned policy outperforms or matches the performance of policies trained solely with heuristic rewards. In simulations, the proposed HEPO method consistently delivers superior task performance across various tasks, demonstrating improved returns in robotic locomotion, helicopter, and manipulation tasks.
strength_weakness: strength_weakness: The paper would benefit from a more thorough theoretical analysis. Including a theoretical guarantee on the convergence rate, an examination of how the selection of heuristics impacts the HEPO method, and an analysis of the effect of the alpha parameter would enhance the robustness and depth of the study.
questions: questions: Could you please explain how the heuristic is obtained in the implementation? Additionally, it would be helpful to see the computational cost of solving the constrained optimization problem. Is it possible to include a comparison of your method with reward shaping and policy cloning methods? For equation 5, when rearranging equation 4, could you clarify where the term J(pi_H) goes?
limitations: limitations: While the HEPO method shows empirical success, it lacks a thorough theoretical analysis, including a formal guarantee on the convergence rate and the effects of heuristic selection and the alpha parameter on performance. Additionally, the computational cost of solving the constrained optimization problem is not addressed, which could impact the feasibility of the approach in large-scale or real-time applications. Furthermore, the study does not compare HEPO with other reinforcement learning techniques such as reward shaping and policy cloning, which would provide a more comprehensive evaluation of its effectiveness.
paper_summary: paper_summary: In this paper, the authors address the problem of reward design in reinforcement learning. Specifically, it is common practice to add heuristics in the reward to help the training and a lot of manual engineering is needed to balance this heuristic and the main reward. They propose to modify the standard RL objective to a constrained optimization objective in order to incorporate heuristics in the reward function in a principled way. 

Their formulation consists in optimizing both the task reward and the heuristic reward while constraining the policy to have a better performance in the task reward compared to a policy that is solely maximizing the heuristic reward. This constraint ensures that the policy is not only exploiting the heuristic but also solves the task.

The method is evaluated on a set of robotic control tasks where heuristics are already provided.

I acknowledge reading the authors rebuttal and updated my score.
strength_weakness: strength_weakness: The data collection part of the algorithm is not super clear. My understanding is that both policies are used and they collect the same amount of trajectories for each, why would the amount of data be the same and not more? The theoretical implications of this data sharing part should be clarified as well. It is not obvious to me that equation 7 and 8 actually lead to the correct policy improvement step. 

No theoretical analysis of the algorithm is provided. I am not sure that detailed theorem and proofs would be needed but at least some insights or discussion about the convergence. There is a brief mention that it would not converge to the optimal policy in the limitations but no proof is given. 

Why not include EIPO in the first experiment? Overall, there is no direct comparison with EIPO. The authors comment on certain aspects during the ablation study but it would have been even stronger to add a line for EIPO. 

In the second experiment some results appear to be missing which makes me question the validity of table 1 as well. I hope the author can clarify this in the rebuttal.

One experiment that is missing and could have been interesting is to study the performance of the algorithm depending on the quality of the heuristic. 

I suggest merging the related work section and section 3.3.
questions: questions: -	Why doesn’t the algorithm need more data since you need to collect data with both policies?
-	Equation (7) and (8) introduce an off-policy aspect to the algorithm, is convergence still guaranteed?
-	Why use J_random in the formula to measure performance? Why not use a percentage of improvement over $J_{H-only}$?
-	What happened with H4-9-10-12 in figure 2?
-	On the same problem with different heuristic, how would the performance of HEPO look like depending on the asymptotic performance of $\pi_H$
limitations: limitations: The current algorithm seems limited to on policy methods. It also applies to problems where we can make a clear distinction between task reward and heuristic rewards.

Comments: title: Thank you for the timely response
Thank you for the detailed response and the additional experiments.
update the figure
comment: Thank you for your responses. I have updated my rating.
Your answers addressed my questions. I have no further comments.
I think you answered all of my questions quite clearly. 

I would advise to modify the table 1 from the paper with the full results and add the plots from the rebuttal to the appendix as they are quite useful in understanding the performance of the method. I will update my score.
We missed one curve in the rebuttal PDF and have updated Figure 7(a) in the following link: https://imgur.com/jzxWoq6. Note that we didn't train EIPO and HEPO for 2 billion frames since it will take around two weeks to finish.