Going Beyond Heuristics by Imposing Policy Improvement as a Constraint
Authors: Chi-Chang Lee
Zhang-Wei Hong
Pulkit Agrawal
Venue: NeurIPS 2024 poster (2024)
Keywords: Deep reinforcement learning
Primary area: reinforcement_learning
TL;DR: We propose a modification to existing RL algorithms to improve the performance when trained with heuristic rewards.

Abstract: In many reinforcement learning (RL) applications, incorporating heuristic rewards alongside the task reward is crucial for achieving desirable performance. Heuristics encode prior human knowledge about how a task should be done, providing valuable hints for RL algorithms. However, such hints may not be optimal, limiting the performance of learned policies. 
The currently established way of using heuristics is to modify the heuristic reward in a manner that ensures that the optimal policy learned with it remains the same as the optimal policy for the task reward (i.e., optimal policy invariance). 
However, these methods often fail in practical scenarios with limited training data. We found that while optimal policy invariance ensures convergence to the best policy based on task rewards, it doesn't guarantee better performance than policies trained with biased heuristics under a finite data regime, which is impractical. In this paper, we introduce a new principle tailored for finite data settings. Instead of enforcing optimal policy invariance, we train a policy that combines task and heuristic rewards and ensures it outperforms the heuristic-trained policy. As such, we prevent policies from merely exploiting heuristic rewards without improving the task reward. Our experiments on robotic locomotion, helicopter control, and manipulation tasks demonstrate that our method consistently outperforms the heuristic policy, regardless of the heuristic rewards' quality.
Code is available at https://github.com/Improbable-AI/hepo.

Going Beyond Heuristics by Imposing Policy Improvement as a Constraint
Chi-Chang Lee' Zhang- Wei Hong²* Pulkit Agrawal?
Improbable AI Lab Massachusetts Institute of Technology
Abstract
In many reinforcement learning (RL) applications, incorporating heuristic rewards alongside the task reward is crucial for achieving desirable performance. Heuristics encode prior human knowledge about how task should be done, providing valu- able hints for RL algorithms. However, such hints may not be optimal, limiting the performance of learned policies. The currently established way of using heuristics is to modify the heuristic reward in a manner that ensures that the optimal policy learned with it remains the same the optimal policy for the task reward (i.e., optimal policy invariance). However, these methods often fail in practical scenarios with limited training data. We found that while optimal policy invariance ensures convergence to the best policy based on task rewards, it doesn't 't guarantee better performance than policies trained with biased heuristics under finite data regime, which is impractical. In this paper, we introduce a new principle tailored for finite data settings. Instead of enforcing optimal policy invariance, we train a policy that combines task and heuristic rewards and ensures it outperforms the beuristic-traines policy. As such, we prevent policies from merely exploiting heuristic rewards with- out improving the task reward. Our experiments control, and manipulation tasks demonstrate that our method consistently outper- forms the heuristic policy, regardless of the heuristic rewards' quality. Code is available https: //gi thub. com/ Ittp://bpbbbb/////aaaaa
robotic locomotion, helicopter
Introduction
Reinforcement learning (RL) [1] is powerful framework for learning policies that can surpass human performance in complex tasks. However, training RL policies with sparse or delayed rewards often ineffective. Instead of relying solely on sparse task rewards that indicate an agent's 's success or failure, it is common to augment the sparse reward with heuristic reward terms that provide denser reward supervision speed up and improve the performance of RI. policies [2, 3]. Shining examples of the success and necessity of heuristic reward terms are complex robotic object manipulation [4] and locomotion [5-8] tasks. However, heuristics impose human assumptions that may limit the RL. algorithm. For example, heuristic reward function may encourage robot to walk like human, yet there could be faster walking policies that don't 't resemble human gait.
The key question is how to learn a policy π that outperforms one trained solely on heuristics (i.e., heuristic policy T Practitioners tackle this problem by tuning the balance between the task objective (TT) and the heuristic objective H(x) in the augmented training objective ((T) + (H)).. where À controls the balance between the two objectives for a policy æ. However, requires careful tuning of À to make the policy outperform the heuristic policy; otherwise, the algorithm might prioritize heuristic rewards while neglecting the task objective.
indicates equal contribution. National Taiwan University, Taiwan. Improbable Al Lab, MIT, Cambridge, USA.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Tuning À to balance both objectives is time-consuming We desire an algorithm that finds policy that outperforms the heuristic policy without requiring such tuning for any given heuristic reward function. Classic methods [2, 9-14] modify heuristic rewards to align the augmented objective's 's optimal policy with the one for the task objective (i.e., optimal policy invariance), theoretically ensuring that with infinite data the policy outperforms the heuristic. However, in practice, these modified heuristics often fall short on complex robotic tasks compared to policies trained solely on heuristic objectives, as demonstrated in our study (Section 4.1) and prior work [10].
In this paper, we challenge the prevailing paradigm by questioning whether optimal policy invariance is the appropriate objective to prevent heuristics from limiting RL agent's 's performance in the finite data regime. As optimal policy invariance ensures convergence the optimal policy with infinite data it may not be practical in many weal-world settings. We propose an alternative paradigm that, in every step of training, imposes the constraint of improving task performance beyond policy trained solely on heuristic rewards (i.e., [TT 2 J(TH) )). This condition J(xr) 2 ((xH) ) guarantees that the learned policy æ outperforms the heuristic policy TH, effectively surpassing -usanndesigned heuristics. Additionally, such policy improvements can be verified and achieved using many existing
deep RL algorithms [15-18] in finite data settings.
Therefore, we enforce the policy improvement condition J(T) 2 J(TH)) as a constraint, preventing the policy from exploiting heuristic rewards during training. We propose the following constrained optimization objective:
max J(x) + (T)) subject to J(z) 2 ((TH)
IT
Optimizing this objective each iteration allows learning a policy performing better than or equal to policies trained only on heuristic rewards. It prevents capitalizing on heuristic rewards at the expense of task rewards. Moreover, it enables adaptively balancing both rewards over time instead of using a fixed coefficient A. Our contribution is an add-on to existing deep RL algorithms to improve RL algorithms trained with heuristic rewards. We evaluated our method on robotic locomotion, helicopter, and manipulation tasks using the IsaacGym simulator [19]. The results show that our method led to superior task rewards and higher task-comples completion success rates compared to the policies solely trained with heuristic rewards, even when heuristic rewards are ill-designed.
2 Preliminaries: Reinforcement Learning with Heuristic
Reinforcement Learning (RL): RL a popular paradigm for solving sequential
problems [1] where the problems are modeled as an interaction between an agent and an unknown environment [1]. The agent aims to improve its performance through repeated interactions with the environment. At each round of interaction, the agent starts from the environment's 's initial state and samples the corresponding trajectory. At each timestep within that trajectory, the agent perceives the state s,, takes action a₂ T(-ls:) according to its policy æ, receives a task reward = r(st, at). and transitions to a next state 8141 until reaching terminal states, after which a new trajectory is initialized from so, and the cycle repeats. The agent's goal is to learn a policy π that maximizes the expected return J(T) in trajectory as below:
J(x) F Σ (r)),,
where y denotes a discount factor [1] and F.z [-] denotes taking expectation over the trajectories sampled by æ. In the following, we term J as the true task objective, as it indicates the performance of policy the task.
In many tasks, learning a policy to maximize the true objective / is challenging because rewards may be sparse or delayed. This lack of feedback makes policy optimization difficult for RL algorithms. To address this, practitioners often use a heuristic reward function h with denser reward signals to facilitate optimization, aiming to learn policy that performs better in J. The policy trained to maximize the expected return of heuristic rewards is called the heuristic policy тy. The expected return of heuristic rewards, termed the heuristic objective H, is defined as:
RL with Heuristic:
H(TH) = E
2 th(s,, ax)
(2)
m=0 0

---

where h(s;, az) is the heuristic reward at timestep & for state and action
3 Method: Improving Heuristic Policy via Constrained Optimization
Optimizing both task J and heuristic H objectives jointly could lead better task performance than training solely with J or H, but needs careful tuning on the weight coefficient among both objectives in max, (=) + (T). Without careful tuning, the policy may learn to exploit heuristic rewards H and compromise performance of J. The goal of this paper is to mitigate the requirement of tuning this coefficient to balance them in order to improve task performance.
Problem statement:
Key insight- Leveraging Heuristic with Constraint: We aim to use the heuristic objective H for training only when it improves task performance J and ignore it otherwise. Rather than manually tuning the weight coefficient À to balance both rewards, we introduce a key insight: impose a policy improvement constraint (i.e., J(x) 2 J(TH)) during training. This prevents RL algorithms from exploiting heuristic rewards H at the expense of task rewards J. To achieve this goal, we introduce the following constrained optimization objective:
max ( + (T) subject to (TT) 2 ((H)).
IT
This constrained objective (Equation 3) results in an improved policy over the heuristic policy тH; leading us call this framework Creuristic-Fnhnnce Policy Optimization (HEPO). practical algorithm to optimize this objective is presented in Section 3.1, and its implementation on widely- used RL. algorithm [15] in robotics is detailed in Section 3.2.
3.1 Algorithm: Heuristic-Enhancee Policy Optimization (HEPO)
Finding feasible solutions for the constrained optimization problem in Equation is challenging due the nonlinearity of the objective function J with respect æ. One practical approach is to convert it into the following unconstrained min-max optimization problem using Lagrangian duality:
min max L(TT at,, where L(T,, (x) == ((T) + ((T) + a (J(z)) ((m)))
(4)
a20
where the Lagrangian multiplier this min-max problem by a gradient descent-ascent strategy, alternating between optimizing π and a.
R+, We can optimize the policy π and the multiplier a for
Enhanced policy x: The optimization objective for the policy can be obtained by rearranging Equation as follows:
max (1 + [T)])())+H()))
(5)
where (1+a) + (T)) = E. Σ ((1+ a:)) + h(s),
This represents an unconstrained regular RL objective with the modified reward at cach step as (1+ + a)r(s,, a.) + h(s,,, at), which can be optimized using any off-the-shelf deep RL algorithm. In this modified reward, the task reward r(s,, az) is weighted by the Lagrangian multiplier a, reflecting the potential variation in the task reward's 's importance during training as a evolves. The interaction between the update of the Lagrangian multiplier and the policy will be elaborated upon next.
Lagrangian Multiplier a: The Lagrangian multiplier a is optimized for Equation 4 by stochastic gradient descent, with the gradient defined as:
V.L(T, = J(x)= (mm))-
Notably, V V.L(T, a) is exactly the performance gain of the policy i over the heuristic policy тH on the task objective J. By applying gradient descent with V.L(T, a)), when J(r) > J(TH) and a represents the weight of the task
thus V.L(T, a) > 0. the Lagrangian multiplier a decreases. reward in Equation 5, it indicates that when outperforms nH, the importance of the task reward diminishes because и already achieves superior performance compared to the heuristic policy "h regarding the task objective J. Conversely, when J(T) < J(TH), ), increases, thereby emphasizing the importance of task rewards in optimization. The update procedure for the I Lagrangian multiplier offers an adaptive reconciliation between the heuristic reward h and the task reward r.
3

---

3.2 Implementation
We present a practical approach to optimize the min-max problem in Equation using Proximal Policy Optimization (PPO) [15]. We selected PPO because it is widely used in robotic applications involving heuristic rewards, although our HEPO framework not restricted PPO. The standard PPO implementation involves iterative stochastic gradient descent updates over numerous iterations, alternating between collecting trajectories with policies and updating those policies. We outline the optimization process for each iteration and provide summary of our implementation in Algorithm 1.
Training policies π and πh: Instead of pre-training the heuristic policy тH, which requires additional data and reduces data efficiency, we concurrently train both the enhanced policy и and the heuristic policy æ, allowing them to share data. For each iteration i, we gather trajectories and Th using the enhanced policy x¹ and the heuristic policy THi respectively. Following PPO's implementation, we
compute the advantages A7 (st, at), A, (st, at), A7, (st, qz). and A (st, 0z) for the task reward r and heuristic reward h with respect to n' and THz. We then weight the advantage with the action probability ratio between the new policies being optimized (i.e., xf+1 and TH( !) and the policies collecting the trajectories (i.e., n¹ or Tht). Finally, we optimize the policies at the next iteration i + 1 for the objectives in Equations and 8:
(1++)) (s,, + A%. (st, an)]
n arg max EE E
(ag(s))
(T(e((s))
(1+ at) + A7 (se, a))]
E )
(Enhanced policy)
a # (as ss))
((ce)s))
i+1 TH < arg max E Ts w
Ali (st, an)
+
(8)
T (ax (t))
T (acl(8))
E
T'(ay A (st, as)
(Heuristic policy).
Maximizing the advantages will result in a policy that maximizes the expected return for a chosen reward function, as demonstrated in PPO [15]. This enables to maximize the objective J and H.
We estimate the advantages Ag (Se, dz)) and AR (st, ar) (or A (z)) (z) and A (st, ax)) using the standard PPO implementation with different reward functions. Therefore, we omit the details of the advantage's is clipped surrogate objective in PPO, and leave them in Appendix A.
Although PPO is an on-policy algorithm, the use of off-policy importance ratio correction (i.c., the action probability ratios between two policies) allows us to use states and actions generated by another policy. This enables us to train π using data from тH and vice versa. Both policies ा and лH are trained using the same data but with different reward functions. Note that collecting trajectories from both policies does not require more data than the standard PPO implementation. We collect half the trajectories with each policy, π and TH, for total of B trajectories (see Algorithm 1). Then, we update both π and ни using all B trajectories.
Optimizing the Lagrangian multiplier at To update the Lagrangian multiplier a, we need to compute the gradient in Equation 6, which corresponds to the performance gain of the enhanced policy π over the heuristic policy тн on the task objective J. Utilizing the performance difference lemma [20, 16], we relate this improvement the expected advantages over trajectories sampled by the enhanced policy π as (TT) - ( Eg [A* r(st, at)]. However, this approach only utilizes half of the trajectories at each iteration since it exclusively relies on trajectories from the enhanced policy л. To leverage trajectories from both policies, we also consider the performance gain in the reverse direction as - (J(mH) J(T)) -Ex,, (A"r(s a:)]. Consequently, we can estimate the
gradient of a using trajectories from both policies, as illustrated below:
VaL(n,a) = ([T) ((TH) E- [A, (z).
(9)
( - (()) E. (sc,
(10)
At each iteration i, we estimate the gradient of a using the advantage A (st, a.) and A (*t, on the trajectories sampled from both x' and ThH. and update with stochastic gradient descent as follows:
1 t- 2
(Erwat
[A an) - EnNNY (A5"(er,a)) [A (st,
(11)
4

---

where E R is the step size. The expected advantage in Equation 11 are estimated using the generalized advantage estimator (GAE) [21].
Algorithm 1 Hcuristic- EnttristtieEnhaneedd Policy Optimization (HEPO)
1: Input: Number of trajectories per iteration B
2: Initialize the enhanced policy n², the heuristic policy The and the Lagrangian multiplier a i denotes iteration index
3: for i. 0... do
4:
Rollout B/ trajectories T by x²
5: Rollout B/ trajectories ти by THH
6:
m'+1 <- Train the policy n² for optimizing Equation using both and
7: "H
-
Train the policy TH for optimizing Equation 8 using both and ти
8:
a
Update the Lagrangian multiplier by gradient descent (Equation 9) using and ти
9:
end for
3.3 Connection to Extrinsic-Intrinscc Policy Optimization [22]
Closely related to our HEPO framework, Chen et al. [22] proposes Extrinsic- Intrinssi Policy Opti- mization (EIPO), which trains policy to maximize both task rewards and exploration bonuses [23] subject to the constraint that the learned policy и must outperform the task policy æy trained solely task rewards. HEPO and EIPO differ in their objective functions and implementation of the constrained optimization problem. Additional information can be found the Appendix, cover- ing the objective formulation (Appendix A.1), implementation tricks (Appendix A.2), and detailed pseudocode (Appendix A.3).
Exploration bonuses [23] can be viewed as heuristic rewards. The main difference between HEPO and EIPO's optimization objectives lies in constraint design. Both frameworks require the Iearned policy to outperform reference policy nee (i.c., (xx > (TTet) but different reference policy. EIPO uses the task policy ии as the reference policy nef because they aim for asymptotic optimality in task rewards. If the constraint is satisfied with Ty being the optimal policy for task rewards, the learned policy æ will also be optimal for task rewards. contrast, HEPO uses the heuristic policy TH trained solely on heuristic rewards since HEPO aims to improve upon it.
HEPO simplifies the implementation. Both HEPO and EIPO train two policies with shared data, but EIPO alternates the policy used for trajectory collection each iteration and has complex switching rule, which introduces more hyperparameters. HEPO collects trajectories using both policies together at each iteration, simplifying implementation and avoiding extra hyperparameters.
4 Experiments
We evaluate whether HEPO enhances the performance of RL algorithms in maximizing task re- wards while training with heuristic rewards. We conduct experiments on 9 tasks from IsaacGym (ISAAC) [19] and 20 tasks from the Bidexterous Manipulation (BI-DEX) benchmark [24]. These tasks rely on heavily engineered reward functions for training RL algorithms. Each task has task reward function r that defines the task objective to be maximized, and a heuristic reward function h that defines the heuristic objective H. provided in the benchmarks to facilitate the optimization of task objectives J. We implement HEPO based on PPO [15) and compare with the following baselines:
H-only (heuristic only): This is the standard PPO bascline provided in ISAAC. The policy is trained solcly using the heuristic reward: max, (T)) The heuristic reward functions ISAAC and BI-DEX designed to help RL algorithms maximize the task objective J. This baseline is crucial to determine if an algorithm can surpass a policy trained with highly engineered heuristic rewards.
J-only (task only): The policy is trained using only the task reward: max, JTT). This baseline demonstrates the performance achievable without heuristies. Ideally, algorithms that incorporate heuristics should outperform this baseline.
5

---

J+H (mixture of task and heuristic): The policy is trained using a mixture of task and heuristic rewards: max, J=x + (x), with À balancing the two rewards. As [22] shows, proper tuning of À can enhance task performance by balancing both training objectives.
Potential-based Reward Shaping (PBRS) [2]: The policy is trained to maximize F =x 7'1, + yhy+1 - h where rt and by are the task and heuristic rewards at timestep I. PBRS guarantees that the optimal policy is invariant to the task reward function. We include it as a baseline to examine if these theoretical guarantees hold in practice.
The policy is trained to maximize E, o + (1- 3.) where 3 is a coefficient updated at each iteration to balance heuristic rewards during different training stages. The scheduling mechanism is detailed in [10] and our source code provided in the Supplementary Material.
HuRL [10]:
EIPO [22]] The policy is trained using the constrained objective: max, JTT) + H(T) s.t. J(T) 2 J(mY), where πy is the policy trained with task rewards only. EIPO is similar to HEPO but differs in formulation and implementation, detailed in Section 4.3.
Each method is trained for 5 random seeds and implemented based on the open-sourced implementa- tion [25], where the detailed training hyperparameters can be found in Appendix A.4.
Metrics:
Based on the task success criteria in ISAAC and BI-DEX, we consider two types of task reward functions r. (i) Progressing (for locomotion helicopter robots) and (ii) Goal-reaching (for manipulation). In progressing tasks, robots aim to maximize their traveling distance or velocity from an initial point goal-reaching tasks, robots aim to complete assigned goals by reaching specific goal states. Here, task rewards are binary, with a value of indicating successful attainment of the goal and 0 otherwise. Detailed descriptions of our task objectives and total reward definitions are provided in Appendix C.
destination. Thus, movement progress is defined as the task reward. In
4.1 Benchmark results
Setup: We aim to determine HEPO achieves higher task returns and improves upon the policy trained with only heuristic rewards (H-only) in the majority of tasks. In this experiment, we use the heuristic reward functions from the ISAAC and BI-DEX benchmarks. To measure performance improvement over the heuristic policy, we normalize the return of each algorithm X using the formula (Jx - Jrandom)/ (JJHooonly - randdm)) where Jx. JH--nly. and Jrandom denote the task returns of algorithm X, the heuristic policy, and the random policy, respectively. In Figure 1, present the interquartile mean (IQM) of the normalized return and the probability of improvement for each method across 29 tasks, following [26]. IQM, also known the 25% trimmed mean, is robust estimate against outliers. It discards the bottom and top 25% of runs and calculates the mean score of the remaining 50%. The probability of improvement measures whether an algorithm performs better than another, regardless of the margin of improvement. Both approaches prevent outliers from dominating the performance estimate.
Results: The results in Figure indicate that policies trained with task rewards only (J-only) generally perform worse than those trained with heuristics, both in terms of IQM of normalized return and probability of improvement. PBRS does not improve upon J-only, demonstrating that the optimal policy invariance guarantee rarely holds in practice. Both EIPO and HuRL outperform J-only but do not surpass H-only, demonstrating that neither approach can improve upon the heuristic policy. Policies trained with both task and heuristic rewards (J+H) perform slightly worse than those trained with heuristics only (H-only), possibly because the weight coefficient balancing both rewards is too task-sensitive to work across all tasks. HEPO, however, outperforms all other methods in both IQM of normalized returns and shows a probability of improvement over the heuristic policy greater than 50%, indicating statistically significant improvements as suggested by Agarwal et al. [26]. Complete learning curves are presented in the Appendix B. Additional results on various benchmarks and RI. algorithms are provided in Appendix B.3, demonstrating that HEPO is effective in hard-exploraaiio tasks using exploration bonuses [23] and with RL algorithms beyond PPO.
4.2 Can HEPO be robust to reward functions designed in the wild?
Setup: We envision to develop an RL. algorithm that can effectively utilize heuristic reward functions, thereby reducing the time costs associated with reward design. We simulate reward design scenarios

---

J-only
H-only
J+H
HuRL
PBRS
EIPO
НЕРO (ours)
1.2
1:13
0.17
0.98
Normalized returr
0.50
0.84
Algoritim X
0.8
0:70
0042
0.30
0.6
-0.13
0.4
0.27
0.29
0.17
0.2
0.62
0.02
0.0
0.0
0.1
0.2
0.3
0.4
0.5
0.6
Algorithm
P(X H-only)
(a) IQM over ISAAC BI-DEX
(b) Probability Improvement over ISAAC + BI-DEX
Figure 1: (a) The vertical axis represents the interquartile mean (IQM) [26] of normalized task return across 29 tasks. HEPO outperforms the policy trained solely with a heavily engineered heuristic reward function (H-only) and other methods, demonstrating that HEPO makes better use of heuristic rewards for learning. (b) The horizontal axis shows the probability that algorithm X outperforms the policy trained solely with heuristic rewards (H-only). HEPO achieves 62% probability of improvement over the heuristic policy on average, with the lower bound of the confidence interval above 50%, indicating statistically significant improvements over the heuristic policy.
and evaluate the algorithm's performance when trained with heuristic reward functions created under real-world conditions. Unlike the highly engineered reward functions in ISAAC, participants were asked to design their heuristic reward functions within a short time frame. This approach assesses the algorithm's effectiveness when trained with less refined heuristic reward functions. Participants were asked to iterate on their reward design by writing the reward function, training a policy with a given RL algorithm, reviewing the videos and learning curves, refining the reward, and repeating the process. We selected the FrankaCabinet task for this study because its original heuristic reward function is heavily engineered and consisting of many terms. The task of Franka Cabinet is training robot arm to open cabinet. We recruited twelve graduate students with varying levels of proficiency in machine learning and roboties and divided them into two groups. One group used HEPO to iterate on heuristic rewards, while the other group used PPO. This approach ensures that the designed heuristic reward functions participant was instructed to edit the heuristic reward function to help RL algorithms maximize the task return. We used the same task reward metric in Section 4.1. We then used the final versions of their heuristic reward functions to train HEPO and PPO, and reported the normalized return of the learned policies. Note that we adhered to the normalization scheme outlined Section 4. based on the performance of the policy trained with the original heuristic reward function. This approach allows us to observe any performance drop when training policies with less engineered heuristic reward functions.
not specialized for one algorithm and ineffective for another. Each
Quantitative results: Figure shows that НEРO achieves significantly higher (lower con- fidence bound above the baselines' upper confi- dence bound) average normalized returns than PPO trained only heuristic rewards (PPO (H- only)) in 9 out of 12 heuristic reward functions. Additionally, Table indicates that across heuristic functions, HEPO achieves higher in- terquartile mean (IQM) of normalized returns and has a statistically significant probability of outperforming PPO (H-only) with lower confi- dence bound greater than 0.5. This suggests that even when trained with poorly designed heuris- tic reward functions. HEPO performs better than PPO (H-only). Notably, PPO (H-only) that is trained with H2, H5, and H6 achieves normal-
Algo. X
IQM
P(X > H-only)
H-only
0.44 (0.37, 0.52)
0,50 (0.49, 0.51)
HuRL
0.00 (0.00, 0.00)
0.33 (0.32, 0.33)
PBRS
0.00 (0.00, 0.00)
0.22 (0.21, 0.22)
НЕРO
0.94 (0.85, 1.03)
0.73 (0.73, 0.74)
Table 1: IQM and probability of improvement (PI) over H-only (P(X > H-only)) with 95% confidence intervals across 12 heuristic reward functions (Sec- tion 4.2). H-only uses PPO with heuristic rewards. HEРO achieves higher normalized returns and statistically significant PI greater than 0.5, indicat- ing it significantly outperforms H-only.
7

---

:1:60
2.12
1:19
2,13
1.68
2.01
Normalize
0.96
0.53
0.66
0.36
0.01
0.09
0.00
0.00
0.00.00000.00000 0.07
0.00.0.00
0.15
H12
2.11
2.09
1.59
1.62
Normaized
0.78
0.00
0.00
0.00
1.000.000.000
0,00
0.00 www.000..000000
Figure 2: Normalized task return of PPO (H-only) and HEPO that trained with heuristic reward function H1 to H12 designed by human subjects in the real world reward design condition. HEPO achieves higher task return than PPO (H-only) in 9 out of 12 tasks. This shows HEPO is robust to possibly ill-designed heuristic reward functions and can leverage them to improve performance.
ized returns below 1, while HEPO achieves returns greater than or close to 1. Since returns are normalized using the performance of the PPO policy trained with the well-designed heuristic reward function in ISAAC, return below 1.0 indicates performance drop for PPO (H-only) when using potentially ill-designed heuristic rewards. contrast, HEPO can improve upon policies trained with carefully engineered heuristic reward functions, even when trained with possibly ill-designed heuristic reward functions.
We aim to understand why PPO's performance declines when trained with heuristic reward functions H2, H5, and H6. These functions are similar to the original heuristic reward in FrankuCabinet, but with different weights for each term. For example, in H5, the weight of action penalty
Qualitative observation:
whereas in the original heuristic reward function is 7.5. This suggests that HEPO might handle poorly scaled heuristic reward terms better than PPO, which is sensitive to these weights. The heuristic reward functions H2 and H9 had an incorrect sign for the distance component, which caused the policy to be rewarded for moving away from the cabinet instead of toward it, making the learning task more challenging.
4.3 Ablation Studies
Expanding on the discussion of relation to relevant work EIPO [22] in Section 3.3, our goal is to examine the implementation choices of HEPO and illustrate the efficacy of each modification in this section. HEPO differs from EIPO primarily two aspects: (I) the selection
reference policy Tie in the constraint J (T) 2 JiTee),, and (2) the strategy for utilizing policies to gather trajectories. Both studies are conducted on standard locomotion and manipulation tasks, such as Ant, FrankaCabinee and AllegroHand. In addition, we provide further studies on the sensitivity to hyperparameters in Appendix B.2.
Selection of reference policy in constraint: HEPO and EIPO both enforce performance improve- ment constraint J(r) Z -(Tet) during training. HEPO uses heuristic policy и as the reference (The H-only), while EIPO uses task-only policy (л J-only). However, relying solely on policies trained with task rewards as references may not suffice for complex robotie tasks, as they often perform much worse than those trained with heuristic rewards. We compared the performance of HEPO with different reference policies in Figure 3a. The result shows that setting Tref = J-only (EIPO) improves the performance over the task-only policy J-only while notably degrading perfor- mance, sometimes even worse than H- H-only, suggesting it's insufficient for surpassing the heuristic policy.
Strategy of collecting trajectories: We use both the enhanced policy π and the heuristic policy тh simultaneously to sample half of the environment's trajectories (referred to as Joint). Conversely, EIPO switches between æ and ли using a specified mechanism, where only one selected policy samples trajectories for updating both æ and ли within the same episode (referred to as Alternating). This study compares the performance of these two trajectory rollout methods. We modify HEPO to

---

J-only
H-oaly
HEPO(Ra --only)
HE H-only)
Ant
PrankaCabinet
AllegroHand
Normalized return
2.0
1.0
1.5
0.5
0.5
0.0
2000
1000
1500
5000
10000
15000
Iteration
Iteration
Iteration
(a) Comparison of reference policy Tea choice in HEPO's constraint J(r) 2 ((Ter)
J-only
H-only
HEPOVAllemssing)
HEPO(join))
Ant
FrankaCabinet
AllegroHand
Normalized return
1.00
2.0
2.0
0.75
1.5
1.0
0.25
0.5
0.3
0.00
0.0
0.0
0
1000
2000
500
1000
1500
0
5000
10000
15000
Iteration
Iteration
Iteration
(b) Comparison of trajectory collecting strategies
Figure 3: (a) We show that using the policies trained with heuristic rewards (J-only) is better than using the policies trained with task rewards (J-only) when training HEPO. (b) HEPO(Joint) that collects trajectories using both policies leads to better performance than H[EPO(Aleernating) that alternates between two policies to collect trajectories. See Section 4.3 for details
gather trajectories using the Alternating strategy and present the results Figure 3b. The findings indicate that Alternating results in a performance drop during mid-training and fails to match the performance of HEPO(Joint). We hypothesize that this occurs because the batch of trajectories collected solely by one policy deviates significantly from those that another policy can generate (i.e., high off-policy error), leading to less effective PPO policy updates. contrast, Joint samples trajectories using both policies, preventing the collected trajectories from deviating too much from each other.
Related Works
Reward shaping: Reward shaping has been a significant area, including potential baase reward shaping (PBRS) [2, 27], bilevel optimization approaches [28-30] reward model learning, and heuristic-guided methods (HuRL) [10] that schedule heuristic rewards. Our method differs as is policy optimization method agnostic the heuristic reward function and can be applied those shaped or learned rewards.
Constrained policy optimization: Recent work like Extrinsic-Intrings Policy Optimization (EIPO)
[22] proposes constrained optimization by tuning exploration bonuses to prevent exploiting them at the cost of task rewards. Extensions [31] balance imitating a teacher model and reward maximization. Our work differs in balancing human-designed heuristic rewards and task rewards, improving upon policies trained with engineered heuristic rewards. We also propose implementation enhancements over EIPO [22] (Section 4.3).
6 Discussion & Limitation
this paper, we showed that HEPO is robust to the possibly ill- designed heuristic reward function in Section 4.2 and also exhibit high-probability improvement over PPO when training with heavily engineered heuristic rewards in robotic tasks in Section 4.1. Moving forward, when users need to integrate heuristic reward functions into RL algorithms. HEPO can potentially be a useful tool to reduce users' time on designing rewards since it can improve performance even with under-engineered heuristic rewards.
НEРO for RL practitioners:
Limitations: While HEPO shows high-probabiling performance improvement over heuristic policies trained with well well-designed heuristic reward, one limitation is that HEPO does not have guarantee

---

to converge to the optimal policy theoretically. One future work can be incorporating the insight in recent theoretical advances on reward engineering [3] make a convergence guarantee.
Acknowledgements
We thank members of the Improbable AI Lab for helpful discussions and feedback. We are grateful to MIT Supercloud and the Lincoln Laboratory Supercomputing Center for providing HPC resources. This research was supported in part by Hyundai Motor Company, Quanta Computer Inc., an AWS MLRA research grant, ARO MURI under Grant Number W91 10722333--0 DARPA Machine Common Sense Program, ARO MURI under Grant Number The views and conclusions contained in this document are those of the authors and should not be interpreted representing the official policies, either expressed or implied, of the Army Research Office or the United States Air Force or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes, notwithstanding any copyright notation herein.
and ONR MURI
under Grant Number
1-0001422-1-211-1
Author Contributions
Chi-Chang Lee:
Co-led the project, led the implementation of the proposed algorithms and baselines, and conducted the experiments.
Zhang-Wei Hong: Co-led the project, led the writing of the paper, sealed up the experiment infrastructure, and conducted the experiments.
Pulkit Agrawal: Played a key role in overseeing the project, editing the manuscript, and the presentation of the work.
References
[1] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. 2018.
[2] Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transfor- mations: Theory and application to reward shaping. In Ieml, volume 99, pages 278-287, 1999.
[3] Abhishek Gupta, Aldo Pacchiano, Yuexiang Zhai, Sham Kakade, and Sergey Levine. Unpacking reward shaping: Understanding the benefits of reward engineering on sample complexity. Advances Neural Information Processing Systems, 35:15281-11221 2022.
[4] Tao Chen, Jie Xu, and Pulkit Agrawal. system for general in-hand object re-orientation. Conference on Robot Learning, 2021.
[5] Gabriel B Margolis, Ge Yang, Kartik Paigwar, Tao Chen, and Pulkit Agrawal. Rapid locomotion via reinforcement learning. Robotics: Science and Systems, 2022.
[6] Gabriel B Margolis and Pulkit Agrawal. Walk these ways: Tuning robot control for general- ization with multiplicity of behavior. In Conference on Robot Learning, pages 22-31. PMLR, 2023.
[7] Ashish Kumar, Zipeng Fu, Deepak Pathak, and Jitendra Malik. RMA: rapid motor adaptation for legged robots. In Robotics: Science and Systems XVII, Virtual Event, July 12-16, 2021, 2021.
[8] Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning quadrupedal locomotion over challenging terrain. Science robotics, x(47)xeabc5986, 2020.
[9] Sam Michael Devlin and Daniel Kudenko. Dynamic potential- reward shaping. //th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2012), pages 433 440. IFAAMAS, 2012.
[10] Ching-An Cheng, Andrey Kolobov, and Adith Swaminathan. Heuristic-guided reinforcement learning, Advances in Neural Information Processing Systems, 18:13550-13560 2021.
10

---

[11] Sam Devlin, Logan Yliniemi, Daniel Kudenko, and Kagan Tumer. Potential based difference rewards for multiagent reinforcement learning. In Proceedings of the 2014 international conference on Autonomous agents and mulli-agent systems, pages 165-172, 2014.
[12] Adam Eck, Leen- Kiat Soh, Sam Devlin, and Daniel Kudenko. Potential- based reward shaping for finite horizon online pomdp planning. Autonomous Agents and Mulli-Agent Systems, 30: 403 445, 2016.
[13] Babak Badnava, Mona Esmaeili, Nasser Mozayani, and Payman Zarkesh-Ha. new potential- based reward shaping for reinforcement learning agent. In 2023 IEEE 13th Annual Compuling and Communication Workshop and Conference (CCWC), pages 01-06. IEEE, 2023.
[14] Grant C Forbes and David L Roberts. Potential-based reward shaping for intrinsic motivation (student abstract). In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 23488-221 23388, 2024.
[15] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint
2017.
[16] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Proceedings of the 32nd International Conference on Machine Learning (ICML-15), pages 1889-1897, 2015.
[17] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv:180..01290, 2018.
[18] Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, Stig Pe- tersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 2015.
[19] Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, and Gavriel State. Isaac gym: High performance gpu-based physics simulation for robot learning, 2021.
[20] Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In In Proc. 19th International Conference on Machine Learning. Citeseer, 2002.
[21] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. International Confer- ence of Representation Learning, 2015.
[22] Eric Chen, Zhang-Wat Wei Hong*, Joni Pajarinen, and Pulkit ( equal contribution) Agrawal. Redeeming intrinsic rewards via constrained optimization. Advances Neural Information Processing Systems, 35:4996-5008 2022.
[23] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations, 2019. URL
attps://openrewiew sostt/forumidddiH11..
[24] Yuanpei Chen, Yaodong Yang, Tianhao Wu, Shengjie Wang. Xidong Feng, Jiechuan Jiang, Zongqing Lu, Stephen Marcus McAleer, Hao Dong, and Song-Chun Zhu. Towards human-level bimanual dexterous manipulation with reinforcement learning. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
[25) Denys Makoviichuk and Viktor Makoviychuk. rl-games: A high-performance framework for reinforcement learning.
games, May 2021.
[26] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Belle- mare. Deep reinforcement learning at the edge of the statistical precipice. Advances Neural Information Processing Systems, 34, 2021.

---

[27] Sam Devlin and Daniel Kudenko. Dynamic potential-basee reward shaping. In Adaptive Agents and Multi Multi-Agent Systems, 2012.
[28] Yujing Hu, Weixun Wang, Hangtian Jia, Yixiang Wang, Yingfeng Chen, Jianye Hao, Feng Wu, and Changjie Fan. Learning to utilize shaping rewards: A new approach of reward shaping. Advances in Neural Information Processing Systems, 53:15951-15941, 2020.
[29] Dhawal Gupta, Yash Chandak, Scott Jordan, Philip S Thomas, and Brumo C da Silva. Behavior alignment via reward function optimization. Advances in Neural Information Processing Systems, 36, 2024.
[30] Zeyu Zheng, Junhyuk Oh, and Satinder Singh. On learning intrinsic rewards for policy gradient methods. Advances in Neural Information Processing Systems, 31, 2018.
[31] Idan Shenfeld, Zhang- Wei Hong, Aviv Tamar, and Pulkit Agrawal. TGRL: An algorithm for teacher guided reinforcement learning. In Proceedings of the 40th International Conference on Machine Learning. 2023.
[32] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference Learning Representations, 2014.
12

---

A Implementation Details
A.1
Full Derivation
We will detailedly describe the update of the enhanced policy (a in Equation 7) and the heuristic policy (Th in Equation 8) at each iteration.
A.1.1 Notations
Ei 0 (r)ss,a)))ss = st (T)()) = E(s
(st) = F(erae)
E 7'1(8 (c) 8t)
A7 at) = r(s;,)) (5(+1) (st)
(7() = h(s,, a+) + Vi (s++1) Vi
BнEPо: the buffer to store samples collected by x²
Bhi the buffer store samples collected m|y-
A.1.2 Enhanced Policy Update
Given a value, -(1 derived using the arguments of the maxima in Equation 4, which can be re-written as follows:
n++1_ arg max [(T) + H(T)
{1 (1+ +[)))
arg max
+ (1 (() + H())) - 2 (1 + + +(('))
= arg max
(12)
En (1 + a)A, a.) + A (st, ai)
arg max
Fr
(1 + a)
(st, at) A (s1,
arg max
E
U (st, ax)
+ デ Em U (st, (st, }
2
arg max
Ex
U (st.
ax)
+ Fr U (st, (s), }
where Us! and & are defined as follows:
Us (st, 0z) = (1 + a) (st, ae) + A (st at) U (st, 04) = (1 + a)4," (st, a.) + A (st, at)
(13)
To efficiently achieve the update process in Equation 12, we aim to utilize previously collected trajectories for optimization, outlined in Equation 7. Here, refer to [15], using those previously collected trajectories to form lower bound surrogate objectives, J (x)) and j (I)) as alternatives
13

---

of F (st, 4:) and Fr Uu (st, at)] ]
derive r'+1:
8
min x((r(st) U (s,,
(T)
Σ
BuEPol
(((a(18))
(aalss)) T (1)))))))
clip
e, 1 +
U (st,
(14)
min x (e) U (s),
jo (T)
(T) =
Σ
|B
Tig (1sl88))
EB
π ((a))s)) Tir (ax(s)))
U Ua (s)
clip
- e, c,l+e
where E (s,, [x)] HEPo ( and E, [U (st, az)l 2 HH (T) always hold; e E [0, 1] denotes threshold. Intuitively, this clipped objective (Eq. 14) penalizes the policy that behaves differently from xt' or Tyy because overly large or small the action probability ratios between two policies are clipped.
A.1.3 Heuristic Policy лH Update
i+1 H
is derived using the arguments of the maxima of (x), which can be re-written as follows:
++1 H
arg max [m()))
1 ([((T) [1((TT)
arg max n
11 (m(T1)
(15)
1 Eg AA (or, a+)
+ マ و و A (se. an)
1
arg max
arg max
{
[ AA7 (Se, at)
+ lr (st,
( an)}
Similarly, we again rely on the approximation from [15] derive lower bound surrogate objective for both [=[A (8e,ar) and EE, [AA (st, az)] as follows:
7" min ((al8e))
A1 (T)
Σ (St,0e) BuErO
A (sc,
BHEPO|
x (arl(r)
(16)
π (arl35))
A (st
clip
1 - c, c,1 e
T' (Taaa (t))
f (TT) = |Bal
Σ
min x(a)(s))
8i (st,
(Ty (acls))
(e.,or)EBn
1=0
(17)
(as((s)) T (arls))
clip
e, 1,
A (sc, 1, +
where E, (st, (s),a))) 2 AT (T) and E,, [A (Nt, az)l 2 Am (T) always hold. Different from vanilla heuristic training, instead of solely collecting trajectories from Th. we collect trajectories from both x and गH enrich sample efficiency.
A.2
Implementation Tricks
A.2.1 Sample Sharing for Value Function Update
In practice, obtaining real value functions for training is not feasible. We estimate the value function using collected trajectories, but this approach tends to fail because the value function becomes biased toward the policy responsible for trajectory collection.
14

---

To prevent error information from the estimated value function interfering with the training procedure, we share the trajectory samples within the BHEPO and B₁ buffers to update our value functions:
rr(e, a1) + /V, ($141) ((s))?
V
f- arg min
Σ
(18)
BHEPO + |B
(1))))) Bu
[h(s,, a1) + +V (8(+1) - ((s)))
Vx 1-1
V/2
<- arg min?
Σ
(19)
BBIEO + |BH
(sch(ar E B
V
11
arg aigmin min
Σ
r(st, + vV²T ($441) ((e))1
(20)
| + +BHH
(e)r(or,a)))011) BBH
V
i-1
arg min
Σ
|h(s,, a.) +,V0"' ($c+1) - (ss))
(21)
[BEEPo| + BHH
(s..h(91re)):444)) BumoUBy
A.2.2 Smoothing Lagrangian Multiplier a Update
The Lagrangian multiplier a determines the desired constraint information during training. However, in practice the gradient tends to become explosive. To stabilize the update procedure, accumulate previous gradients and adopt the Adam optimizer [32] as follows:
g(a) + med
Σ
Ag (s2, at)
Σ
A (&t, an) }
BIEEO
BB
(E),
(B
(22)
a - AdamOpt (
where K is the number of previous K advantage expectation records that we take into account. To smooth the current a gradient for each update, we calculate median the previous records. In experiments, assigned K value of 8.
A.3 Overall Workflow
Algorithm 2 Detailed Heuristic-Enhanced Policy Optimization (HEPO)
1: Initialize policies (T xHH) and values (V,',, V. V.Th vi
2: for -1... do
denotes iteration index
3: # ROLLOUT STAGE
4: Collect trajectory buffers (BIEPO, Bu) using (It', TH))
5: Compute (A7' (s), A7, (st, at)) via GAE with (VT' V. V(s, ar) BhEPO
6: Compute (A7" (st, A," (st, a:) via GAE with (V b) V(s ar) Bи
7:
Compute (J (JHEPO=
j HHEPO) based on Equation 14
8:
Compute (ix*, A*") based on Equation 16
9:
10:
UPDATE STAGE
11:
x
arg max,
JHEro (TT) juij
:+1
[ (T) i (T)
arg max,
Update (V.*', Va"" vr Vrn) ","",,, based on Equation 18
14: Update based Equation 22
15:
end for
15

---

A.4 Training details
Following the PPO framework [15], our experiments are based on continuous action actor-critic algorithm implemented in rl_games [25], using Generalized Advantage Estimation (GAE) [21] to compute advantages for policy optimization. For PPO, we employed the same policy network and value network architecture, and the same hyperparameters used in IsaacGymEnvs [19]. We also include our source code in the supplementary material. In HEPO, we use two policies for optimization, with each policy maintaining the same model configurations as those used in PPO. Below we introduce HEPO-specific hyperparameters used in our experiments Section 4.1. The hyperparameters for updating the Lagrangian multiplier in HEPO are listed as follows:
Table 2: HEPO Hyperparameters
Name
Value
Initial
0.0
Step size y7 of a (learning rate)
0.01
Clipping range of da (atayta)
1.0
Range of the a value
00,
For baselines, search for hyperparamters À for J H in Ant, FrankaCabinet, and AllegroHand, as shown in Section B.2. We set À = 1 for all the experiments because it shows better performance on the three chosen environments. For HuRL [10], follow the scheduling setting provided in their paper.
16

---

B Supplementary Experimental Results
B.1 All Learning Curves on the task objective J
We present all the learning curves in Figure 4.
Joonly
H-only
J+H
HuRI
EIPO
HEPO
quadespter
3020
5030
-200
-750
520
2530
-1000
Itecation
Iberation
Itrration
Allegralbend
PranlosCatine
7530
0.4
5000
2000
0.00
0.3
2053
4005 MAO
2003
10302
3200 [beration
10000
13000
320
1002
1300
2000
2010
5000
9002
5030
Dlecstion
Iteration
Studowéland
Shadmmttnzipp
ShadowtiinnUpiaaaDoon
0.38
0.30
0.3
0.00
0.34
0.2
0:32
0.00
6.32
0.00
5002
10000
15303
2000
4200
5920
0262
10000
2302
4030
2000
1000
1000
2200
4000 6000 Dersdon
50200
Deeratinn
Itarationn
Htratton
Shadon(CandlaattaCaa
6.0
2.4
02
0.3
0.5
0.0
08
2003
30363
1920
L5000
1000
1000
Tteestion
Theration
Iteration
Shalon
0.5
133
1.2
2003
4002
1000
2000
10303
2000
4100
5020
0003
10000
2303
6000
2000
5000
1000
2:00
4100
6000
2002
50200
Shacowiltandiattee
0.100
Stadonclandi stUncersem
Stattottinndtwwe
0.2
0:
02
Drestton
tharation
Seerttoon
baddPeelllekkk
0.20032
0.20075
0.3004
0.5
0.5
04
6.000
04
02
0.10035
0.3000
2002
4002
6000
2003
10383
2000
5020
9302
LE000
2000
10002
2300
9000 50200
lieeation
Rerution
Hirrtioo
Ixcutua
StadootiandSwttch
0.000
0.535
3.5
0.0
0.10
8.
0.000
2003 4002
600
1000
10300
2000
4100
5020
0003
LE0CO
02
04
0.6
2.8
100
08
02
044
2.6
02
1.0
Figure 4: All learning curves in Section 4.1
B.2 Sensitivity to hyperparameters
In this section, we aim to verify HEPO's sensitivity to two main types of hyperparameters: (1) the weight of the heuristic reward in optimization (denoted X) and (2) the learning rate for updating
17

---

A
HEPO
]+I
0.01
0.1
Heuristic normalizm lzzed relarn
Ant
CrankaCabinet
AllegroHand
2.5
2.0
1.00
2.0
1.5
0.50
1.0
0.25
0.5
0.5
0.00
0.0
0.0
1000
2000
1000
1500
5000
10000
15000
Iteration
Iteration
Iteration
Figure 5: Sensitivity X
Lagrangiaa multiplier stop size (r)
0.01
0.0001
Heuristic normalized return
Ant
FrankaCabinet
AllegroHand
1.00
2.0
0.75
1.5
0.50
1.0
0.25
0.5
0.5
0.0
1000
2000
5000
1000
1500
5000
10000
15000
Tteration
Iteration
Iteration
Figure 6: Sensitivity alpha learning rate
Similar to Section 4.3, we conducted our experiments on the Ant, FrankaCabinet, and AllegroHand tasks.
B.2.1 Sensitivity to the Value
Both HEPO and J+H can set scaling coefficient to weight the heuristic reward in optimization, such that the objective becomes ((T) + WV (T)) This scaling coefficient can be used to balance both objectives. In this study, compare HEPO and J+H their performance sensitivity to the choice of X, exhaustively training both HEPO and J+H with varying values. Note that though the formulation of HEPO does not depend on X, reward in НEРO. In our experiments, we did not optimize À for HEPO but for the baselines trained with both rewards (J+H). In Figure 5, found that J+H is sensitive to À in all selected tasks, while HEPO performs well across a wide range of values. This indicates that HEPO is robust to the choice of X.
still set À coefficient scale the heuristic
B.2.2 Sensitivity to the Learning Rate for Updating a
HEPO's robustness relies on the a update, as it reflects the necessary constraint information at each iteration. Similar Appendix B.2.1, setting different initial values of a is equivalent to using different À values for our estimation, since both can be rewritten the ratio between H(x) and J(T), Both of these parameters indicate the necessary constraint information for conducting multi-objective optimization. In this study, we aim verify whether HEPO yield comparable improvement gaps under different initial values of a, thus providing a more robust optimization procedure.
As shown in Figure 6, we observe that HEPO is also robust to the choice of a's initial values, similar the results in 5. 5.
B.3 Additional results
Generality of HEPO: We also demonstrated HEPO can be implemented over the other RL algorithms
in addition to PPO. We integrated HEPO into HuRL's SAC codebase. Despite HuRL using tuned hyperparameters as reported in its paper [10], HEPO outperformed SAC and matched HuRL on the
18

---

most challenging task in Figure 7b using the same hyperparameters from Section 4 of our manuscript, showing the generality of HEPO on different RL algorithms.
Better than EIPO on the most challenging task: Comparing HEPO and EIPO on the most challenging task, Montezuma's Revenge, reported in EIPO's paper [22] using RND exploration bonuses [23] (as used in the EIPO paper), Figure 7a shows that HEPO performs better than EIPO. Also, HEPO matched PPO trained with RND bonuses at convergence (2 billion frames) reported in [23] using only 20% of the training data, demonstrating drastically improved sample efficiency.
How quality of heuristic reward functions impact HEPO? We believe that Figure 2 reveals the relationship between HEPO's 's performance and the quality of the heuristic reward. The policy trained with only heuristic rewards (H-only) represents both the asymptotic performance of in HEPO and the quality of the heuristic itself. We found a positive correlation (Pearson coefficient of 0.9) between the average performances of H-only and HEPO in Figure 2 results, suggesting that better heuristics lead to improved HEPO performance. Figure 7c provides more details.
Puffermenee of hgpo 1
ConTHEPO, H-only) 0.9
P20
EIPO
HEPO
Reachet v2
2030
AeeDgg tails vertur
Averng task
5030
-300
0.0
0.3
1.0
1.5
2.0
2030
Porformance of only
6.0
0.5
3.0
2.0
2.5
3.0
-820
54
350
200
(c) Correlation between the performance HEPO and H- only.
(a) EIPO v.s. HEPO on PPO
(b) HuRL v.s. HEPO on SAC
Figure 7: (a) Comparison of HEPO and EIPO [22] on the most challenging Atari task, Montezuma's Revenge, shown in the EIPO paper [22]. Both are implemented on top of EIPO's PPO codebase using RND exploration bonuses [23] as heuristic rewards H,, as suggested in [22]. HEPO outperforms
EIPO, achieving the performance (denoted as dashed line) similar to PPO trained with RND at convergence (2 billion frames) reported in [23] in five times fewer frames. (b) HEPO matches HuRL's
task using HuRL's SAC codebase [10], despite HuRL being tuned for this task and HEPO using the same hyperparameters from our Section 4. This also highlights HEPO's generality in different RI. algorithms. (c) HEPO's performance is positively correlated with that of the heuristic policy trained with heuristic rewards only (H-only), suggesting that HEPO's effectiveness will improve as the quality of heuristic rewards increases.
performance on the most challenging Sparse-Reacee
C Environment Details
As depicted in Section we conducted our experiments based on the Isaac Gym (ISAAC) simu- lator [19] and the Bi-DexHands (BI-DEx) benchmark [24]. The selected task classes in ISAAC can be partitioned into groups Locomotion Tracking (Anymal), Locomotion Progressing (Ant and Humanoid), Helicopter Progressing (Ingenuity and Quadcopter), and Manipulation Tasks (FrankaCabinet, FrankaCubeStack, ShadowHand, and AllegroHand), In addition, BI-DEX pro- vides dual dexterous hand manipulation tasks through ISAAC, reaching human--eve sophistica- tion of hand dexterity and bimanual coordination. Their tasks include ShadowHandOver, Shad- owHandCatee Underarm, ow HaHdaaddooo wo Catch U nderarm, Shadow Handl laaanniiitnnnan iitt Shadow HondDDorOOpeeeenaaarr Shad- ow wwWHnndDoorOpendd Ouwwdaaarddddddddddrra.aaa Shadow CoaaddDorrrllloooeeeo Shadow HandSpin.. Shadow HandU pside Down, Shadow BlocksssStakkk Shadow HandBoteann ShadowHHandGree wanenndrrraraanaaaaaaaeeeeee Shadow Handdeaaaaaeee Shadow HandPen. Shadow Shadow Hwwaaeeeiiennaattteeeeennnnenaae Shadow HandScissors, Shadow HandSwingCup.
HandHanddCace Underrrmm Shadow Hww.llaaanaahhahaae
Shad-
For Locomotion Tracking, our emphasis lies in assessing the precision of velocities in linear and angular motions, ensuring that the robot responds closely to the assigned values. To this end, we define tracking errors as our task rewards. For Locomotion Progressing and Helicopter Progressing, our emphasis lies evaluating the progress made by the robots reaching the assigned destination from a given start point. To this end, we define movement progress as our task rewards. For Manipulation tasks and all tasks within the BI-DEX benchmark, our emphasis lies whether and how quickly the robotic hands can successfully complete the assigned missions, reaching the desired goal states. To
19

---

this end, we define task rewards using a binary label, assigning a value of to indicate successful attainment of the goal state and otherwise.
The following are our reward function definitions, which include heuristic and task reward terms in Python style:
Isaac Gym Locomotion Tracking Task: Anymal
compute_ srapute_anyai_rrewaae
root_states,
comnanda, torques,
contact_forces,
knee_indices,
lengthe,,
res_scales,
base_inder,
max_episoce we_gepppodde_etgth
(reward, reset, feet, in asc, fest_ time, epésode auss)
type: (Tensor, Temsor, Tensor, Tensor, Tessor, Tensor, Dictleir, float], 581, Tupte(Tensor, Tensor, Tensor]
# propare quantéties (T900: returs from aba ?) base_quat root_states[;, 3:7] base_lin_val guat_rotatt_invvrsaeaaee base_ang_vel vel ggut_rotate__nvvrseeeeee (base. root_states 10:13])
rost_states[;, 7:10])
i velocity tracking reward
Lin_vel_erros torch.sum www.tttoc...cuaaacccoooo..o square :2] base hane_lin_well:: :2]), dim-1)
verrror torch, aquare (commnddad 2] base_ang vel-vell:; 2])
rew_lin_vel_gg torch exp(-lin__el vel__vee__rrrrrr error/0..2) rew_scaled("ioo.el.co res_ang_vel. torch. wrrccheepp-angg..e
rau_ecaleen"mng.vol_.c
forgue penalty
res_torque torch. sum(torch. iquare dim-1) rev_ccales("torquu
rotal_ wo..ii__eelxyy rex_ang. vel,2 reu_torque
kotal_revart torch clip(total_reweat 0., None) tracking.revard veir_vel_error errorr
agents
base_index, :], din-1) > reaet reaet torch. (ny nororrchhoonmcaaee (contact. rorceaCi, inee_indices, :], dix=2) 1., din=1)
reset norch..ar(canttattffooo.
tins_out epieode__enggile >- epx_epie length 1 serminal reward for tine-osts reset reset time_out
heuristic_resard, task_reward total_revard detach roward
return heuristic_revard, tavk_revard, reset
20

---

Isaac Gym Locomotion Progressing Task: Ant
def computa_ant__
oba_buf, reset__uuf,
progress_but actions, up_veight, heading_veight veight, potentials,
pev.ppotentiaee,
actions_cost__cal
mergy_cost_scclee
ternination_hoight,
death_cose lex_et
):
type: (Tensor, Tensor, Tensor, Tensor, float, flost, Tensor, Tensor, float, flost, float, float, float, float) >> Tuple[Tensor, Tesser, Tensor]
reward direction headed heading.eeight.too weight_tenatrr torch.onea. ines_1ike but Lt, 11]) hending.weight heading 11] 0.8)
besding, veight, heading. obe_bul[:
# aligning azés ant and ensirenmest up_reward torch geros (headiinn reward)
up_revard torch. (hare [:, 10] 0.93, up-revard up_neight, up_reward)
energy penaldy actions_cort torch. sarch.sumacctions ** 2, dim--1) electricity dof lot_ torch. aurchsssu(obsbbrr 12:20] 0.99, dim--1)
abb.rcch.baaaattiooooooaa (actionn
[s, 20:28]), dimm-1)
for duration of staying ai<ue alive_revane torch onee onee, like_llleeppptttill 0.5 progress_reward potentials prev_poteatiale
total_ravmt seaarr progreas_ alive_revard up_resard heading.
actions_coe' coste__cee__salll actions_cost ceetgy_cost_ecale electricity.cost dontet_
+
# adjuet reward for folles agents total_rest
whwrechbrrobbbuueeee death_cost, total_revard)
termination,heighn oott_
reset
reset torch where (obs buf(: bermination_heiieee torch, ones
caee_eeeeeeeeeeeeeeeeeeeee (resett reset_ buf)
torch. (progaapoor but >- max_opisode_lange torch. (reoeett_rr reset)
heuristic_renne rreaard rewadd total_ progress_reward
return houristic_rovart teearre
21

---

Isaac Gym Locomotion Progressing Task: Humanoid
oba_buf,
reset_buf,,
progress_but
actions,
up_veight,
medding_veight,
potentials,
pev..potentiale,
actions_cott__cale,
mergy_cost_scalle
jotns__at_hittt.cott.enm
max_notor_offort,
motor _ffor_afforts,
ternination,height
death_ cost,
max wwwsoode_ennnteeee
):
type: (Tensor, Tessor, Tensor, Tensor, float, flost, Tensor, Tensor, flast, float, float, float, Tensor, float, flaat, float) Tuple[Vensor, Tensor, Tenson]
headed heading_veights weisatttettrtttterrrrrrer torch ones_ like__ike buf [:, 11]) + heading_xeight heading_ 11] 0.8)
from
whare 11] 0.8, beading, beight_tensso, heading_voight obs_buf
reward for being ypright up_reward torch. geros_ wiw._liieeiii (hadhhadinng up up_rew
[uf 10] 0,93, up_reward up_weight, up_revard)
actions_cost sum(actions dimm-1)
energy cost reward
afforr_t_t__t notor_efforta max_notor_ettie
cost limii,at,tiiit.cosss (obs_bbuuf 12:33]) 0.96) scaled_cost dof_at_lini baf_at__init_cott 4 netor_efforte netor_effort_ratio (O))uesze()), dim--1)
(obe,, bofbbbbb[[[:; buf 12:38]) 0.98) / 0.02
electricity.com cost torch. sun (torch abs (actions obs_buf[;, 33:54]> notor notor_effort_retia (0), dig--1)
8 reward for duratson of being alive
torch.ones, borch oetp__iilppottetiiall progress_revare revard potentials prev_potentials
cotal_revard progress_ revard slive_revard up_reward Bedding_revare actious_cost_Ecale actions_cost eoergg_cost_scale olectricity_coss dof_at_limit_coo
adjust revard for follen agents total_revard torch where death_cost, total_revard)
ternimation,heigh: ones (ne__
reset
reset torch. vhere (obs buf(:, termination height, torch, ones, aeee__ikeeeeeeeeeeeeeeeeeeeeeeee (raseerrrett_tt reset_bat
torch. vhare sporreprrora max_opisode_lenge torch. (resset__bbt) reset)
heuristic hearistic_revard, task_reward sotal_revare progress_reward
return houristic_rovard tasarre
22

---

Isaac Gym Helicopter Progressing Task: Ingenuity
def
reward(rott.posttatti target_rood sarget_rrott_ppre poot_ponitions, rost_quuata, root__invals, root, root_angvedd angvela, reaet_buf, progress_but, buf, nax_ type: (Tensor, Tensor, Tensor, Tensor, Jensor, Tensor, Tensor, float) Tuple(Tensor, Tensor, 4 Tensor]
distance to target target_dist torch. torch.sqrttee (taaeett roott positioon rotttttoon root_ponitions). cum(())) poa_rexard
(1.0 target_diatt target_diat)
sprightness
upa quat_axii (roor quats, 2)
tiltage terch. aba upac...,
up_revard 5.0 / (1.0 tiltage tiltage)
spsnwing
spinnage torch (root, angvele(.... abe 2]) epinnago_revard
spinnage spinnage)
combined rewerd
sprigness spinking only matter when close to target resard poa_reward + poa_reward (up_revard + opinnage_rooard
resets due to misbehavior ones lnes_
die terch, zerca, like (reset_bar? buf)
2eroa_
die torch. (tereetarg dist 8.0, ones, die)
whera whw..rrrrrt...pttton......... 2] 0.5, ones, dia)
epseode length
reset torch. torch.vhere cpore(progress..btt >- lexs_opisod__engeee
1, ones, die)
heuristic_revard, task_resard
pos_revard
return houriatic_reward, task_revard, reaet
Isaac Gym Helicopter Progressing Task: Quadcopter
compute_quae quadcopter_reeeeeee cerot_prrooooooiittte. root_quate, rost_linvele, root_angvels, roset_buf,
progress_but buf, spppodde_ennttha. type: (Tensor, Tessor, Tensor, Tensor, Tessor, Tensor, float) Tuple[Tensor, Tensor, Tensor]
distance to target
diet torch. eqro(root crtesoot_.oeitione_....
pott.poeitione_....
root_ poot_pooiitoone..... 1] root_poritionss.....
root poot_ppoiiioon 2]) root_posittons [..., rea_re 1.0 / (1.0 dist target_diat)
sprightness upa (rit_ quatta, 2) tiltage abs abs ups(...,
up_reward 1.0 (1.0 tiltage tiltage)
spsnning spionage torch abrchhabee(ie (root, angrroot.ageeel...... 2]) spinnage_revard
(1.0 spinnage apinnage)
combined reward
# sprigness and spinning only matter when close to the target resard pes_resard poa_resard (upare apionago_rroaad)
resets dse to misbehavsor
torch ones_like (reset_bb))
die torch, 2eros_ like (reset.buf)
3.0, ones, die) die torch. ahere ahera chotpprrott.poiiioooo........... 2] < 0.3, ones, die)
shere
resets due to epssode length
reset torch where shere >- max_ max_episode_lengta 1, ones, die) rear,s task_reward resard, pos_rovard return rewri tevarde reset
23

---

Isaac Gym Manipulation Task: FrankaCabinet
def computa, nanxa_revard,
buf, progress.but, buf, weeet_goalbuu, successes, consecuiive_succssse,, Tranks_graep_pos. drawer_grasp.pos, franke, greep_root drawer, seaaer_rrapproo. Tranka_lfingor, tranka_rtinger_ppr
successes, actions, cabinet_dof_pos,
tripper_.orvard_axte sanser_insard_are gripper_up.arte drazer
draser_up.axis, up.axis,
sist_revard_ecale, rot_revard_eeale, booni__andle_reward..eo Inner_dist_reeard.d.
open_revard_ecale, action.pmalty.scale distX_ max.eptaode_aangth
type: (Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tenaur, Tensor, Tensor, Tensor, Tensor, Ienson, Iensor, Jensor, Jensor, Iensor, snt, float, float, float, flost, float, float, 4 float, float) >> Tuple[Tensor, Tensor, Tensor, Tensor]
distance from hand the drawer d barch.normttranaggg...
draver stavar-graspppaa, p=2, din=-1)
1.0 (1.0
dist_rewarz -- dist_revard
dist_reward
0.02, dist_reward dist_rovard)
axist tf_vector (recoor(rankk..reepp.ooo fripper_.orvard_..
tf_vector tf_voctor(drauerggrapp (draver_grasp_.ro droser_invar__axte))
axia3 tf. "e_vector (frankaa.aaa graaaaaaaarraaooo.... pripper_up_axia)
V__vector (eecoorr(rseerr.rree...o. upsser_up_exis)
bam(axisl wiwseviieeenneeeeee cnva, 1, 3), vierviinennnneaa alignment foruard assa for gripper
squeeze aqueeze
dot2 torch (am(exia3 veexies.vievumuu
alignment up vieu(mun_enva,, 1, 3), axist: view(nun,evaa 3, 1)) 1))..aaueeeeeeee. squeeze(-1). aqueeze(-1)
gripper
reaard for matchsng the orsentation of the hand to the drower (fingers wrapped) corch:eigu(dot2)
rot_rexard
(torch.signe sign((o)))
boeus if Left finger around soound_anndle..mm around handle_rewasse torch, where (i_trennk__iinge_pppeeeeee
drauer
right
torch. zeros_like(de baros_1ike(rot.raaaar
staver_graep_podee 2]. 2] draser-grasp_
(taraatranka_rrfineer
around_ bta_hhannl__errdddd 0.5, heuudd_aardlerevaae around handle_reward)
# reward for distance of each finger from the draver finger_dist, diser_ddat__rrarddd ifinger_dis dist torch abs Bainukaa_iiigepppooo rfinger_de dist torch, abrch (ttfeenka_riiiger.perre. finger_ Lingor_dist_rovard torch. where (frankaa Laannnka_.iiiingooo_..... 2] draver_grasp_posl: 2),
draver.graep.poe 2])
draver_grasp_poal; 2])
torch. where (franka_
2] draver ts 2),
(0.04 lfinger_dist) (0.04 ifinger_dist).
S finger_dist ; renger_
regsiarszation the actions (sunsed for each environsent)
action_penalty torch. sun(actione 2, dim--1)
# how for the cadinet has been opened est around handlo_resare cabinet_dof_podd 3]
open_revard cabsnet_dof_porl] draver_top... foint
rewarde rest_revaad__call dist_reward rot_revard_scale rot_revard
around brnule reaaddeerevar__eeec scale around handle_rovard open_ scale open_reward
shagr_ddist_rewadd..oo.. ringer_dsatt.raard action_pemalty_ccala action.pemalty
boeus for opening
propenly
resarda torch. (eaatteebtett.do poa[:, 3] 0.01, revarda 0.5, revards)
revarde torch. (eereece wof.ecabbeeed.oepppeeooeeo doo_poss [;, 0.2, rewards around handle_aadddee... revard,, rewarde)
revards torch. shere (cbbincabitt
0.39, revards (2.0 around handll___aeeeeeeeeeee rewards)
prevent style opening
resards torch, shere(franka, Lfangrr
lfanger_posdf pos 0] drawer drawer_grasp.pee paarrrrrapppppo... 0] distk_
torch. (ie__iik
revards torch, (wwaeetranee_rraaee wwi.eer__poooee 0] graxer_graapp.por pos[:, 0] dfftx,
torch. like_
-1, revards)
reset if drower is open or man length resched
Shore(cabinet.dor_ppos 3] torch.onea. (1eccess))( successes)
geal_reach reach torch. caarr(ccabnet_dof...o zarch. crroa_likeereeett.c goall_or))
3] 0.39, torch, ones mee__
reset_buf buf torch, where shere(progress_uuk >- bes_egi torch, (i2eeee buf), refet_buf)
consecutiva, auccessse Sarrhhohheaaraane > successea resat_buf, consecutive puristiic_revar, return heuristic reset_buf, taea_reward, consecutive succeessee
rewarde, geal_reach

---

Isaac Gym Manipulation Task: FrankaCubeStack
def computa,_ ranxa_
reaet_buf, buo, reset_goal_buf, actione, states, revard_settiing, nax_episode__eee Lenngt
type:
(Tensor, Tensor, Tensor, Tensor, Dict(str, Tensor), Dict(str, float], float) Tupie
Tensor, Tenson]
Conpute per-esv physical paraneters
target_height "autteslcubeBBazee cubed_eize - states ["cuuee_sizee] cubeB_size ("atee("cubeB_eizg
atates("cubol_nize"
distance from d torch. (orchao pttpaaaaaooooooeaatiiiiaaaaaae
csbed
dim--1)
d_1f torch norm ["cubaA_ppaa"] states["eat ("aaa_llaaa____aaeea soaa__f--pos_____ din=-1)
torch.noru(states ("susaattteeeeeeppppppee ("cubet_ppa"" states r'e""_"...a" din--1)
dist_revard torch. taab(10.0
torch.
(d d_1f + d_rf) /
# reward for lifting cubed cubeA height "catea("ubee_.oa_ [:, 2] setting "taiiiioe"ttaleeeiiihh" auboA_lifted (cubek_height cubsA_size) 0.04
lifa_revard cubeA_1ifted
# how closely aligned cubed ss to cubell (only provided cubed is lifted) 2or (staata[""uubel to_ectt.oooeB_p
offaet[:, 2] (cubed_siza cabeB_aize) / d_ab torch (aclsstttttssuube__leeee tacee_____ube__poo poo___poeop_ offeet, din--1) align_revard
tanh(10.0 d_ab)) cuboA_lifted
align
dist_revard torch max max(dist_rovard align_revard)
# ffiaa revard for stacking esccessfully fosty if cubel ss close target height and corresponding location, and gräpper is not grosping) aubeA_alig__uubl
(torch.norm (arrch.nrmmstttees (staattt" taapraa_ttocubB_pooa"o
:2], dime-1) 0.02)
cubed_on_ cubeB torch. (cubel, height target_height) < 0.02
seaper__vay_freee cubed stack_revard cubeeeeligmmcublll cubeell sripper.wwa_.fron_ccm
Conpose revards
We either provide the stack reward the align dist reward revarde torch, wbere(
stack_revard, seward_settinga "irrsstack_ccallr
stack_reward,
rexard_satting (ar ["y_diat__ccale" diat_rreard toard lift_reeard reward_ settinge[
srale"" slign_revard,
Compute
reset_bur torch. share bat >= Lex_eopiade__lngtt 1), Carch.ones wike_t reaet_bur)
goal_reach torch maaresstack_revard
torch onee ooee_
torch
zeros serchrzzzs____eeeeeeeeee leroos_sseerrrettgalll..o
heuristic_reward, task_reward resards, geal_reach return houristic_rovard, tack_revard, buf
25

---

Isaac Gym Manipulation Task: Shadow
def computa__hare
hatrrta__and_daerrrrre revard(
rev_buf, reset_buf, reset.goal_ progress,buf, successes, consecutive_ max, calsode_lengtee cist__reaar__caa float, rot, rot_revard_scale: float, rot_ops: float,
object_pos, object_rot, target_poe, target_rot,
actions, pazton_ponalty.acaaee fleat, toceess_ollerane:: float, reach_ flost, fall_dist: flost, fall, falllyenalty: flont, vax_consecativvaaacccec
int, av_fastor: fleat, ignore_z_rat: bool
Distance frow the hand to the object goal_dist norm(object_poo target_pos, p=2, din--1)
ignore_2_rot: succoss_toleraace 2.0 saccass_tolerane
Orsentation sligneent for quat_ (targett
goal
quat_diff (objjecctooe_errtt
rot_dist asin(torch.... aein(torch.clamp(tordd... (orm(quat_ 0:3], p-2, dim--1), max-1.0))
rest_rev goal_dist rwatd_rward
1.0/(torch..o..... (bott_dd rot_eps) rot_resard_ecale
action,penalty sun(actione 2, din--1)
is: position fall penalty
oriestation alignnent regularization
revard dist_rew penaon, petion..enalty..caal
Find out which enve hst the goal and spdate successes coant goal_reach
abe(rot_ success_tolerance, onee, wtt.eeei
a reset_goal_buf) successes successes geal_reach
Success bonus: orientation withis 'success, folerance" of gool orsentation shera_goal__eact
reach_goal_bonug, reward)
goal larger threshold revard torch, shere(goal,dist >" fall_dist, reward fall_ponalty, revard)
penalty:
Check fermination condstions, ssclading aszimus success namber resets torch, where(goal,,d diet fall_dist, torch, ones like(resettrtbiit ruset_buf) wtw_ccccecuvvveeeeeeerte
8 Resel progress buffer on goal enve sf moz, consecutive, successes > progress_but torch, whore(torch. (brtt__dist)) succeso_tolerance. 4 torch zeros_like (ipoooooeeppo buf), progress_hut)
resets torch where (succeases max, connaccutive_aucceeee torch. ones ones_ likerrett)))))))) reseta)
resets torch. (progreeee buf >- pax_episode__engee lenggt torch, like_ei))()eeet))) resets)
Apply penalty successes 0:
reaching the goal
if nax_ caaa_coccecuteee.. saarreuuivve_eucceseee
where (progrepr max_opisode
fall pall__ynalty, revard)
reaets torch sun (reaets)
finiched_coas_ ccaceccceee torch, sun(successes
float())
torch. wwwree(uun_eeet.....
ar.factoo--iniiaaad_.oo. wwwccessee/nur_reeaeeee (1.0
b av wow.fectttrrnnssuutttte.evee suceuutiiee_uceessssss casseutive__ucccssen
heuristic resard, task_resard return heuristic_rewate reseta, rask_revard, progress_buf, successes, cons_ Bucceaes
goal_reach
26

---

Isaac Gym Manipulation Task: AllegroHand
def computa__hare hand_aaraadddddd
rev_buf, reset_buf, reset.goal_ progress_buf, successes, consecutive_ max, calsode_lengtee cist__reaar__caa float, rot, rot_revard_scale: float, rot_ops: float, actions, paztoo_ponalty_acaaee fleat, toceess_ollerane:: float, reach_ flost, fall_dist: flost, fall. falllyenalty: flont, vax_consecativvaaacccec
object_pos, object_rot, targot_poe, target_rot,
int, av_fastor: fleat, ignore_z_rat: bool
Distance frow the hand to the object goal_dist norm(object_poo target_pos, p=2, din--1)
ignore_2_rot: succoss_toleraace 2.0 success_tolerance
Orsentation sligneent for quat_ (targe quat_diff (objjecctooeberrtt
goal
rot_dist asin(torch.... arin(torch.clamp(tordd.. (orm(quat_ 0:3], p-2, dim--1), max-1.0))
rest_rev goal_dist rwatd_rward
1.0/(tarch..ooo... (bot,,ddiitt) rot_eps) rot_revard_ecale
action,penalty sun(actione 2, dim--1)
is: position fall penalty
orientation alignnent
regularization
revard dist_rew rot_rev action_penalty peall..penalyy.ecclee
Find out which eave hst the goal and spdate successes coant goal_reach
onee abe(rot_,ist) success_tolersnce, lrec.ooee.ikkeeeetett. (ireetttttreeelallll....
a reset_goal_buf) successes successes geal_reach
Success bonus: orientation withis 'success, folerance" of gool orsentation shera_goal_react
reach_goal_bonug, reward)
goal larger threshold revard torch, shore(goal,dist >" fall_dist, reward fall_ponalty, revard)
penalty:
Check fermination condstions, ssclading aszimus success namber resets torch, where (oorl fall_dist, torch, ones sess_liee(rese (ike(resett_bu)) reset_buf) wtt_cccceuuvvveeeceeeett
8 Resel progress buffer on goal enve sf moz, consecutive, successes > progress_but torch, (torch, abs ( torch.zaroe zeros_ wtw.o_ (progrrorr buf), progress_hut. huf) reaets torch. (succeases max, nax_ Soonaccutiveaaucceaee
succeso_tolorance.
torch, onee_ like__iik(reseta)) reseta)
tined_out progress_bve buf max_opisode_lengt 1 torch. where(tined_out, torch. mike_iiikrrrsetta resets)
reaching goal 1f www.cooseeurvve_ecceeeeee rexard torch whereetined_oog reward 0.5 fell_penalty, reward)
Apply penalty successscceesssseeee 0:
RuR resers torch cunesetts)
finished,
torch. fich.suumaannn sun(sccessse
float())
cons, successes
resets consecutive_auccesen
resetssta
son.trrtwcconeeetttte.e..aac..e heuristic revard, reward rexard, gesh_resch
heuristic_rowant
progress_buf,
27

---

Bi-DexHands: ShndovHandOver
def computa__aare haae__aanndrdddddd_eeee
rev_buf, reset_buf, reset.goal_But progress,buf, succesees, consecutive.succeesss max, episeeepsoeeeeeee wwt.eeppioeeedddeeeeeeeeeeeeeeee length:: cist_revard_scar float, rot, rot_revard_scale: float, rot_ops: float,
object_pos, object_rot, target__pee, target_rot,
actions, aetton_ponalty_acaae. fleat, toccess_tllerane:: float, reach_goal_ flost, fall_dist: flost, fall, pall_penalty: flont, max_consecativv_aaccceer
int, av_fastor: fleat, ignore_z_rat: bool
Distance from the hand to the object
goal_dist norm(targot_poc object_pos, p=2, din--1) if ignore_2_rot:
auccesa_tolarance
aucceas_talerane
Orientation alignment for the cube is hand and goal cube
quat_diff quat_mul (ueteeullobecttrro. vut_cconjuatettarget.u rot_dist 2.0 torch aein(torch siiattorc..llapptteot..... clamp(torch, dorn(quutddifff(; 0:3), p-2, dim--1), max-1.0))
dist_rev goal_dist
Total reward position distonce
orientation alignsent actson regularization bonus
4 fall penalty
torch. exp(-0.2=(diet._r diet_rewarr__ecll rot_diet))
goal update successes
geal_ maal_reseaa torch shera (borch.aba (ggaal_l_datt)) <= 0, torch. miea_ (raeattttaat but), reset_goal__urr) successes torch (euceeeuceesaee 0, torch. chara(geal_daat 0.03, torch. wtteeeonnnneeeecaaaeeeee (ikecmcccaasess) successes), succasses)
Success bonus: revard torch, where shore - revard + neach_goxl_bonus, revard)
within
of gool orsentation
Fall penatty; distance to the goat sa larger than threashoid reward torch, (os 2] 0.2, reward fall tenallynal,y, reward)
# Check eny fermination condstions, including aazimus success number 0.2,
whore(object_poa(::
ones (reeetetttt.)) reset_buf)
if www.cccceuuvveeeecceecoec successes 0:
Reset progreas buffer goat saus Kf maz, consecative. successes progreas_ buf torch where(torch. (borch..barrot......o........t abe(rot_diet) < tuccess_tolerance, + torch zeros like torch. (progreer buf), progress_buf)
aas_censeeutive_erceeee successess,,, torch. ones_ lik___iitrresets). resets)
resets torch, where where (progrese_ buf >- wet.ppidd_eeetttteee torch sorch.ozee_ (ike resets)
Apply penalty for not reacking the goal iitaaeuttiv_succeeeee reward torch. where (progress, but >- pax_episode__ese length, revard 0.5 fell_penslty, revard)
torch. vhare(rasets
canerutiva_aucceeet mean
geal_reach chore 0.03, onecloones__e (ees_liihe(ucccssses)))))) zeros, Like (successes)) heuristic_ resard, task_reward revard, goal_reach return rewristic_reward, resa_re reaets, geal_resete, progress_ buf, successea, successee
28

---

Bi-DexHands: ShadowHaandCuchhUnneennn
def computa__hare hatrrta__and_drerrrrre
rev_buf, reset_buf, reset.goal_But progress_buf, successes, consecutive_suceee max, eww.oeepsoeeeeeee saasode_length: dust_revard_scom float, rot, rot_revard_scale: float, rot_ops: float,
object_pos, object_rot, target__pee, target_rot,
actions, aetion.ponalty.ccalee fleat, toceess_ollerane:: float, reach_ flost, fall_dist: float, fall, fenllyenalty: flont, aas_consecativo_acccceee
int, av_fastor: fleat, ignore_z_rat: bool
Distance from the hand to the object goal_dist norm(targot_poc object_pos, p=2, din--1)
ignore_2_rot:
succoss_toleraace 2.0 seccass_tolerace
Orsentation sligneent for conjuaaaa (targe quat_diff postt_mullobiiee (objecct___ rot_dist
goat
asin(torch. clamp(torch corn(quat_difffe 0:3], p-2, dim--1), max-1.0))
rest_reu goal_dist
Total reuard das posstion distence orsestation aligneent action regularization success bonus fall penally
revard torch. 2
rist_rewaad_scale rot_dist))
Find out which enus hit the goal and update successes coant
geal, resear torch shere (torch. abs abe(gol__diat) 0, torch onea_ ooe__ reset_gosl_burf torch. (succeaae torch where (gere 0.03, torch. oneblooes__ii mne__Like(uuccesee) successes), aucceases)
Fall penalty: distance to the goal is larger than threashold 0.1, rezard fall_penalty, revard)
resord shere(abjact.paa__
Check eny fermination conditions, Smcloding sasímus success number <hera
if souceeestree_eucccesse 0:
successes
progress buffer goal enve if maz, consecutive. successee progress_ buf torch,
<- success_tolerance, torch, zaros_ (ippoooraaapr....saaaaa baf), progress_hut)
resets torch. where (succeases >- maz, succeeeeeeeecceeeeeeeeaeae torch ones (ise__iie(reset))) reseta)
resets torch, where where (progrese__bu buf max_episode lengeh, birch.oae,,ikkereestee resets)
Apply penaliy for not reaching the goal
revard torch. vhare (prograss bat > www.oopsode__enntte revard 0.5 fall_ponalty, revard)
torch Viereere > 0, successes resets, gusseuutive_eucceest
goal_reach torch, (goal_goelddirt 0.03,
torch. onss__ihhamuuccesaae torch zoro___)he(tueccsssss)))
heuristic revard, tasa_reward rexard, goal_reach
return houristic_rovard, task_revard, resets, geal_resete, progress_ successes, cons_ruccesses
29

---

Bi-DexHands: ShadwrHaanCCuchhvverZnnndraa
def computa__aate hww_haaandddddddde
rev_buf, reset_buf, reset_goal_but progress buf successes, consecutive__uccee maz_episode_ length: flost, object_pos, object_rot, target_poe, target__ot, beft_ a right_hand_ hand_annddaaa base_pos.
diat_ float, rot_reeard_scale float, rot_eps: float, actions, perion.penllty.eclee:: fleat,
auccess_tolerance float, reach_goal, bacthggggabbuuuaaae bonus: float, fall_dist: float, fall.penalty: float, ww__conaecutivv_sucaee str
int, av_factor: float, ignore_s_rot: bool, device:
# Distance from the hand to the object
goal_diat coreehoormtargattppo objact_poa, p=2, din=-1) igmere_z_rot:
succesa_tolarant 2.0 succass_talerance
Orientation alignnent for the cuèe in hand and goal cube
quat_diff quat_mul(bjeet_rott rot_dist 2.0 torch. asin(torch..cc....... aaia(torch.clamp(toree norm dirf, 0:3], p-2, dim=-1), max-1.0))
dist_rev goal_dist
Total
posstion
alignnent
regularization
fall penaliy
torch. (opp-..r(dist_re) dist_reward_scat rot_dist))
Find out which enve the goel and spdate successes coant
torch. shera (torch. aba(goal_dtat) 0, torch. (herettttaraa reset_goal__utt) successes torch where (succesaee where(goal_dist 0.03, onee likeeeucccessee) succesees), successes)
esv right_ hand base, dist torch. nornbboorrrrrht..oe http://aaaaddbaa__ppaae dtype-torch. flost, device-device), p-2, dim--1) laft__and__aa
condstions, including
Corch.tensor (Co.0, 0.0, 0.5],
torch.tensor(Do.. -0.8, 0.5], dtype-torch. flost, device-device), p-2, din--1)
Corch.nommiiartthand.bbaa
resets torch. shere hand_base_distt >- 0.1, torch. cres, (te___tttt buf), reset_but) resets shera sandd_ass__di11 0.1, torch 1nss_112aaresetta) resats)
reaets torch. where(abject.poaf: 2] <- 0.3, torch. lixe__ix)))eseta) resets)
nax_ wowsecutive_euceeeeeeeeeeeeeeeee
# Reset progress buffer goal
progrees_bus buf torch, where (tooce sbero__ddet <- succese_tolerance
torch. like buf), progress_buff
resets torch. where (suceaazee > commaeuttive_aucceaeee torch. cwws__nneeeeeeeeeeeeeeeeeeeeeeee like_litrr)sssts)) resets)
reaets torch. where (progress_ buf max_epiaade__enge torch. 1nea, reseta)
Apply penalty for not reaching the goal taa___cosccurvve__eeeeeee rexard = torch where (progress, buf >- opissode__egghha revard 0.5 fall_penalty, revard)
successesss torch. shara(raseta > 0, successea resets, canseutiiv_auucceeeet geal_reach torch where (iit 0.03,
torch, oorch.ones_11 sese_like((ee wwteeoosssseeeeeeeeeeeeeeeee (successs))) torch zeros like (itrs__iiieeeeeeeeeee
hearistic revard, task_reward resard, goal_reach houristic_raward, task_revard,
geal_resets, progress_bui
30

---

Bi-DexHands: ShadowHaandCacchhrreen
def computa__haie haa__aanndrd_ddddeee
rev_buf, reset_buf, reset_goal_buf, progrees_buf, successes, consecutive_
object_pos, object_rot, target__poe, target_rot, left left_hand_ppos a right_ hand_pos, left_hand_ diat__reaar_ float, rot_reeard_scale float, rot_eps: float, actions, setion.ponalty.ecale:: fleat, auccess_tolerance float, reach_goal, march_goalbbous.. bonus: float, fall_dist: float, fall.penalty: str
maz_episode_ length:: seft__haddbbase_pee base.poo, right_hand, htw___aan_ba__pp___ee base_pos,
aak_conaecutivo.uuceee int, av_factor: float, ignore_s_rot: bool, device:
Distance from the hand to the object goal_diat torch.
norm(targat_pos torch.normsarggt..uoo
objact_ppa, p=2, din=-1)
ignore_2_rot:
success_
success.
Orientation alignnent for www.uugaaeetargget..ea. quat_diff quat_mul (abjectt_rr.t quat_
goal
rit_ 2.0 torch. asin(torch. aeia(torchhclammttorch. cottoorcc..oomggpege dorttquat,tifff: 0:31, p-2, dim=-1), nax-1.0))
res goal_dist
# Total reward is: position distance fall penalty
orientation alignnent action regularization success bonus
resard torch.
blat_revard_acall rot_diat))
and update Find which ahe.ectorcc..bbbaeeelll (baagll__iatt) <- 0, torch, onea_ (raet,i teset_goal_but))
geal_reseta torch, where where (torch.
successes torch. where (successes -- 0, 0.03, torch. ones, (ie___iik((uccessee)))))))))) successes), succeases)
whara
Fall penalfy: distance the goal is larger than threashotd
torch. shera(object.paall,
0.2,
fall_penalty, reward)
conástione, including base_pes bes worch.tensor tenooor(-oorr,,,, -0.65, 0.5], dtypestorch. float, devica-derice)) p=2, dime-1) left hand, basdd_aae___d__e torch norcletorall + dtype-torch. flost, device-device), p-2, din--1)
env
right_hand_bant based_bae__das dist torch. nornnrightt
torch. te-.or([-O.3,, -1.15, 0.5],
resets torch, where band_base, dast_dist.io >- 0.1, torch ones lieeee buf), reeet_buf)
(left. sraett_hand_baaeddeiee 0.1, otes__ibe( resets)
reaets torch. (oafettbjet..paa:: 2] 0.2, torch. carcchonee__111 ones_ (ike_ reseta)
www.ecottieeeccccccoo.o consecutivv_eecceeeee succeesee
Reset progress buffer os goal envs if maz, consecutive. successes progress_ torch. shore(torch. abrtt__ddrroo
auccess_tolorance,
b torch zeros. like (progrersss buf), progress_buff where resets torch. where (prograss__irg buf >= max_eplsode_ longth, torch. torch (ikeratt))) resets)
resets
max_ www.e//cttieeecceeeeeee..
enes_ likerreets) resets)
Apply penoliy not reaching the gosl cax_consecutive sececuereeeeeeeesesssssess torch. vhare (porgaproor baf max_eptsode_langth revard 0.5 fall_panalty, revard)
cons_successes successes
shere(resets 0, geal_reach torch. vhere(goal.diat <= 0.03,
resets, cansoutiiv__auccesae
torch ones saee_likeesucee (succesees)) torch torch.2eros, leecheeeee_..eeeeeeeeeeee (ieeeeesek((
puristiccrevatte task_reward reward, goal_reach
return heurdsticrrevadd tank_revard, resets, geal_resets, progress_buf, successes, cons_nuccesses
31

---

Bi-DexHands:
def computa__hare haraa_haanddrddddd_eeeeeeeeeeeee
rev_buf, reset_buf, reset_goal_buf, progrees_buf, successes, consecutive max_episode, lengghh: a anpect_anohher_se _rot, snget_another_per barget_anotber_to clat__eeard__chh float, rot_reeard_scalee float, rot_eps: fleat, actions, setion.penslty.ecle:: fleat, auccess_tolerance float, reach_goal, fall.penalty: float, aak_conaecttiv_auceeee
object_pos, object_roo, targot__po,, target_rot, object_another_po
laagggggaaaaaaaaaa boaus: float, fall_dist: float,
int,
float, ignore_s_rot: bool
from goal_dist torch. nomm(aaget..pp. object_pos, p-2, din--1) ignore_2_rot: succoss_tolerrace 2.0 acceess_tolerance tolerance
object
goal_ www.anntteer.ec
norm anoter.. otject, p-2, din--1)
if ignore_2_roe 22_ot;
success_tolerance
success_tolerance
Orsentation alsgnnent for the cube ss and geat cube
mut__conjugateetarge quat_diff quat_mu rot_ 2.0 torch. cetantttrcc.ccappttrre..... norn (ouatquu_..fffff 0:31, p-2, dim=-1), max=1.0))
quat_ anot_rnotherrc.m diff quat_mul (objeect_ansthhe_.oo ansther_ror rot_ansther_com
asin(torch. (ta(ttrrch.clamptorcch aorm(quat, iiiiatt_nntthrrdde [:, 0:3], p=2, dim--1),
max=1.0))
dist_rev goal_dist
Total (opp(-0-(ddst...... baat_revard_scall rot_dist)) +
posstion fall penalty
alignnent
regularization
torch xxp.-0.2-0 anothe
rest_reward_ecale rot_another__dae _dist))
goal update
goal_resee realte torch where abe
(bere <- 0, torch onea ouee_like (ollt reset_goxl_but)
(succeszes successes torch. (hara_gool_diie successes), successes)
aaal_aantthordirs 0.06, tarch, .mc..c...lll(cccccesss))))
Fall penalfy: distance the goal is larger than threashotd revard torch, thore(object_ponl:. 2] <- 0.2, revard falllpenalty, reward) revard torch, where(object_nnotaa
2] <- 0.2, revard + fall_penslty, reward)
# Check eav termination condstions, imcluding mazieus success sumber 0.2, torch. oechooonss_] siee_likerreaet. buf)) reaet_bur)
reaets torch. (hore resets torch. (obre
2] 0.2, torch like (reeett. buf), resets)
mas_consecutive_co
8 Reset progress buffer on goal enve sf moz, consecutive, successes > progress_but torch, thore(torch..o + torch baros__ikeeppll (progress buf), progress_huf) huf) resets torch where (successes max, consecutiveeeceemmcoo..m successse,, torch ones, liketres)ta), resets)
success_tolorance
torch, where (progreeee buf >- nax_episode__engee length, torch. ones, like_ resets)
reaching the goal if wtt_cooeeurrvee_eeeceec
Apply penalty successos 0:
torch, (progress,
nax_ mex_oopisode_legthh
fall_ponalty, revard)
cons cons_ successes torch Where(resets 0, successes resets, consecative_auccessee)) mean()
goal_reach torch (here goal_ another_dist 0.06, torch. oneo ont.___cccc((ccoossss)))) torch. 2oros__iicctuccccsssss))))
hearistic_revard,
goal_reach
return houristic_rovard, task_revard, resets, geal_resets, progress_ buf, successes, cons_ successes
32

---

Bi-DexHands: Stadowfaand
def computa__anee harrrt___and__rerrrrre
rev_buf, reset_buf, reset.goal_But progress buf successes, consecutive.auceene max_episode lengthh: + ret_riigt.handle.pss haft__aand_pa, right_hand.pos,
object_pos, object_rot, target_poe, target_rot, pot_left,
handle_pos,
dist_revard_scalee flost, rot_revard_ecaae: flost, ret_ops: float,
actions, action aetior.ponalty_ccale. fleat, :occeaa_oleerane:::
goach_gol_bbous: fell.penalty: float, wtw_cooeecutiv_eeeeeeee.. int, av ar_factor: flost, ignore___rot: bool
fall_diat:
# Distance from the hand to the object goal_diat porehnoom(targat..o.. objjcct_paa, p=2, din=-1)
# goal_ dist target_pos(;, 2] objest. 2]
right_hand_dis torch ror bagdt_handle_ppr right, rightt_aad_ hand__os, p-2, dim--1) left_ seft_handddiss torch torch. aorftpot_eeft_teeeee haadeeee left, baad_ p-2, dim--1)
right_hand hand_anddddda _isttreev right_hand feght_handddee dist left, hant__iat__re heft_handddias
Totat reuard da: posstion déstence oriestation aligneent action regularization success bonus
ffal penally
up_rew torch. zeros_ sakee_lkkeeee likeh (rightt_____ briight_bandddee _and__istt_eee
up_rew (righthriht_band bast_
tarch. (aaeettttt_aandddee 0.008,
3+(0.385 goal_dist), up_rev), up_rev)
revard 0.2 beight_handddiet.rree dist. + up_rev
resets torch. where 2] <= 0.3, torch.cnes, mmes__ikorreseetbbiie reset_buf) (here
0.2, lnee__ttk))))ets) resets)
reaets torch caaftellet_aanddd.ee >= 0.2, torch.onea. 1nea__1Xe(rvaatta) resets)
goot update successes torch. Shere - 0, uhere 0.05, ones, (euccessee))e successes), successes)
resets shere (progress_ but max max_eeptadd__engg torch. anea_11kaarrsaet resets)
goal_resetz torch zeroe (ikeeeeeeesss) 1orccceeeeo,__eeeeeeeeeeess
cons succeesses torch chere(reseta > 0, successes regets, sansecutive_successes mean() geal_reach where whore
torch, torch.ones__1t anns_lthh(mcccassen torch. zeroa_ saros__iieemuccccseees
heuristic_reward, taak_reward rexard, goal_reach return heurtatic_revard, task_revard, reaets, gesetr, progrese.but. buf,
33

---

Bi-DexHands: Shadow/HandDaorrOeeenllvvvd
def computa__hare hatrrt__aand__rerrreee
rev_buf, reset_buf, reset_goal_but progrees_buf, successes, max_episode_ length:: flost, object__oos object_zot, target_poe, target_Eot, door_left__ie hew.deeeeedddeppc u door wer__right__aaae sighht_andle_pps.
hand_hand_pps, right_hand_por.
hanpt_hand_tfpp right, ha right,
haght_haand_1__ps right_band,th.paa.
laft_hand_#f _f heft_hand_mf_ppon left_hand haft_hand_ left_ haft_hand_th_ppr. diat_reeard_scall
rot_revard_scale: float, rot_epa: float,
actione, priion_pocall__ecllee fleat,
duccess_tolerance float, reach_goal benus: fall_dist: flost, fall_ penalyyy float, max_consecutivv_encceeee
int, av av_fastor: fleat, ignore_a_rot: bool
Distance from the hand the object
goal_dist (trrgetett.. objact_pon, p=2, din--1)
right__est hand_ finger_ dist (torch, norn(dbor_rihht. handle__andde_ee Biight_anddff.ee p-2, din--1) torch. worch.nornddoor..ooooooo morn www.doorrrrghhhhatttaaeee
right_band pos, pr2, din-
torch, norn
torch htt.bar_rrigtthhadll.._oo
fight_hand_rf p-2, dim--1)
1o-pa, p-2, din--1)
torch. (aom hig
right_hand_th th_pa_ pr2, din--1))
hand__aan_fft finger_diatt
(torch. norm(door_iete _et___tttnaddepo ieft_hand_ff.ppm p=2, dime-1)
torch nornr 1 handle torch norm left__aandle_por
_a_ead_____ppp,_________________ p-2, dia--1)
beft_band_rf_ppe p=2, dim--1) +
norn htt.d/ooor_otthddlllldl loft_handlee_os
left_hand_ lfpo_, pr2, dim-.1)
torch norn (door. handt_handeppos
left_hand_th,__e th_po, p=2, dim--1))
right_ haahhandd________eeee_aeaaa_a_sd right_hand_ffinerr left_hand dee seft_hend_fee (ingerdinnerdai
# Total reward is: position distance orientation alignnent action regularization + success bonus
fall penaliy
torch. corch.zeros_jp aeros_like (rigght (right__aannn baad___andddat_ree
up_rew torch, where (right band__aad____ finnddffnge__dd dist 0.S5,
torch. (dborddoo_rrghe (et.oorriggt..haadl..eedlee paaddee_oeess 1] door__eftt 1]) up_rew), up_rew)
rezard right_ hand_diat__eed left_hand_die hat__aaddddddrrr up_rew
resete torch, where(right band hand__aandedeee finger dist 1.5, torch, corch.ones, like_etterreeetbbuu reset_buf)
torch. where falfofthaad__fggeee diist >= 1.5, torch. (irssettss) resets)
enus goal update successes successes torch. (successss == torch chere(torch. (baor wighthgtthhl.eeppee. 13 door__eftt_ door_ il) 0.5, torch mae__like successes), successes)
torch. where (prograssss but max_opisode_ torch. like resets)
goal_resets zeros (reset))
cons successes torch. where goal_reach torch shore(torch, abe aidoorrrigtt_hamdlee (irc.coe___eeeeeccsssss))))) torch, lics_ zorch. heuristic_reward task_revard,
successes resets, consecutive_succe successea) nean()
1] door left heft__handle_poe 1]) >- 0.5,
heuristic_revard, task_reward goal_resete, progress_buf, buf, successes, successes
goal_reach
34

---

Bi-DexHands: ShadowHHannDoorreennotttwa
def computa__hare hatrrt___and_rrerrrree
rev_buf, reset_buf, reset_goal_ progrees,buf, successes, consecutive_ max_episode_ length: flost, object__pos, object_zot, target_poe, target_rot, hoor_left_hane let__dleeeddee.po u door wer__rigtt_haaae sarr_rigtthhnnle_pee handle_pos.
laft__and_ppo. right_hand_por. right_hand_tf.u tfpos right, ha right
Light_hamd_15_pps right_land,th.pas,
laft_hand_tf_pp left_hand_nn_ppo loft_hand hat__hand_ff.po,, left_ left_hand_thhppr pana_th__po, diat.reward__ccm actione, peiion.peallly_eclee fleat,
rot_revard_scale: float, rot_epa: float,
seccess_tolerance float, reach fall_penalty" float, aax_coonecuutvooeucccessss int, av av_fastor: fleat, ignore_z_rot: bool
fall_dist: flost,
right_hand__iee _aitt dist (torch. (eimddoorrigttthhanddee + norch.norn (doorrrighthttt____. rightthanddleppo right, - pop p=2, din--1)
right_hand_tff_p.. p-2, dimm-1)
norn rightrrrrigtthhaddl....e
right_hand_rff.o rf,a, p-2, din--1)
torch norn rorg hit.oorrrggtthhhdle...p..p.
right_ ht..hhaddllfpll. 1f_po8, p=2, dim--1)
norm (door_rriritt haait__aadd___________
right_hand,de tosps, pr2, dim--1))
soft__han__fann
left tf.__o p=2, dim--1) torch. norn left_ hewtooadeeee handle__po left_ po0,, p-2, dim--1)
wingerinndi__s _istt
(torch. commdorr_lef__aaae
band_paad_rf_pos,, p=2, dim--1) 4 torch. norn left_ haaddhaane__p left_hand 1f_a p-2, din--1)
torch. norn
handle.pos left_band, ph,, p-2, dim--1))
right_ _isthhaddnddddnndddeaa dist_ree riihh__hand_finge waageringrrrddee haft__handddiat_tce baft_hand_fiie aaandd_innrrriaaa .iit
Total reward position distonce
alignsent regularization bonus
+ falll penaliy
up_rew torch, zeroe_ (right bargghtt__addddeee ditt_diet__rex
up_rew terch. where(right__aee saatt__an___fgiddd dist 0..5,
torch. charo faaadd_figee_ddaie 0.5,
torch. aba (dbor_rightthhaddeeee.
1]
door_ 13).
2, up_rex), up_rew)
right.hand_dist..t -eft__and_ddattrer up_rew
(right, band_finger dist 1.5, oateeooeeeeeeeeee bres__ikerreeet_beee beset_buf) resets torch, saarreootthhad_fineeeeee
> 1.5, torch. worch.ones__lkeeeeeeeee likerrott)))s resets)
Fidd out which hst the goal and spdate successes coant
successes torch where (euccessee (wtoodrrrrghthaaalleee torch. chera(torch. chere
1] htw.ioftthnndoopolll 1])
4 0.5, torch, onee _ikeelkkeee see___ike(succeeet (successess) successee), successes)
resets torch. (progenno but > max_ max_episode__err length, torch. ones (ie__likarrsssts)) resets)
goal_reseta torch (iree_l
torch. share(ranets
consecutiva. _emcritsve_cccessss mmann()
goal_reach = torch where abs (ddoo_o__rihtt_kke haatt__hnnee_________ 1] door left, beto__eet_handee_pppe handle_pos[i, 1]) >- 0.5,
torch. onw.oooessttlee (cueccesess, torch. zeros_ wet.o__ (ruccessss))))))
heuristic_reense task_reward resard, goal_rea reach geal_resets, progress_bus buf, successes, cona_successes
return mouristic_reaard task_ravard, resets,
35

---

Bi-DexHands: ShadorHaandDDoorCossaaaaaad
def computa__hare hatrrt__aand_rrerrraee
rev_buf, reset_buf, reset_goal_but progrees.buf, successes, consecutive_ max_episode_ length:: flost, object__pos, object_zot, target_poe, target_rot, soor_left_hanl let.ddeeeeddee.po u door wer__rigtt__aaae sarr_rigtthhnnle_pee handle_pos.
laft__and_ppo. right_hand_por.
handtt_and_t__p right, ha right,
hight_hand_15_pps right_land,th.pps.
laft_hand_tf_ppl left_hand_nn_ppo left_hand hat_ left_ left_hand_th.par tana_th__po, diat.reward__cal actione, peiion.peaally.ealee fleat,
rot_revard_scale: float, rot_epa: float,
saccess_tolerance float, reach fall_penalty" float, aax_cconecuutvooaucccessss int, av av_fastor: fleat, ignore_z_rot: bool
fall_dist: flost,
right_hand__iee áist (torch. (eimddoorrigitthhanddee
hight_hand_#f_pos, p-2, dimm-1) _f_o,, p=2, din--1)
+ norch.norn (dogrdrrgggrrtt.____ rightthanddeepps right, har___andd__p_p_e -aight_band
norn hightrrrrigttthadde....e
bight,handd.rf.u rf,a, p-2, din--1)
torch norn rora
wit.borrrggtthhhdleepppp handle__os right_ hww..aadd.pppp. 1f_po8, p=2, dim--1)
norm (door_rriiitt hastt_handl____________
right_hand,d tospo, pr2, dim--1))
soft__han__faan
wingerinnd_r_sse _istt
(torch. (cnor___eeet__ haabor__et__aamle_ppoo left_ tf.__o p=2, dim--1)
torch. norn left. hewtooodeeeeteeeee_e_eoo handle_pdee.... left_ poo, p-2, dim--1)
bafd_paaddrf_pos,, p=2, dim--1) 4 torch. norn left_ haaddhaanee_p left_hand 1f_a p-2, din--1)
torch. norn
(doorddor____t left handle.pos left_band, ph,, p-2, dim--1))
right_ histhaanddddddnndddeaa dist_ree right__han__fing wwagerinndrddaa haft__handddiat_tce baft_hand_ffae fanddd_fnnrrrdaa _iit
Total reward position distonce
alignsent regularization bonus
+ falll penaliy
up_rew
2eroo_ (right dist_rist_
up_rew terch. where (rere(right__naat saatt__ann__fiie finger
finger_dinsssseeeeeeeeeeeeepssse dist 00.5,
whare(left_haneee faaaadd_iiee_____e 0.5,
torch. (ddor www.ttttannde..ooee......
door_ haoo__eef_haadee__roo
1] 2, up_rew), up_rev)
revard 2 right_baad_dde hand_aadddda dist_rev www.bbendddddddeeeeeeee
up_rew
resets torch. whightright seaght__ann___aaee finger dist > 1.5, torch, ones_ liee_sitrrasett_bii reset_buf)
resets torch. shera htw.haannddddddiiiiee ringer_diat >= 1.5, torch, onch.onnss_ll 1nes (reenet))))) reseta)
Find out which enue hit the goal and spdate successes cound torch. vhere (successes -- 0,, torch (torch. abs (door hww.grhttthhhhle...eeeeeaa _polle 1] door door__eef__anndee_poff 1])
ones_like (successes), successes), successes)
reaets torch, where bprggrees.bbu but max_epiaode, max.epaadee..ngttaa torch.onea, onea__ike reseta)
goal_resets torch, zeros_ torch.zero__Liierreeees. (ike((reeeees)
sans_suceesses torch where > successes * resets, consecutive, door htt.__eandde.portte..oooooo
(ereeuutivv_ucceseeee)) succesees)
geal_reach vhore(torch. (mucccosa)))))))) torch. zeroa_ (irro__tika(mcccasees
aarch.ones__ikee
heuristiic_rvard, teek_reward resard, goal, reach retura pouriatic.revare task_revard, reaeté, geseeresete, progrese_but. buf,
36

---

Bi-DexHands: Shadow Shadow-aanadDorrClseeuuwaaa
def computa__hare haw__aannddrdrrraeee
rev_buf, reset_buf, reset_goal_ progress,buf, successes, consecutive_ max_episode_ length: flost, object__pos, object_zot, target_poe, target_Eot, hoor_left_hanee peaaddeee.aadcc.po u door www__rigtt_haaae sight_handle_pes.
laft__and_ppos right_hand_por.
hannt__and_tf_po right, hand__tf right,
hight_hand_15_pps right_land,thhpps.
jaft_hand_tf_ppl left_hand_nn_ppo left_hand hat_ left_ left_hand_th.par tapa_th__pa, diat.reward__cal actione, peiion.peallly.ealee fleat,
rot_revard_scale: float, rot_epa: float,
saccess_tolerance float, reach fall_penalty" float, aax_cconecuutvooeucccessss int, av av_fastor: fleat, ignore_z_rot: bool
fall_dist: flost,
hight_hand_#f_pos, p-2, dimm-1) + norch.norn (doordrrggggrtt.____ rightthanddeepps right, har___andd__p_p_e -aight_band nf_ps, p=2, din--1)
right_hand__iee _iit (torch. (eimddoorrigitthhanddee
norn righttrriittthh.all....
bight,_andd.rf.u rf,a, p-2, din--1)
torch norn rora
wit.borrrggtthhhdle.pppp handle_pos right_ hwwd.aadd.pppl. 1f_po8, p=2, dim--1)
norm (door_rriiitt haatt_haadl____________
right_hand,ff tospo, pr2, dim--1))
soft__han__faan
wingerinnd_r_sse _iitt
(torch. (cmor___eeet__ haabor_eet__aamle_ppoo left_hand tf.__o p=2, dim--1)
torch. norn left_ hewtooodeeeeteeeee_e_eoo handle_pdee.... left_ poo,, p-2, dim--1)
band_paaddrf_pos,, p=2, dim--1) 4 torch. norn left_ haaddhaane__p left_hand 1f_a p-2, din--1)
torch. norn
(doorddor____t left, handle.pos left_band, ph,, p-2, dim--1))
right_ histhaanddddddnndddeaa dist_ree #ight__haddffinee wwagerinndrddaa haftt_handddiat_cco baft_hand_ffae aaand__ingrrriaa .iit
Total reward position distonce
alignsent regularization bonus
+ falll penaliy
up_rew
2etooe (right dist_riet_
up_rew terch. where (regh(right__naet saatt__ann__fiie finger
finger_dinss dist 0..5,
chare(left_haneee faaaddd_iier_d_e_e 0.5,
torch. (doorrrihthtt www.rrtttaaanne.polll...
door_ heor__eef_
1] 2, up_rew), up_rev)
revard 6 right_hand_ baand_istt_re
left_bend hecr_hand__ditee diet_rexx up_rew
resets torch. wrightright__ti frrgght_aan__faeee finger _iit > 3, torch. enes_ lire___tttrertt_etee buf)), reset_but) resets
haand_ringerddtt >= forch.ones_11kee (1k(srettt)))) resats)
Find out which enue hit the goal and spdate successes cound torch. vhere (successes -- 0, torch (torch. abs (door hww.grhttthhhhle...eeeeeaa
1] door, bbor_ieft_bandle_ppeff: 1])
ones_like (euccesses), successes), successes)
reaets torch, where (progress_buu but max_eepaade__eneee torch.onea, onea_ reseta)
geal_ realts torch, zeros_ zeros_]ikcc(reeete)
torch where > successes resets, snssecutive_successe mean() storce ridborrright_hhadl.... catiionn____aee torch.ones__lkaacnccce... torch. zeros_ (at..a_.
geal_reach
door www.__handlo.posi......
Heuristiiereevadd, teea_reward resard, goal_reach retura heuristic.revard, task_re reaeté, gese_resete, progrese_but. buf,
37

---

Bi-DexHands: ShadowHandSpin
def computa__hare haad_aanaddddddde
rev_buf, reset_buf, reset_goal_but. progress_buf, succesees, consecutive.successse max, www.eepip_eeeeeeeee length:: cist__revard_cce float, rot, rot_revard_scale float, rot_ops: float, actions, perton_ponalty.ccale fleat, toccess_ollerane:: float, reach_goal_ flost, fall_dist: flost, fall, palllpenalty: flont, vax_consecativvaaacccec
object_pos, object_rot, targot_ppe, target_rot,
int, av_fastor: fleat, ignore_z_rat: bool
Distance from the hand to the object gisl_dist norm(object.ppo target_pos, p=2, din--1)
ignore_2_rot:
succoss_tolerance 2.0 auccess_tolerance
Orsentation alsgneent
Modsfied so pen se sponeirical; since only rotate around the ozse,
quat_diff_1 quat_mllobjjee (object..rot aaa__coojuaateaargttooo
rot_dist_1 = 2.0 torch. wwtnnttrcc..langgg clancccaacctcccc camnqqat_diif_lll 0:31, p-2, dim--1), max=1.0)
quat_diff_2 quat_ robject,rot, quat_ conj_,.co (rgaaattiip.ooeeetaaiooo ttetttttoontarggttuuoo
rot_dist_2- 2.0 torch. asintarrchhhcccc clamp cainnaaa_ddif__a 0:3], pr2, dim=-1), max-1.0))
rot_dist torch min (rot__diistt_,, rot_diat__)
dist_rev goal_diet dist_revard_scnt rot_res (torrht (ba rot__pa) rot_revard_
action.penalty torch sun(actione 2, dim--1)
Total reuard is: posstion distance orientation aligneent + action regularization + success bonus fall penally
revard dist_rev rot_rev action_penalty, * action.pomaly..calle
Find out which enus Ast the gool and spdate auccesses coant resler torch, (root, (boo.___diet) suceess_tolerance torch. ones ote.e_ reaet.goal_bur)
successes successes geal_reseta
Success bonus: orientation is withis 'success, tolerance" of goal orientation rezard terch, chora(goal_reaate
meach_goal_homug resard)
Fall penalfy: distance the goal is larger than threshold revord torch. dgoal fall_dist, reward fall_penalty, revard)
conástions, including resets torch. shere(goal_dist fall_dist torch, Sarch.ones__ie mes__iix(rreet__bff reset_but) nax_ cww.ecccecttte.cco buffer goat
Reset successes succcccasto
consecutsve, successes
progress_buf torch, where(torch..eo absrt_.bbs
success_tolerance, torch. zaroo_rrossslllee (irogrrr baf), progress_buf)
resets torch where (successes >- max www.///cutieeesuueeeeeeeeooeeeee successes, torch. enes piee_likereeeete resets)
resets
where shere (progresz, but max_ mar_opisode_longth. 1, cerch.ones__ie likeooss))))ssts,, resets)
Apply penally for not reaching the goat
wtw.co/euuvvveeeceeeeeeeeeeeeeee succeeses resard torch, whare where (progrepss buf >- max.opisode_ resard 0,5 fall fall_penalty, revard)
nun resets torch sum(reset)) finished, cons_successes torch. (succcesses resets. fleat)))
cons successes torch where aaare(nun_reseee resete 0, httpar/eeeeisseooooooueeeeeeeeee f geal_reace real_reach torch, share(torch. abr crrch.ones_Ikkareeatt..e
(1.0
consecutive conecutive_euccessess)
saccess_oleramca,
torch.zeras 1eros__11
cuuristicrevardd tasa_r reward, goal_reach return heuristic_rovard, rask_revard, resets, geal_resets, progress_but, but, successes, cona_successes
38

---

Bi-DexHands: ShadowHaandJssddeDorr
def computa__hare haa___aanndddddddde
rev_buf, reset_buf, reset_goul_but, progrees_buf, successes, consecutive max, epwsoeepsoeeeeeee www.eepppieeoeteeeeeeeeeeeeeeeee length::
object_pos, object_rot, sargot__pe,, target_rot, dist_reward sevard_scale: float, rot, rot_revard_scale: float, rot_ops: float,
actions, pazton_ponalty.acaaee fleat, toceess_ollerane:: float, reach_ flost, fall_dist: flost, fall, falllyenalty: flont, vax_consacativoaacccoe
int, av av_fastor: fleat, ignore_z_rat: bool
Distance frow the hand to the object goal_dist norm(object_poo target_pos, p=2, din--1)
ignore_2_rot:
succoss_toleraace 2.0 success_tolerance
Orsentation sligneent for quat_ (targe
goal
quat_diff poatt_mulobbiie (objjecctooeberrot
rot_dist asin(torch.... aein(torch.clamm(tordd. (orm(quat_ 0:3], p-2, dim--1), max-1.0))
rest_rev goal_dist rwatd_rward
1.0/(tarch..ooo... (bot,_ddiitt) rot_eps) rot_revard_ecale
action,penalty sun(actione 2, din--1)
ds: position fall penalty
orientation alignnent
regularization
revard dist_rew rot_rev action_penalty setion.penally.ecale
Find out which enve hit the goal and spdate auccesses coant goal_reseta
(toreetorch. abo(rot_diet) success_ a reset_goal.butf successes successes goal_resets
Success bonus: orsentation withss "success, tolerance" of gool orsentation (goal
revard reach.goal_ revard)
threshold revard torch, shere(goal_dist >" fall fall, reward fall_penalty, revard)
penalty:
goal larger
Check fermination condstions, ssclading aszimus success namber resets torch, where(goal,,d fiet fall_dist, torch, ones like(reseteeebiit ruset_buf) wtw__cccecuvv__eeeeeeeeeee
8 Resel progress buffer on goal enve sf moz, consecutive, successes > progress_but torch, whore(torch. (ist succeso_tolerance, 4 torch zeros_ zaros_like (ppoooogrss buf), progress_hut) huf) resets torch where (succeases max, connacuttive.uuceeeee torch. ones. ones_ like(rett)))))))) reseta)
resets torch. (progreeee buf >- pax_episode__engee lenggh torch, liee_ei))()eeets)) resets)
Apply penalty successes 0: if nax caaa_cocrecuteee.. whore (progrepr
reaching the goal
max_opisode_lengi
hall__enalty, revard)
reaets torch sun (reaets)
finiebed_ccns__ucceeeee torch sun(cuccesses
float())
cons_successes torch. whare(nun_resets
ww_factor-finisbed.cc
cacaassee/nuuuttee (1.0
wo ay puccessee,, consecutive (ecuutiiveecccosses))
geal_reach whore abs(rot_dist) success_tolerance ones (rase.._tt.
heuristic_revard, task_reward resard, gosl_resch resch geal_resete, progress_but. buf,
return heuristic.revard, resa_revard, reseté,
39

---

Bi-DexHands: HhadoowHandBoockttuck
def computa__hare band_revard(
rev_buf, reset_buf, reset_goal_but progrees,buf, successes, consecutive_ maz_episode_ lengthh: flost, object_pos, object_zot, target_poe, target_Eot, block_righs, u block_left bback_lefthhandl__pes
haft__aan_ppas right_hand_por.
hantt_han__tpppo right, ha hight_hand_rf_ppo
hight_hamd_15_pps right_hand,th.pa.,
laft_hand_tf_pp left_hand__f_ppo
hat__hand_ff.po,, Jaft_hand_th_pon.
diat_reward__cake actione, priion.peaally_ecllee fleat, success_tolerance, float, reach_goal benus: fall_dist: flost,
rot_ravard_scale: float, rot_epa: float,
fall penalty: float, aas_consscuttiv_eacceeee int, av av_fastor: fleat, ignore_a_rot: bool
Distance from the hand the object stack_post = poack.pos2 target. clonee
stack_poell;,
stack_pos2[: 1] - 0.1 stack.poaiE: [;, 2] 0.05
geal_diati torch norn(atack.post Maeck_latt_aandee.. p-2, dim=-1)
goal_dist2 nornletackk. block_right.Daddee.pos p-2, din--1)
right_ fight_hand__ie hand__inger_dddi (torch. hbw.:ccckkkghhhhaaallee...
right_ fapo, p-2, dim--1) rright "ight_hand__t.. of_d__ p=2, dim--1)
torch. norn Baalock_right_haae
torch seen(boooo__rihthheeeee
right_ hand_rf.ja rf. p-2, din--1)
tarch. norn(block_rrghtt. handle__andlepps
haght__han_1f_ppn p=2, dim--1)
torch. norch..oorrmill (blockk rriglock_rihh.... htw.rrgggghhhhhe.pp....
right_hand_th,,p p-2, din--1))
left hand finger_ diet wo torch, morn(block_ _ handle_pos
diet
(torch. norm(block____ lt____lee_____________ handle_pos
fttd_ban__ff_poe p-2, dim--1) _oo_, p-2, din
normbblock_lttt handlahpnddeps left_hand_rr.pps,, p-2, dim--1)
torch. norm (block, btaplocck_eef__dtdldeeee bofft_andlee.po
baft_baadd_ffppt
din--1)
torch norn left hendeed
thtthea p-2, diu--1))
right_hand, hand, hast_dist____paeea dist_rev rew - right, hta.hh_finneeeeeeeeeeeeeeeeeeeee finger_ diet left hand dist_rev left
Totat reward da: posstion déstence oriestation alignsent
regularization success bonus
fall penally.
up_rew torch, zeros, (right hand_ dant__iss_res)
up_rew tarch. (rere btigggt___dd__iier____oo dist 0.5,
torch where(left, hingediienededdd_eeeeeeee diet 0.5,
goal_disti goal_dist2) 2, up_rew), up_rex)
stack_ree rev torch geros_ www.:eeerggt..anddddr..
stack_rev torch. where (goal, dist2 0.07,
torch. chere (0.05 moli-torch.co torch.trc...eeee worchh.baaetaack..po. abatttack..pn 2] block_left_haneee bacck_loftthamdd..paat. 2]))
0.07,
rstack_rew)
resard 1.5 1.55- dee jaft_hand_diat__a + up_rev stack_rew
torch, where(right, hastt___and__dd__edde <e 0, torch. (ieeeiikeeeeeeeeeeee reset_buf)
resets torch, where (right, saand_fingor_dde dist > 0.75, torch. ones, oite (ikerrsssss resets)
(resata), torch. whera(left__hnme htw.baanndd_n_dddee___ >= 0.76, tarch.
enus
goal update
successes torch. (succeausss =s
torch uhere(etack_rew 1, torch, 1iee_ successes), successes)
resets torch, shore > max_ max_epissde__orree torch ones like resets)
geal_resett torch, zeros zeros_likeeeeeeeeeeeee (ire__likerresats))
successes torch vhere(reseta > 0, successes reseta, consecutive, succeessese_e goal_reach torch where torch. orrch.ones__lie saes_like(uuuccesss torch zeros like torch (successs))))
hearistic_revard, task_reward resard, goal_reach return heuriatic_ravard, task_revard, resets, geat_resets, progress_buf,
40

---

Bi-DexHands: ShadowHandBotllCaa
def computa__anne hares_aanndrr_rdeeae
rev_buf, reset_buf, reset_goal_but progress,buf, successes, consecutivve_ maz_episode_ _llnnghh flost, object_pos, object_zot, target_ target_rot, bottle_cap.pos. u bettle_pos, bottle_cap.rd laft__and_pps, right_hand_por.
ha right, mf right_hand_rffpom
right_hand,1f__on right_band,tà.pes,
dist_revard_scale: float, rot rot_revard_scale, float, rot_epa: float, actiona, aetion.penalty.eee sucoees_toleren tolerance: flost, reach_gosl_ flost, [sll_dist: flost, talllpenalty: float, pax_coasectivvo_eccesss int, av_factor: fleat, ignore_x_rot: ):
float,
right, left_hand_dise torch poo left_hand_poss p-2, dim--1)
handt_hand____ hand_dist torch. torch.normbbatteeeccapppo right, poa, p=2, din=-1)
right_hand hand Mandd_inger_ddee (torch (orn pap_pos right_ tt.. p-2, dim--1) torch morm(bottle_cap_ugg right_ hand af
sauud_df_pppt__ _oe, p=2, din--1)
torch cormtbotlle.ccppFF.. cap_pos right bight_band_rf_.o,, p=2, dim--1)
G terch. norn cornthottle_cam cap-ppo right_ _f_ps, p=2, dim--1)
torch norn (ort(bottee_capppp
right, pos p=2, dim--1))
right, hand dist_rriteeeerr.. ree right,_ han. htn.:hann__iinrre... Tinger_dist left hend_ dist.rev left_band_
ds: position fatt penaliy. up.rew torch. zeros, zeros_
alignnent
regularization success
up_rew torch, sbere (rightt, Lingend <= 0.3, torch. up bottle_pos, p-2, dim--1) 30, up_rew)
right_ baats_han_ddssttrree
hand_diat_ree up_rev
resets torch, where (borttlbeee_caapppoooe (s, 2] 0.5, torch, ones like Likeeteett_bbl reset_buf) (ies_ resets)
reaets torch, caaeeellft__enddddee >- 0.2, onee_ resets)
# Find out which enus hit the goal and update successes count (succaaas successes torch where w norn(bottle.con
bottle_pos, p-2, din--1) 0.03, torch. onee_ seee_llie(succceees successet), succesees)
resets torch, where (progreeee buf >- max_ epieedde_ torch onee like resets)
geal_reseta torch. zeres_ (aro__)i
cons successes torch Where(resete geal_reach torch, storee norm(bottle_cap.. battle_pos, pr2, dim--1) 0.03,
consecutive _uccceesesee
torch. ttt.s__n torch. wtr.o__
couristic_eevarde task_reward reward, goal_ree reach goal_resets, progross_buf, successes, sacceseees
return heuristic_revard, task_revard, resete,
41

---

Bi-DexHands: ShadowwHandGnnsgAnddlloan
def computa__hare harrtt___aad___aaaaaaae
rev_buf, reset_buf, reset.goal_but progrees,buf, successes, consecutive.eucceest maz_episode_ lengghh: flost, object_pos, object_zot, target_poe, target_rot, block_right. u block_left lew___efaaadlppss.
laft__and__pa. right_hand_por.
haght_hand_tfppo right, ma right,
Light_hand_1f__ps right_hand,th.pes,
laft_hand_tf_pen left_hand_nf_ppo left_hand haft_hand_ hatt_hand_thhpo.. diat.revard_scall actione, priion_pecally_ecllee fleat,
rot_revard_scale: float, rot_epa: float,
success_tolerance float, reach_ benus: fall talllpenalty: float, aax_coonecuutvv_accccess int, av av_factor: fleat, ignore_z_rot: bool
fall_dist: flost,
Distance from the hand the object right_hand www_hann_diieee fingrrfinarr_ttt torch. nor.ooookk....ggttt..... rarnttllok.rigtt.baadee...op
(torch. norack hww.rriggghhhhee.ppp. handle_ right_hand__f.. hhan__ff.pos. p=2, dim=-1) right, pos, p-2, din--1)
torch (tw.cboookkkghhhheeceeeeeeee
tf.,,,, p=2, din--1)
righ handleee
rinht_band__15 1f_po,, p=2, dim--1)
torch, norch baalacck_iggthhaadllae
bight_hand_th... thopo, p=2, din--1))
lend_ finger_ dist (torch. het.t
left_band,ff.pos p-2, dim--1)
torch. morch..ormbblckk... left_handde__ppn
air__hand_f__pe_, p-2, din--1)
torch norn (blockk__ll htw.eettttaaae.p.. left fert_hand_rf_pp ropa,, p-2, dime-1)
left_hard_ pos, p-2, din--1)
torch, left headeleeeeeetateppeeeee
left hand_ thpo_ p=2, dim--1))
right_hand, hww_hannd_d__ _isttree torch. 6x1 x right_hand_ ringer _diat_
hand
left ditt_ Norch.exp(-11 left_ finge__ooeeddiees
up_rew torch. 2eros_ (right, handt_idtdddteee up_rew torch, exp(-10
block leftkllef___aatee hewteetttaaleeeee p-2, din--1))
right_ dint__ist_rro beft_hand_dist_ree up_rew
resets torch, where (righttihht__aee btt___addddddeeeeeeeeeeeeeeeeeee dist_rew 0, torch, onee, saee_like(reede like reset_ shereerrggtt.andddrine reaets torch, Ser_leeft_eedd__eeeeeeeee
tnee_t reseta)
>- 1.5, torch. ones, sme__lieezesetae resets)
# Find out which enus hit the goal and update successes count (suceaaae successes torch. ohere nooreettooonbbbbbboeo norn rightokkk.ght,.banll bandle_ppse left,_ hefd.e p-2, dim--1)
0.2, torch ones saee_likedee (succosess))) successoe), successes)
resets torch, where (progreeee buf >- max_ spisode_lengtee torch onee, like_ resets)
geal_resee resets torch. zeres_ (iae__llk(rresttt
successes torch Where(resete goal_reach reach torch, shere(torch norm(block handee_eeeea llock_left_hae bockklleft p-2, dim--1) 0.2,
consecutive. consecutive__ucceesee) mean()
heuristic_rezard torch, onee (successsee)es) torch zeros_likee like (successes))
goal_reach return cerristic_rosadd, task_revard, resets,
realtr progress_bre but, successes, cons_succeset
42

---

Bi-DexHands: ShadowHandkettle
def computa__hare hand_aaeardddddde
rev_buf, reset_buf, reset.goal_but progress_ successes, maz_episode lennghh: flost, object_pos, object_zot, target_poe, target_Eot, bantleebandle_pr u bucket_ cager__andde__pp kettle_spout_pprt
lafd__and_ppa. right_hand_por,
ha right, mf right,
bightthhand_1__pp right_land,th.pa..
left_hand _ppos haft_hand__f_ppo bfft_hand_rffpos haft_hand_1f.po,,, saft_hand_th_por diat_reward__cake actione, priion_pcaall__ecllee float,
rot_revard_scale:
rot_epa: float,
success_tolerance. float, reach_goal benus: fall penalyyy float, sss_cooneeuttvv__ucccessse int, av av_factor: fleat, ignore_a_rot: bool
fall_dist: flost,
Distance from the hand the object findrrinnarr__aa right_hand www_hanndd_ieee torch wwwnnoottleeeeeeeeeeee bon.eeee......eeppo.....o.o bight_baand_ff_pp pos, p-2, din--1)
(torch norochhnorm(kattii harriii__aande_pppe handle_pos pea right_ hantt_han__ff.. mann, p-2, din=-1)
torch morm(kettle,,... handle_peo hot.teetiii.haaaleee..o
hard_adddrfff of pos, p=2, din--1)
harile__aand__ppe right hand__aan___.... _f_po, p-2, dim--1)
hop right_hand_tt saadd_t__poss p-2, din--1))
land_ finger_dist
www.aaannddnnndddeeeee torch. norn(backet_ haanckket__annd__e_ee____ left
left, heft_handdff.pos p-2, din--1)
_ps, p=2, dim--1)
torch norn (wwckeettaandeeeeppppee handle_pos
haft_hand_rf-pps p-2, din=-1)
norn (orrkbbeett_bbae baakeet__andl_ppppee pos
bett_eed__1_ppo p-2, din--1)
torch. norm (bucket, handle (backett___aadelleeeee
left_ hatt_haad_th_pps, p=2, dim--1))
right_hand hand hhndhdann________addeaaaaaaaaasd _isttrrew = right_handd ht__hhad__fgge fingeri fingerdds_d dist left hitt_hand_diate dist.rev left_hand ringeedddddddnndddddee
# Total reuard ia: posstion disfence orientation aligneent + action regularization + success bonus
fall penally
up_rew torch. zeros, seros_like "rright_bandddittt
up_rew (rire biigghhhbbd__nneeeeeee hand_finger_diie 0.7,,
torch. uhere(Left_ wtw._aanndddieedddeeeee 0.7,
0.5 torch hot..nttttceeeehaeleee..pp keltle__pout..po, p-2, dim--1) up_rex), up_rew)
up_rev rightt_handddiet
noft__aad_diee_rre
resets shera(tuckat_hhand hanickatthanll.pofff
0.2, torch. (ieeeatttr))) resat_buf)
Find out which enue hit the goal and spdate successes count
torch. where (successes =-
torch (torch, www.oobbuket.taaaaeee ones_like (successs) successos), successes)
kettle, kettle_epout.ppo, epout._pos, p-2, dim--1) < 0.05,
reaets torch, where progress_bbu but epx_epiaode_ torch.onea, (ike__sie reseta)
goal_resets torch zeros_ sarch:.ero__aikecreeeet (ike((reeetts)
sans_suceesses torch where > successes resets, srseecttivesucceesee mean() geal_reach vhoreetorch. sorm(bucckt_hanccco.oo kettle_spout_pel pr2, din--1) 0.05, Barch.onea_likamiii torch. zeroa_
heuristiic.evard, teek_reward resard, reach retura meuriatic_revard, task_revard, reaeté, gese_resete, progrese_but. buf,
43

---

Bi-DexHands: ShadowHandPen
def computa__hare haw.daaadddddddaae
rev_buf, reset_buf, reset_goal_but progress,buf, successes, consecutive_ max maz_episode_ lengghh: flost, object_pos, object_zot, target_poe, target_rot, p w led.e_eanndl.por.
hand__and__ps, right_hand_pos,
hanht_hand_tf_pa right, ha right_hand_rf.pon
fight__had______0 right_hand,th.pa..
left_hand_ff _apo_ laft_hand__f_ppo loft_hand_rf.ppe haft_hand_ haft_hand_th_ppr. diat_reeard__call
rot_revard_scale: float, rot_epa: float, actione, priion_pecally_ecllee fleat,
success_tolerance float, reach_goal_ benus: fall pelllpen float, max_consecutivooaacccesss int, av av_factor: fleat, ignore_a_rot: bool
fall_dist: flost,
Distance from the hand the object right_hand www_hann__innnn fingrr_nnarr_atttt Chorch.norm(panrrtt htt.pp__rigtthhanle.ppe handle_po right_ hat_t_and__tppp, p-2, dim--1) torch.norn rorn
right,hand_ng pomd__f_poo. sos,, p-2, din--1)
torch moru(pen_ right, handle_poe rightthand_rff hand_rf_poe. p-2, dim--1) +
rightt_aand_ pos, p=2, dim--1)
torch, parch.nornnpar aporn(een_rtppe rog right_
p-2, dim--1))
hand fingerfiirrrddie dist (torch. het_ppp__oft.bandeee.
left_ pos, p-2, din--1) torch. porch.norn(paa) (prn la hafd__andle..po left_hand__d p=2, dim--1)
torch.norn poennppn_ieft.... jewfeettaadee.orr haft__aad_.rfppp p=2, dine-1)
Left_hand_11f 1f, p-2, diu--1)
torch, norn. corm(pen hawdeee_adle_ppr
hand_aandtta pos, p=2, dim--1))
hand_ right_hand hand hiadhdand________aadaaaaaaaaaaad _isttree torch. 6xp = right_hand_ ringer _diat)
left hand,dditt.ree torch. exrche faad___e____ [inger_ditt)
# Total reuard is: posstion distance orsentation aligneent + action regularization success bonus fall penally
up_rew terch. zeros, zeror_like (right, frright_bandddittt. band_dist__eee)
up_rew (righ_trigt_bbndd fand_finger_dist 0.75,
torch. (loft__le__aadd www._fineerrdiee < 0.75,
torch norm(pen, hitpppee_rrhhh_adddeeeee.e. up_rew), up_res)
htn_leef__haddl..ppp p-2, din--1) 0.8,
up_rex begght_handddiet.reee
resets (right, sand_diat_rex 0, torch, like__ikerresst_t_o reset_bur) bur)
resets torch. where (rreerrght__aann band_ finger, dist >- 1.5, torch, ones oaee_llkeeette (reeett)))), resets)
resets torch, where where(left. (left_hend hendd_fingerdddee finger_dist >- 1.5, torch, ones, onee_ (ike_ee)))))) resets)
Find out which enve hst the goal and update successes cound (successee torch. share (torch. fiiippnn_righthhadllle. 1.5, torch. mee__1ieeesuceeee
pen_left_hane haft_haaddo.pos, p-2, dim--1)
auccessea), successes)
resets torch, where where (progeesere max_episode snx_episode__egttaaa
resets)
goal_resets torch, zeroe likee
Lans_successes torch. share(reseta successea resets, consecutiva_auccceest geal_ torch where(torch, norm(pen, right pen_left_ handlee_aa p-2, din--1) 1.5,
torch ones (succoss))))) torch. lertrroooo__kaeeeeeeee scros__iicc(ucccosseee))
heuristic_revard, task_reward
goal_reach
heuriatic_reward, task_revard,
geal_resets, progress_buf, buf, successes, cons_successes
44

---

Bi-DexHands: ShadhowHandPshhhBlooo
def computa__hare www__aaandddddddee
rev_buf, reset_buf, reset_goal_but progrees_buf, succegees, consecutive_euccessse. max_episode_ length: flost, object_pos, object_rot, left_target_poee left_ targ__taree__ott wa saaht_target_por, right_targes targe_targ,t_ot.. block_ laft_ right_hand,pos,
bock__eft_hhnclle
hagtt_aand_tf_po.. hanht_hand__rt pos, haght__hnd_rr_ppa right_ haat__han___________ right_hand,th.pas,
left_hand ff _fpos haft_hand__f_pos left_hand_rf.pos hat__hand_lf.po.. loft_hand_th top Hiat_reward__cakk actione, paiion_peally_ecall.. fleat,
rot_revard_scale: float, rot_epa: float,
saccess_tolerance float, reach_goal, benus: fall. penalty: float, pax_cooneccutvvaauuccesss int, av av_factor: fleat, ignore_a_rot: bool
fall_dist: flost,
Distance from the hand the object left._gall gaal--oll_diss torch. sergclatt._argett.oo black, laft, handtetaaaddeppppla p=2, dim--1) blook.right p-2, din--1)
rogbt.goal_dist torcb.nommtrigii..areet..
right_ torch, morn(block,rightt www.hrrrhhhhhheeeppper - right_hand_nt. no_pos, p=2, dim--1)
(torch. rertbblock.iggh.hbaade handdt_aandl_ppp right_ foppo, p-2, dim
torch norn heanmboloo..rihhhheeeee..
right_ pos, p=2, din--1)
tarch. caan(bboc..right_hanndd
hand_ p-2, dim--1)
torch nornnbl bralocckriggthbeee haggtt_annde_per right, Hight_hand__te poa, p=2, dim--1))
left hand finger_ (torch. norm(block_eed) baftleandeeepp left bett_band_ff_poo p-2, dim--1)
+ torch norn(block_ _ handle_pos left sfftt_han_______ _os,, p=2, din--1)
torch. norm(block_lfftt hos left_hand_rr.pp p-2, dim--1)
terch. torch tor
suft_hand_if_pat p=2, dim=-1)
heft_hand,_thppo th.po, p-2, din--1))
right_hand hand, hant_dist. dist_rree ree 1.2-1-right, her-1-iighthhand__fre finger_ierrdde left hand dist_rev 1.2-1eleft_ htr_efttaan_ffineedddiii
ds: position
alignnent
regularisation success
fall penally.
up_rew torch, torch. zeroe, zeroe (rroe_ baadddie_eedddeeerree
up_rew 5+left.goal_dist Beright.goal_ddat
right_hande dist_rees beft__han_ddistrree up_rew
resets (rightright_tiie haigggttaaand_fngeerr finger_nnnsssr 1.2, torch. onea_ like__ttker reset_buf)
reaets torch where (ledteeeeedddiineeeeeeee
>- 1.2, torch. mie__lik(reeset)) resets)
Find out which hit successes torch. vhere (eucceesee -
and update
count
chere(torch, (leett_gool__dii tarch. (torcht (trgght.aaaatighttttoo. abt(tggtt.ggaal goal_ddstt___ <= 0.1, torch,
corch oneo like__ika (succossess
wrrch.ones_1i 1ie___1te 0.5), auccessea), succesaes)
resets torch, where (pore(progrese_ber max_ max_episode_lonttaa orrch.oaee_iikereeee resets)
goal_resets torch, zeroe leieel
cons_successes torch. share(reseta 0, successea reseta, consecutiva .nean()
geal_reach 0.5 (torch. sbere aba (ealeett_geal,dist)) 0.1,
terch, like (successee), torch, zeroe_ 2eros__iik
torch.there (bt.ooth..bbrrrigee..o Brrigghtggal_dittt tarch. zecos (eroos_
0.1,
heuristic_revatte task_reward reward, goal_reach return heuristic_rovard, task revard, resets, goal_resets, progress_but, successes, cons_successes
45

---

Bi-DexHands: Shadaw
def computa__hate hww.daaddddddddddd
rev_buf, reset_buf, reset_goal_buf, progrees_buf, successes, consecutive.auceet maz_episode, lengghh: a aepect_anthher_se _rot, Saget__nntherr_pe Sarget_anotber.to clat__eeerr__cct float, rot_reeard_scalee float, rot_eps: float, actions, setior.penalty.ecale:: fleat,
object_pos, object_roo, targot__poe, target_rot, object_another_po
auccess_tolerance float, reach_goal, laallgggaaaaaaaaaae boaus: float, fall_dist: float, fall.penalty: float, aak_consecutivo_uuccoo
int,
float, ignore_s_rot: bool
from goal_dist torch. nomm(taggt...p. object_pos, p-2, din--1)
object
ignore_2_rott success_tolerrace 2.0 aucess__tolerance tolerance
goal_
norm (taretarget,t..mmc another.o abject,.notheerppo. p-2, din--1)
if ignore_2_roe 22_ot;
success_tolerance
success_tolerance
Orsentation alignnent for the cube ss and geat cube quat_diff
caat_conjgaatetaargee.. rot_ 2.0 torch. cesiatttrch.caap(ttrrrr.. norn (oratquu__diffff 0:31, p-2, dim=-1), max=1.0))
quat_another diff quat_mul ansther_roo rot, cagjocaaaatttttargee.. antteerr_ootheerroo) rot_ansther__ee _ist max=1.0))
asin(torch. (torcht aam_qqua_annther.... 0:3], p=2, dim--1),
dist_rrv goal_dist diat_reward_ecale goal_another_ dist_ravvar_accae ret_rex 1.0/ aborchhbbbbrroot (bstt (rot,dist) rot_epe) rot_revard_scale (torch abe rot_ rot_revard_scan
action_ ponalty
cantioiions 2, dim--1)
Total reward ie: position distance orsentation alignsent action regularizstion success bonus fall penalty revard dist_rex rot_rev action_penalty action.pemally.ecall
Find out which enus hit the goal and update successes count geal_resets torch. shere (torch. (bott_.distt) 0.1, 1iea_1ika(reaat goaltttt_ooabbr)), reset_goal_ geal_reseta torch. where (torch. abe (rot_,rothhee reset_goal__op)
0.1,, torch ones,, like (resett.gooll.but)
successes successes
goal_reses resete
Success bonus: orientation is withis 'success, tolerance" of goal orientation rezard torch, (gorl rresta
revard reach.goal_ recard)
Fall penaliy: distance the goal is larger than threashold resard torch. shere 2] <- 0.2, resard fall_penalty, reward) revard torch, where(object_auotteeee
2] <- 0.2, recard fall_penalty, reward)
# Check sav termination condstions, inctading saaceus success sueber reaets torch, where (oere 2] 0.2, torch. ones miee_likereese buf), reaet_bur)
torch, where sojereeoebeettteeer.....
2] 0.2, torch (iee(tett_ resets)
# Reset progress buffer on goal esus if maz, consecuiive. successes > progress_bit torch. shere(torch..rmm (rott + torch zeros_ (prooorr buf), progress_buf) recets torch where (successes maz, munsecutive_succeeee torch ones like(reseta),, resete)
success_tolerance
torch. where (progrerrrseeeeee buf >- sax_episode__engee length, torch, liee__ike(resets)) resets)
Apply penalty successes 0:
reaching goat
if max_consseurive__nue
torch. (prro(progress >= max_cptssod__annttt resard fall_ponalty, revard)
Run regets torch.eurreseed (resete)
fimiched_ waw.dede_cccceccsseeeeeeeeee torch. sun(ccccesses
fleat())
ww_tactor=iinisbed.co hat.aaaae...ooaeeocccoeeeeeeeeed (1.0 consetttttcnseetttttt..oo successes, consecutive, sounuttive_cccssess))))
shere(nun_reaets
geal_reach 0.5 (tarch. where aba 0.1, torch.ones_ stacchoms__.ikerreee.ooo......
torch, zeroe lel.lkeereeee pereset_ggaalbbutt
torch. shere torch.
0.1, wrrch.onee_likeeeee aet____iierreeet_ggol_l..
hearistic revenittcceeerdd task_reward resard, goal_react return hauriat:c_ravvrdd, task_reuard, resets,
rese_r, progress_bul,
46

---

Bi-DexHands: ShadowHandSeisoor
def computa__hare
samputa_haad_raee [
rev_buf, reset_buf, reset_goal_but, progress_buf, successes, consecutive_succeeett maz_episode, _engghh: flost, object_pos, object_rot, target_poe, target_rot, ecissore_right, wa scissors_left. hiasoors_eftthhmmle.pooo object_dof_pos.
laft__andd_pas, right_hand_por. hight_hand,tf.pon
hight_hann_rf_poo
fightthaad__f___0 right_land,th.paa.
left_hand _ppos, haft_hand__f_poo. boft_hand_rf.pos laft_hand_1f.pos, laft_handdth.por. siat_reward__cake actione, peiion.peaall_.eallee flost, success_tolerance float, reach_goal_ benus: fall_dist: flost, fall penalyyy float, aox_coonecutivo_auucessss int, av av_fastor: fleat, ignore_z_rot: bool
rot_revard_scale:
rot_epa: float,
Distance from the hand the object right_hand www_hann__figee norn(ecissore_ handle. poa right,band, aand__f_ppo,, pos,, p-2, din--1)
(arrrhhnnmmsccsssssrrrrr. handle_pos rightt_had__tt hand_aad_dopp p-2, dim=-1)
torch norm(scissore, rightt, handle_pos po rightthand, hand poee, p-2, dim--1)
righttbband_le
pos, po p=2, dim--1)
torch, norn (sclasorrsss waciaaaorrrrrr_hhaaa__eee handle_pss toc right_hand had__ann___hhh thopa, p-2, dim--1))
land_ finger_distt
pos left_ pop, p-2, din--1) torch (sciiscssr,, laarrsllla....ee heft__andde_pppp handle_pos left_hand__f _ee, p=2, dim--1)
(torch. http:csssorr_eott.hhoaoo.e
torch norn norn (sciasooraaieff. hafft_aandee_pr
baft__and_rf_pom p=2, dim--1)
left_ po,e p-2, din--11
torch norm (scissor, seaeissoorsseettt.ooooee handle_pos pomd left_hand_th, poo, p=2, dim--1))
right_hand haahhanddddd _isttrre = right_handd ht__hhad__fggg fingeri fingerddstt dist left hand_dist__eee hett__han___ngee
# Total reuard is: posstion disfence orientation aligneent action regularization success bonus
fall penally
up_rew torch. zeros, paros_like(e (rightihht__aaat baat___andddi__eeee
up_rew (righ bandd_aandddddddeeeeeeee 0.7,
torch, harr((ort_hann_fiiiee fiager_ditt 0.7,
(0.59 object_dof_pos[:: 0]) 5, up_rew), up_rev)
revard 2 up_rew right, hand_aiittdeseee left_ hand dist_rew
resets torch. where there(up_ree rep_rev -0.6, torch. ones (iseeetttereeeee_eee 1.75, torch. (skaaratts)) resats)
reset_but)
shere (rightright_ banggtt findddffnger_di >- 1.75, onee_ (iee_likereestt)) resets)
reaets torch. where Sesleef_hand_rnnneeee
Find out which hit resets torch, where (progeeroggee__buu >- max_episode__engee length, torch onee (aee_]iie(reeets))) resets)
and update
count
successes torch. wwwreeauccesseaceeeeeee =
torch shere(chfect_ddf.poo.. successee)
-0.3, torch. (aee(ccss)) successes),
geal_resets torch. saros_like(resats)
cons_successes (ajjsabbjst_ddof_p _oo______p_ 0] -0.3,
0,
resets, consocutiva_auccert mean()
goal_rean reach torch. torch. ones onea_like
heuristic_revary task_reward beee_likesuuseee (successes), torch.2eros, seroa_1iie((uuceesess)))
gosl_reach
leuristic_revard task_revard,
bo goal_resets, progress_buf, successes, cons_ conc_ssce
47

---

Bi-DexHands: ShadowHandSwingCup
def computa__hare
saapputt_hand__aueeeeee r
rev_buf, reset_buf, reset_goal_but progress,buf, successes, consecutive. maz_episode_ lennghh: flost, object_pos, object_zot, target_poe, target_rot, ciap w cup_left_ _eff__annle_por.
laft_hand_ppa. right_hand_pos, hight_band_tf_pp right, mf right
hight__andd_1.... right_hand,th.paa.
baft_hand_tf_pp laft_hand_nf_ppo left_hand_rf.ppe haft_hand_ haft_hand_thhpor..
diat_reward_scall actione, priion_pecally_ecllee float,
rot_revard_scale: float, rot_epa: float,
success_tolerance float, reach_geal_ benus: fall_dist: flost, fall penalyyy float, sss_connecuttie_ecccceess int, av av_factor: fleat, ignore_a_rot: bool
Distance from the hand the object right_hand www_hann__iaee finn__finar__itt torch.
(torch.normccup right__riht_haadd
right_ ha___atdd___p., p-2, dim--1) right,hand,af, pf pos,, p-2, din--1)
torch morm(cup_ right, handle_poe right_hand_rff hand_rf_pos. p-2, dim--1) +
right_handdee.os right_handd_ff.
pos, pop,, p=2, dim--1)
torch, norchhnornncau
(cup_righttttt...... right_handle_pos right_ hand_th_pps p-2, dim--1))
left_ fangedfiirrrddeee (torch. htt.:cuppeftbbadiiiie
left_ po0, p-2, din--1) torch. (erch.. lottt__p___t_aaaaeee_ee han.thaaddee.ppp left_hand__d p=2, dim--1)
torch.norn
haft__hand_f_ppp p=z, dime-1)
porm(cup_leff he heft_hand_1ee po, p-2, diu--1)
torch. corn hendllandeeee_or hand_aandtta pos, p=2, dim--1))
Orientation alégnsent for the cube so hand and goal cube quft_diff quat__ulloojett__e (object_ rot_dist 2.0 torch asinntorch.....o...... ctapeccc.caaapptorrriacc nerrechh.oorngggoo (quat_difff 0:3], p=2, dim=-1), max=1.0))
right, hand left hand_ hand_aanndddedeeeeeeeeae_iiiiiie dist_rev
right_ win.ffiigeedee diet
shand_finger
rot_rew 1.0/ 1.0/((trrc (torch. abs(rot.diat) rot_eps) rot_revard_acale
Total reward posstion
alégnsent regularísation
+ fall penally
up_rew zeroe_ zeros_like(rot_rog
up_rew torch, where sandd_finge_ddde dist 0.4,
sarch.shhraalaft fingefiinee__daa 0.4, rot_reN, up_rev), up_rew)
revard Brggtt_band_dist_teee left_hand_dist_ree
whore(object_poaf::
0.3, ones, (ike(resetttbbu))) reset_buf)
Find which enua gool and update auccesses coant
successes torch. vhere (eucceeses --
torch. chere(rot_dist 0.765, torch, cnes, wew.:llkkooscccccc successes), successes)
(progresz_ buf max_ max_eepiodd__ongtha ones, (reesests)) resets)
goal_reseia torch.2eros 2eroa, aaroa_like Sere__lkkeeeeeete (resets)
cns__successes torch where(resets > successes resets, sasecutiva_auccesee mean)
geal_reach torch. vhard 0.786, torch.ones_ onas__ like (xucceessee))
souristicreeaad, task_reward reward, goal_reach reach return heuristic_rovard, task_revars resets, goal_resets, progress_buf, successes, cons_successes
48

---

Bi-DexHands: ShadowHandSwich
def computa__hare haa___anddd_ddddde
rev_buf, reset_buf, reset_goal_but progrees.buf, successes, consecutive.succeest max_episode__er _ennghh: flost, object_pos, object_zot, target_poe, target_rot, righth w seitch_left _eft_haaddde_pp.
hand, right_hand_pot,
haght_hand_tfpp right, mf right,
hight_hand__1__pp right_hand,th.pas,
left_hand_ hfpos, left_hand_nf_ppr loft_hand_rf.ppe haft_hand_1f.po,,, baft_hand_th.por,, diat_reward_scake
rot_revard_scale: float, rot_epa: float,
actione, priion_poaall__ecllee fleat, success_tolerancer float, reach_goal fall, palllpenalty: float, max_consecutivv_accceees int, av av_fastor: fleat, ignore_z_rot: bool
fall_dist: flost,
Distance from the hand the object right_hand www_hannnd_aaaa sann__fingr_daaa torch. wornlesitch..ightt.tt (avichhh_rr right, hig.errtttttthheee.ppeeeee_ handle_poa rightt_had tf p=2, dim--1)
(torch.norm(svitenn
po. right, pos, p-2, ditt--1)
torch norm(sxitck higgithd_iighthhade.......
right. biiht__aan__rt band_rf_poo p=2, dim--1)
right, hand, pos, p-2, din--1)
torch, sorch..ornnsswteee (switchh,rsgtt_hnndld handle_pon right_ haatt_han__hhppp tho p=2, dim--1)) left_ ff.oo, p-2, dim--1) torch. (wwttntccchcheeeeee htt.taaddedtttbddeee_eeee baandle_pos left_band_nf___p___ p-2, dim--1)
left_ www.baanndddeeeddeeeee finter_dist (torch. boomssttchh.lefffhalll
torch. norn (switch_ htttt
hand_ p=2, din=-1) +
norn(evitch beft_bandle_ppe pos
beft__bad_11..pp 10,p0, p-2, din--1)
torch, norm norm (switch, left htn.hileee left_ baft_hand_th_pos p=2, din--1))
right_hand haahhanddddd _isttree = right_handd ht__hand__f finger, finger_dstt dist left hirt_hhad_dddaae dist.rev left_hand ringerr
# Total reuard ia: posstion distance orsentation aligneent action regularization success bonus fall penally
up_rew torch. zeros, tre baadghht_baddddttt_t_te
up_rew (1.4- rigttthh..rgtt.haaaaa
het.dlleeetthheeeeeeteeee 2]))
revard rightt hadt_distdddddrd
hett__hen__ddi___t_ up_rew
reaets torch. (here ban_gtt__eddddd_eeeee <- 0, torch onee Like_llkkeree (reeeteett,but) reset_bur)
# Find out which enus hit the goal and update successes count (suceaauss successes torch. where ( rightthh haagttt__endlleeeeleeea pos(: 2] switch, het_illeeetaadee_eo__. 2])
0.05, torch, ones oaes_ sue__iii((ssseeeses))))))) successes), successes)
resets torch, where where (progreee.r buf >- max_eepeode__eggeee length, torch onee, likelllieeeeeer resets)
geal_ moll_resets torch. zeres_ saros_likarresatt)
successes torch (hereeresete > successes
consecutive. successes) mean()
geal_reach reach torch, 4 (svitch_right, (eviich_rigtthhhadlop..... basdle 2] svitch_left. suitch_ 2)) >- 0.06,
torch, onee lile (successes), torch zeros saxos_lite like (successee) resard, goal_reach
heuristic_revard,
roeaard task_revard, resets,
gpal_resets, progress_bie buf, successes, cons_successn
49

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: This paper aims to explore alternatives for improving task performance in finite data settings using heuristic signals. Our experiments robotic locomotion, helicopter, and manipulation tasks demonstrate that this method consistently improves performance, regardless of the general effectiveness of the heuristic signals. We are confident that our abstract and introduction sections accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of our approach are illustrated in Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
50

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
We have shown our derivation details and limitations in Section A.1 and
Section
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
We have provided detailed derivation and implementation descriptions (includ- ing the simulation environments, hyperparameters, and reward definitions for training and evaluations) in our Appendix section. Additionally, we have also provided our source code in our Supplementary Materials.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
51

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We have provided detailed implementation descriptions (including the simula- tion environments, hyperparameters, and reward definitions for training and evaluations) in Appendix section. Additionally, have also provided source code in our Supple- mentary Materials. The adopted simulation environments are all well-known and available online.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurlPS code and data submission guidelines (https: / wwtpcccppppcccpeeeeeeeec.eeeeeee gww.d//////eeeeeeeeo SrodeSubmissionPPP
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have provided detailed implementation descriptions (including the simula- tion environments, hyperparameters, and reward definitions for training and evaluations) in the experiment and Appendix sections. Additionally, we have also provided our source code in Supplementary Materials, including all the training and environment configurations.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
52

---

Answer: [Yes]
Justification: All of our experimental results were obtained by random seeds. We have provided the mean and standard deviation for our experimental results.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [No]
Justification: But each training procedure can be performed on single GeForce RTX 2080 Ti device. The required computational resources for all the simulation benchmarks are listed on their respective websites.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the lins?
NeurIPS Code of Ethics https: : http///eurrp..p//
Answer: [Yes]
Justification: We have carefully examined the ethical guidelines and verified that our work fully adheres to all the principles and requirements.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
53

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper does not discuss both potential positive societal impacts and negative societal impacts of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymenn of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper has no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
54

---

Throughout this paper, we have provided proper citations and references for all utilized repositories, benchmark simulations, and models/algorihms uphold transparency and ensure appropriate attribution.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We will furnish comprehensive documentation our released code, elu- cidating its usage and providing information about the original source. Additionally, we have ensured that any code modificd from external sources is subject to licenses that permit modification and redistribution.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: No, but we engaged 12 participants in devising reward functions as part of the experiments detailed in Section 4.2.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
55

---

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
56


Reviews:
[{"review_id": "swxEQQvyrG", "reviewer": "Reviewer_inih", "summary": "The paper presents a novel approach to improve reinforcement learning (RL) by incorporating heuristic signals. The authors propose a constrained optimization method that uses heuristic policies as references to ensure that the learned policies outperform heuristic policies on the exact task objective. The method, named Heuristic-Enhanced Policy Optimization (HEPO), was tested on various robotic tasks, demonstrating consistent performance improvements over standard heuristic methods.", "questions": "- Please check weaknesses.\n- How does HEPO perform with other deep RL algorithms, such as Soft Actor-Critic (SAC)?\n- What specific heuristics are used in the progressing tasks, and how are the rewards sparse/delayed?\n- In figure3: Is the T-only ablation equivalent to the EIPO algorithm or a modified version of HEPO?\n- Line 129 is missing a reference to the application of PPO in robotics with heuristic rewards.", "limitations": "The authors address the limitations and potential negative impacts in their paper.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper introduces a simple method to improve reinforcement learning (RL) algorithms using heuristic rewards, by shifting focus from ensuring optimal policy invariance to simply outperforming heuristic policies.\n- The method is validated on various environments, showing consistent performance improvements.\n- The paper is clearly written.", "weaknesses": "- The claim that the contribution is an add-on to existing deep RL algorithms is not fully substantiated, as the authors only evaluate HEPO using Proximal Policy Optimization (PPO). Demonstrating HEPO's performance with other deep RL algorithms, such as Soft Actor-Critic (SAC), would help validate this claim.\n- The method is very similar to Extrinsic-Intrinsic Policy Optimization (EIPO), with the main difference being the reference policy used. The authors should provide more details on this difference, including the number of additional hyperparameters introduced in EIPO and a comparison of its complexity in terms of time with HEPO.\n- The benchmarks used in this paper differ from those in previous works like EIPO [6] and HuRL [7]. The authors should address how HEPO would perform on those benchmarks to ensure a fair comparison.\n- The paper does not include challenging tasks such as Montezuma's Revenge.\n- It is unclear whether the reward functions and heuristics used in the experiments are designed by the authors or sourced from ISAAC and BI-DEX benchmarks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for appreciating our idea of shifting the focus of optimal policy invariance, as well as our extensive evaluation and clear writing. We address the remaining comments below:\n\n> Q1: Demonstrating HEPO's performance with other deep RL algorithms, such as Soft Actor-Critic (SAC), would help validate this claim.\n> \n\nWe integrated HEPO into HuRL's SAC codebase and found that HEPO outperforms SAC and matches HuRL's performance on the most challenging task (see Figure 7(b) in the rebuttal PDF). HuRL used optimized hyperparameters for this task, as reported in its paper, while HEPO used the same hyperparameters as in Section 4.1 of our manuscript. Other tasks in HuRL's paper require pre-trained value functions as heuristic reward functions, but the checkpoints were unavailable online.\n\n> Q2: The method is very similar to Extrinsic-Intrinsic Policy Optimization (EIPO), with the main difference being the reference policy used. The authors should provide more details on this difference, including the number of additional hyperparameters introduced in EIPO and a comparison of its complexity in terms of time with HEPO.\n> \n\nWe acknowledged the similarities between EIPO and HEPO and discussed the differences in Section 3.3 of our manuscript. The difference in the choice of reference policy doesn’t add more hyperparameters or time complexity.\n\n**Hyperparameters:** Both HEPO and EIPO share the same number of  hyperparameters: 1. Initial value of the Lagrangian multiplier and 2. Learning rate of the Lagrangian multiplier.\n\n**Time complexity:** It’s the same for both methods:\n\n- **Trajectory rollout:** Both rollout $B$ trajectories (number of parallel environments), involving the same number of forward passes on the policy networks. Thus, both have the same time complexity in this stage.\n- **Policy update:** Both train two policies on the collected trajectories at each iteration, so HEPO does not introduce additional complexity during training.\n\n> Q3: The benchmarks used in this paper differ from those in previous works like EIPO [6] and HuRL [7]. The authors should address how HEPO would perform on those benchmarks to ensure a fair comparison. The paper does not include challenging tasks such as Montezuma's Revenge.\n> \n\nHEPO outperformed both HuRL and EIPO on the most challenging tasks reported in their papers. For HuRL, see Q1. For EIPO, we tested both methods on Montezuma's Revenge using RND exploration bonuses (also used in EIPO's paper), which are crucial for good performance on this task. Figure 7(a) in the rebuttal PDF shows that HEPO significantly outperforms EIPO, with no overlap in their confidence intervals at the end of training. Additionally, HEPO achieves the same performance as training PPO with RND bonuses at convergence (2 billion frames), as reported in RND's paper, while using only 20% of the training data, demonstrating significantly improved sample efficiency.\n\n> Q4: It is unclear whether the reward functions and heuristics used in the experiments are designed by the authors or sourced from ISAAC and BI-DEX benchmarks.\n> \n\nAs described at the beginning of Section 4 in our manuscript, our heuristic reward functions are all based on the source code from Isaac Gym [1] and Bidexterous Manipulation [2]. More details can be found in their respective papers and repositories.\n\n[1] Makoviychuk, Viktor, et al. \"Isaac gym: High performance gpu-based physics simulation for robot learning.\" arXiv preprint arXiv:2108.10470 (2021).\n\n[2] Chen, Yuanpei, et al. \"Towards human-level bimanual dexterous manipulation with reinforcement learning.\" Advances in Neural Information Processing Systems 35 (2022): 5150-5163.\n\n\n> Q5: What specific heuristics are used in the progressing tasks, and how are the rewards sparse/delayed?\n> \n\nIn progressing tasks, large rewards can be sparse or delayed because making progress often requires actions that don't immediately advance the robot. For example, in Humanoid tasks, the robot must balance itself to move quickly and get large rewards. However, achieving balance requires a sequence of actions, which delays the large reward associated with fast movement.\n\n> Q6: In figure3: Is the T-only ablation equivalent to the EIPO algorithm or a modified version of HEPO?\n> \n\n\"HEPO ($\\pi_{\\text{ref}}$ = T-only)\" is an ablated version of HEPO whose reference policy is trained solely with task rewards. This differs from EIPO only in the trajectory collection method, as shown in Figure 3(b). To clarify, \"T-only\" should be changed to \"J-only,\" indicating the policy is trained only with task rewards. We will correct this typo in the next manuscript.\n\n> Q7: Line 129 is missing a reference to the application of PPO in robotics with heuristic rewards.\n> \n\nThanks for pointing this out. We plan to add the following references on robotics [1, 2].\n\n[1] Chen, Tao, et al. \"Visual dexterity: In-hand reorientation of novel and complex object shapes.\" *Science Robotics* 8.84 (2023)\n\n[2] Lee, Joonho, et al. \"Learning quadrupedal locomotion over challenging terrain.\" *Science robotics* 5.47 (2020)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AlUYJfiibc", "reviewer": "Reviewer_15cn", "summary": "This paper introduces a constrained optimization approach for reinforcement learning, ensuring that the learned policy outperforms or matches the performance of policies trained solely with heuristic rewards. In simulations, the proposed HEPO method consistently delivers superior task performance across various tasks, demonstrating improved returns in robotic locomotion, helicopter, and manipulation tasks.", "questions": "Could you please explain how the heuristic is obtained in the implementation? Additionally, it would be helpful to see the computational cost of solving the constrained optimization problem. Is it possible to include a comparison of your method with reward shaping and policy cloning methods? For equation 5, when rearranging equation 4, could you clarify where the term J(pi_H) goes?", "limitations": "While the HEPO method shows empirical success, it lacks a thorough theoretical analysis, including a formal guarantee on the convergence rate and the effects of heuristic selection and the alpha parameter on performance. Additionally, the computational cost of solving the constrained optimization problem is not addressed, which could impact the feasibility of the approach in large-scale or real-time applications. Furthermore, the study does not compare HEPO with other reinforcement learning techniques such as reward shaping and policy cloning, which would provide a more comprehensive evaluation of its effectiveness.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper is straightforward and easy to follow, with a clearly stated motivation. It introduces an innovative constrained optimization approach that effectively leverages heuristic signals to enhance task performance in reinforcement learning. The effectiveness of the HEPO method is demonstrated through its consistent superiority across diverse tasks such as robotic locomotion, helicopter control, and manipulation, even with limited training data. Additionally, the proposed method simplifies the reward design process by dynamically balancing heuristic and task rewards without the need for manual tuning, showing significant improvements over traditional heuristic-only and mixed reward approaches.", "weaknesses": "The paper would benefit from a more thorough theoretical analysis. Including a theoretical guarantee on the convergence rate, an examination of how the selection of heuristics impacts the HEPO method, and an analysis of the effect of the alpha parameter would enhance the robustness and depth of the study.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We're thrilled that the reviewer love our paper and appreciate their compliments on the clarity of our presentation, the novelty of our proposed method, and the thoroughness of our experimental evaluation. The following are our responses to the remaining comments:\n\n> theoretical guarantee on the convergence rate\n> \n\nHEPO can be considered a constrained RL problem, where the constraint is that the policy’s expected return must improve over the reference policy (i.e., $J(\\pi) \\geq J(\\pi_{\\text{ref}})$). Therefore, existing theoretical analyses of convergence rates (https://arxiv.org/abs/2407.10775) for constrained RL with expected value as constraints can be applied to HEPO.\n\n> examination of how the selection of heuristics impacts the HEPO method\n> \n\nAt the core of HEPO is training policies by RL with heuristic rewards, which the impact of heuristic selection has been analyzed theoretically in previous work (https://arxiv.org/abs/2210.09579). Empirically, Figure 2 in our manuscript compares HEPO and PPO using different heuristic rewards for the same task reward. The results show that HEPO is less sensitive to heuristic reward selection and outperforms PPO on average. Also, Figure 7(c) in the rebuttal PDF shows that HEPO’s performance scales with the quality of the heuristic (i.e., the performance of training with heuristic rewards only).\n\n> an analysis of the effect of the alpha parameter\n> \n\nIn Appendix B.2.2, we analyzed the effect of the alpha parameter and showed that HEPO is not sensitive to its learning rate.\n\n> Could you please explain how the heuristic is obtained in the implementation?\n> \n\nAs outlined at the beginning of Section 4 in our manuscript, our heuristics are based on the source code from Isaac Gym [1] and Bidexterous Manipulation [2]. Further details are available in their respective papers and repositories. In Section 4.2, we use heuristic reward functions designed by human subjects to evaluate the performance of each method on reward functions of varying quality. More information can be found in Section 4.2.\n\n[1] Makoviychuk, Viktor, et al. \"Isaac gym: High performance gpu-based physics simulation for robot learning.\" *arXiv preprint arXiv:2108.10470* (2021).\n\n[2] Chen, Yuanpei, et al. \"Towards human-level bimanual dexterous manipulation with reinforcement learning.\" *Advances in Neural Information Processing Systems* 35 (2022): 5150-5163.\n\n> Additionally, it would be helpful to see the computational cost of solving the constrained optimization problem.\n> \n\nUpdating the constrained problem in HEPO requires only an additional memory allocation to store the Lagrangian multiplier $\\alpha$. After finishing one epoch of policy updates, we perform a single gradient step to update $\\alpha$, resulting in only a minor computational overhead.\n\n> Is it possible to include a comparison of your method with reward shaping and policy cloning methods?\n> \n\nPlease note that Section 4.1 of our manuscript provided a comparison of well-known reward shaping methods (PBRS [1] and HuRL [2]) and showed that HEPO outperforms both methods.\n\nRegarding policy cloning, we kindly ask the reviewers to clarify the definition of policy cloning. If it refers to behavior cloning that is trained to imitate expert demonstration, this might be outside the scope of our work, as we do not assume a dataset of expert demonstrations. Instead, our focus is on improving RL algorithms' performance when trained with heuristic reward functions.\n\n[1] Ng, Andrew Y., Daishi Harada, and Stuart Russell. \"Policy invariance under reward transformations: Theory and application to reward shaping.\" *Icml*. Vol. 99. 1999.\n\n[2] Cheng, Ching-An, Andrey Kolobov, and Adith Swaminathan. \"Heuristic-guided reinforcement learning.\" *Advances in Neural Information Processing Systems* 34 (2021): 13550-13563.\n\n> For equation 5, when rearranging equation 4, could you clarify where the term J(pi_H) goes?\n> \n\n$J(\\pi_H)$ can be considered a constant for the optimization in Equation 5 since it doesn’t influence the gradient of policy $\\pi$. Therefore, it can be omitted in Equation 5.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wL0J2hfVl5", "reviewer": "Reviewer_P4zd", "summary": "In this paper, the authors address the problem of reward design in reinforcement learning. Specifically, it is common practice to add heuristics in the reward to help the training and a lot of manual engineering is needed to balance this heuristic and the main reward. They propose to modify the standard RL objective to a constrained optimization objective in order to incorporate heuristics in the reward function in a principled way. \n\nTheir formulation consists in optimizing both the task reward and the heuristic reward while constraining the policy to have a better performance in the task reward compared to a policy that is solely maximizing the heuristic reward. This constraint ensures that the policy is not only exploiting the heuristic but also solves the task.\n\nThe method is evaluated on a set of robotic control tasks where heuristics are already provided.\n\nI acknowledge reading the authors rebuttal and updated my score.", "questions": "-\tWhy doesn’t the algorithm need more data since you need to collect data with both policies?\n-\tEquation (7) and (8) introduce an off-policy aspect to the algorithm, is convergence still guaranteed?\n-\tWhy use J_random in the formula to measure performance? Why not use a percentage of improvement over $J_{H-only}$?\n-\tWhat happened with H4-9-10-12 in figure 2?\n-\tOn the same problem with different heuristic, how would the performance of HEPO look like depending on the asymptotic performance of $\\pi_H$", "limitations": "The current algorithm seems limited to on policy methods. It also applies to problems where we can make a clear distinction between task reward and heuristic rewards.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "They propose a simple formalization of the problem of reward design with adding heuristic to a main reward task. This does not address the general problem of reward alignment, but I think it is a very relevant problem in RL for domains with well defined but sparse rewards.\n\nThe derivation of the algorithm is presented clearly. \n\nThe experiments show that the proposed algorithm outperform the baseline in terms of average return. Except from the missing EIPO, the authors choose relevant baseline to compare against. \n\nThe second experiment of reward designed in the wild is interesting. The author attempt to recreate the process of reward design with human participants. \n\nThe statistical analysis of the experimental results is rigorous. \n\nThe ablation study shows the importance of using the policy trained on the heuristic reward as a reference.", "weaknesses": "The data collection part of the algorithm is not super clear. My understanding is that both policies are used and they collect the same amount of trajectories for each, why would the amount of data be the same and not more? The theoretical implications of this data sharing part should be clarified as well. It is not obvious to me that equation 7 and 8 actually lead to the correct policy improvement step. \n\nNo theoretical analysis of the algorithm is provided. I am not sure that detailed theorem and proofs would be needed but at least some insights or discussion about the convergence. There is a brief mention that it would not converge to the optimal policy in the limitations but no proof is given. \n\nWhy not include EIPO in the first experiment? Overall, there is no direct comparison with EIPO. The authors comment on certain aspects during the ablation study but it would have been even stronger to add a line for EIPO. \n\nIn the second experiment some results appear to be missing which makes me question the validity of table 1 as well. I hope the author can clarify this in the rebuttal.\n\nOne experiment that is missing and could have been interesting is to study the performance of the algorithm depending on the quality of the heuristic. \n\nI suggest merging the related work section and section 3.3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer’s positive feedback on the simplicity of our method, the design and statistical rigor of our experiments, and the clarity of our writing. We have addressed the remaining questions below.\n\n> Why doesn’t the algorithm need more data since you need to collect data with both policies?\n> \n\nEach policy collects half the total number of trajectories for training. Suppose we aim to gather $B$ trajectories per training epoch to update both policies $\\pi$ and $\\pi_{\\text{H}}$. HEPO collects $B/ 2$ trajectories using $\\pi$ and $\\pi_{\\text{H}}$, respectively, and updates both policies with these $B$ trajectories.\n\n> Theoretical properties of Eq 7 & 8\n> \n\nHEPO's policy update objectives for $\\pi$ and $\\pi_H$ (Eq. 7 & 8) are derived from PPO's policy improvement objective. The key difference is that HEPO uses data collected by both $\\pi^{i-1}$ and $\\pi_H^{i-1}$ from the previous epoch $i-1$, while PPO uses data from only one policy $\\pi^{i-1}$. More details are in Appendix A. This does not invalidate PPO's policy improvement objective because applying PPO's clipping trick to these equations (as done in our implementation) ensures that the updated policies remain close, preserving the validity of the PPO update. Consequently, HEPO's convergence can be analyzed by recent theoretical work (https://openreview.net/pdf?id=uznKlCpWjV) on PPO's convergence properties with neural network parameterization. \n\n> Limitation on convergence to optimal policy\n> \n\nWe'd like to elaborate this limitation below: Solving HEPO’s constrained objective in Equation 3 of our manuscript,\n\n$$\n\\max_{\\pi} J(\\pi) + H(\\pi) ~ \\text{subject to} ~ J(\\pi) \\geq J(\\pi_{H}),\n$$\n\ndoesn't guarantee that $\\pi$ will converge to the optimal policy for $J$ since $\\pi$ is maximizing the biased objective $J(\\pi) + H(\\pi)$.\n\nHowever, this issue can be resolved by reshaping the heuristic rewards $H$ using potential-based reward shaping (PBRS) for HEPO. PBRS ensures that $\\text{arg}\\max_\\pi J(\\pi) + H(\\pi) = \\text{arg}\\max_\\pi J(\\pi)$, guaranteeing the modified HEPO’s optimization objective is unbiased.\n\nWhile training policies with PBRS often perform worse empirically than training with the heuristic objective $J(\\pi) + H(\\pi)$ (as shown in Section 4.1), our results in Figure 9 of the rebuttal PDF show that combining PBRS with HEPO yields a higher probability of improvement over the policy trained with the heuristic objective (though still worse than HEPO trained without PBRS). This indicates that combining PBRS and HEPO is promising for achieving both good empirical performance and theoretical guarantees.\n\n> Comparison with EIPO\n> \n\nThough EIPO and HEPO share a similar formulation, we didn’t include EIPO in the experiments because its purpose is not to enhance performance with heuristic rewards but to ensure that policies trained with heuristics do not degrade the performance of policies trained solely with task rewards. \n\nThat being said, in Figure 8 of the rebuttal PDF, we compare EIPO and HEPO using the Isaac and Bi-Dex benchmarks (as done in the experiments in Section 4.1). The results indicate that HEPO outperforms EIPO in both IQM of return and probability of improvement. EIPO is implemented according to the original EIPO paper, collecting trajectories by alternating between two policies and training the reference policy solely with task rewards.\n\n> The current algorithm seems limited to on policy methods.\n> \nWe show that HEPO is also effective at off-policy RL, too. Please see \"Generality on SAC\" in the general response.\n\n> Baselines in Table 1\n> \n\nWe didn’t include HuRL and PBRS in Table 1 due to their poor performance in Figure 1. In Table 3 of our rebuttal PDF, we add HuRL and PBRS for comparison, showing that both perform poorly.\n\n> Quality of the heuristic/asymptotic performance v.s. HEPO performance\n> \n\nWe believe that Figure 2 reveals the relationship between HEPO's performance and the quality of the heuristic reward. The policy trained with only heuristic rewards (H-only) represents both the asymptotic performance of $\\pi_H$ in HEPO and the quality of the heuristic itself. We found a positive correlation (Pearson coefficient of 0.9) between the average performances of H-only and HEPO in Figure 2's results, suggesting that better heuristics lead to improved HEPO performance. Figure 7(c) in our rebuttal PDF provides more details.\n\n> Related work and section 3.3.\n> \n\nThanks! We will merge them.\n\n> Why not use a percentage of improvement over $J_{H-only}$\n> \nThe formula $(J - J_{\\text{min}}) / (J_{\\text{max}} - J_{\\text{min}})$ is a standard method to normalize scores in deep RL [1]. Here, we define $J_{\\text{random}} = J_{\\text{min}}$ and $J_{\\text{Honly}} = J_{\\text{max}}$.\n\n**Why not percentage of improvement $J / J_{\\text{max}}$?** It's because $J_{\\text{max}}$ can be negative. For instance, if $J_{\\text{max}} = -2$ and the performance of algorithms A and B are 1 and 10, the normalized scores are $1 / -2 = -0.5$ for A and $10 / -2 = -5$ for B, respectively. This incorrectly suggests algorithm A is better than B despite A's score (1) being worse than B's (10). Subtracting $J_{\\text{min}}$ from both $J$ and $J_{\\text{max}}$ ensures both the numerator and denominator are positive, addressing this issue.\n\n[1] Deep reinforcement learning at the edge of the statistical precipice, NeurIPS 2021\n\n> H4-9-10-12 in figure 2\n> \n\nAs mentioned in Section 4.2 of our manuscript, the heuristic reward functions h12 and h9 had an incorrect sign for the distance component. This error caused the policy to reward moving away from the cabinet instead of toward it, making the learning task more difficult.\n\n**H4 & H10:** Both heuristic reward functions include multiple axis alignment rewards, like aligning the robot's end-effector with the target cabinet's forward axis. The poor performance likely stems from a designer misunderstanding the coordinate system, resulting in rewards that direct the robot away from the cabinet.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We're delighted that the reviewers find our manuscript easy to follow. We sincerely thank them for appreciating the simplicity of our method and the breadth of our experiments. Here, we'd like to summarize the common questions and the new experiments:

### Additional Results

- **Generality on SAC (`#inih`):** We integrated HEPO into HuRL's SAC codebase. Despite HuRL using tuned hyperparameters as reported in its paper [1], HEPO outperformed SAC and matched HuRL on the most challenging task (Figure 7(b) in the rebuttal PDF) using the same hyperparameters from Section 4.1 of our manuscript, showing the generality of HEPO on different RL algorithms.
- **Better than EIPO on the most challenging task (`#inih`):** Comparing HEPO and EIPO on the most challenging task, Montezuma's Revenge, reported in EIPO's paper [2] using RND exploration bonuses [3] (as used in the EIPO paper), Figure 7(a) in the rebuttal PDF shows that HEPO performs better than EIPO. Also, HEPO matched PPO trained with RND bonuses at convergence (2 billion frames) reported in [3] using only 20% of the training data, demonstrating drastically improved sample efficiency.
- **Additional Baselines (`#P4zd`):**
    - **Outperforming EIPO in IsaacGym + Bi-Dex (Section 4.1):** Figure 8 in the rebuttal PDF shows that HEPO outperforms EIPO in both IQM of return and probability of improvement using Isaac [4] and Bi-Dex [5] benchmarks. EIPO was implemented per the original paper [2], alternating between two policies and training the reference policy solely with task rewards.
    - **Outperforming baselines in the experiments of human-designed heuristics in the wild (Section 4.2):** Table 3 in our rebuttal PDF shows HuRL and PBRS perform poorly with human-designed reward functions from Table 1 in real-world scenarios.
- **Synergistic with potential-based reward shaping (PBRS) [6] (`#P4zd`):** By reshaping heuristic rewards $H$ using PBRS for HEPO, we ensure $\text{arg}\max_\pi J(\pi) + H(\pi) = \text{arg}\max_\pi J(\pi)$, guaranteeing the modified HEPO’s optimization objective is unbiased. While training with PBRS often performs worse than the policy trained with $J(\pi) + H(\pi)$ (see Section 4.1), our results in Figure 9 of the rebuttal PDF show that combining HEPO with PBRS improves the probability of improvement over nine IsaacGym tasks. This suggests that combining PBRS and HEPO is promising for both empirical performance and theoretical guarantees in optimizing an unbiased objective that yields the same optimal policy for $J$ asymptotically.

### Clarification

- **Source of heuristic reward functions (`#inih/#15cn`):** Our heuristic reward functions are based on the source code of Isaac Gym [4] and Bidexterous Manipulation [5], as mentioned at the beginning of Section 4 in our manuscript. For more information, please refer to their respective papers and repositories. In the experiments on reward functions designed in the wild (Section 4.2), humans were recruited to write the heuristic reward functions. See Section 4.2 for details.
- **EIPO and HEPO have the same memory/time complexity (`#inih/#P4zd/#15cn`):**
    - **Hyperparameters/Memory:** Both EIPO and HEPO share the same number of hyperparameters: 1. Initial value of the Lagrangian multiplier, and 2. Learning rate of the Lagrangian multiplier. Thus, HEPO does not require additional memory or hyperparameters compared to EIPO.
    - **Time Complexity:**
        - **Trajectory Rollout:** Both rollout $B$ trajectories (number of parallel environments), thus involving the same number of forward passes on the policy networks and resulting in the same time complexity at this stage.
        - **Policy Update:** Both train two policies on the collected trajectories at each iteration, so HEPO does not add complexity during training.

[1] Cheng, Ching-An, Andrey Kolobov, and Adith Swaminathan. "Heuristic-guided reinforcement learning." Advances in Neural Information Processing Systems 34 (2021): 13550-13563.

[2] Chen, Eric, et al. "Redeeming intrinsic rewards via constrained optimization." Advances in Neural Information Processing Systems 35 (2022): 4996-5008.

[3] Burda, Yuri, et al. "Exploration by random network distillation." arXiv preprint arXiv:1810.12894 (2018).

[4] Makoviychuk, Viktor, et al. "Isaac gym: High performance gpu-based physics simulation for robot learning." arXiv preprint arXiv:2108.10470 (2021).

[5] Chen, Yuanpei, et al. "Towards human-level bimanual dexterous manipulation with reinforcement learning." Advances in Neural Information Processing Systems 35 (2022): 5150-5163.

[6] Ng, Andrew Y., Daishi Harada, and Stuart Russell. "Policy invariance under reward transformations: Theory and application to reward shaping." *Icml*. Vol. 99. 1999.

Meta-review: The paper proposes a method, HEPO, for ensuring that heuristics help RL by making RL optimize a constraint objective, where the constraint is given by the value function of a policy trained on heuristic rewards. HEPO is closely related to an existing approach, EIPO, but provides valuable improvements over it in implementation simplicity and reduced number of hyperparameters. The reviewers found the paper easy to read and suggested a number of improvements to the experiments, such as using SAC, that the authors have successfully implemented during the rebuttal stage. The reviewers and the meta-reviewers trust that the authors will include them into the manuscript.

After these new results are incorporated into the paper, the work's only notable remaining weakness (which is addressed in the rebuttals only partly) is the scarcity of its theoretical analysis. However, the metareviewer believes that the paper's algorithmic contributions are sufficiently valuable on their own and fully recommends this paper for acceptance.
Decision: Accept (poster)