PEAR: Primitive Enabled Adaptive Relabeling for Boosting Hierarchical Reinforcement Learning

Authors: Utsav Singh, Vinay P. Namboodiri
Primary area: reinforcement learning
Keywords: Hierarchical reinforcement learning, Learning from demonstrations

Abstract:
Hierarchical reinforcement learning (HRL) has the potential to solve complex long horizon tasks using temporal abstraction and increased exploration. However, hierarchical agents are difficult to train due to inherent non-stationarity. We present primitive enabled adaptive relabeling (PEAR), a two-phase approach where we first perform adaptive relabeling on a few expert demonstrations to generate efficient subgoal supervision, and then jointly optimize HRL agents by employing reinforcement learning (RL) and imitation learning (IL). We perform theoretical analysis to bound the sub-optimality of our approach and derive a joint optimization framework using RL and IL. Since PEAR utilizes only a few expert demonstrations and considers minimal limiting assumptions on the task structure, it can be easily integrated with typical off-policy \RL algorithms to produce a practical HRL approach. We perform extensive experiments on challenging environments and show that PEAR is able to outperform various hierarchical and non-hierarchical baselines and achieve upto 80% success rates in complex sparse robotic control tasks where other baselines typically fail to show significant progress. We also perform ablations to thoroughly analyze the importance of our various design choices. Finally, we perform real world robotic experiments on complex tasks and demonstrate that PEAR consistently outperforms the baselines.

Decision: accept (poster)
Reviewer scores: 6, 6, 8, 5

Reviews and discussion:
<thread>
creation date:
1729592014199

thread_type:
official_review

<thread_object>
creator: reviewer_gcp5

<content>
summary:
This paper proposes a method to improve HRL, utilizing a few expert demonstrations. The key insight is that subgoals selected from the dataset can effectively guide exploration for the policy. An adaptive relabeling method is proposed to select the proper subsequent subgoal based on the Q value of the low-level policy. The relabeled data provides an imitation-based regularization for the high-level policy, encouraging it to output reachable, high-quality subgoals for the low-level policy. Experiments in diverse simulation robotic tasks demonstrate the effectiveness of the method.

strengths:
1. Using a few expert demonstrations to improve goal-based HRL is promising. The proposed adaptive relabeling method for IL regularization is straightforward, well-motivated, and yields good results.
2. The paper includes some theoretical analysis.
3. The experiments cover a variety of robotic tasks, including real-world test.

weaknesses:
1. Inconsistent definition: Section 3 states that the expert data contains only states. But in Section 4.2, the low-level regularization term uses actions from the expert data.
2. Addressing the non-stationarity issue in HRL is a main claim of the paper. However, the proposed method does not resolve this issue. The high-level policy still faces non-stationarity, as the transitions in its replay buffer, which are determined by the low-level policy, continue to change throughout the training process.

questions:
1. Many existing works [1,2,3] also use IL loss from data to regularize high-level policy learning. What differentiates the proposed regularization method from these approaches?
2. Why can we set the threshold of Q to 0 (Section 4.1) in all experiments? I believe this hyperparameter should vary, depending on the reward function specific to each task.

[1] Pertsch, et al. "Accelerating reinforcement learning with learned skill priors." Conference on robot learning. PMLR, 2021.
[2] Shi, et al. "Skill-based model-based reinforcement learning." arXiv preprint arXiv:2207.07560 (2022).
[3] Yuan, et al. "Pre-training goal-based models for sample-efficient reinforcement learning." The Twelfth International Conference on Learning Representations. 2024.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730581267969

thread_type:
official_review

<thread_object>
creator: reviewer_di4o

<content>
summary:
PEAR combines its key feature, adaptive goal relabeling, with IL regularization (either MSE or IRL), and other tricks (e.g. margin classification objective) to beat several prior HRL methods on a standard array of tasks using expert demonstrations.

strengths:
The authors carefully outline their construction of the PEAR algorithm, justifying the usage of each component (goal relabeling, the general algorithm, the joint optimization framework, etc) thoroughly and clearly. The sub optimality analysis provides further credence to their method. The method is novel, and notably outperforms previous HRL works (while, in some cases, removing the need for e.g. hand-made action primitives).

I agree with the author view that the significance of the work should be gauged less by its immediate improvement over other LfD methods, and more by its conceptual groundwork. In this regard, this paper is well-written, the findings are well-presented, and the extensive ablations provide further insight into key aspects of the method.

weaknesses:
As authors note, the method is currently reliant on expert demonstrations. However, many benchmarks exist which include other kinds of demonstrations, including human teleop. While the method may not perform well on these demonstrations just yet (as it is listed as an aim of future work), providing results on suboptimal demonstrations would help demonstrate concretely the strong and weak points of authors' method, and potentially provide insights on why it fails in these settings.

Furthermore, it seems inaccurate to state that PEAR uses only “a handful” of demonstrations, when Fig. 13 shows that generally 50-70+ demonstrations are needed to solve the provided tasks (with the exception of Franka Kitchen, which provides fewer demos).

questions:
Have you tested on suboptimal demonstrations? Are there any interesting findings/results which may point to future avenues for research or failings of the method which should be investigated/improved upon?

Were you able to find notable reasons for why the MSE-regularized learning objective would occasionally outperform the IRL-regularized version? Is there any relationship to task difficulty, data diversity, etc?

In “Algorithm 2: PEAR,” line 8, shouldn’t the lower-level policy’s IL regularization be done with $D_g$? Since we are providing state $s^f$ from the goal dataset $D_g$ and subgoal supervision $s^e_g$ to the goal-conditioned low-level policy, then the policy predicts action $a$, and we regularize this to be close to the dataset action $a^f$ (either with MSE or the IRL objective)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730660944815

thread_type:
official_review

<thread_object>
creator: reviewer_cpmf

<content>
summary:
The authors introduce an algorithm, Primitive Enabled Adaptive Relabeling (PEAR), to address the issue of non-stationary transition and reward functions when implementing HRL.  Like Relay Policy Learning (RPL) (Gupta 2019), PEAR encourages the high level policy to output feasible subgoals using imitation learning.  The main difference from RPL is that instead of imitating the actions from a dataset that occur at fixed intervals, PEAR uses a heuristic to select the latest subgoal that the low level policy can achieve.  The authors show in a variety of experiments that PEAR can outperform several baselines.

Gupta et al. Relay Policy Learning. 2019

strengths:
- The writing was clear and easy to understand.
- The authors included several ablation experiments that provided some important insights.
- The paper included some real world robotic experiments.

weaknesses:
My main concern with this approach is that the contribution seems to be marginal as I do not see a compelling reason why PEAR would consistently perform better than (a) HAC (Levy 2019) with replay buffer augmented with demonstration data and (b) hierarchical behavior cloning, in which pure imitation learning is applied to both a high and low level policies.  

HAC already addresses the problem of nonstationarity through relabeling and subgoal testing.  The problem with HAC is that it does not have a built-in mechanism to explore but this can be remedied with the demonstration data that is provided to PEAR.  An advantage of HAC + demonstration data over PEAR, which would use a pure RL objective, is that if the demonstration data is suboptimal, it would not have an imitation learning regularization term forcing the agent to output suboptimal subgoals.  HAC has also demonstrated it can learn more than two levels of hierarchy.  The results of HER+BC, which I understood to be HER with a replay buffer augmented with demonstration data, was often ultimately able to match the performance of PEAR (see Figure 14), making it more likely that HAC, which is close to a hierarchical version of HER, should be able to match PEAR.  

In addition, it seems that a pure hierarchical imitation learning approach, in which both levels are trained with supervised learning should also work, at least potentially with more data.  The baseline BC may not have worked well because the tasks were too long horizon, but the addition of a high level policy trained with imitation learning should help.

Levy et al. Hierarchical Actor-Crtiic. 2017

questions:
1.  Why is the Q_{Thresh} set to 0?  If the low-level reward is -1 for not achieving the goal and 0 for achieving the goal, the Q value should be negative for any correct action that puts the agent on a path to achieve the goal.  
2.  Is the high level policy only trained with a dataset containing expert trajectories?  Or is it also trained on its own interaction data?  If it is trained on its own interaction data, is there any relabeling done on that?
3.  I did not understand the “margin” component of the objective.  Can you provide another explanation of this component?
4.  Why does the paper characterize the number of demonstrations as a “handful” of demonstrations, when it uses 100 for most tasks?
5.  Have you experimented on any domains with image observations?

I am willing to raise my score if the authors can (i) provide some principled reasons why PEAR should outperform HAC+demonstrations and hierarchical behavior cloning and (ii) provide some answers to the above questions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730733438000

thread_type:
official_review

<thread_object>
creator: reviewer_u7xn

<content>
summary:
The paper presents a novel approach called Primitive Enabled Adaptive Relabeling (PEAR) aimed at enhancing Hierarchical Reinforcement Learning (HRL) for complex long-horizon tasks. The authors propose a two-phase methodology where the first phase involves adaptive relabeling of expert demonstrations to generate subgoals, followed by joint optimization of HRL agents through Reinforcement Learning (RL) and Imitation Learning (IL). The results indicate that PEAR outperforms various baselines in both synthetic and real-world robotic tasks, achieving up to 80% success rates in challenging environments.

strengths:
1. **Innovative Approach:** The use of adaptive relabeling to generate subgoals tailored to the capabilities of the lower primitive is a significant contribution that addresses the non-stationarity issue in HRL.
2. **Theoretical Justification:** The authors provide theoretical analysis that bounds the sub-optimality of their approach, lending credibility to their claims.
3. **Comprehensive Experiments:** Extensive experiments across multiple challenging tasks demonstrate the practical efficacy of PEAR, showing improved performance and sample efficiency over existing methods.
4. **Real-World Application:** The validation of PEAR in real-world tasks enhances the relevance and applicability of the research.

weaknesses:
1. **Expert Demonstrations Requirement:** The first phase of PEAR relies on expert demonstrations to generate subgoals, which raises concerns about the fairness of comparison with other HRL methods that do not require such demonstrations. This could affect the generalizability of the findings.
2. **Lack of Recent Comparisons:** The paper does not include comparisons with several hierarchical reinforcement learning methods published in the last three years. This omission limits the contextual relevance of the results and could misrepresent the state of the art, such as [1], [2].

[1] Kim J, Seo Y, Shin J. Landmark-guided subgoal generation in hierarchical reinforcement learning[J]. Advances in neural information processing systems, 2021, 34: 28336-28349.

[2] Wang, Vivienne Huiling, et al. "State-conditioned adversarial subgoal generation." Proceedings of the AAAI conference on artificial intelligence. Vol. 37. No. 8. 2023.

3. **Citation Issues:** Many citations are sourced from CoRR instead of their formal conference versions. This oversight should be corrected to ensure academic integrity and accurate referencing. For example, the last two references should be cited as follows (Note that there are many other citation errors beyond these two):

Wulfmeier, Markus, et al. "Data-efficient hindsight off-policy option learning." International Conference on Machine Learning. PMLR, 2021.

Zhang, Tianren, et al. "Generating adjacency-constrained subgoals in hierarchical reinforcement learning." Advances in Neural Information Processing Systems 33 (2020): 21579-21590.

questions:
- How do the authors justify the reliance on expert demonstrations in the first phase of PEAR compared to HRL methods that function without such requirements?
- Can the authors provide additional comparisons with recent HRL approaches to strengthen the positioning of PEAR within the current landscape?
</content>
</thread_object>

</thread>
<thread>
creation date:
1733263450132

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to Area Chairs and Reviewers

comment:
We thank all the reviewers for their valuable feedback and insightful comments. We are particularly encouraged that the reviewers consider our proposed approach **novel and promising** (u7xN, di4o and GCp5), our **theoretical analysis insightful** (u7xN and GCp5), our **empirical evaluation and ablations impressive and extensive** (u7xN, Cpmf, GCp5, and di4o), the **paper well-written and well-organized** (Cpmf, di4o), and our **real world experiments credible** (u7xN, Cpmf, GCp5). We address individual questions of reviewers in separate responses. We have uploaded a modified version of our paper based on reviewers' suggestions: [**rebuttal_pdf_link**](https://openreview.net/pdf?id=0nJEgNpb4l). 

We first enlist additional experiments to address the reviewer's concerns for your consideration.

### Additional Experiments.

* **Comparison with state-of-the-art baseline**: In order to address reviewer **u7xN**'s concerns, we implemented recent state-of-the-art approach **SAGA** (State-conditioned adversarial subgoal generation) and critically compared it with PEAR. We demonstrate that PEAR is able to significantly outperform this approach, which shows that PEAR efficiently mitigates non-stationarity in HRL and demonstrates impressive performance over state-of-the-art approaches. We are happy that reviewer acknowledged the efficacy of this experiment, and kindly raised the score to 6.
* **Additional Hierarchical baselines: HAC-demos and HBC**:  In order to address reviewer **Cpmf**'s concerns, we implemented two additional baselines: **HAC-demos** (Hindsight actor critic with expert demonstrations) and **HBC** (Hierarchical behavior cloning) and compared it with PEAR. This comparison shows that *primitive enabled adaptive relabeling* and *joint optimization* are crucial for mitigating non-stationarity in HRL and improved performance. We are happy that reviewer acknowledged the efficacy of this experiment, and kindly raised the score to 6.

We have added the additional experiments and improvements suggested by the reviewers in the submitted manuscript, and provided detailed clarifications to address all the reviewers' concerns in the rebuttal. We also provide a summary of our core contributions below.

### Summary of core contributions:

* In this work, we propose a novel approach PEAR that efficiently mitigates non-stationarity in off-policy HRL for solving complex long horizon tasks.
* Our adaptive relabeling based approach generates efficient higher level subgoal supervision according to the current goal achieving capability of the lower primitive.
* We perform detailed theoretical analysis to bound the sub-optimality of our approach, and to theoretically justify the benefits of periodic re-population using adaptive relabeling.
* We perform extensive experimentation on six sparse robotic environments to empirically demonstrate our superior performance and sample efficiency over prior baselines.
* We show that PEAR demonstrates impressive performance in multiple challenging real world tasks.
* In summary, we propose a novel hierarchical reinforcement learning algorithm that is a promising step towards building practical robotic systems for real-world scenarios.

We greatly appreciate all reviewers' time and effort in reviewing our paper. We hope that our responses and updates to the paper have addressed all the concerns, and solidified **PEAR** as a promising HRL approach for building practical systems to solve complex tasks.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposed a two-stage HRL method that uses expert demonstrations to partition trajectories into subsets of training data for low-level policies. The main claimed contributions are that the method generates efficient subgoal supervision, the method mitigates non-stationarity in HRL, experiments show the work outperforms prior work, and that real-world experiments also demonstrates "impressive performance" in real world tasks. Experiments provide a reasonable amount of evidence of these claims (except the latter, in my opinion, as these experiments were fairly limited). Reviewers were almost all positive about this work, and I found no glaring issues with the paper that were either unmentioned by the reviewers or unsatisfactorily addressed by the authors.