Goal-conditioned Reinforcement Learning with Subgoals Generated from Relabeling

Authors: Xing Lei, Zifeng Zhuang, Xuetao Zhang, Donglin Wang
Primary area: reinforcement learning
Keywords: Goal-Conditioned Reinforcement Learning, Hindsight Experience Replay, Subgoal-based Approach

Abstract:
In goal-conditioned reinforcement learning (RL), the primary objective is to develop a goal-conditioned policy capable of reaching diverse desired goals, a process often hindered by sparse reward signals. To address the challenges associated with sparse rewards, existing approaches frequently employ hindsight relabeling, substituting original goals with achieved goals. However, these methods exhibit a tendency to prioritize the optimization of closer achieved goals during training, leading to the loss of potentially valuable information from the trajectory and low sample efficiency. Our key insight is that these achieved goals, generated from the same hindsight relabeling, can serve as effective subgoals to facilitate the learning of policies that reach possible long-horizon desired goals within the same trajectory. Leveraging this perspective, we propose a novel framework called Goal-Conditioned reinforcement learning with Q-BC (i.e, behavior cloning (BC)-regularized Q) and Subgoals (GCQS) for goal-conditioned RL. GCQS is a innovative goal-conditioned actor-critic framework that systematically exploits more trajectory information to improve policy learning and sample efficiency. Specifically, GCQS initially optimizes a Q-BC objective to facilitate learning policies that reach achieved goals effectively. Subsequently, these achieved goals are redefined as subgoals, which serve to enhance the goal-conditioned policies, thereby predicting better actions to reach the desired goals. Experimental results in simulated robotics environments demonstrate that GCQS significantly enhances sample efficiency and overall performance compared to existing goal-conditioned methods. Additionally, GCQS demonstrated competitive performance on long-horizon AntMaze tasks, achieving results comparable to such state-of-the-art subgoal-based methods.

Decision: reject
Reviewer scores: 8, 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1729382239267

thread_type:
official_review

<thread_object>
creator: reviewer_curz

<content>
summary:
This paper studies a tendency in existing GCRL methods with goal relabelling that prioritises optimisation toward closer achieved goals. To mitigate this bias, the authors propose an actor-critic objective without goal relabelling, incorporating a KL-divergence constraint towards a novel prior policy. This prior policy ensures that the learned policy behaves consistently towards both the final desired goal and intermediate subgoals. In the experiments, the authors demonstrate that the proposed method achieves strong performance with fewer environment interactions, although the experimental design remains contentious.

strengths:
1. The motivation behind the proposed method is insightful and articulated clearly, providing a strong rationale for the approach.
2. The method developed from this motivation is based on a promising and well-supported assumption.

weaknesses:
The main concern with this paper lies in the experimental design:
1. Section 6.1: It lacks comparison with suitable baselines (see Question 5).
2. Section 6.2: This section appears redundant, as GCQS's advantage in sample efficiency is already demonstrated in Figure 5 and Section 6.1.

Some minor issues are noted in the questions section.

questions:
1. The definition of Q-BC should be clarified upon its first appearance (line 80).
2. There is a typo in line 312-313 - 'policy objective that reaching achieved goals'.
2. Section 2 discusses GCWSL methods like GoFar (Ma et al., 2022) and WGCSL (Yang et al., 2022) without mentioning their offline nature. Yang et al. prove the theoretical guarantees in the offline goal-conditioned setting—do these guarantees still hold in the online setting?
3. In Section 5.2, the relabelled (achieved) goal $g'$ is redefined as subgoals $s_g$, but both $g'$ and $s_g$ are used in the text, which causes some confusion. Would it be clearer to consistently use the notation $g'$ throughout the paper?
5. The comparison in Section 6.1 appears unfair, as some baselines, like Actionable Models, WGCSL, and GoFar, are designed for offline goal-conditioned RL and may not perform as well in the online setting.

**References**

Ma, J. Y., Yan, J., Jayaraman, D., \& Bastani, O. (2022). Offline Goal-Conditioned Reinforcement Learning via f-Advantage Regression. NeurIPS.

Yang, R., Lu, Y., Li, W., Sun, H., Fang, M., Du, Y., … Zhang, C. (2022). Rethinking Goal-conditioned Supervised Learning and Its Connection to Offline RL. ICLR.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730528586799

thread_type:
official_review

<thread_object>
creator: reviewer_knzr

<content>
summary:
This paper works on goal-conditioned reinforcement learning tasks and introduces the idea of leveraging subgoals in hindsight relabeling to improve learning efficiency. The proposed method is mainly heuristic, and it is justified with theory on the performance. Empirical results are provided to support the method.

strengths:
The authors appropriately used illustrative figures to explain some concepts in their paper. Motivating examples are provided to demonstrate the problem of previous approaches. The experiments are conducted on multiple tasks and results are compared with multiple baselines.

weaknesses:
There are several main weaknesses in the current paper:

1. on the high level, the necessity of the proposed method is not clear. I understand the key insight the authors aimed to convey using Figure 2. However, from my perspective, isn't this a *property* rather than a *problem* of hindsight relabeling? e.g. when the number of steps required to achieve different goals is uniformly sampled from 1-50, then during the learning process, hindsight relabeling using trajectories having a length of 50 will generate a uniformly distributed length from 1-50, and as the learning proceeds, the policy will learn to achieve closer goals, and averaged trajectory length will be smaller than 50 --- this will lead to a compounding effect in increasingly having more shorter paths. 


2. to solve the above challenge, why is the proposed method "necessary"? For instance, why can not one just to re-sample or down-sample on some of the state-goals? In the current write-up, it is not very clear why the multiple designing factors introduced are well-motivated or necessary. 


3. according to the theory, using a smaller $\eta$ will lead to a tighter bound, does this mean that we need the $\eta$ to be as close to 0 as possible?

4. In Figure 5, the averaging/smoothing method for different figures seems different, and some results lack error bars. The FetchSlide performance is noticeably lower than what has been achieved in the literature.
From my perspective, it is not necessary to demonstrate a method is always better than baselines in all experimental cases. What is more important is to clearly show what are the realistic motivations for introducing the techniques/designing components of the paper, and then use experiments to highlight --- the settings where those challenges exist, the method shines and on the other hand, when those challenges (controllably) alleviated, the performances of different methods would converge. 


Minor:

5. On presentation. In Figure 1, it would be great if the authors could give some concrete examples and make sure the notations are self-consistent. I acknowledge the authors' effort in explaining their method using Figure 1 and it could be a great idea if the clarity can be further enhanced. For instance, what is "QBC" objective in this figure, what does it mean by "subgoals come from achieved goals", and why the policy can take either s and g or s and s_g as its inputs?

questions:
please refer to weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730647767693

thread_type:
official_review

<thread_object>
creator: reviewer_kpwt

<content>
summary:
This paper presents GCQS, which utilizes achieved goals as subgoals and iteratively updates by considering behavior cloning schemes for prior policy and KL-constraint for policy.

strengths:
This paper is generally well-written, easy to follow, and provides promising results on some benchmark problems.
The paper also highlights neglected aspects, such as the issue of short trajectory usage in subgoal generation, and presents theoretical support for the proposed objective.

weaknesses:
The lack of comparison with other benchmark problems, such as complex AntMaze tasks presented in [1, 2], where the original HER showed significantly degraded performance compared to similar methods such as PIG, raises questions about whether omitting high-level planning for subgoal generation is valid for other complex and long-horizon GCRL tasks.

See questions for others.

[1] Kim, Junsu, et al. "Imitating graph-based planning with goal-conditioned policies." arXiv preprint arXiv:2303.11166 (2023).

[2] Kim, Junsu, Younggyo Seo, and Jinwoo Shin. "Landmark-guided subgoal generation in hierarchical reinforcement learning." Advances in neural information processing systems 34 (2021): 28336-28349.

questions:
(1) This is the question for clarification. What is the key difference between the proposed methods compared to existing methods [1,2] which update subgoals throughout training? Specifically, how does relabelling achieved goals as subgoals differ from landmark or subgoal updates in PIG and HIGL? 

(2) This method discards the necessity of high-level planning for subgoal generation but I'm curious whether GCQS works well in long-horizon tasks such as AntMaze [1,2,3] compared to methods utilizing high-level planning, such as HIGL and PIG.

(3) It is unclear how GCQS addresses the issue of short trajectory updates since HER already conducts a similar relabeling scheme. Which component of GCQS handles this? 

(4) How important is the SAC component in Eq.12 for GCQS? This omitted ablation leaves the question of the importance of the BC term.

(5) Presenting more details on the derivation of Eq. 25 from Eq. 24 would be helpful for readers.

[1] Kim, Junsu, et al. "Imitating graph-based planning with goal-conditioned policies." arXiv preprint arXiv:2303.11166 (2023).

[2] Kim, Junsu, Younggyo Seo, and Jinwoo Shin. "Landmark-guided subgoal generation in hierarchical reinforcement learning." Advances in neural information processing systems 34 (2021): 28336-28349.

[3] Lee, Seungjae, et al. "Cqm: Curriculum reinforcement learning with a quantized world model." Advances in Neural Information Processing Systems 36 (2023): 78824-78845.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730687612243

thread_type:
official_review

<thread_object>
creator: reviewer_jyb4

<content>
summary:
This paper studies hindsight relabeling in goal-conditioned reinforcement learning (GCRL). The key finding is that previous GCRL methods have a bias towards closer achieved goals during training, which results in the learned policy being less aligned to reach long-term goals. To address this issue, this paper proposes a novel GCRL method, GCQS, which first learns a goal-reaching policy and then uses the KL-divergence between the learned policy to final goals and goal-reaching policy to subgoals along the trajectories to guide policy optimization for reaching the long-term goals. This method enjoys a performance guarantee and demonstrates better performance than state-of-the-art GCRL baselines.

strengths:
1. The proposed method shows strong empirical results.
2. The method is well-motivated.

weaknesses:
Major questions:

1. If prioritizing the closer achieved goals is the main issue of previous GCRL methods, can we simply use data augmentation to rebalance the relabeling trajectory dataset and improve the performance? I think this would be an interesting result to see and could make the paper more solid.
2. The connection between GCQS and GCWSL: while the authors do not classify GCQS as one GCWSL method, I think there are some similarities to discuss between the two methods. Actually, in offline settings, both Equation 12 and Equation 15 should have a closed-form solution, which is an exponentially weighted form of $\pi_{relabel}$ and $\pi^{prior}$, as discussed in [1]. This makes me wonder if GCQS is still a GCWSL method (like exponential advantage weight mentioned in WGCSL [2]) and if the closed-form solution could further improve the method.
3. Several baselines in the experiments are offline methods, so I am wondering how authors compare them with other online baselines and GCQS to make sure the comparison is fair.

Minor points:

1. The cumulative function should be defined in Theorem 4.1, as it is slightly different from the commonly used probability cumulative function.
2. I believe Section 6.1 has a typo: 16 CPUs -> 16 GPUs.

Despite the questions mentioned above, I am recommending weak acceptance due to the motivation and strong performance of the paper at this stage, however I hope the authors can address my concerns.

[1] Nair, Ashvin, et al. "Awac: Accelerating online reinforcement learning with offline datasets." *arXiv preprint arXiv:2006.09359* (2020).

[2] Yang, Rui, et al. "Rethinking goal-conditioned supervised learning and its connection to offline rl." *arXiv preprint arXiv:2202.04478* (2022).

questions:
There are some questions and concerns, which I have outlined in the previous section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733278155096

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Response to all reviewers

comment:
We thank all reviewers for their constructive feedback! We included new experimental results and revisions in the PDF following the reviewers’ suggestions. Please check out our revised paper. We highlighted the changes in the revised version.

Here, we summarize the major changes:

- Revised the paper to correct unreasonable grammar and inappropriate expressions.

- Added a comparison with such subgoal-based state-of-the-art algorithms, such as HIGL [1], DHRL [2], PIG [3] and BEAG [4]  (Section 6.2). The results from complex, long-horizon Antmaze experiments indicate that GCQS demonstrates performance that is slightly inferior to or comparable with PIG, while outperforming both HIGL and DHRL. Additionally, GCQS achieves twice the training speed of PIG and three times that of BEAG, with significantly reduced memory costs. Extensive experiments across diverse tasks—including Antmaze, the Robotics Fetch series, and the Hand manipulation tasks—highlight the primary contributions of GCQS: addressing the challenges discussed in Section 4 and offering broad generalizability. Notably, the generalizability of GCQS lies in its ability to integrate seamlessly into any HER-based algorithm, delivering superior performance enhancements.

- Included error bar plots in Figure 6 to enhance the visual representation of the results. (Appendix Section D.4)

- Removed the offline baseline “action models” and added detailed explanations for baselines such as WGCSL, GoFar, and DWSL. These changes do not affect the results or main conclusions of the paper. (Figure 6 and Appendix C.1)

- The primary motivation of GCQS has been clarified as effectively leveraging longer trajectory information within the same hindsight framework (Figure 11).

- An ablation study on the non-uniform sampling of longer trajectories, such as weighted sampling or upsampling, for achieved goals (Appendix D.1).

Overall, our new findings and the novel extension framework based on GCAC, GCQS, represent an innovative and interesting contribution to goal-conditioned reinforcement learning (GCRL). Overall, our new findings and the novel extension framework based on GCAC and GCQS represent a significant and innovative contribution to goal-conditioned reinforcement learning (GCRL). The GCQS framework, which is both simple and highly efficient, not only achieves state-of-the-art performance and improved robustness on challenging goal-conditioned benchmarks, but also demonstrates competitive results on the long-horizon AntMaze task, comparable to some advanced subgoal-based methods. The final revised version of the paper can be found in https://pdfhost.io/v/~QVHMgEkR_ICLR2025_GCQS_Rebuttal.

[1]Kim, Junsu, Younggyo Seo, and Jinwoo Shin. "Landmark-guided subgoal generation in hierarchical reinforcement learning." NIPS 2021.

[2]Lee, Seungjae, et al. "DHRL: a graph-based approach for long-horizon and sparse hierarchical reinforcement learning." NIPS 2022.

[3]Kim, Junsu, et al. "Imitating graph-based planning with goal-conditioned policies." ICLR 2023.

[4]Yoon, Youngsik, et al. "Breadth-First Exploration on Adaptive Grid for Reinforcement Learning." ICML 2024.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper studies a tendency in existing GCRL methods with goal relabelling to prefer optimizing towards closer goals. To mitigate this bias, the paper regularizes the policy to take similar actions for the final desired goal and intermediate subgoals. Experiments demonstrate that the proposed method achieves strong performance with fewer environment interactions.

Reviewers appreciated the clear motivation/writing/figures and strong empirical results. Some reviewers appreciated the theoretical support for the paper.

Multiple reviewers suggested comparing to a baseline that rebalanced the training data to skew the distribution over short-horizon vs long-horizon goals. Reviewers had questions about the relationship with some prior work (e.g., GCWSL, PIG, performance on FetchSlide), and recommended including additional baselines designed for the online setting. Reviewers had a few questions about the paper presentations, such as how results were smoothed and averaged across seeds, whether there is an inherent tradeoff in prioritizing short vs long horizon tasks.

Taken together, my recommendation is that the paper be rejected.