Goal-Reaching Policy Learning from Non-Expert Observations via Effective Subgoal Guidance
Authors: RenMing Huang
Shaochong Liu
Yunqiang Pei
Peng Wang
Guoqing Wang
Yang Yang
Heng Tao Shen
Venue: CoRL 2024 (2024)
Keywords: Goal-Reaching
Long-Horizon
Non-Expert Observation Data

Abstract: In this work, we address the challenging problem of long-horizon goal-reaching policy learning from non-expert, action-free observation data. Unlike fully labeled expert data, our data is more accessible and avoids the costly process of action labeling. Additionally, compared to online learning, which often involves aimless exploration, our data provides useful guidance for more efficient exploration. To achieve our goal, we propose a novel subgoal guidance learning strategy. The motivation behind this strategy is that long-horizon goals offer limited guidance for efficient exploration and accurate state transition. We develop a diffusion strategy-based high-level policy to generate reasonable subgoals as waypoints, preferring states that more easily lead to the final goal. Additionally, we learn state-goal value functions to encourage efficient subgoal reaching. These two components naturally integrate into the off-policy actor-critic framework, enabling efficient goal attainment through informative exploration. We evaluate our method on complex robotic navigation and manipulation tasks, demonstrating a significant performance advantage over existing methods. Our ablation study further shows that our method is robust to observation data with various corruptions.

Goal-Reaching Policy Learning from Non-Expert Observations via Effective Subgoal Guidance
Renming Huang', Shaochong I Liu', Yunqiang Pei',
Peng Wang" Guoqing Wang'.*, Yang Yang'. Hengtao Shen Schooo of Computer Science and Engineering,
University of Electronic Science and Technology of China
School of Computer Science and Technology. Tongji University 'Donghai Laboratory, Zhoushan, Zhejiang Corresponding author trenming13@gmail.oo com, p. wang5@hotmail com, gqkang0420Questc. edu
com,
Abstract: In this work, we address the challenging problem of reaching policy learning from non-expert, action-free observation data. Unlike fully labeled expert data, our data is more accessible and avoids the costly pro- of action labeling. Additionally, compared online learning, which often involves aimless exploration, data provides useful guidance for more efficient exploration. To achieve our goal, we propose a novel subgoal guidance learning strategy. The motivation behind this strategy is that long-horizon goals offer lim- ited guidance for efficient exploration and accurate state transition. We develop a diffusion strategy-based high-level policy to generate reasonable subgoals as way- points, preferring states that more easily lead the final goal. Additionally, we learn state-goal value functions encourage efficient subgoal reaching. These components naturally integrate into the off-policy actor-critic framework, abling efficient goal attainment through informative exploration. We evaluate our method on complex robotic navigation and manipulation tasks, demonstrating a significant performance advantage over existing methods. Our ablation study fur- ther shows that our method robust to observation data with various corruptions.
goal-
Keywords: Goal-Reaching, Long-Horizon, Non-Expert Observation Data
Introduction
Learning goal-reaching policy [1, 2] from sparse rewards holds great promise as it incentivizes agents to achieve variety of goals, acquire generalizable policies, and obviates the necessity for meticulously crafting reward functions. However, the lack of environmental cues poses significant challenges for policy learning. Online learning methods [3, 4. 5, 6] typically rely exploring all potentially novel states to cover areas where high rewards may exist, which can lead to inefficient exploration, especially in tasks with long horizons [7]. Alternatively, some methods [8, 9, 10, I1] resort to behavior cloning on external expert data or pre-training with extensive offline data using offline reinforcement learning to obtain an initial policy for online fine-tuning [12]. However, these approaches rely on fully labeled data, which is often expensive or impractical collect, and faces distribution shift challenges [13] during online fine-tuning.
In this work, we tackle long-horizon goal-reaching policy learning from a novel perspective by leveraging non-expert, action-free observation data. Despite the absence of per-step actions and the lack of guaranteed trajectory quality, this data contains valuable information about states likely lead to the goal and the connections between them. By extracting such information, we can effectively guide exploration and state transition, achieving higher learning efficiency compared
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

to pure online learning. Additionally, the accessibility of this data minimizes labeling costs and broadens data sources, rendering our approach practical.
However, this setting poses significant challenges. The absence of action labels precludes the direct learning of low-level per-step policies, while using the final goal the reward may result in guidance decay, particularly in long-horizan tasks. To address this, we propose the Efficient Goal- RealliReehing policy learning with Prior Observations (EGR-PO) method, which employs hierarchical approach, extracting reasonable subgoals and exploration guidance from action-free observation data to assist in online learning.
Our method involves learning a diffusion model-based high-level policy generate reasonable sub- goals, acting waypoints for reaching the final goal. Additionally, learn another state-goal value function [2] to calculate exploration rewards, thereby encouraging efficient achievement of subgoals and ultimately reaching the final goal. During the pre-training phase, the state-goal value function assists the high-level policy in generating optimal subgoals from non-expert observation data, and the subgoals, acting as nearer goals, addressing issues of guidance decay and predic- tion inaceuracy [14] encountered by the state-goal value function when dealing with long-horizon final goals. Furthermore, our method seamlessly integrates into the off-policy actor-critic frame- work [15]. The high-level policy generates subgoals serving as guidance for the low-level policy, while the state-goal value function calculates informative exploration rewards for every transition, fostering effective exploration.
In summary, our contributions are as follows: (1) We offer a fresh perspective on long- long-horizon goal-reaching policy learning by leveraging non-expert, action-free data, thereby reducing the need for costly data labeling and making our approach practical. (2) We introduce EGR-PO, a hierarchical policy learning strategy comprising subgoal generation and state-goal value function learning. These components work in tandem to facilitate effective and efficient exploration. (3) Through extensive empirical evaluations on challenging robotic navigation and manipulation tasks, we demonstrate the superiority of our method over existing goal-reaching reinforcement learning approaches. Ablation studies further highlight the desirable characteristics of our method, including clearer guidance and robustness to trajectory corruptions.
2 Related Work
Goal-Reaching Policy Learning with Offline Data. In many real-world reinforcement learning settings, it is straightforward to obtain prior data that can help the agent understand how the world works. Various types of prior data have been widely explored to tackle goal-reaching challenges, including fully labeled data, video data, reward-free data, and expert demonstrations. Fully labeled data serves as either experience replay [16] during online learning or an offline dataset for pre- training [12, 17, 18, 19, 20, 21], enabling the acquisition of pre-trained policy. Furthermore, the pre-trained policy can be employed an external policy to guide exploration [22, 23, 10] or directly fine-tuned online [11, 8]. However, online fine-tuning typically encounters distribution shift [24, 9] and overestimation [8. 1 challenges. On the other hand, video data provides wealth of information that can be directly utilized for learning policy [25, 26, 27], learning state representation
for downstream RL [28, 29, 30), guiding the discovery of skills [31], and learning world models for planning and decision decision-making [32, 33, 34, 35]. Moreover, recent research has showcased the potential of reward-free data [36] in accelerating exploration through optimistic reward labeling [6]. In the case of expert demonstration data, imitation learning is commonly employed learn stable policy [37, 38, 39, 40, 41].
Online Learning via Exploration. Sparse rewards hinder learning efficiency, making desired goals challenging to achieve. To address this, boosting exploration abilities allows agents to cover unseen goals and states, facilitating effective learning through experience replay. One standard approach is to add exploration bonuses to encourage exploration to unseen state. Exploration bonuses seek to reward novelty, quantified by various metrics such as density model [42, 43, 44], curiosity [3, 45), model error [46, 47, 5]. even prediction error to randomly initialized function [4]. Gorl-directed exploration involves setting exploratory goals for the policy to pursue. Various goal-selection meth-

---

Prior Observations
Do
Update
High Policy Update
Online Collection
9
[et, Re43s
Actor-Critic Update
A(sts Books ghو
Mick 16
Gg
Besh
(eat)
Eepleottce Resocck
@(+ (gts.o,))
((see
9
Rush
V(a,g)
x (t)4
Iraadie Dut
Elladlght
(u) Subgoal Guided Policy Learning from Prior Observations
(b) Efficient Online Learning with Subgoal Guidance
Figure I: Overview of EGR-PO. (a) Our method is composed of two key learning components: a state-goal value function designed for informative exploration and high-level policy to generate reasonable subgoals. (b) Integrating the two components into the actor-critic method, where the learned state-goal value function provides exploration rewards to encourage meaningful exploration, and the reasonable subgoals provide clear guidance signals.
ods have been proposed, such as frontier-based [48], learning progress [49, 50J, goal difficulty [51], "sibling rivalry" [52], value function disagreement [53], go-explore framework [54, 55]. Our method falls under goal-directed exploration, but our reasonable goals are learned from prior observations.
3 Preliminaries
Problem Setting. We investigate the problem of goal-conditioned reinforcement learning, which is defined by Markov decision process M = (S,A, p. [56], where S denotes the state space, A denotes the action space, μ E P(S) denotes an initial state distribution, p E S x A + P(S) denotes a transition dynamics distribution, and r(s, g) denotes a coal-conditioned reward function. We as- sume that we have an additional observation dataset Do that consists of state-only trajectories т, = (So, s1))..,8T) Our goal is relay on Do help learn an optimal goal-conditioned policy T(als,0)
that maximizes (xx = Eg-ple))r-ppp (r) EEi= ('r(st,g) g)] with (r) = {=(s)) I x(a 8t, g)++(++1 18, az), where y is discount factor and p(g) is goal distribution. In our method, the policy is formulated as hierarchical policy π T[a s. g) x² (gsubl s. s. x (als,
Implicit Q-learning. Kostrikov et al. [9] proposed a method called Implicit Q-Learning (IQL. that circumvents the need to query out-of-sample actions by transforming the max operator in the Bellman optimal equation into expectile regression. Specifically, IQL trains an action-value function ((s, a) and a state value function (s) with the following loss:
Lv = (s.a)>D [L5
V(s))] fQ = (e,,,,"'>> [(r(e.)) + +VV(s') Q(s, a)) (1)
where r(s.a) represents the reward function, Q represents the target Q network [57], and L3 denotes the expectile loss with a parameter 7 belonging to the interval [0.5, 1). The expectile loss L (1r) is defined |- 1(x 0)]2², which exhibits asymmetric square loss characteristic by placing greater emphasis on penalizing positive values compared to negative ones.
Ad Hauntag-WWeighter Regression (AWR). AWR [58] considers policy optimization as a maximum likelihood estimation problem within an anpectaaioo-Maximiizan advantage values weight the likelihood, thereby encouraging the policy select actions that lead to large Q values while remaining close to the data collection policy. Given a collection dataset D, the objective of extracting policy with AWR is formulated follows [12, 60, 61]:
[59] framework. utilizes the
Jn (0) = (e (exp(B A(s, a)) log no, (a]s))
(2)
where B E R denotes inverse temperature parameter, A(s, a) Q(s, a) V(s) represents the extent to which the current action is superior to the average performance.
Efficient Goal-Reaching with Prior Observations (EGR-PO)
The overview of our method illustrated in Figure 1. In Section 4.1, we extract guidance compo- nents from observations, which involves training a state-goal value function V(s,g) g) to encourage informative exploration and learning high-level policy (L.ooss g) that generates reasonable sub- goals. In Section 4.2, we illustrate how the learned components collaborate enhance the learning of the online low-level policy T/g (ax) [ar]st. g sh..). Algorithm presents sketch of our method, imple- mentation details can be found in the Appendix.
3

---

Algorithm Efficient Goal-Reaching with Prior Observations
1: Input: Observation dataset Do
2: while not converged do
3: Sample batch ss. ...' Do
4: Update state-goal value network (s,gg) with Equation (3) # Train state-goal value function
5: Sample batch geub. g) ~ Do
6: Update high-level policy x (Asubl 8, with Equation (7) # Extract high-level policy
7: end while
8: for each environment step do
9: Execute action a ~ T' (als,, s. gous) with subgoals goub ~ x
Calculate exploration reward rg with Equation (9), store transition the replay buffer D
11:
# Actor Critic Style Update
12:
Sample transition mini-batch = {(s, a, sr, [sub)) D with hindsight relabeling
13:
Update Q network and Q2 network by minimize Equation (8) and Equation (10)
14:
Extract online policy n² with Equation (11)
15:
end for
4.1 Subgoal Guided Policy Learning from Prior Observations
Learning state-goal value function for informative exploration. Learning value function from prior data encounters overestimation [8] caused by out-of- distriibbbutinn we adopt the IQL approach, which prevents querying out-of-distribution "actions". Specifically, we utilize the action-free vari- ant [28, 62] IQL to learn the state-goal value function, denoted as V(s, g):
Lv g)>Do [L5(r + V Vss,g))]
The learned state-goal value function is unreliable with increasing distance, as discussed in Sec- tion 5.3. Relying solely on for long-horizon tasks ineffective and potentially harmful. To address this, we learn another policy that generates nearby and reasonable subgoals, which enhance the accuracy of predicted values and provide clearer guidance signals.
Learning to generate reasonable subgoals. We use the diffusion probabilistic model to perform behavior cloning for subgoal generation. Previous studies [63, 64) have demonstrated the robustness of diffusion probabilistic models in policy regression. We represent our diffusion policy using the reverse process of conditional diffusion model follows:
A Th (s = Po (gaud (. = Ng O, 1) po (o Deubs 8,g).
II ==1
(4)
We follow DDPM [65] and train the e-conditional model by optimizing the following objective:
JBC (Ø)
[|le cv(Vargeut vI ā,e, s,g, i)
(5)
where e is noise following a Gaussian distribution ~ I), U is a uniform distribution over the discrete set as {1....NN] Following HIQL [14], we sample goals g from either the future states within the same trajectory or random states in the dataset. Similarly, we sample subgoals Jsub from either the future k-step states Si+* within the same trajectory.
However, due to the prior data is not from expert, the subgoals generated through behavior cloning are not necessarily optimal. With the state-goal value function learned by Equation 3, we can im- prove the policy with the following variant of AWR [58]:
( E(s. exp(( Á(s, Ieub: g)) log n (goub s. |sg)],
approximate A(s, Suub: g) as Volgsub, g) Vo(s, g), which assists extract subgoals with higher advantage without deviating from the data distribution. The final objective function is a linear com- bination of behavior cloning and policy improvement:
arg (i) arg min(JBc(0) - a (j())),
where a is hyperparameter used to control the diversity of subgoals.

---

(a) SowyerReach
(b) Reach
(c) PickAndPlace
(d) Maze2d
(c) CALVIN
(1 AntMuze
Figure 2: We study the robotic navigation and manipulation tasks with sparse reward.
4.2
Efficient Online Learning with Subgoal Guidance
The learned high-level policy and state-goal value function naturally integrate into the off-policy actor-critic paradigm [15, 66, 67, 68, 69]. Actor-critic framework simultaneously learns the action- value function Q [70] and the policy network Th [71]. The action- value network is trained with temporal difference [72]:
Lq= o. [|r r ((s',''')) Q(s, -(s
where Q is the target network, is the environmental reward. We introduce additional exploration rewards ry encourage informative exploration. The design of the exploration reward function is based on the learned state-goal value function:
Rs(s,6 s' g) tanh(n (Vo(s',g) - Vo(s, g))).
where 17 is scaling factor. Different from previous methods [4, 73, 74, 75, 76], we do not directly add rg to the environmental reward r. The exploration reward has taken the future into consideration, thus, we introduce an additional guiding Q function Qy a, g). which directly approximates the exploration reward:
fa, Rearr sht IND [1Q, a, giws) rall
(10)
The online low-level policy is updated by simultaneously maximizing Q and Qg: T = arg max (s.ohh...) M-..a....DDaattt ( a..h...) (o 1Q(s,
sluss) + 3 Qg a, ghall. where 3 is used to control the strength of guidance.
(11)
5 Experiments
Our experiments delve into the utilization of non-expert observation data expedite online learning in goal-reaching tasks, particularly focusing addressing challenges associated with long-horizon objectives. We evaluate the efficacy of our methods challenging tasks with sparse rewards, as depicted in Figure 2. These tasks encompass manipulation tasks such as SawyerReach [77] and RetchReach [78], as well as long-horizon tasks like FetchPick [78], CALVIN [79], and two navigation tasks [80] of varying difficulty levels. Detailed information regarding our evaluation environments can be found in the Appendix. Our experiments aim to provide concrete answers to the following questions:
Is our method more efficient compared other online methods and can it compete with offline-olliee methods learned from fully labeled data?
(2) Does the efficiency of our method stem from informative exploration?
Do reasonable subgoals make the guidance clearer in our method?
(4) Is our method robust to insufficient, low-quality, and highly diverse observation data?
5.1 Comparison with previous methods
Baselines. We compare approach against various online learning methods, including the actor- critic method Online [69], exploration-based methods such RND [4] and ExPLORe [6], well
5

---

Oniiee
MAD
OC3L
HER
EsPLONe
105
0.55
ultra play
large pay
Succusse Bar5
large
0.8
24
98
5/2
2980
0.2 64
53 04
Eriforment
Staps
Envronment
Enviroment Stape
Envrrrmen Staps
Sawye
Succes P400
FetchReach
maze2d-medium
0.8
Aoc Ratum
Succes
8.4
%
67
044
E.C.
3.0
2og
6.4
0.5
2.000
07
04 24
08
Figure 3: Comparison with online learning methods robotic manipulation and navigation tasks.
Shaded regions denote the 95% confidence intervals across 5 random seeds. Best viewed in color.
AWNICC
C0L
SPOT
CoI-QL
REX
Ours
Calvin
Suoess katt
411 antttteeeeellll Ita.slly
Fettch/cceAndeP ace
maze dolattge
0.0
Aesrage Return
0.4
80
6.0
Erviatment Steps
Envirommnt Steps 1n6
Encronment Staps
Envronment Stops lei
Ensterment Steps lnt
Sawwer Beach
Aeerye Sicees
antonaze-umazt
FatchRuach
maze mediuu
Avergee Return
10
0.2
0.4
Env Yoveeneent Stape
Envionment
Steps
Envronmenr Stope
Envionment Steps 3wt
Envirrment Steps
Figure 4: Comparison with offline pre-training and online fine-tuning methods.
Shaded regions
denote the 95% confidence intervals across random seeds. Best viewed in color.
as data-efficient methods like HER [81]. GCSL [82], and RIS [83]. Additionally, we compare against offline-online methods, including naively online fine-tuning methods [13] like AWAC [12] and IQL [9], pessimistic methods CQL [8] and Cal-QL [11], policy-constraining method SPOT [24], and policy expansion approach PEX [10]. We set the update data ratio (UTD) [16] to for fair policy updates. Learning curves are presented in Figure 3 and Figure 4.
Figure depicts the performance curves of our approach compared to various online learning meth- ods in navigation and manipulation tasks. Our method shows significant improvements learning efficiency and policy performance, particularly in challenging tasks with long-horizon challenges such FetchPickAndPlace, AntMaze-Ultee and CALVIN, In these tasks, the agent requires intel- ligent exploration rather than exhaustive exploration of all states, which would be time-consuming Notably, our approach achieves rapid convergence and surpasses the performance of previous meth- ods while maintaining stability and demonstrating superior efficiency. Figure 4 illustrates a compar- ison between our approach and offline-onliln methods. Our method quickly reaches performance levels comparable to prior methods while outperforming them in terms of long- long-horizon tasks.
5.2 Does the efficiency of our method stem from informative exploration?
We evaluated the impact of our method on the state coverage [6] of the agent in the AntMaze do- main to investigate whether its effectiveness primarily stems from informative exploration. This evaluation allows us determine the effectiveness of incorporating non-expert action-free observa- tion data in accelerating online learning. We assess the state coverage achieved by various methods in the AntMaze task, with specific focus on their exploration effectiveness in navigating the maze. Figure provides a visual illustration of the state visitation on the antmaze-large task. Remarkably, our method guides the agent to explore, prioritizing states that can lead to the final goal.
To provide more quantitative assessment, utilize the "weighted state coverage" metric. We divide the map into grid and assign importance values to each state based their distance to the goal, as depicted in Figure 6. The weighted coverage metric reflects the average importance of states

---

Online
RND
EXPLORe
Ours
SW S0224 20
Figure 5: Visualizations of the agent's exploration behaviors on antmaze-large. The dots are uni- formly sampled from the online replay buffer and colored by the training environment step. The visualization results are obtained by sampling 512 points from a maximum of 120K environment steps. The results show that Ours achieves higher learning efficiency via informative explorations.
antmaze-large
Weighted Coverage
100
antmaze urnaze
Weighted Coverage
100
metmmaze-maduumpplay...
Weighted Coverage
100
antmaazelarge-playa
Cnl
75
RMS
75
75
50
Our
50-
50
25
25
25
0
0.0
0.5
10
0
0.0
0.5
1.0
0
0.0
0.5
1.0
Environment Stapsle6
Environment Steps 1e6
Enviranment Steps 1e6
Figure 6: Left: Visualization of state importance, which is determined by distance the goal. Right: Weighted coverage for 3 AntMaze tasks. The weighted coverage represents the average importance every 1K steps. Higher weighted coverage emphasizes exploration as more valuable. encountered every 1K steps. Higher weighted coverage indicates greater emphasis on exploring important states. In comparison, RND [4] and ExPLORe [6] prioritize extensive exploration but may not necessarily focus on crucial states for task completion. In contrast, our approach concentrates exploration on states more likely lead to the goal.
5.3 Do reasonable subgoals make the guidance more clear?
The learned state-goal value function may erroneously create the perception that can effectively guarantee informative exploration. However, the learned state-goal value function proves to be imperfect due to the following reasons: (1) Estimating long-horizon goals is likely to introduce higher degree of noise. (2) As the distance increases, the gradient within the state-goal goal value function progressively diminishes in prominence. We visualize these errors in the Figure 7, which highlight the limited guidance provided by relying solely on the value function. However, by setting subgoals that offer nearby targets the current state, we can effectively alleviate the aforementioned errors, especially in the context of long-horizon tasks. In Figure 7(b), the exploration rewards derived from the subgoals demonstrate notable distinctions and enhanced precision, rendering the learning signal more evident and discernible. The ablation study "w/o subgoals" is presented in the Appendix.
5.4 Is our method robust to different prior data?
To further evaluate the capability of our method in leveraging prior data, we modified the animaze- large-play-v2 dataset to evaluate our approach under different data corruptions. We primarily con- sider the following scenarios and report the results in Figure 8.
Diversity: We evaluate the sensibility of our method to the diversity offline trajectories under two variations of antmaze-large dataset: catmazz-large-aae of starting points fixed of endpoints, and antmazz-arre--ivvess, where the agent navigates from random starting points random endpoints.
where the agent navigates from a fixed set
Limited Data: We verify the influence of the quantities of offline trajectories on our performance by removing varying proportions of the data, where 10% denotes only 100 trajectories are preserved. Insufficient Coverage: We assess the dependence of our method on offline data coverage by retain- ing partial trajectories from the animaze- antmaze-large-paa large-play dataset. We divide it into three regions: Begin, Medium and Goal. We conduct ablation experiments by removing data from each region. Incomplete Trajectories: We verify the robustness of our method incomplete trajectories by di- viding each offline trajectory into segments of varying lengths. We consider three different levels of segmentation: 2 divide, 3 divide and 4 divide. The trajectory lengths are reported in Figure 8(d).
7

---

5
-25
-75
Std of
Distance between s and (a)
(b)
Figure 7: (a) Visualization of the standard deviation of V(+,)): As the distance between the state
and the goal increases, the learned value function becomes noisy. (b) Top: Relying solely on long- horizon goals leads to unclear and erroneous guidance. Bottom: Subgoals make guidance clear.
The arrows represent the gradient of ((,g), reflecting guidance for policy exploration.
(a) Diverse
(b) 10% Data
(c) Insufficient Coverage
(d) Trajectory Length
LHF
300
Diide
Divid
420
1000
Awerage Sucger Rate
Diversity
Rate
Limited Data
Insufficiett Coverage
Rate
Incomp ete Trajectories
1.0
1.0
1.0
0.9
0.8
0.8
Eegi
0.8
2
0.6
0.6
0.6
0.6
0.4
0.4
Avarage
Auarage 0.00
0.00
0.0
Enviromment Steps le5
Environment Steps le5
Environment Steps
Enviranment Steps le5
Figure 8: Visualizations of four different types of prior observation data and evaluation results on
them. Top:
Visualization of data characteristics. Bottom: Evaluation results. (a) Complexity and
Diversity. (b) Limited Dataset Regime. (c) Insufficient Coverage. (d) Incomplete Trajectories.
Our method demonstrates robustness in handling diverse datasets, even in scenarios with limited data. remains effective and stable even when data extremely searce. The absence Begin and Medium data does not significantly impact the learning our policy when there is insufficient coverage. However, challenges arise in online learning when there is inadequate coverage of the Goal region, highlighting the importance of goal coverage. When dealing with data consisting of trajectory segments, our method excels due to its strong trajectory stitching capability. The above comparison emphasizes the robustness of our method across different datasets, encompassing di- verse prior data and low-quality data with varying levels of corruption.
Conclusion
In conclusion, our proposed method, EGR-PO, addresses the challenging problem of long-horizon goal-reaching policy learning by leveraging non-expert, action-free observation data. Our method leams a high-level policy to generate reasonable subgoals and a state-goal value function to en- courage informative exploration. The subgoals, serving as waypoints, provide clear guidance and enhance the accuracy of predicting exploration rewards. These two components naturally integrated into the actor-critic framework, making it straightforward apply existing algorithms. Our method demonstrates significant improvements to various corrupted datasets, enhancing the practicality and applicability of our approach.
existing goal-reaching methods and shows robustness

---

Acknowledgements
This work was supported in part by the National Natural Science Foundation of China under grant U23B2011, 62102069, U20B2063 and 62220106008, the Key R&D Program Zhejiang un- der grant 1009,,, and the New Cornerstone Science Foundation through the XPLORER PRIZE.
References
[1] L. Kaelbling. Learning to achieve goals. In IJCAI, volume 2, pages 1094-8 8. Citeseer, 1993.
[2] T. Schaul, D. Horgan, K. Gregor, and D. Silver. Universal value function approximators. In International conference machine learning, pages 1312-1320. PMLR, 2015.
[3] D. Pathak, P. Agrawal, A. Efros, and Darrell. Curiosity-driven exploration by self- supervised prediction. International conference on machine learning, pages 2778-2787.
PMLR, 2017.
[4] Y. Burda, H. Edwards, A. Storkey, and O. Klimov. Exploration by random network distillation. In Seventh International Conference on Learning Representations, pages 1-17, 2019.
[5] T. Lin and A. Jabri. Mimex: Intrinsic rewards from masked input modeling. Advances in Neural Information Processing Systems, 36, 2024.
[6] Q. Li, J. Zhang, D. Ghosh, A. Zhang, and S. Levine. Accelerating exploration with unlabeled prior data. Advances in Neural Information Processing Systems, 36, 2024.
[7] K. Pertsch, O. Rybkin, F. Ebert, S. Zhou, D. Jayaraman, C. Finn, and S. Levine. Long-horizon visual plamning with goal-conditioned hierarchical predictors. Advances in Neural Information Processing Systems, 33:17321-17333, 2020.
[8] A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative q-learning for offline reinforce- ment learning. Advances in Neural Information Processing Systems, 33:1179-1199 2020.
[9] I. Kostrikov, A. Nair, and S. Levine. Offline reinforcement learning with implicit q q-learning In Deep Workshop NeurlPS 2021, 2021.
[10] H. Zhang, W. Xu, and H. Yu. Policy expansion for bridging offline-to-online reinforcement learning. In The Eleventh International Conference on Learning Representations, 2022.
[11] M. Nakamoto, Zhai, A. Singh, M. Sobol Mark, Y. Ma, C. Finn, Kumar, and Levine. Cal-ql: Calibrated offline pre-training for efficient online fine-tuning. Advances in Neural Information Processing Systems, 36, 2024.
[12] A. Nair, A. Gupta, M. Dalal, and S. Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09 09359, 2020.
[13] S. Levine, A. Kumar, G. Tucker, and J. Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020.
[14] Park, D. Ghosh, B. Eysenbach, and Levine. Hiql: Offline goal-conditioned with latent states actions. Advances in Neural Information Processing Systems, 36, 2024.
[15] V. Konda and J. Tsitsiklis. Actor-critic algorithms. Advances in neural information processing systems, 12, 1999.
[16] Ball, L. Smith, Kostrikov, and S. Levine. Efficient online reinforcement learning with offline data. In International Conference Machine Learning, pages 1577-1594, PMLR, 2023.

---

[17] S. Lee, Y. Seo, K. Lee. P. Abbeel, and J. Shin. Offline-to-onlie oolline reinforcement learning via balanced replay and pessimistic q-ensemble. In Conference on Robot Learning, pages 1702- 1712. PMLR, 2022.
[18) A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstra- tions. arXiv preprint arXiv: wet.:: 2017.
[19] T. Xie, N. Jiang, H. Wang, C. Xiong, and Y. Bai. Policy finetuning: Bridging sample-efficient offline and online reinforcement learning. Advances in neural information processing systems, (1227395-27007 2021.
[20] T. G. Rudner, C. Lu, M. A. Osborne, Y. Gal, and Y. Teh. On pathologies in kl-regularized re- inforcement learning from expert demonstrations. Advances Neural Information Processing Systems, 34:28376-2888 283899 2021.
[21] D. Tirumala, A. Galashov, H. Noh, L. Hasenclever, R. Pascanu, J. Schwarz, G. Desjardins, W. M. Czamecki, A. Ahuja, Y. W. Teh, et al. Behavior priors for efficient reinforcement learning. Journal of Machine Learning Research, 23(221):1-63 2022.
[22] V. Campos, P. Sprechmann, S. S. Hansen, A. Barreto, S. Kapturowski, A. Vitvitskyi, A. P. Badia, and C. Blundell. Beyond fine-tuning: Transferring behavior in reinforcement learning. In ICML 2021 Workshop on Unsupervised Reinforcement Learning. 2021.
[23] I. Uchendu, T. Xiao, Y. Lu, B. Zhu, M. Yan, J. Simon, M. Bennice, C. Fu, C. Ma, J. Jiao, et al. Jump-start reinforcement learning. In International Conference on Machine Learning, pages PMLR, 2023.
[24] J. Wu, H. Wu, Z. Qiu, J. Wang, and M. Longg Supported policy optimization for offline re- inforcement learning. Advances in Neural Information Processing Systems, 35:31278-31291, 2022.
[25] D. Schmidt and M. Jiang, Learning to act without actions. In International Conference on Learning Representations, 2024.
[26] C. Wen, X. Lin, J. So, K. Chen, Q. Dou, Y. Gao, and P. Abbeel. Any-point trajectory modeling for policy learning. arXiv preprint arXiv:2401.00025. 2023.
[27] I. Radosavovic, T. Xiao, S. James, P. Abbeel, J. Malik, and T. Darrell. Real-world robot learning with masked visual pre-training. In Conference Robot Learning, pages 416-426. PMLR, 2023.
(28) D. Ghosh, C. A. Bhateja, and S. Levine. Reinforcement learning from passive data via latent intentions. In International Conference on Machine Learning, pages 11321-11339. PMLR, 2023.
[29] B. Baker, I. Akkaya, P. Zhokov, J. Huizinga, J. Tang, A. Ecoffet, B. Houghton, R. Sampedro, and J. Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. Advances in Neural Information Processing Systems, 2122439--26655 2022.
[30] D. Brandfonbrener, O. Nachum, and J. Bruna. Inverse dynamics pretraining learns good rep- resentations for multitask imitation. Advances in Neural Information Processing Systems, 36, 2024.
[31] M. Tomar, D. Ghosh, V. Myers, A. Dragan, M. E. Taylor, P. Bachman, and S. Levine Video- guided skill discovery. In ICML 2023 Workshop The Many Facets of Preference Based Learn- ing. 2023.
[32] Y. Du, M. Yang, P. Florence, F. Xia. A. Wahid, B. Ichter, P. Sermanet, T. Yu, P. Abbeel, J. B. Tenenbaum, et al. Video language planning. arXiv preprint arXiv:2310.10625, 2023.
10

---

[33] R. Mendonca, S. Bahl, and D. Pathak. Structured world models from human videos. arXiv preprint arXiv:2308. 10901, 2023.
[34] H. Liu, W. Yan, M. Zaharia, and P. Abbeel. World model on million-length video and language with ringattention. arXiv preprint arXivv2402.08268, 2024.
[35] S. Yang, J. Walker, J. Parker-Holder, Y. Du, J. Bruce, A. Barreto, Abbeel, and D. Schu- urmans. Video as the new language for real-world decision making. arXiv preprint 240.:3990..7139, 2024.
[36] H. Hu, Y. Yang, J. Ye, Z. Mai, and C. Zhang. Unsupervised behavior extraction via random intent priors. Advances Neural Information Processing Systems, 36, 2024.
[37] A. Edwards, H. Sahni, Y. Schroecker, and C. Isbell. Imitating latent policies from observation.
In International conference on machine learning, pages 1755-1763. PMLR, 2019.
[38] Paul, J. Vanbaar, and A. thyy-Chowh.ur. Learning from trajectories via subgoal discovery. Advances in Neural Information Processing Systems, 32, 2019.
[39] Z. Luo, J. Mao, J. Wu, T. Lozano-Pérez, J. B. Tenenbaum, and L. P. Kaclbling. Learning ra- tional subgoals from demonstrations and instructions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 12068-123 -2078, 2023.
[40] L. X. Shi, A. Sharma, T. Z. Zhao, and C. Finn. Waypoint-based imitation learning for robotic manipulation. In 7th Annual Conference on Robot Learning, 2023.
[41] C. Wang. L. Fan, J. Sun, R. Zhang, L. Fei-Fei, D. Xu, Y. Zhu, and A. Anandkumar. Mimicplay: Long-horizon imitation learning by watching human play. In 7th Annual Conference on Robot Learning, 2023.
[42] M. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos. Unifying coset exploration and intrinsic motivation. Advances in neural information processing systems, 29, 2016.
[43] Z.-X. Xu, X.-L. Chen, L. Cao, and C.-X. Li. A study of count-based exploration and bonus for reinforcement learning. In 2017 IEEE 2nd International Conference Cloud Computing and Big Data Analysis (ICCCBDA), pages 425 4299. IEEE, 2017.
[44] V. H. Pong, M. Dalal, S. Lin, A. Nair, S. Bahl, and S. Levine Skew-fit: state-covering self- supervised reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning, pages 7783-7792. 2020.
[45] D. Pathak, D. Gandhi, and A. Gupta. Self-supervised exploration via disagreement. In Inter- national conference on machine learning, pages 5062-5071. PMLR, 2019.
[46] R. Houthooft, X. Chen, Y. Duan, J. Schulman, F. De Turck, and P. Abbeel. Vime: Variational information maximizing exploration. Advances in neural information processing systems, 29, 2016.
[47] J. Achiam and S. Sastry. burseise-based intrinsic motivation for deep reinforcement learning. arXiv preprint arXin:1703.01732. 2017.
[48] B. Yamauchi. Frontier-based exploration using multiple robots. In Proceedings of the second international conference on Autonomous agents, pages 47-53, 1998.
[49] A. Baranes and P.-Y. Oudeyer. Active learning of inverse models with intrinsically motivated goal exploration in robots. Robotics and Autonomous Systems, 61(1):49-73, 2013.
[50] V. Veeriah, J. Oh, and S. Singh. Many-goals reinforcement learning. 17Xiv:1806.09605, 2018.
arXiv preprint

---

[51] C. Florensa, D. Held, X. Geng, and P. Abbeel. Automatic goal generation for reinforcement learning agents. In International conference on machine learning, pages 1515-1528. PMLR, 2018.
[52] A. Trout, S. Zheng, C. Xiong, and R. Socher. Keeping your distance: Solving sparse reward tasks using self-balancing shaped rewards. Advances in Neural Information Processing Sys- lems, 32, 2019.
[53] Y. Zhang, P. Abbeel, and I.. Pinto. Automatic curriculum learning through value disagreement. Advances in Neural Information Processing Systems, 33:7648-7659, 2020.
[54] S. Pitis, H. Chan, S. Zhao, B. Stadie, and J. Ba. Maximum entropy gain exploration for long horizon multi-goal reinforcement learning. In International Conference Machine Learning, pages 7750-7761. PMLR, 2020.
[55] E. S. Hu, R. Chang, O. Rybkin, and D. Jayaraman. Planning goals for exploration. CoRL 2022 Workshop on Learning, Perception, and Abstraction for Long-Horizon Planning, 2022.
[56] R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT press, 2018.
[57] Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv: 2013.
[58] X. B. Peng, A. Kumar, G. Zhang, and S. Levine. Advantage- regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv: 1rXiv:1910.00/1 2019.
[59] T. K. Moon. The expectation- maximization algorithm. IEEE Signal processing magazine, 13 (6):47-60, 1996.
[60] G. Neumann and Peters. Fitted q-iteration by advantage weighted regression. Advances in neural information processing systems, 21, 2008
[61] R. Yang, Y. Lu, W. Li, H. Sun, M. Fang, Y. Du, X. Li, L. Han, and C. Zhang. Rethink- ing goal-conditioned supervised learning and its connection to offline rl. In ICLR 2022-10th International Conference on Learning Representations, 2022.
[62] H. Xu, L. Jiang, L. Jianxiong, and X. Zhan. A policy-guidde imitation approach for offline reinforcement leamiin. Advances in Neural Information Processing Systems, 35:4085-4098, 2022.
[63] Z. Wang. J. Hunt, and M. Zhou. Diffusion policies as an expressive policy class for offline re- inforcement learning. The Eleventh International Conference on Learning Representations, 2022.
[64] C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137, 2023.
[65] Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6833 2020.
[66] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. 2015.
Continuous control with deep reinforcement learning. arXiv preprint
[67] S. Fujimoto, H. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. International conference on machine learning, pages 1587-1596. PMLR, 2018.
[68] V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforeement learning. In International conference machine learning, pages 1928-1937. PMLR, 2016.
12

---

[69] T. Haarmoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[70] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep rein- forcement learning. nature, 518(7540):529-333 2015.
[71] R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour. Policy gradient methods for rein- forcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
[72] R. S. Sutton. Learning to predict by the methods temporal differences. Machine learning, 3:9-44, 1988.
[73] D. Kim, J. Shin, P. Abbeel, and Y. Seo. Accelerating reinforcement learning with value- conditional state entropy exploration. Advances in Neural Information Processing Systems, 36, 2024.
[74] Z. Wu, Lu, J. Xing, Y. Wu, R. Yan, Y. Gan, and Y. Shi. Pac: Reinforcement learning from external knowledge for efficient exploration. In The Twelfth International Conference Learning Representations, 2023.
[75] Bruce, A. Anand, B. Mazoure, and R. Fergus. Learning about progress from experts. The Eleventh International Conference on Learning Representations, 2022.
[76] B. Mazoure, J. Bruce, D. Precup, R. Fergus, and A. Anand. Accelerating exploration and representation learning with offline pre-training, arXiv preprint arXiv:2304.00046 2023.
[77] R. Yang, Y. Lu, W. Li, H. Sun, M. Fang. Y. Du, X. Li, L. Han, and C. Zhang. Rethinking goal- conditioned supervised learning and its connection to offline rl. In International Conference on Learning Representations, 2021.
[78] M. Plappert, M. Andrychowicz, A. Ray, B. McGrew, B. Baker, G. Powell, J. Schneider, J. To- bin, M. Chociej, P. Welinder, et al. Multi-goal reinforcement learning: Challenging robotics environments and request for research. arXiv preprint 18X2.:180.00946: 2018.
[79] O. Mees, L. Hermann, E. Rosete-Beatt Beass, and W. Burgard. Calvin: A benchmark for language- conditioned policy learning for long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters, 73):7327-7334, 2022.
[80] J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.0722 2020.
[81] M. Andrychowiez, F. Wolski, A. Ray, Schneider, R. Fong, Welinder, McGrew, J. To- bin, O. Pieter Abbeel, and Zaremba. Hindsight experience replay. Advances neural information processing systems, 30, 2017.
[82] D. Ghosh, A. Gupta, A. Reddy, J. Fu, C. Devin, B. Eysenbach, and S. Levine. Learning to reach goals via iterated supervised learning. In 9th International Conference on Learning Representations, ICLR 202/, 2021.
[83] E. Chane-Sanc C. Schmid, and I. Laptev. Goal-conditioned reinforcement learning with imag- ined subgoals. In International Conference on Machine Learning, pages 1430-1440. PMLR, 2021.
[84] Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learn- ing using nonequilibrium thermodynamics. In International conference machine learning, pages 2256-2265. PMLR, 2015.
13

---

[85] Z. Xiao, K. Kreis, and A. Vahdat. Tackling the generative learning trilemma with denoising diffusion gans. In International Conference on Learning Representations, 2021.
[86] Y. Song, Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-based generative modeling through stochastic differential equations. In International Conference Learning Representations, 2020.
[87] A. Gupta, Kumar, C. Lynch, Levine, and K. Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. In Conference Robot Learning. pages 1025-1037. PMLR, 2020.
[88] D. Hendrycks and Gimpel. arXiv: 08416, 2016.
Gaussian error linear units (gelus).
arXiv preprint
[89] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. 1r12../41..666 2014.
arXiv preprint
[90] J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv preprint arXiv:160..06450, 2016.
[91] Zhang, M. Janner, Y. Li, Rocktäschel, Grefenstette, Y. Tian, et Efficient plan- ning in a compact latent action space. In The Eleventh International Conference on Learning Representations, 2022.
[92] L. X. Shi, J. J. Lim, and Y. Lee. Skill-based model-based reinforcement learning. In Conference on Robot Learning, pages 2262-2272. PMLR, 2023.
[93] M. Fang, T. Zhou, Y. Du, L. Han, and Z. Zhang. cariculuu-ggidee hindsight experience replay. Advances in neural information processing systems, 32, 2019.
[94] A. Nair, B. McGrew, M. Andrychowicz, W. Zaremba, and P. Abbeel. Overcoming exploration in reinforcement learning with demonstrations. In 2018 IEEE international conference robotics and automation (ICRA), pages 6292-6299, IEEE, 2018.
14

---

A Implementation Details
A.1 Diffusion policy
Diffusion probabilistic models [84, 65] are a type of generative model that learns the data distribution q(x) from a dataset D :- (os It represents the process of generating data as an iterative denoising procedure, denoted by (,;2-1122 where is an indicator of the diffusion timestep. The denoising process is the reverse of a forward diffusion process that corrupts input data by gradually adding noise and is typically denoted by g(z]|==-1)- The reverse process can be parameterized as Gaussian under the condition that the forward process obeys the normal distribution and the variance is small enough: pe (2,-1(z;) = M( (z,, i), E.), where 14o and Σ are the mean and covariance of the Gaussian distribution, respectively. The parameters A of the diffusion model are optimized by minimizing the evidence lower bound of negative log- logeliikl of pa (zo), similar to the techniques used in variational Bayesian methods: 0" = arg ming - Eu [lg Po (2o)]- For model training. a simplified surrogate loss [65] is proposed based on the mean jsg of pa (x where the mean is predicted by minimizing the Euclidean distance between the target noise and the generated noise: Lenoose (0) E,,, [|e- eo (z,, i) []. where E NO I).
Specifically, our diffusion policy is represented as Equation (4) via the reverse process of condi- tional diffusion model, but the reverse sampling, which requires iteratively computing Eø networks N times, can become a bottleneck for the running time. To limit N to a relatively small value, with Smin = 0.1 and fmax = 10.0, we follow [85] to define:
B = a; 1 c 9min ) (..5 Smin) N2
(12)
which is a noise schedule obtained under the variance preserving SDE of [86].
A.2 Goal distributions
We train our state-goal value function and high-level policy respectively with Equation (3) and (7), using different goal-sampling distributions. For the state-goal value function (Equation (3)), we sample the goals from either random states, futures states, the current state with probabilities of 0.3, 0.5, and 0.2, respectively, following [28]. We use Geom -?) for the future state dis- tribution and the uniform distribution over the offline dataset for sampling random states. For the high-level policy, we mostly follow the sampling strategy of [87]. We first sample a trajectory (80: 81,.... st...., from the dataset Do and a state 8 from the trajectory. we either (i) sample g uniformly from the future states 8t, (fg > -) in the trajectory and set the target subgoal Jwub to kmim((+)..g)) or (ii) sample y uniformly from the dataset and set the target subgoal to "min((-k:2")
A.3 Advantage estimates
Following [14), the advantage estimates for Equation (6) is given as:
k-1
A(st. 8tt.))) 9) =
9) + r(sr g) Vo
Σ F=1
(13)
where we use the notations ie and 6 to incorporate the edge cases discussed in the previous para- graph (i.e., k= = min(k, L, - 1) when we sample g from future states, k- = min(k; T - t) when we sample g from random states, and & = smin(t+*,T)). Here, Sy * g and 8: & š144 always hold except for those edge cases. Thus, the reward terms in Equation (13) are mostly constants (under our reward function r(s,g) g) = 0 (if = g). -1 (otherwise)), as are the third terms (with respect to the policy inputs). As such, we practically ignore these terms for simplicity, and this simplification further enables us to subsume the discount factors in the first terms into the temperature hyperpa- rameter B. We hence use the following simplified advantage estimates, which we empirically found to lead to almost identical performances in experiments:
Á(s, 9eub. g) Velgeut, g) Vo(s.))
(14)
where we use goub to represent 8,4ū under various conditions.
15

---

Table Hyperparameters.
Hyperparameter
Value
Batch Size
1024
High-level Policy MLP Dimensions
(256, 256)
State-Goal Value MLP Dimensions
(512,512,512)
Representation MLP Dimensions
(512,512.512)
Nonlinearity
GELU [88]
Optimizer
Adam [89]
Learning Rate
0,0003
Target Network Smoothing Coefficient
0.005
AWR Temperature Parameter
1.0
IQL Expectile
0.7
Discount Factor
0.99
Diversity of Subgoals or
0.5
B Hyperparameters
We present the hyperparameters used in our experiments in Table I, where we mostly follow the network architectures and hyperparameters used by [28, 14]. We use layer normalization [90] for all MLP layers and we use normalized 1 10-dimensional output features for the goal encoder of state-goal value function make them easily predictable by the high-level policy, discussed in Appendix A.
For the subgoal steps k; we use k = 50 UArra,, k 15 (FetchReach, Pitch and SawyerReach), or k = 25 (others). We sample goals for high-level or flat policies from either the future states the same trajectory (with probability 0.7) or the random states in the dataset (with probability 0.3). During training, we periodically evaluate the performance of the learned policy at every 20 episode using 50 rollouts.
C Ablation Study Results
Subgoal Steps. In order to examine the impact of subgoal step values (k) on performance, we conduct an evaluation of our method on AntMaze tasks. We employ six distinct values for k E {1,5, 15, 25, 50, 100}. The results, depicted Figure 9, shed light on the relationship between k and performance outcomes. Remarkably, our method consistently demonstrates superior performance when k falls within the range of 25 50, which can be identified the optimal range. Our method exhibits commendable performance even when & deviates from this range, except in cases where k is excessively small. These findings underscore the resilience and efficacy of our method across various subgoal step values.
Ablation on Subgoals and Exploration Guidance. To demonstrate how subgoals and exploration guidance contribute to efficient policy learning for geal-reaching tasks, we conduct ablation ex- periments where we remove each component separately. The results, shown in the Figure 10, highlight the crucial importance of subgoal setting, the absence of subgoals hinders the resolu-
Rate
antmaze-umaze
antmaze-medlum-pla
1.0
antmaze- arge:play
antmaze-ultra-play
0.8
0.8
0.6
6.6
0.6
0.6
0.4
0.4
0.4
0.4
0.2
2000
708
08
0.00
0.0
02
Environment 1e6
Environment 1e6
Enviromeet
lee
Environment le6
Figure 9: Ablation study of the subgoal steps k. Our method generally achieves the best perfor- mances when & between 25 and 50. Even when k not within this range, ours mostly maintains reasonably good performance unless k is too small (i.e., < 5).
16

---

Werrgg Retum
2.4
2.4
5-2
0.6. 0.5
5-4
0.2
0.4
0.
0.4
0.2
0.4
05
3.0
0.0
5.4
0.6
Les
Figure 10: Ablation study on Subgoals and Exploration Guidance. The result shows that the crucial importance of subgoal setting. Additionally, incorporating exploration guidance facilitates the policy in efficiently reaching subgoals, resulting in further improvements in learning efficiency. Shaded regions denote the 95%coofidence intervals across 5 random seeds.
tion of long-horizon tasks. Additionally, incorporating exploration guidance facilitates the policy in efficiently reaching subgoals, resulting in further improvements in learning efficiency. Overall, our findings indicate that including both subgoal setting and exploration guidance enables our approach leverage the benefits of both, leading to efficient learning efficiency.
Visualization of End-Effector Trajectories in the FetchReach Task. We fixed the goal at farther location from the starting point, thereby increasing the task difficulty. In the Figure 11, the black dot represents the starting point, and the green dot represents the final goal. As shown in the visualization, other methods explore outward evenly from the start point. Despite incorporating active exploration, they struggle to reach the goal point due to the vast state space. Our method, on the other hand, avoids aimless exploration and efficiently reaches the goal point, which aligns with conclusions.
Online
RND
ExPLORe
Ours
10
L0
01
91
57
85
12
12
66
04
Figure 11: Visualization of End-Effector Trajectories in the FetchReach Task. The dots are uniformly sampled from the online replay buffer and colored by the training environment step.
D Environments
SawyerReach enviromment, derived from multi-world, involves the Sawyer robot reaching target position with its end-effector. The observation and goal spaces are both 3-dimensional Cartesian coordinates, representing the positions. The state-to-goal mapping is a simple identity function, ((s) s, and the action space 3-dimensional, determining the next end-effector position.
FetchReach and FetchPickAndPlace environments in OpenAl Gym feature 7-DoF robotic arm with a two-finger gripper. In FetchReach, the goal is to touch a specified location, while Fetch- PickAndPlace involves picking up a box and transporting it to a designated spot. The state space comprises 10 dimensions, representing the gripper's position and velocities, while the action space 4-dimensional, indicating gripper movements and open/close status. Goals are expressed as 3D vectors for target locations.
Maze2D is coal-conditioned planning task, which involves guiding a 2-DoF ball that can be force- actuated in the cartesian directions of x and y. Given the starting location and the target location, the policy is expected to find a feasible trajectory that reaches the target from the starting location avoiding all the obstacles.
AntMaze is class of challenging long- morizoo navigation tasks where the objective is to guide an 8-DoF Ant robot from its initial position to specified goal location. We evaluate the performance
17

---

in four different difficulty settings, including the "umaze", "medium" and "large" maze datasets from the original D4RL benchmark. While the large mazes already pose a significant challenge for long-horizon planning, we also introduce an even larger maze "ultra" proposed by [91]. The maze in the AntMaze-Ultra task twice the size of the largest maze in the original D4RL dataset. Each dataset consists of 999 length- 1000 trajectories, in which the Ant agent navigates from an arbitrary start location to another goal location, which does not necessarily correspond the target evaluation goal. At test time, to specify goal for the policy, we set the first two state dimensions (which correspond to the x-y coordinates) to the target goal given by the environment and the remaining proprioceptive state dimensions to those of the first observation in the dataset. At evaluation, the agent gets a reward when reaches the goal.
CALVIN is another long-horizon manipulation environment features four target subtasks. We use the offline dataset provided by [92], which is based on the teleoperated demonstrations from [79]. The dataset consists of 1204 length-499 trajectories. In each trajectory, the agent achieves some of the 34 subtasks in an arbitrary order, which makes the dataset highly task-agnostic [92]. At test time, to specify goal for the policy, we set the proprioceptive state dimensions to those of the first observation in the dataset and the other dimensions to the target configuration. At evaluation, the agent gets a reward ofl whenever it achieves a subtask.
E More Related Work
Learning Efficiency. Introducing relabeling can enhance learning efficiency. HER [81] relabels the desired goals in the buffer with achieved goals in the same trajectories. CHER [93] goes step further by integrating curriculum learning with the curriculum relabeling method, which adaptively selects the relabeled goals from failed experiences. Drawing from the concept that any trajectory represents a successful attempt towards achieving its final state, GCSL [82], inspired by supervised imitation learning, iteratively relabels and imitates its own collected experiences. [94] filters the
actions from demonstrations by Q values and adds supervised auxiliary loss the RL objective to improve learning efficiency. RIS [83] uses imagined subgoals to guide the policy search process. However, such methods are only useful if the data distribution is diverse enough to cover the space of desired behaviors and goals and may still face challenges in hard exploration environments.
F Baseline Introduction
F.1 Online learning baselines
Online: A standard off-policy actor-critic algorithm [69] which trains an actor network and critic network simultaneously from scratch that does not make use of the prior data at all.
RND: Extends the Online method by incorporating Random Network Distillation [4] as a novelty bonus for exploration. given an online transition (s, (s,a, r,a,,,,s) s'), and RND feature networks fo a), f(s, a), we set
f(s.a) r+ + (lfo(s,,)) al,
(15)
and use the transition (s, a, f, ) in the online update. The RND training is done the same way as in our method where a gradient step is taken on every new transition collected.
HER: Combines Online method with Hindsight Experience Replay [81] to improve data efficiency by re-labeling past data with different goals.
GCSL: Trains the policy using supervised learning, leading to stable learning progress.
RIS: This method [83] incorporates a separate high-level policy that predicts intermediate states halfway to the goal. By aligning the subgoal reaching policy with the final policy, RIS effectively regularizes the learning process and improves performance in complex tasks.
18

---

This approach learns reward model from online experience, labels the unlabeled prior data [6] with optimistic rewards, and then uses it concurrently alongside the online data for down- stream policy and critic optimization.
ExPLORe:
F.2 offline-online baselines
AWAC: AWAC combines sample-efficient dynamic programming with maximum likelihood policy updates, providing simple and effective framework that is able to leverage large amounts of offline data and then quickly perform online fine-tuning of reinforcement learning policies.
IQL: Avoiding querying out-of-sample actions by converting the max operator in the Bellman opti- mal equation into expectile regression,and thus learn a better Q Estimation.
CQL: CQL imposes additional regularizer that penalizes the learned Q-function out-of- distribution (OOD) actions while compensating for this pessimism actions seen within the train- ing dataset. Assuming that the value function is represented by function, Qo the training objective of CQL is given by
min a (Dow E (Qo(s, 1 + -- ((2o(s,0)BB*((000)
Conservative regularizer R())
(16)
where B (s, a) is the backup operator applied to delayed target Q-network, Q: 8 ( a)
r(s, a) + a conservative regularizer that aims to prevent overestimation in the Q-values for OOD actions by minimizing the Q-values under the policy [a]ss, and counterbalances by maximizing the Q-values of the actions in the dataset following the behavior policy nT..
a')]. The second term is the standard TD error. The first term R(0) is
This method learns conservative value function initialization can speed up online fine- tuning and harness the benefits of offline data by underestimating learned policy values while en- suring calibration. Specifically, Calibrating CQL constrain the learned Q-function Q to be larger than value function V via a simple change to the CQL training objective. Cal-QL modifies the CQL regularizer, R(0) this manner:
Cal-QL:
E
aoe Daww mmax (Qo(s, a). V(s))] - EE,,aD [Qo(s,a)),
(17)
where the changes from standard CQL are depicted in red.
This work constrains the policy network in offline reinforcement learning (RL) not only be within the support set but also avoid the out-of-distribution actions effectively unlike the standard behavior policy through behavior regularization.
SPOT:
This work introduces policy expansion scheme. After learning the offline policy, it is in- cluded as a candidate policy in the policy set, which further assists learning the online policy. This method avoids fine-tuning the offline policy, which could disrupt the learned policies, and instead allows the offline policy participate in online exploration adaptively.
PEX:
19


Reviews:
[{"review_id": "WYFen4HvV0", "reviewer": "Reviewer_onEe", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "Proposes an approach from learning from sub-optimal data by learning to propose reasonable subgoals. This is used with a learned goal-conditioned policy. Experiments are conducted on some mujoco environments involving arms and mazes for locomotion.", "summary_of_recommendation": "Due to the concerns regarding subgoal generation, the exploration reward, and missing comparisons, I vote for rejection.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 2, "robotics_focus": 2, "quality_of_limitations": 1, "questions_for_rebuttal": "1. Please address the questions regarding subgoal generation listed above.\n2. Please address questions regarding exploration reward above, add comparisons requested.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qjiBYG1Yiv", "reviewer": "Reviewer_qsNz", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposed a hierarchical policy learning strategy combining subgoal generation and state-goal value function learning.", "summary_of_recommendation": "Overall, the method is efficient with sufficient experiments provided. Although I worried about its significance in real world, I recommend accepting the paper, but will not argue for my recommendation if other reviewers have a different opinion.", "technical_quality": 3, "clarity_of_presentation": 2, "potential_impact": 2, "robotics_focus": 2, "quality_of_limitations": 2, "questions_for_rebuttal": "Please refer to the weaknesses.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0qiUS0f5ru", "reviewer": "Reviewer_RXHf", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduces a goal-conditioned RL framework that can utilize prior data for guiding the learning procedure. Specifically this paper (i) learns a state-goal value functions from prior action-free data and use it for informative exploration and (ii) learns a high-level policy with diffusion models that can generate nearby subgoals. Proposed components are integrated into actor-critic algorithm. Experiments are conducted on a variety of state-based benchmarks.", "summary_of_recommendation": "This paper is of good quality but its relevance to real-world robotics is not high. I'm in a borderline between weak accept / weak reject evaluations for this paper, I'd like to read the reviews of other reviewers and read the rebuttal response from the authors before making a final recommendation decision.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 2, "robotics_focus": 2, "quality_of_limitations": 1, "questions_for_rebuttal": "- Please address my weaknesses in the main review.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Initially the reviewers raised concerns regarding the exploration rewards used and random subgoal sampling, among others. The rebuttal submitted by the authors addressed these concerns to a large extent and now all reviewers uniformly recommend acceptance.
Decision: Accept