Goal-Reaching Policy Learning from Non-Expert Observations via Effective Subgoal Guidance

Authors: renming  huang
shaochong  liu
yunqiang  pei
peng  wang
guoqing  wang
yang  yang
hengtao  shen
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-06 16:09:02

Abstract: In this work, we address the challenging problem of long-horizon goal-reaching policy learning from non-expert, action-free observation data. Unlike fully labeled expert data, our data is more accessible and avoids the costly process of action labeling. Additionally, compared to online learning, which often involves aimless exploration, our data provides useful guidance for more efficient exploration. To achieve our goal, we propose a novel subgoal guidance learning strategy. The motivation behind this strategy is that long-horizon goals offer limited guidance for efficient exploration and accurate state transition. We develop a diffusion strategy-based high-level policy to generate reasonable subgoals as waypoints, preferring states that more easily lead to the final goal. Additionally, we learn state-goal value functions to encourage efficient subgoal reaching. These two components naturally integrate into the off-policy actor-critic framework, enabling efficient goal attainment through informative exploration. We evaluate our method on complex robotic navigation and manipulation tasks, demonstrating a significant performance advantage over existing methods. Our ablation study further shows that our method is robust to observation data with various corruptions.

Main paper: 
 introduction: 
 Learning goal-reaching policy [1, 2] from sparse rewards holds great promise as it incentivizes agents to achieve a variety of goals, acquire generalizable policies, and obviates the necessity for meticulously crafting reward functions. However, the lack of environmental cues poses significant challenges for policy learning. Online learning methods [3, 4, 5, 6] typically rely on exploring all potentially novel states to cover areas where high rewards may exist, which can lead to inefficient exploration, especially in tasks with long horizons [7] . Alternatively, some methods [8, 9, 10, 11] resort to behavior cloning on external expert data or pre-training with extensive offline data using offline reinforcement learning to obtain an initial policy for online fine-tuning [12] . However, these approaches rely on fully labeled data, which is often expensive or impractical to collect, and faces distribution shift challenges [13] during online fine-tuning.In this work, we tackle long-horizon goal-reaching policy learning from a novel perspective by leveraging non-expert, action-free observation data. Despite the absence of per-step actions and the lack of guaranteed trajectory quality, this data contains valuable information about states likely to lead to the goal and the connections between them. By extracting such information, we can effectively guide exploration and state transition, achieving higher learning efficiency compared to pure online learning. Additionally, the accessibility of this data minimizes labeling costs and broadens data sources, rendering our approach practical. However, this setting poses significant challenges. The absence of action labels precludes the direct learning of low-level per-step policies, while using the final goal as the reward may result in guidance decay, particularly in long-horizon tasks. To address this, we propose the Efficient Goal-Reaching policy learning with Prior Observations (EGR-PO) method, which employs a hierarchical approach, extracting reasonable subgoals and exploration guidance from action-free observation data to assist in online learning.Our method involves learning a diffusion model-based high-level policy to generate reasonable subgoals, acting as waypoints for reaching the final goal. Additionally, we learn another state-goal value function [2] to calculate exploration rewards, thereby encouraging efficient achievement of subgoals and ultimately reaching the final goal. During the pre-training phase, the state-goal value function assists the high-level policy in generating optimal subgoals from non-expert observation data, and the subgoals, acting as nearer goals, addressing issues of guidance decay and prediction inaccuracy [14] encountered by the state-goal value function when dealing with long-horizon final goals. Furthermore, our method seamlessly integrates into the off-policy actor-critic framework [15] . The high-level policy generates subgoals serving as guidance for the low-level policy, while the state-goal value function calculates informative exploration rewards for every transition, fostering effective exploration.In summary, our contributions are as follows: (1) We offer a fresh perspective on long-horizon goal-reaching policy learning by leveraging non-expert, action-free data, thereby reducing the need for costly data labeling and making our approach practical. (2) We introduce EGR-PO, a hierarchical policy learning strategy comprising subgoal generation and state-goal value function learning. These components work in tandem to facilitate effective and efficient exploration. (3) Through extensive empirical evaluations on challenging robotic navigation and manipulation tasks, we demonstrate the superiority of our method over existing goal-reaching reinforcement learning approaches. Ablation studies further highlight the desirable characteristics of our method, including clearer guidance and robustness to trajectory corruptions. 
related work: 
 Goal-Reaching Policy Learning with Offline Data. In many real-world reinforcement learning settings, it is straightforward to obtain prior data that can help the agent understand how the world works. Various types of prior data have been widely explored to tackle goal-reaching challenges, including fully labeled data, video data, reward-free data, and expert demonstrations. Fully labeled data serves as either experience replay [16] during online learning or an offline dataset for pretraining [12, 17, 18, 19, 20, 21] , enabling the acquisition of a pre-trained policy. Furthermore, the pre-trained policy can be employed as an external policy to guide exploration [22, 23, 10] or directly fine-tuned online [11, 8] . However, online fine-tuning typically encounters distribution shift [24, 9] and overestimation [8, 11] challenges. On the other hand, video data provides a wealth of information that can be directly utilized for learning policy [25, 26, 27] , learning state representation for downstream RL [28, 29, 30] , guiding the discovery of skills [31] , and learning world models for planning and decision-making [32, 33, 34, 35] . Moreover, recent research has showcased the potential of reward-free data [36] in accelerating exploration through optimistic reward labeling [6] . In the case of expert demonstration data, imitation learning is commonly employed to learn a stable policy [37, 38, 39, 40, 41] .Online Learning via Exploration. Sparse rewards hinder learning efficiency, making desired goals challenging to achieve. To address this, boosting exploration abilities allows agents to cover unseen goals and states, facilitating effective learning through experience replay. One standard approach is to add exploration bonuses to encourage exploration to unseen state. Exploration bonuses seek to reward novelty, quantified by various metrics such as density model [42, 43, 44] , curiosity [3, 45] , model error [46, 47, 5] , or even prediction error to a randomly initialized function [4] . Goal-directed exploration involves setting exploratory goals for the policy to pursue. Various goal-selection meth- ods have been proposed, such as frontier-based [48] , learning progress [49, 50] , goal difficulty [51] , "sibling rivalry" [52] , value function disagreement [53] , go-explore framework [54, 55] . Our method falls under goal-directed exploration, but our reasonable goals are learned from prior observations.3 Preliminaries Problem Setting. We investigate the problem of goal-conditioned reinforcement learning, which is defined by a Markov decision process M = (S, A, µ, p, r) [56] , where S denotes the state space, A denotes the action space, µ ∈ P (S) denotes an initial state distribution, p ∈ S × A → P(S) denotes a transition dynamics distribution, and r(s, g) denotes a goal-conditioned reward function. We assume that we have an additional observation dataset D O that consists of state-only trajectories τ s = (s 0 , s 1 , . . . , s T ). Our goal is relay on D O to help learn an optimal goal-conditioned policy π(a|s, g) that maximizesJ(π) = E g∼p(g),τ ∼p π (τ ) [ T t=0 γ t r(s t , g)] with p π (τ ) = µ(s 0 ) T -1 t=0 π(a t | s t , g)p(s t+1 | s t , a t ), where γ is a discount factor and p(g) is a goal distribution. In our method, the policy is formulated as a hierarchical policy π(a|s, g) = π h (g sub |s, g) • π l (a|s, g sub ).Implicit Q-learning. Kostrikov et al. [9] proposed a method called Implicit Q-Learning (IQL) that circumvents the need to query out-of-sample actions by transforming the max operator in the Bellman optimal equation into expectile regression. Specifically, IQL trains an action-value function Q(s, a) and a state value function V (s) with the following loss:EQUATION) where r (s,a) represents the reward function, Q represents the target Q network [57] , and L τ 2 denotes the expectile loss with a parameter τ belonging to the interval [0.5, 1). The expectile loss L τ 2 (x) is defined as |τ -1(x < 0)|x 2 , which exhibits an asymmetric square loss characteristic by placing greater emphasis on penalizing positive values compared to negative ones.Advantage-Weighted Regression (AWR). AWR [58] considers policy optimization as a maximum likelihood estimation problem within an Expectation-Maximization [59] framework. It utilizes the advantage values to weight the likelihood, thereby encouraging the policy to select actions that lead to large Q values while remaining close to the data collection policy. Given a collection dataset D, the objective of extracting a policy with AWR is formulated as follows [12, 60, 61] :EQUATIONwhere β ∈ R + 0 denotes an inverse temperature parameter, A(s, a) = Q(s, a) -V (s) represents the extent to which the current action is superior to the average performance. 
efficient goal-reaching with prior observations (egr-po): 
 The overview of our method is illustrated in Figure 1 . In Section 4.1, we extract guidance components from observations, which involves training a state-goal value function V (s, g) to encourage informative exploration and learning a high-level policy π h ϕ (g h sub |s, g) that generates reasonable subgoals. In Section 4.2, we illustrate how the learned components collaborate to enhance the learning of the online low-level policy π l θ (a t |s t , g h sub ). Algorithm 1 presents a sketch of our method, implementation details can be found in the Appendix. Update high-level policy π h ϕ (g sub |s, g) with Equation ( 7) # Extract high-level policy 7: end while 8: for each environment step do 9:Execute action a ∼ π l (a|s, g h sub ) with subgoals g h sub ∼ π h (g h sub |s, g)10:Calculate exploration reward r g with Equation ( 9), store transition to the replay buffer D Update Q network and Q g network by minimize Equation ( 8) and Equation (10) 14:Extract online policy π l with Equation (11) 15: end forLearning state-goal value function for informative exploration. Learning a value function from prior data encounters overestimation [8] caused by out-of-distribution, we adopt the IQL approach, which prevents querying out-of-distribution "actions". Specifically, we utilize the action-free variant [28, 62] of IQL to learn the state-goal value function, denoted as V (s, g):EQUATIONThe learned state-goal value function is unreliable with increasing distance, as discussed in Section 5.3. Relying solely on it for long-horizon tasks is ineffective and potentially harmful. To address this, we learn another policy that generates nearby and reasonable subgoals, which enhance the accuracy of predicted values and provide clearer guidance signals.Learning to generate reasonable subgoals. We use the diffusion probabilistic model to perform behavior cloning for subgoal generation. Previous studies [63, 64] have demonstrated the robustness of diffusion probabilistic models in policy regression. We represent our diffusion policy using the reverse process of a conditional diffusion model as follows:EQUATIONWe follow DDPM [65] and train the ϵ-conditional model by optimizing the following objective:EQUATION)where ϵ is noise following a Gaussian distribution ϵ ∼ N (0, I), U is a uniform distribution over the discrete set as {1, ..., N }. Following HIQL [14] , we sample goals g from either the future states within the same trajectory or random states in the dataset. Similarly, we sample subgoals g sub from either the future k-step states s t+k within the same trajectory.However, due to the prior data is not from expert, the subgoals generated through behavior cloning are not necessarily optimal. With the state-goal value function learned by Equation 3, we can improve the policy with the following variant of AWR [58] :EQUATIONwe approximate Ã(s, g sub , g) as V θ (g sub , g) -V θ (s, g), which assists extract subgoals with higher advantage without deviating from the data distribution. The final objective function is a linear combination of behavior cloning and policy improvement:EQUATIONwhere α is a hyperparameter used to control the diversity of subgoals. The learned high-level policy and state-goal value function naturally integrate into the off-policy actor-critic paradigm [15, 66, 67, 68, 69] . Actor-critic framework simultaneously learns the actionvalue function Q [70] and the policy network π l θ [71] . The action-value network Q is trained with temporal difference [72] :EQUATION)where Q is the target network, r is the environmental reward. We introduce additional exploration rewards r g to encourage informative exploration. The design of the exploration reward function is based on the learned state-goal value function:EQUATIONwhere η is a scaling factor. Different from previous methods [4, 73, 74, 75, 76] , we do not directly add r g to the environmental reward r. The exploration reward has taken the future into consideration, thus, we introduce an additional guiding Q function Q g (s, a, g), which directly approximates the exploration reward:EQUATION)The online low-level policy is updated by simultaneously maximizing Q and Q g :EQUATIONwhere β is used to control the strength of guidance. 
experiments: 
 Our experiments delve into the utilization of non-expert observation data to expedite online learning in goal-reaching tasks, particularly focusing on addressing challenges associated with long-horizon objectives. We evaluate the efficacy of our methods on challenging tasks with sparse rewards, as depicted in Figure 2 . These tasks encompass manipulation tasks such as SawyerReach [77] and FetchReach [78] , as well as long-horizon tasks like FetchPickAndPlace [78] , CALVIN [79] , and two navigation tasks [80] of varying difficulty levels. Detailed information regarding our evaluation environments can be found in the Appendix. Our experiments aim to provide concrete answers to the following questions:(1) Is our method more efficient compared to other online methods and can it compete with offline-online methods learned from fully labeled data?(2) Does the efficiency of our method stem from informative exploration?(3) Do reasonable subgoals make the guidance clearer in our method?(4) Is our method robust to insufficient, low-quality, and highly diverse observation data?Baselines. We compare our approach against various online learning methods, including the actorcritic method Online [69] , exploration-based methods such as RND [4] and ExPLORe [6] , as well as data-efficient methods like HER [81] , GCSL [82] , and RIS [83] . Additionally, we compare against offline-online methods, including naïvely online fine-tuning methods [13] like AWAC [12] and IQL [9] , pessimistic methods CQL [8] and Cal-QL [11] , policy-constraining method SPOT [24] , and policy expansion approach PEX [10] . We set the update data ratio (UTD) [16] to 1 for fair policy updates. Learning curves are presented in Figure 3 and Figure 4 .Figure 3 depicts the performance curves of our approach compared to various online learning methods in navigation and manipulation tasks. Our method shows significant improvements in learning efficiency and policy performance, particularly in challenging tasks with long-horizon challenges such as FetchPickAndPlace, AntMaze-Ultra, and CALVIN. In these tasks, the agent requires intelligent exploration rather than exhaustive exploration of all states, which would be time-consuming. Notably, our approach achieves rapid convergence and surpasses the performance of previous methods while maintaining stability and demonstrating superior efficiency. Figure 4 illustrates a comparison between our approach and offline-online methods. Our method quickly reaches performance levels comparable to prior methods while outperforming them in terms of long-horizon tasks.We evaluated the impact of our method on the state coverage [6] of the agent in the AntMaze domain to investigate whether its effectiveness primarily stems from informative exploration. This evaluation allows us to determine the effectiveness of incorporating non-expert action-free observation data in accelerating online learning. We assess the state coverage achieved by various methods in the AntMaze task, with a specific focus on their exploration effectiveness in navigating the maze.Figure 5 provides a visual illustration of the state visitation on the antmaze-large task. Remarkably, our method guides the agent to explore, prioritizing states that can lead to the final goal.To provide a more quantitative assessment, we utilize the "weighted state coverage" metric. We divide the map into a grid and assign importance values to each state based on their distance to the goal, as depicted in Figure 6 . The weighted coverage metric reflects the average importance of states encountered every 1K steps. Higher weighted coverage indicates a greater emphasis on exploring important states. In comparison, RND [4] and ExPLORe [6] prioritize extensive exploration but may not necessarily focus on crucial states for task completion. In contrast, our approach concentrates exploration on states more likely to lead to the goal.The learned state-goal value function may erroneously create the perception that it can effectively guarantee informative exploration. However, the learned state-goal value function proves to be imperfect due to the following reasons: (1) Estimating long-horizon goals is likely to introduce a higher degree of noise. (2) As the distance increases, the gradient within the state-goal value function progressively diminishes in prominence. We visualize these errors in the Figure 7 , which highlight the limited guidance provided by relying solely on the value function. However, by setting subgoals that offer nearby targets to the current state, we can effectively alleviate the aforementioned errors, especially in the context of long-horizon tasks. In Figure 7 (b), the exploration rewards derived from the subgoals demonstrate notable distinctions and enhanced precision, rendering the learning signal more evident and discernible. The ablation study on "w/o subgoals" is presented in the Appendix.To further evaluate the capability of our method in leveraging prior data, we modified the antmazelarge-play-v2 dataset to evaluate our approach under different data corruptions. We primarily consider the following scenarios and report the results in Figure 8 . Diversity: We evaluate the sensibility of our method to the diversity of offline trajectories under two variations of antmaze-large dataset: antmaze-large-play, where the agent navigates from a fixed set of starting points to a fixed set of endpoints, and antmaze-large-diverse, where the agent navigates from random starting points to random endpoints. Limited Data: We verify the influence of the quantities of offline trajectories on our performance by removing varying proportions of the data, where 10% denotes only 100 trajectories are preserved.Insufficient Coverage: We assess the dependence of our method on offline data coverage by retaining partial trajectories from the antmaze-large-play dataset. We divide it into three regions: Begin, Medium and Goal. We conduct ablation experiments by removing data from each region. Incomplete Trajectories: We verify the robustness of our method to incomplete trajectories by dividing each offline trajectory into segments of varying lengths. We consider three different levels of segmentation: 2 divide, 3 divide and 4 divide. The trajectory lengths are reported in Figure 8(d) . Our method demonstrates robustness in handling diverse datasets, even in scenarios with limited data. It remains effective and stable even when data is extremely scarce. The absence of Begin and Medium data does not significantly impact the learning of our policy when there is insufficient coverage. However, challenges arise in online learning when there is inadequate coverage of the Goal region, highlighting the importance of goal coverage. When dealing with data consisting of trajectory segments, our method excels due to its strong trajectory stitching capability. The above comparison emphasizes the robustness of our method across different datasets, encompassing diverse prior data and low-quality data with varying levels of corruption. 
conclusion: 
 In conclusion, our proposed method, EGR-PO, addresses the challenging problem of long-horizon goal-reaching policy learning by leveraging non-expert, action-free observation data. Our method learns a high-level policy to generate reasonable subgoals and a state-goal value function to encourage informative exploration. The subgoals, serving as waypoints, provide clear guidance and enhance the accuracy of predicting exploration rewards. These two components naturally integrated into the actor-critic framework, making it straightforward to apply existing algorithms. Our method demonstrates significant improvements over existing goal-reaching methods and shows robustness to various corrupted datasets, enhancing the practicality and applicability of our approach.  
 
 Appendix: We present the hyperparameters used in our experiments in Table 1 , where we mostly follow the network architectures and hyperparameters used by [28, 14] . We use layer normalization [90] for all MLP layers and we use normalized 10-dimensional output features for the goal encoder of state-goal value function to make them easily predictable by the high-level policy, as discussed in Appendix A.For the subgoal steps k, we use k = 50 (AntMaze-Ultra), k = 15 (FetchReach, FetchPickAndPlace, and SawyerReach), or k = 25 (others). We sample goals for high-level or flat policies from either the future states in the same trajectory (with probability 0.7) or the random states in the dataset (with probability 0.3). During training, we periodically evaluate the performance of the learned policy at every 20 episode using 50 rollouts.Subgoal Steps. In order to examine the impact of subgoal step values (k) on performance, we conduct an evaluation of our method on AntMaze tasks. We employ six distinct values for k ∈ {1, 5, 15, 25, 50, 100}. The results, depicted in Figure 9 , shed light on the relationship between k and performance outcomes. Remarkably, our method consistently demonstrates superior performance when k falls within the range of 25 to 50, which can be identified as the optimal range. Our method exhibits commendable performance even when k deviates from this range, except in cases where k is excessively small. These findings underscore the resilience and efficacy of our method across various subgoal step values.Ablation on Subgoals and Exploration Guidance. To demonstrate how subgoals and exploration guidance contribute to efficient policy learning for goal-reaching tasks, we conduct ablation experiments where we remove each component separately. The results, as shown in the Figure 10 , highlight the crucial importance of subgoal setting, as the absence of subgoals hinders the resolu-Figure 9 : Ablation study of the subgoal steps k. Our method generally achieves the best performances when k is between 25 and 50. Even when k is not within this range, ours mostly maintains reasonably good performance unless k is too small (i.e., ≤ 5). tion of long-horizon tasks. Additionally, incorporating exploration guidance facilitates the policy in efficiently reaching subgoals, resulting in further improvements in learning efficiency. Overall, our findings indicate that including both subgoal setting and exploration guidance enables our approach to leverage the benefits of both, leading to efficient learning efficiency.Visualization of End-Effector Trajectories in the FetchReach Task. We fixed the goal at a farther location from the starting point, thereby increasing the task difficulty. In the Figure 11 , the black dot represents the starting point, and the green dot represents the final goal. As shown in the visualization, other methods explore outward evenly from the start point. Despite incorporating active exploration, they struggle to reach the goal point due to the vast state space. Our method, on the other hand, avoids aimless exploration and efficiently reaches the goal point, which aligns with our conclusions.Online ExPLORe RND Ours SawyerReach environment, derived from multi-world, involves the Sawyer robot reaching a target position with its end-effector. The observation and goal spaces are both 3-dimensional Cartesian coordinates, representing the positions. The state-to-goal mapping is a simple identity function, ϕ(s) = s, and the action space is 3-dimensional, determining the next end-effector position.FetchReach and FetchPickAndPlace environments in OpenAI Gym feature a 7-DoF robotic arm with a two-finger gripper. In FetchReach, the goal is to touch a specified location, while Fetch-PickAndPlace involves picking up a box and transporting it to a designated spot. The state space comprises 10 dimensions, representing the gripper's position and velocities, while the action space is 4-dimensional, indicating gripper movements and open/close status. Goals are expressed as 3D vectors for target locations.Maze2D is a goal-conditioned planning task, which involves guiding a 2-DoF ball that can be forceactuated in the cartesian directions of x and y. Given the starting location and the target location, the policy is expected to find a feasible trajectory that reaches the target from the starting location avoiding all the obstacles.AntMaze is a class of challenging long-horizon navigation tasks where the objective is to guide an 8-DoF Ant robot from its initial position to a specified goal location. We evaluate the performance in four different difficulty settings, including the "umaze", "medium" and "large" maze datasets from the original D4RL benchmark. While the large mazes already pose a significant challenge for long-horizon planning, we also introduce an even larger maze "ultra" proposed by [91] . The maze in the AntMaze-Ultra task is twice the size of the largest maze in the original D4RL dataset. Each dataset consists of 999 length-1000 trajectories, in which the Ant agent navigates from an arbitrary start location to another goal location, which does not necessarily correspond to the target evaluation goal. At test time, to specify a goal g for the policy, we set the first two state dimensions (which correspond to the x-y coordinates) to the target goal given by the environment and the remaining proprioceptive state dimensions to those of the first observation in the dataset. At evaluation, the agent gets a reward of 1 when it reaches the goal.CALVIN is another long-horizon manipulation environment features four target subtasks. We use the offline dataset provided by [92] , which is based on the teleoperated demonstrations from [79] .The dataset consists of 1204 length-499 trajectories. In each trajectory, the agent achieves some of the 34 subtasks in an arbitrary order, which makes the dataset highly task-agnostic [92] . At test time, to specify a goal g for the policy, we set the proprioceptive state dimensions to those of the first observation in the dataset and the other dimensions to the target configuration. At evaluation, the agent gets a reward of1 whenever it achieves a subtask.Learning Efficiency. Introducing relabeling can enhance learning efficiency. HER [81] relabels the desired goals in the buffer with achieved goals in the same trajectories. CHER [93] goes a step further by integrating curriculum learning with the curriculum relabeling method, which adaptively selects the relabeled goals from failed experiences. Drawing from the concept that any trajectory represents a successful attempt towards achieving its final state, GCSL [82] , inspired by supervised imitation learning, iteratively relabels and imitates its own collected experiences. [94] filters the actions from demonstrations by Q values and adds a supervised auxiliary loss to the RL objective to improve learning efficiency. RIS [83] uses imagined subgoals to guide the policy search process. However, such methods are only useful if the data distribution is diverse enough to cover the space of desired behaviors and goals and may still face challenges in hard exploration environments.Online: A standard off-policy actor-critic algorithm [69] which trains an actor network and a critic network simultaneously from scratch that does not make use of the prior data at all.RND: Extends the Online method by incorporating Random Network Distillation [4] as a novelty bonus for exploration. given an online transition (s, a, r, s ′ ), and RND feature networks f ϕ (s, a), f (s, a), we setEQUATION)and use the transition (s, a, r, s ′ ) in the online update. The RND training is done the same way as in our method where a gradient step is taken on every new transition collected.HER: Combines Online method with Hindsight Experience Replay [81] to improve data efficiency by re-labeling past data with different goals.GCSL: Trains the policy using supervised learning, leading to stable learning progress.This method [83] incorporates a separate high-level policy that predicts intermediate states halfway to the goal. By aligning the subgoal reaching policy with the final policy, RIS effectively regularizes the learning process and improves performance in complex tasks.ExPLORe: This approach learns a reward model from online experience, labels the unlabeled prior data [6] with optimistic rewards, and then uses it concurrently alongside the online data for downstream policy and critic optimization.F.2 offline-online baselines AWAC: AWAC combines sample-efficient dynamic programming with maximum likelihood policy updates, providing a simple and effective framework that is able to leverage large amounts of offline data and then quickly perform online fine-tuning of reinforcement learning policies. where the changes from standard CQL are depicted in red.SPOT: This work constrains the policy network in offline reinforcement learning (RL) to not only be within the support set but also avoid the out-of-distribution actions effectively unlike the standard behavior policy through behavior regularization.PEX: This work introduces a policy expansion scheme. After learning the offline policy, it is included as a candidate policy in the policy set, which further assists in learning the online policy. This method avoids fine-tuning the offline policy, which could disrupt the learned policies, and instead allows the offline policy to participate in online exploration adaptively.8th Conference on Robot Learning (CoRL 2024), Munich, Germany. 


Decision: Initially the reviewers raised concerns regarding the exploration rewards used and random subgoal sampling, among others. The rebuttal submitted by the authors addressed these concerns to a large extent and now all reviewers uniformly recommend acceptance.

Reviews: review: title: title: Review
paper_summary: paper_summary: Proposes an approach from learning from sub-optimal data by learning to propose reasonable subgoals. This is used with a learned goal-conditioned policy. Experiments are conducted on some mujoco environments involving arms and mazes for locomotion.
main_review: main_review: 1. Subgoal generation 

Suggoals are generated using a combination of the action-free state-goal value function, and advantage weighted imitation of states that either appear in the future of the trajectory, or are randomly picked from the dataset. Why are randomly sampled goals reasonable subgoals? These could be arbitrarily far away from the current state, making them infeasible. Further, if the trajectory is sub-optimal, then picking future states in the trajectory will not necessarily be reasonable. If the action-free state-goal value function is unreliable with increasing distance, then it in unclear why resorting to states that are potentially arbitrarily far away or from suboptimal trajectories is the right correction. There is prior work that trains world models on potentially suboptimal prior data [1,2], enabling accurate long horizon prediction and showing efficient exploration. These works have not been mentioned or compared against.   


2. Exploration reward 

Exploration is induced in goal conditioned policy learning using a specific reward defined in Equation 9, which compares the state-goal value function at the current and next states. It is not immediately clear why this formulation should promote exploratory behavior, and is not discussed in the paper. If the motivation relates to a notion of uncertainty, the authors should compare to prior work on exploration using uncertainty estimation, via prediction error [3], model disagreement [4], and world model disagreement for long horizon prediction [1,2]. This comparison is important since the exploration ability of the method is stressed as an important point, and this is the key part of the algorithm that encourages exploration.

[1] : Planning to Explore via Self-Supervised World Models, Sekar et al.
[2] : Discovering and Achieving Goals via World Models, Mendonca et al.
[3] : Curiosity-driven Exploration by Self-supervised Prediction, Pathak et al.
[4] : Self-Supervised Exploration via Disagreement, Pathak et al.
questions: questions: 1. Please address the questions regarding subgoal generation listed above.
2. Please address questions regarding exploration reward above, add comparisons requested.
limitations: limitations: 1
review_summary: review_summary: Due to the concerns regarding subgoal generation, the exploration reward, and missing comparisons, I vote for rejection.
title: title: Goal-Reaching Policy Learning from Non-Expert  Observations via Effective Subgoal Guidance
paper_summary: paper_summary: This paper proposed a hierarchical policy learning strategy combining subgoal generation and state-goal value function learning.
main_review: main_review: This paper proposes a framework that combines subgoal generation and state-goal value function learning. The quality of the writing is good, and its motivation is presented well.
Clarity: The framework is clear. However, some details need to be provided. For instance, the reward function of each environment is not provided. If each the reward is a sparse reward, the success of each environment needs to be defined. 
Originality: There are some similar works, but the proposed approach is original.
Significance: The main idea is not very far from HER, and many similar works. The task seems very simple if the observations are all the low dimensional states.
Strengths:
The paper is well structured.
The article conducted a lot of comparative experiments and verified the results in multiple environments.
Weaknesses：
Some details about the task are missing.
The way of generation of prior data is not provided.
The ablation study is weak. The offline methods (like HER) directly use the prior data need to be included.
questions: questions: Please refer to the weaknesses.
limitations: limitations: 2
review_summary: review_summary: Overall, the method is efficient with sufficient experiments provided. Although I worried about its significance in real world, I recommend accepting the paper, but will not argue for my recommendation if other reviewers have a different opinion.
title: title: Review
paper_summary: paper_summary: This paper introduces a goal-conditioned RL framework that can utilize prior data for guiding the learning procedure. Specifically this paper (i) learns a state-goal value functions from prior action-free data and use it for informative exploration and (ii) learns a high-level policy with diffusion models that can generate nearby subgoals. Proposed components are integrated into actor-critic algorithm. Experiments are conducted on a variety of state-based benchmarks.
main_review: main_review: Strengths
- In general, the quality of paper is good with a clear writing. Experiments are thorough and supporting the claims made in the paper.
- The paper is dealing with the practical and important setup of utilizing action-free prior data for accelerating reinforcement learning.

Weaknesses
- The paper is missing experiments in a setup more relevant to real-world robotic setup, for instance, all the experiments are conducted in a state-based benchmarks without vision inputs. Can this method be scalable to setups with visual observations?
- One major limitation I think is that high-level policy is fixed after training with prior observations. While the proposed idea can be robust to some scenerios where prior data is not perfect in Section 5.4, not updating the high-level policy can severely limit the asymptotic performance of the proposed idea, especially when the prior data consists of small amounts of initial exploratory data.
- Can analysis similar to Figure 5 be conducted for manipulation tasks where the gain from the current idea is significant? For instance, what would the end-effector trajectories or joint positions look like throughout the training?
- Minor: move a part that describes how goals are sampled (line 135-137) to maybe line 128, as there's no description on what g means for Equation (3)
questions: questions: - Please address my weaknesses in the main review.
limitations: limitations: 1
review_summary: review_summary: This paper is of good quality but its relevance to real-world robotics is not high. I'm in a borderline between weak accept / weak reject evaluations for this paper, I'd like to read the reviews of other reviewers and read the rebuttal response from the authors before making a final recommendation decision.