Learning to Reach Goals via Iterated Supervised Learning
Authors: Dibya Ghosh
Abhishek Gupta
Ashwin Reddy
Justin Fu
Coline Manon Devin
Benjamin Eysenbach
Sergey Levine
Venue: ICLR 2021 Oral (2021)
Keywords: goal reaching
reinforcement learning
behavior cloning
goal-conditioned RL

Abstract: Current reinforcement learning (RL) algorithms can be brittle and difficult to use, especially when learning goal-reaching behaviors from sparse rewards. Although supervised imitation learning provides a simple and stable alternative, it requires access to demonstrations from a human supervisor. In this paper, we study RL algorithms that use imitation learning to acquire goal reaching policies from scratch, without the need for expert demonstrations or a value function. In lieu of demonstrations, we leverage the property that any trajectory is a successful demonstration for reaching the final state in that same trajectory. We propose a simple algorithm in which an agent continually relabels and imitates the trajectories it generates to progressively learn goal-reaching behaviors from scratch. Each iteration, the agent collects new trajectories using the latest policy, and maximizes the likelihood of the actions along these trajectories under the goal that was actually reached, so as to improve the policy. We formally show that this iterated supervised learning procedure optimizes a bound on the RL objective, derive performance bounds of the learned policy, and empirically demonstrate improved goal-reaching performance and robustness over current RL algorithms in several benchmark tasks. 

Published as a conference paper at ICLR 2021
LEARNING TO REACH GOALS VIA ITERATED SUPERVISED LEARNING
Dibya Ghosh UC Berkeley
Abhishek Gupta* UC Berkeley
Ashwin Reddy UC Berkeley
Justin Fu UC Berkeley
Coline Devin
Benjamin Eysenbach Carnegie Mellon University
Sergey Levine UC Berkeley
UC Berkeley
ABSTRACT
Current reinforcement learning (RL) algorithms can be brittle and difficult to use, especially when learning goal-reaching behaviors from sparse rewards. Although supervised imitation learning provides simple and stable alternative, requires access to demonstrations from a human supervisor. In this paper, we study RL algorithms that use imitation learning to acquire goal reaching policies from scratch, without the need for expert demonstrations or value function. In lieu of demon- strations, we leverage the property that any trajectory is a successful demonstration for reaching the final state in that same trajectory. We propose simple algorithm in which an agent continually relabels and imitates the trajectories it generates to progressively learn goal- -eal-rea behaviors from scratch. Each iteration, the agent collects new trajectories using the latest policy, and maximizes the likelihood of the actions along these trajectories under the goal that was actually reached, so as to improve the policy. We formally show that this iterated supervised learning procedure optimizes a bound on the RL objective, derive performance bounds of the learned policy, and empirically demonstrate improved goal roal-nen performance and robustness over current RL. algorithms in several benchmark tasks.
INTRODUCTION
Reinforcement learning (RL) provides an elegant framework for agents to learn general-purpose purpose behaviors supervised by only a reward signal. When combined with neural networks, RI has enabled many notable successes, but our most successful deep RI. algorithms are far from a turnkey solution. Despite striving for data efficiency, RL algorithms, especially those using temporal difference learning, are highly sensitive to hyperparameters (Henderson et al. (0018) and face challenges of stability and optimization (Tsitsiklis & Van Roy| 1997] van Hasselt et al. 2018 Kumar et al., 2019b), making such algorithms difficult to use in practice.
If agents are supervised not with a reward signal, but rather demonstrations from an expert. the resulting class of algorithms is significantly more stable and easy to use. Imitation learning via behavioral cloning provides a simple paradigm for training control policies: maximizing the likelihood of optimal actions via supervised learning. Imitation learning algorithms using deep learning mature and robust; these algorithms have demonstrated success in acquiring behaviors reliably from high-dimensional sensory data such as images (Bojarski et al. 2016 Lynch et al. 2019). Although imitation learning via supervised learning is not a replacement for RL. the paradigm is limited by the difficulty of obtaining kinesthetic demonstrations from supervisor the idea of learning policies via supervised learning can serve as inspiration for RL agents that learn behaviors from scratch.
In this paper, we present a simple RL algorithm for learning goal- goal-directed policies that leverages the stability of supervised imitation learning without requiring an expert supervisor. We show that when learning dial-directed behaviors using RL, demonstrations of optimal behavior can be generated from sub-optimal data in a fully self-supervised manner using the principle of data relabeling: that every trajectory a successful demonstration for the state that it actually reaches, even if is sub-optimal
First two authors contributed equally. Correspondence di bya www.ghheehhhheeehgoooeoooo key.eerrreeeeeee.....ee

---

Published as a conference paper at ICLR 2021
for the goal that was originally commanded to generate the trajectory. A similar observation of hindsight relabelling was originally made by Kaelbling (1993), more recently popularized in the deep RL literature (Andrychowicz et al. 2017). for learning with off off-policy value-based methods and policy-gradien methods (Rauber et al. 2017). When goal-relabelling. these algorithms recompute the received rewards though a different goal had been commanded. In this work, we instead notice that goal-relabelling to the final state in the trajectory allows an algorithm to re-interpret an action collected by a sub-optimal agent as though it were collected by an expert agent, just for a different goal. This leads to a substantially simpler algorithm that relies only on supervised imitation learning primitive, avoiding the challenges of value function estimation. By generating demonstrations using hindsight relabelling, we are able to apply goal-conditioned imitation learning primitives (Gupta et al. 2019 Ding et al., 2019) on data collected by sub-optimal agents, not just from an expert supervisor.
We instantiate these ideas as an algorithm that we call goal-conditioned supervised learning (GCSL). At each iteration, trajectories are collected commanding the current goal-conditioned policy for some set of desired goals, and then relabeled using hindsight to be optimal for the set of goals that were actually reached. Supervised imitation learning with this generated "expert" data is used to train an improved condiconditioned policy for the next iteration. Interestingly, this simple procedure provably optimizes a lower bound on a well-defined RL objective; by performing self- imiitatiionnnn on all of its own trajectories, agent can iteratively improve its own policy to learn optimal goal-reaching behaviors without requiring any external demonstrations and without learning a value function. While self-imitation RL algorithms typically choose small subset of trajectories to imitate (Oh et al., 2018 Hao et al. 2019) or learn a separate value function to reweight past experience (Neumann & Peters 2009 Abdolmaleki et al., 2018 Peng et al., 2019), we show that GCSL learns efficiently while training on every previous trajectory without reweighting, thereby maximizing data reuse.
The main contribution of our work GCSL, simple goal-reaching RL algorithm that uses supervised learning to acquire policies from scratch. We show, both formally and empirically, that any trajectory taken by the agent can be turned into an optimal one using hindsight relabelling, and that imitation of these trajectories (provably) enables an agent to (iteratively) learn goal-reaching behaviors. That iteratively imitating all the data from a sub-optimal agent leads to optimal behavior is a non-trivial conclusion; we formally verifyy that the procedure optimizes lower-bound on a goal-reaching RL objective and derive performance bounds when the supervised learning objective is sufficiently minimized. In practice, GCSL is simpler, more stable, and less sensitive to hyperparameters than value-based methods, while still retaining the benefits of off- oef-poliy learning. Moreover, GCSL can leverage demonstrations (if available) to accelerate learning. We demonstrate that GCSL outperforms value-based and policy gradient methods on several challenging robotic domains.
2 PRELIMINARIES
Goal reaching. The goal reaching problem is characterized by the tuple A, T, ((g)) where S and A are the state and action spaces, T [ss'|s, a) the transition function, p(so) the initial state distribution, I' the horizon length, and p(g) is the distribution over goal states g E S. We aim to find a time-narying goal-conditioned policy (- (s, g, h): S x S x [T] + A(A), where ((A) the probability simplex over the action space A and h is the remaining horizon. We will say that goal is achieved if the agent has reached the goal at the end of the episode. Correspondingly, the learning problem is to acquire policy that maximizes the probability of achieving the desired goal:
J(x)) = Fgyp((
[Pr, (ST = g)]
Notice that unlike a shortest-pate objective, this final meal-timmstep objective provides no incentive to find the shortest path to the goal. We shall see in Section that this notion of optimality is more than simple design choice: hindsight relabeling for optimality emerges naturally when maximizing the probability of achieving the goal, but does not when minimizing the time to reach the goal.
The final timestep objective is especially useful in practical applications where reaching a particular goal is challenging, but once a goal is reached, it is possible to remain at the goal. When reaching the goal is itself challenging, forcing the agent to reach the goal as fast as possible can make the learning problem unduly difficult. In contrast, this objective just requires the agent to eventually reach, and then stay at the goal, a more straightforward learning problem. In addition, the final timestep objective is useful when trying learn robust solutions that potentially take longer over shorter solutions that
2

---

Published as a conference paper at ICLR 2021
Cullact nillouis
Reiabel
Behaviceal clening relaboled dats
ao(ala, B)
[[8]] dbr B) ...... (8f B]
max E B(nag)) log (,
xo(als.
[ Ogr A) ( oi A]]
C)
pe our
D
larvl procesa
Figure 1: Goal-conditioned supervised learning (GCSL): The agent learns how to reach goals by sampling trajectories, relabeling the trajectories to be optimal in hindsight and treating them expert data, and then performing supervised learning via behavioral cloning.
have lower probability of success. There are certain drawbacks to this formulation: it is not as appropriate when the environment stochasticity prevents the agent from remaining at goal, and it requires knowing an upper bound on the time takes to reach goal.
Goal-conditioned RL. The goal reaching problem can be equivalently defined using the nomen- clature of RL as a collection of Markov decision processes (MDPs) {Mo.ges. Each MDP Mg is defined as the tuple (S, .A, Tg. gg p. T). where the state space, action space, initial state distribution, and horizon as above. For each goal, a reward function is defined as (s) L(s = g). Using this notation, an optimal goal-conditioned policy maximizes the return in an MDP Mg sampled according the goal distribution,
J(T) Bg-p(n) [Erw,y [rg())]]]-
Since the transition dynamics equivalent for different goals, off off-policy value-based methods can use transitions collected for one goal to compute the value function for arbitrary other goals. Namely, Kaelbling (193)) first showed that if the transition (s, a, s',)) was witnessed when reaching specific goal it can be relabeled to (s, a, s', Tg (s)) for an arbitrary goal g E S if the underlying goal reward function is known. Hindsight experience replay (Andrychowiez al., (017) considers specific case of relabeling to when the relabeled goal is another state further down the trajectory.
Goal-conditioned imitation learning. If an agent is additionally provided expert demonstrations for reaching particular goals, behavioral cloning is simple algorithm to learn the optimal policy by maximizing the likelihood of the demonstration data under the policy. Formally, demonstrations are provided as dataset of expert behavior D* {TI, 72.... from expert policy π², where each trajectory т; = {s) ab: aj, .f......... ... optimal for reaching the final state the trajectory. Given parametric class of stochastic, tme-varying policies II, the behavioral cloning objective is to maximize the likelihood of actions seen in the data when attempting to reach this desired goal.
BBC = arg max E,-," [log ग (ag 8c, = 8T, T - -(]] =ETI
for 0 S ST.
LEARNING
POLICIES WITH SELF-IMITATION
In this section, we show how imitation learning via behavior cloning with data relabeling can be utilized in an iterative procedure that optimizes a lower bound on the RL objective. The resulting procedure, in which an agent continually relabels and imitates own experience, not an imitation learning algorithm, but rather an RI algorithm for learning goal-reaching from scratch without any expert demonstrations. This algorithm, illustrated Fig. I is simple and allows us to perform off-policy reinforcement learning for goal reaching without learning value functions.
3.1 GOAL-CONDITIONED SUPERVISED LEARNING
We can attain the benefits of behavioral cloning without the dependence on human supervision by leveraging the following insight: under last-timestep optimality (Equation []. a trajectory that fails to reach the intended goal is nonetheless optimal for reaching the goal it actually reached. As result, trajectory from a sub-optimal agent can be re-interpreted by goal-conditioned behavior cloning as an optimal trajectory for reaching a potentially different goal. This insight will allow us to convert sub-optimal trajectories into optimal goal reaching trajectories for different goals, without the need for any human supervision.
3

---

Published as a conference paper at ICLR 2021
Algorithm 1 Goal-Conditioned Supervised Learning (GCSL)
1: Initialize policy TT( | s. s. At)
2: Initialize dataset (((s, a, 4:)
3: for k 1,,,3,.... do
4: Sample g ~ p(g), collect data with TT( ',g)-
5: Log trajectory т (so: ao, $1, a1. ST, OT)
6: Add tuples D- to dataset D
D sec Eg
7: T++1 f- arg max ED [log (a s, g, h)]
8: end for
More precisely, consider trajectory {81, 41 82, d2....,,TT: "r,..,,,,a"a ar) obtained by commanding the policy Na (a s. 9, /t) to reach some goal For any time step t and horizon h, the action az in state 8, is likely to be good action for reaching 5t+h in time steps (even if it is not a good action for reaching the originally commanded goal g). and thus can be treated as expert supervision for Tp(- St. stth, h)). To obtain concrete algorithm, we can relabel all time steps and horizons in trajectory to create an expert dataset according to
D- {(st oz, 9 stth, h) :t, 0, 1 h
with states St. corresponding actions at, the corresponding goal set to future state st+h and matching horizon h. Because the relabeling procedure valid for any horizon, we can use any valid combination of (st, St+h. h)) tuples as supervision, for total of (2) optimal datapoints of (s, a, g. h) from a single trajectory. This idea is related to data-relabeling for estimating the value function (Kaelbling
1993; AAdrychowicz et al. 2017; Rauber et al. 2017). but our work shows that datarrela can also be used to re-interpret data from a sub-optimal agent as though the data came from an optimal agent (with a different goal).
We then use this relabeled dataset for goal-conditioned behavior cloning. Algorithm summarizes the approach: (1) Sample a goal from target goal distribution p(g). (2) Execute the current policy xx (alss, g, h) for 7 steps in the environment to collect potentially suboptimal trajectory (3) Relabel the trajectory (Equation. 3 add (2) new expert tuples (st, ae; 8t+h; h) the training dataset. (4) Perform supervised learning on the entire dataset to update the policy Tals, g, h) via maximum likelihood. We term this iterative procedure of sampling trajectories, relabeling them, and training policy until convergence
condiionit supervised learning (GCSL). This algorithm can use all of the prior off- off-policy data in the training dataset because this data continues to remain optimal under the notion of goal- goal-reaching optimality that was defined in Section 2 but does not require any explicit value function learning. Perhaps surprisingly, this procedure optimizes a lower bound on an RL objective, as we will show in Section 3.2
The GCSL algorithm (as described above) can learn to reach goals from the target distribution p(g) simply using iterated behavioral cloning. This goal reaching algorithm off-policy, optimizes simple supervised learning objective, and is easy to implement and tune without the need for any explicit reward function engineering or demonstrations. Additionally, since GCSL uses a goal- conditioned imitation learning algorithm as sub procedure, if demonstrations or off off-policy data are available, it is easier to incorporate this data into training than with off-policy value function methods.
3.2 THEORETICAL ANALYSIS
We now formally analyze GCSL to verify that solves the goal-reaching problem, quantify how emvor in approximation of the objective manifest in goal-reaching performance, and understand how relates to existing RI. algorithms. Specifically, we derive the algorithm as the optimization of a lower bound of the true goal-reaching objective, and we show that under certain conditions the environment, minimizing the GCSL objective enables performance guarantees on the learned policy.
We start by describing the objective function being optimized by GCSL. For ease of presentation, we make the simplifying assumption that the trajectories are collected from a single policy Volds and that relabelling is only done with goals at the last timestep (g s.). GCSL performs gonl-condi behavioral cloning on a distribution of trajectories (T) = Eg-p(o) [T))) resulting in the following objective:
4

---

Published as a conference paper at ICLR 2021
(x) = R Bewwoiaat)
log T(a = 0z/s 8t, 9 8T,
T
Our main result shows that, under certain assumptions about the off-policy data distribution, optimiz- ing the GCSL objective JUCSI (x) optimizes lower bound on the desired objective, J(T). Theorem 3.1. Let JOCSL and J be as defined above. Then,
[TT 2 (T) 4T 1)a² + C.
Where a ga (T [Tual h)) and C is constant independent of
The proof is in Appendix B.I This theorem provides a lower-bound on the goal-reaching objective with equality for the optimal policy; akin many proofs for direct policy search methods, the strongest guarantees are provided under on on-policy data collection (a = 0). The analysis raises two quantify the tightness of the bound given by Theorem 3.1 and what does optimal solution to the GCSL objective imply about performance on the true objective?
questions:
The tightness of the bound depends
choices in the algorithm: how off-policy data from wold is used to optimize the objective, and how the relabeling step adjusts the exact distribution of data being trained on. We find that the looseness induced by the relabeling factors: 1) the proportion of data that must be relabeled, and 2) the distance between the distribution of trajectories that needed be relabeled and the distribution of trajectories that achieved the desired goal and were not relabeled. If either of these quantities is minimized to zero, the looseness of the bound that stems from relabeling also goes to zero. We present this analysis formally in Appendix(B.2
be controlled by two
Even when data is collected from an off-policy distribution, optimizing the GCSL objective over the full state space can provide guarantees on the performance of the learned policy. We write n² to denote policy that maximizes the true performance J(T), and म* to denote the policy that maximizes the GCSL objective JGCSL over the set of all policies. The following theorem provides such performance guarantee for deterministic environments (proof in Appendix B.3):
Theorem 3.2. Consider an environment with deterministic dynamics and a data-collection policy Told with full support. If g. Drv
Drv(=(a))) (T(a)s, g, g,h,, h), 7 (a s, g, h)) S e, then J(=²*) ((t) < éT.
This theorem states that in environment with deterministic transitions, the policy that maximizes (T) (T) also maximizes the true performance J (TT). Furthermore, if the GCSL loss is approximately minimized, then performance guarantees can be given as a function of the eтror across the full state space. Whereas Theorem 3.1 shows that GCSL always optimizes a lower bound on the RL objective when iteratively re-collecting data with the updated policy, Theorem 3.2 shows that in certain environments, simply optimizing the GCSL objective from any off-policy data distribution without iterative data collection can also lead convergence.
the GCSL objective JCasI
RELATED WORK
Our work studies the problem of goal-conditioned RI. (Kaelbling 1993) from sparse goal-reaching rewards. To maximize data-efficiency in the presence of sparse rewards, value function methods use off-policy hindsight relabeling methods such as hindsight experience replay (Andrychowicz et al.) 2017) to relabel rewards and transitions retroactively (Schaul et al. 2015 Pong et al. 2018)). Despite the potential for learning with hindsight, optimization of goal-conditioned value functions suffers from instability due to challenging critic estimation. Rauber et al. (2017) extends hindsight relabelling to policy gradient methods, but hampered by high-variance importance weights that emerge from relabelling. Our method also relabels trajectories in hindsight, but does so in completely different way: to supervise an imitation learning primitive learn the optimal policy. Unlike these methods, GCSL does not maintain estimate value function, enabling a more stable learning problem, and more easily allowing the algorithm to incorporate off-policy data.
Hussein et al. 2017) via behavioral cloning (Pomerleaul 1989). Recent works have also considered imitation learning with goal relabeling for learning from human play data (Lynch et al. 2019
GCSL is inspired by supervised imitation learning (Billard al.) 2008
Gupta et al. 2019) or
demonstrations (Ding et al. (2019). While GCSL is procedurally similar to
Lynch et al. (019)) and
Ding et al. (2019), it differs crucially the type of data used to train the policy GCSL I trained
5

---

Published as a conference paper at ICLR 2021
Figure 2: Evaluation Tasks: We study the following goal-reaching tasks: (from left right)
2D navigation, robotic pushing, Lunar Lander, robotic door opening, dexterous object manipulation.
on data collected by the agent itself from scratch, not from an expert or (noisy) optimal supervisor. The fact that the same algorithmic procedure for training on optimal demonstrations can be applied iteratively using data from sub-optimal agent to learn from scratch is non-trivial and constitutes one of our contributions.
GCSL has strong connections to direct policy search and self-imitation algorithms. Direct policy
search methods (Mannor et al., 2003 Peters Schaal 2007 Theodorou et al., 2010; Goschin
ct al. 2013 Norouzi et al. 2016 Nachum et 2016) selectively weight policies or trajectories by their performance during learning, as measured by the environment's reward function or a learned value function, and maximize the likelihood of these trajectories using supervised learning. Similar algorithmic procedures have also been studied in the context of learning models for planning (Pathak et al., 2018 Saaviovv al.) 2018 Eysenbach
al., 2019). GCSL is also closely related to self- imitation learning, where a small subset of trajectories are chosen to be imitated alongside an RL objective (Oh et al., 2018) [Ha et al. 2019), often measured using a well-shaped reward function. However, GCSL neither relies on a hand-shaper reward function nor chooses a select group of elites, instead using goal relabeling to imitate every previously collected trajectory for higher data re-use and sample efficiency. coall-oondiionee self-imitation learning methods when combined with meta-controller, have been shown by Ecoffet al. (2020) learn policies much faster in single-task sparse reward setting. This line of work presents avenue for using GCSL to aid learning in the single-task setting as well.
EXPERIMENTAL EVALUATION
In our experiments, we comparatively evaluate GCSL on number of goal-conditioned tasks. We focus answering the following questions:
Does GCSL effectively learn goal-conditioned policies from scratch?
2. Can GCSL learn behaviors more effectively than standard RL methods?
3. Is GCSL less sensitive to hyperparameters than value-based methods?
4. Can GCSL incorporate demonstration data more effectively than value-based methods?
5.1 EXPERIMENTAL FRAMEWORK
We evaluate GCSL on five simulated control environments for goal-reaching: 2D room navigation, object pushing with a robotic arm, the classic Lunar Lander game, opening door with a robotic arm, and object manipulation with a dexterous 9 DoF robotic hand (referred to as claw manipulation), shown in Figure (Environments from/Nar et al. 2018 Ghosh al. |019) Ahn et al. 2019) details in Appendix A. These tasks allow us to study the performance of our method under variety of system dynamics, in settings with both easy and difficult exploration. For each task, the target goal distribution corresponds to uniform distribution over reachable configurations. Performance is quantified by the distance of the agent to the goal at the last timestep. We present details about the environments, evaluation protocol, hyperparameters, and an extended set of results in Appendix A We have additionally open-sonee open-sourced our implementation at aatttps:: //tp:::://iiee /tt:://gittte com/ www.aghoshhhhccoooooooooo.o.h /css
For the practical implementation of GCSL, we parameterize the policy as a neural network that takes in state, goal as input, ignoring the horizon, and outputs a distribution over actions. Although in general, the optimal policy does vary with the horizon, in environments where it is possible to stay the goal, a Markovian policy that reaches, then stays at the goal can be near-optimal. Empirically, we also find that Markovian policies exhibit more coherent exploratory behavior than horizon-van horizon-varying

---

Published as a conference paper at ICLR 2021
Four Rooms
Sawyer Pushing
Lunar Lander
Distance
0.50
0.15
0.10
0.25
0.05
0.0
0.5
1.0
1.5
2.0
0.5
1.0
0.0
0.5
1.5
2.0
1e5
1e6
1e5
Door Opening
Claw Manipulation
0.4
1.5
Distance
GCSL
0.2
PPO
0.5
TD3-HER
0.0
0.0
0.5
1.0
0.0
0.5 Timesteps
1.0
Timesteps
1e6
1e6
Figure 3: On a majority of tasks, GCSL performs well or better compared to more complex RL algorithms like PPO (Schulman et al., 2017) or TD3-HER (Andrychowicz et al. 2017). Shaded regions denote the standard deviation across 5 random seeds (lower is better).
policies. In our main results, the GCSL policy network ignores the horizon provided to the policy, design decision we ablate in Section 5.3 Implementation details for GCSL are in Appendix A.1
5.2 LEARNING GOND-CONDD POLICIES
We first evaluate the effectiveness GCSL. for reaching goals on the domains visualized in Figure 2 covering variety of control problems spanning robotics and video games. To better understand the performance of our algorithm, we provide comparisons to value-based methods utilizing hindsight experience replay (HER) (Andrychowicz established families of RL algorithms for solving goal- coal-conditioned tasks. In particular, we compare against TD3-HER, off-policy temporal difference RL algorithm that combines TD3 (Fujimoto ctt al. 2018) (an improvement on the DDPG method used by Andrychowicz et al. (2017)) with HER. TD3-HER requires significantly more machinery than GCSL: while GCSL only maintains policy, TD3-HER maintains policy, value function, target policy, and target value function, all of which are necessary for good performance. We also compare with PPO (Schulman et al. (017)) state-of-the-ant on on-policy policy gradient algorithm that does not leverage data relabeling, but is known to provide more stable optimization than off-policy methods and perform well on typical benchmark problems. Details for the training procedure for these comparisons, hyperparameter and architectural choices, as well as some additional comparisons are presented Appendix A..
al. 2017), and policy-gradiens methods, two well
show The results in Figure 3 show that GCSL generally performs well or better than the best performing prior RL method on each task, only losing out slightly to PPO on the door opening task, where exploration is less of challenge. GCSL outperforms both methods by large margin the pushing and claw tasks, and by a small margin on the lunar lander task. These empirical results suggest that GCSL, despite its simplicity, represents stable and appealing alternative to significantly more complex RL methods, without the need for separate critics, policy gradients, or target networks.
5.3 ANALYSIS OF LEARNING PROGRESS AND LEARNED BEHAVIORS
To analyze GCSL, we evaluate its performance in a number of scenarios, varying the quality and quantity of data, the policy class, and the relabeling technique (Figure 5) Full details for these scenarios and results for all domains are in Appendix
First, we study how varying the policy class can affect the performance GCSL. Section 5.1 we hypothesized that GCSL with Markovian policy would outperform time-narying policy. Indeed, allowing policies be vire-ng domains like Lunar Lander; on domains requiring more exploration like the Sawyer pushing task, exploration using time-varying policies is ineffective and degrades performance.
Varring Policy" in Figure 55) speeds up training on
7

---

Published as conference paper at ICLR 2021
1.0
Success Raiio
0.5
GCSL TD3
0.0
Four Rooms
Sawyer Pushing
Claw Manipulation
Lunar Lander
Door Operiing
env
Distribution of final performance of GCSL and TD3-HER across nine hyperparameter configurations in each environment (see Section for details). Higher values indicate better performance, and tightly clustered distributions indicate lower sensitivity to hyperparameters. GCSL is more performant and robust to hyperparameters than TD3-HER.
Figure 4: Hyperparameter Robustness:
To investigate the impact of the data-collection policy, consider variations that collect data using fixed policy or train only on on-policy data. When collecting data using a fixed policy ("Fixed Data Collection" in Figure 5}. the algorithm learns much slower, suggesting that iterative data collection is crucial for GCSL. By forcing the data to be on-policy ("On-Policy" in Figure 5). the algorithm cannot utilize all data seen during training. GCSL still makes progress in this case, but more slowly. We additionally consider limited-how -hriied relabeling, in which only states and goals that are at most 3 steps apart are relabeled, similar to proposals in prior work (Pathak al., 2018 Savinov al.) 2018). Limiting the horizon degrades performance ("Limited relabeling" in Figure indicating that multi-horizon relabeling important.
Finally, we discuss the concern that since GCSL uses final-timestep optimality, may provide significantly dif- ferent behaviors than shortest-path optimality. While in theory, GCSL can learn round-about trajectories or oth- erwise exhibit pathological behavior, find that empirical benchmarks, GCSL learns fairly direct goal- reaching behaviors (visualized in Appendix C.2). Since even the tire-narying policy shares network parameters for different horizons, hypothesize that the policy is constrained to produce behaviors that are roughly con- sistent through time, resulting in directed behaviors that resemble shortest-path optimality.
Lander: Ablations
GCSLL
Fixed Data Collection
Limited
Distance (m)
On Policy
Tary-Varying Pollay
0.0
0.5
1.0
1.5
2,0
Environment Steps
165
Sawyer Pushing: Ablations
5.4 ROBUSTNESS TO HYPERPARAMETERS
(m)
0.100
Our next experiment tests the hypothesis that GCSL is more robust hyperparameters than value-based RL methods like TD3-HER. The intuition is that, while dy- namic programming methods are known be quite sen-
0.075
0.050
0.025
sitive to hyperparameters (Henderson et al.) 2018), su- pervised learning techniques seem more robust. We ran sweep across nine hyperparameter configurations, varying network capacity (size the hidden layers [250, 500, 1000)) and frequency of gradient updates (gra- dient updates per environment step in [1, 2, 4)). We com- pared both GCSL and TD3-HER and plotted the distribu-
0.2
0.4
0.8
Environment Steps
Figure 5: Ablations of GCSL on Lunar Lan- der and pushing. Other domains Appendix
A.4
tion of final timestep performance across all possible configurations in Fig. We observe that the distribution of performance for GCSL is more tightly clustered than for TD3-HER, indicating lower sensitivity to hyperparameters. We emphasize that GCSL has fewer hyperparameters than TD3-HER: since GCSL does not learn value function, does require parameters for the value function architecture, target update frequency, discount factor, or actor update frequency.
5.5 INITIALIZING WITH DEMONSTRATIONS
As GCSL can relabel and imitate trajectories from arbitrary sources, the algorithm amenable to initialization from logs of previously collected trajectories or from demonstration data collected by
8

---

Published as a conference paper at ICLR 2021
an expert. In this section, we compare the performance of GCSL bootstrapped from expert demon- strations to TD3-HER. Both methods can in principle utilize off off-po demonstrations: however, our results in Figure 6 show that GCSL benefits substantially more from these demonstrations, While value-based RI. methods are known to struggle with data that is far off-poliey (Kumar et al. 2019a). the simple supervised learning procedure in GCSL can take advantage of such data easily.
In this experiment, we provide the agent with a set of demonstration trajectories, each for reaching a different goal. GCSL adds this data to the initial dataset, without any other modifications to the algorithm. For TD3-HER, we incorporate demonstrations following the setup of Ve- cerik et al. (2017). Even with these measures, the value function in TD3-HER still suffers degraded performance and error accumulation during pre-training- When expert demonstrations are provided for the robotic pushing envi- ronment (Figure 6 GCSL progressively improves faster than when from scratch, but TD3 is unable to improve substantially beyond the original behavioral-clonee policy. We hypothesize that the difference in performance largely occurs because of the instability and optimism bias present when training value functions using demonstrations.
Sawyer Pushing with Demos
GCSL TD3HER
0.12
0.10
Distanc
0.08
0.06
0.0
0.2
0.4
0.6
1.0
Environment Steps
teb
Figure 6: Demonstrations: GCSL incorpo-
rates expert demonstrations more effectively than TD3-HER.
DISCUSSION
We proposed GCSL, a simple algorithm that uses supervised learning on its own previously collected data iteratively learn goal-reaching policies from scratch. GCSL lifts several limitations of previous goal-reaching methods: it does not require a hand-defined reward, expert demonstrations, or the need to learn a value function. GCSL often outperforms more complex RL algorithms, is robust to hyperparameters, uses off-policy data, and can incorporate expert demonstrations when they are available. The current instantiation of GCSL is limited in exploration, since relies primarily on the stochasticity of the policy to explore: a promising future direction would be to selectively reweight the sampled rollouts to promote novelty-seeking exploration. Nonetheless, GCSL is simple, scalable, and readily applicable step towards the fully autonomous learning of goal-directed agents.
ACKNOWLEDGEMENTS
This research was supported by an NSF graduate fellowship, Berkeley DeepDrive, the National Science Foundation, the Office of Naval Research, and support from Google, Amazon, and NVIDIA. We thank Karol Hausman, Ignasi Clavera, Aviral Kumar, Marvin Zhang, Vikash Kumar for thoughtful discussions, insights and feedback on paper drafts.
REFERENCES
Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum posteriori policy optimisation. arXiv preprint arXiv: 1006.:88000069000 2018.
Michael Ahn, Henry Zhu, Kristian Hartikainen, Hugo Ponte, Abhishek Gupta, Sergey Levine, and Vikash Kumar. Robel: Robotics benchmarks for learning with low-cost robots, 2019.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience replay. In Advances in Neural Information Processing Systems, pp. (048-5058 2017.
Aude Billard, Sylvain Calinon, Ruediger Dillmann, and Stefan Schaal. Robot programming by demonstration. Springer handbook of robotics, pp. 1371-1394, 2008.
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D. Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, Xin Zhang, Jake Zhao,

---

Published as a conference paper at ICLR 2021
and Karol Zieba. End to end learning for self-driving cars. CoRR, abs/1604.033116,, 2016. URL httpa//..rggab///16e4ee/ httpa/aas/1111..31
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. CoRR, (bbs/1606.01540, 2016. URL httphtttp///xxivorrrt
abs/1606. 01540)
Yiming Ding, Carlos Florensa, Mariano Phielipp, and Pieter Abbeel. Goal conditioned imitation learning. Advances in Neural Information Processing Systems, 2019.
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. First return then explore, 2020.
Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. Search the replay buffer: Bridging planning and reinforcement learning. arXiv preprint arXiv:190..052.0 2019.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. arXiv preprint arXiv: 2018.
Dibya Ghosh, Abhishek Gupta, and Sergey Levine. Learning actionable representations with goal conditioned policies. In International Conference on Learning Representations, 2019.
Sergiu Goschin, Ari Weinstein, and Michael Littman. The cross-entropy method optimizes for quantiles. In International Conference on Machine Learning, pp. 1193-1201, 2013.
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy leamn ing: Solving long-horizon tasks via imitation and reinforcement learning. CoRR, abs/1910.11956 2019. URL http:
org/ abs 111//991.... 11956
Xiaotian Hao, Weixun Wang, Jianye Hao, and Y. Yang. Independent generative adversarial self- imitation learning in cooperative multiagent systems. AAMAS, 2019.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. Thirty-Second AAAI Conference on Artificial Intelli- gence, 2018.
Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne, Imitation learning: A survey of learning methods. ACM Computing Surveys (CSUR), 50(2):21, 2017.
Leslie Pack Kaelbling. Learning to achieve goals. In International Joint Conference on Artificial Intelligence (IJCAI), pp. 1094- 1098, 1993.
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Proceedings of the Nineteenth International Conference Machine Learning, ICML '02, pp. 267-274, San Francisco, CA, USA, 2002. Morgan Kaufmann Publishers Inc. ISBN 1-55860-873-7.
URL http:ttt/// //cl. acm. org/ citatitttticooo
Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off off-policy qelearning via bootstrapping error reduction. CoRR, 1ttp//006.0999 2019a.
Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off off-policy qelaarning via bootstrapping error reduction. arXiv preprint arXiv: 1w0../0000000 1906.0094, 2019b.
Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. arXiv preprint urXiv:/903.011 2019.
Shie Mannor, Reuven Y Rubinstein, and Yohai Gat. The cross entropy method for fast policy search. In Proceedings of the 20th International Conference on Machine Learning (ICML-03), pp. 512-519, 2003.
Ofir Nachum, Mohammad Norouzi, and Dale Schuurmans. Improving policy gradient by exploring under-appreciated rewards. arXiv preprint arXiv: 1.0922,, 2016.
10

---

Published as a conference paper at ICLR 2021
Ashvin V Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. In Advances in Neural Information Processing Systems, pp. 9191-9200, 2018.
Gerhard Neumann and Jan R Peters. Fitted q-iteration by advantage weighted regression. In Advances in neural information processing systems, pp. 1177-1184, 2009.
Mohammad Norouzi, Samy Bengio, Navdeep Jaitly, Mike Schuster, Yonghui Wu, Dale Schuurmans, et al. Reward augmented maximum likelihood for neural structured prediction. In Advances In Neural Information Processing Systems, pp. 1723-1733 2016.
Junhyuk Oh, Yijie Guo, Satinder Singh, and Honglak Lee. Self-imitation learning, In International Conference on Machine Learning, pp. 3875-3884, 2018.
Deepak Pathak, Parsa Mahmoudieh, Guanghao Luo, Pulkit Agrawal, Dian Chen, Yide Shentu, Evan Shelhamer, Jitendra Malik, Alexei A Efros, and Trevor Darrell. Zero-shot visual imitation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 2050-2053, 2018
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off- off-policy reinforcement learning. arXiv preprint arXiv: 197.000777 2019.
Jan Peters and Stefan Schaal. Reinforcement learning by reward-weighted regression for operational space control. Pruceedings of the 24th international conference on Machine learning, pp- 745-750. ACM, 2007.
Dean A Pomerleau. Alvinn: An autonomous land vehicle in neural network. In Advances in neural information processing systems, pp. 305-313, 1989.
Vitchyr Pong. Shixiang Gu, Murtaza Dalal, and Sergey Levine. Temporal difference models: Model- free deep for model-based control. arXiv preprint arXiv:1802.0908) 2018.
Paulo Rauber, Avinash Ummadisingu, Filipe Mutz, and Juergen Schmidhuber. Hindsight policy gradients. arXiv preprint arXiv: 1.0600, 2017.
Stephane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Geoffrey Gordon, David Dunson, and Miroslav Dudik (eds.), Proceedings of the Fourteenth International Conference Artificial Intelligence and Statis- fics, volume 15 of Proceedings of Machine Learning Research, pp. 627-635. Fort I Lauderdale, FL. USA, 11-13 Apr 2011. PMLR. URL JRRLtttp:: hett::://prrceeednngg. ml wrwpprress/e.. ptt.rrre//eee.ee/esrr//// html
Nikolay Savinov, Alexcy Dosovitskiy, and Vladlen Koltun. Semi-parametti topological memory for navigation. arXiv preprint arXiv:1803.00 00653, 2018
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators. In International conference on machine learning, pp. 1312-1320, 2015.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 1889-1897, Lille, France, 07-09 Jul 2015. PMLR.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1 006347,377 2017.
Evangelos Theodorou, Jonas Buchli, and Stefan Schaal. A generalized path integral control approach to reinforcement learning. journal of machine learning research, 11(Nov)3313731188 2010.
John N Tsitsiklis and Benjamin Van Roy. Analysis of temporal-diffffrrene Icarning with function approximation. In Advances in neural information processing systems, pp. 1075-1081, 1997.
Hado van Hasselt, Yotam Doron, Florian Strub, Matteo Hessel, Nicolas Sonnerat, and Joseph Modayil. Deep reinforcement learning and the deadly triad. ArXiv, abs/1812 05//1822.02655 2018.
11

---

Published as a conference paper at ICLR 2021
Mel Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nicolas Heess, Thomas Rothörl, Thomas Lampe, and Martin Riedmiller. Leveraging demonstrations for deep reinforcement learning robotics problems with sparse rewards, arXiv preprint 17771:770777777 2017.
12

---

Published as a conference paper at ICLR 2021
A EXPERIMENTAL DETAILS
A.1 GOAL-CONDITIONED SUPERVISED LEARNING (GCSL)
GCSL iteratively performs maximum likelihood estimation using a dataset of relabeled trajectories that have been previously collected by the agent. Here we present details about the policy class, data collection procedure, and other design choices.
policy using neural network which takes as input state and goal (not the horizon), and returns probabilities for a discretized grid of actions of the action space. The neural network concatenates the state and goal together, and passes the concatenated input into feedforward network with two hidden layers of size 400 and 300 respectively, outputting logits for each discretized action. Empirically, we have found GCSL to perform much better with larger choices of neural networks; however, we use this two-layer neural network for fair comparisons to TD3-HER. The GCSL loss optimized using the Adam optimizer with learning rate a x 10- with a batch size of 256, taking one gradient step for every step in the environment.
We parameterize
When executing in the environment, the first 10000 environment steps are taken according to uniform random action selection, after which the data-collection policy is the greedy policy: = arg max, x (a|s8,) g). The replay buffer stores trajectories and relabels on the fly, with the size of the buffer subject only to memory constraints. To clarify, instead of explicitly relabeling and storing all (2) possible tuples from a trajectory, we instead save the trajectory and relabel at training time. When sampling from the dataset, a trajectory is chosen at random, start index and goal index t' >> t sampled uniformly random, and the tuple corresponding to this state and goal are relabelled and sampled.
A.2 RL COMPARISONS
We provide comparisons to two goal-reaching RL methods: TD3-HER and PPO. While GCSL is able to efficiently optimize the last-timestep objective, RL methods that learn value functions perform poorly when optimizing this objective. We include results of RL methods optimizing the last-timestep objective for completeness in Figure 9 discounted sum of returns:
but our main RL comparisons instead optimize the classical
(TT = Egrpls) [E Σ (s)ll
(4)
ST
where reaching the goal leads to termination, the reward function remains the same, (s) = 1 (s g) and we use > 0.99. We log other metrics for these algorithms in Figure 12, and have included variety of possible metrics.
TD3-HER (Fujimoto et al. 2018 Andrychowicz et al., 2017p: To efficiently learn value func- tion, we use hindsight relabelling. Specifically, a transition ((s,g), g), a, (s',g) gets relabeled to ((s,g') a, (')) where g' = g with probability 0.1, s s' with probability 0.5, and g' 8 for some future state in the trajectory s, with probability 0.4. As described in Section 2 the agent receives a reward of and the trajectory ends the transition is relabeled to g' =', and otherwise. Under this formalism, the optimal value function, V* g) x TT(s,.), where ( g) the minimum expected time to go from to g. Both the Q-function and the actor for TD3 are parametrized as neural networks, with the same architecture (except final layers) for state-based domains as those for GCSL. We found the default values of learning rate, target update period, and number of critic updates to be the best amongst our hyperparameter search across the domains (single set of hyperparameters for all domains). Since GCSL. uses discretized actions, we additionally compared to a version of TD3-HER that also uses discretized actions (results in Figure The performance of TD3-HER w/ discretized actions decreases performance on the Claw task, increases the Door Opening task, and has the same performance on the other three environments.
PPO (Schulman et al. 2017): Because PPO is on-policy RL algorithm, we cannot relabel goals, unlike in GCSL or TD3-HER. Instead, we provide a surrogate e-ball indicator reward function: r(s.g) = [(d(ssg) g) < e), where e is chosen appropriately for each environment. We emphasize that this reward function makes the policy optimization problem for PPO much easier, since while the other two methods only can check if two states are exactly equal, PPO also has access to the distance
13

---

Published as a conference paper at ICLR 2021
(within an e-ball). To maximize the data efficiency of PPO, performed coarse hyperparameter sweep over the batch size for the algorithm. Just as with TD3, we mimic the same neural network architecture for the parametrizations of the policies GCSL.
A.3 TASK DESCRIPTIONS
For each environment, the goal space is identical the state space; each trajectory in the environment lasts for 50 timesteps.
2D Room Navigation (Ghosh et al. 2019) This environment requires an agent to navigate to points in an environment with four rooms that connect to adjacent rooms. The state space has two dimensions, consisting of the cartesian coordinates of the agent. The agent has acceleration control, and the action space has two dimensions. The distribution of goals p(g) is uniform the state space, and the agent starts in a fixed location in the bottom left room.
Robotic Pushing (Ghosh al., 2019) This environment requires Sawyer manipulator to move freely moving block in an enclosed play area with dimensions 40 cm x 20 cm. The state space is 4-dimensional, consisting of the Cartesian coordinates of the end-effector of the sawyer agent and the Cartesian coordinates of the block. The Sawyer is controlled via end-effector position control with diceesiiinnional action space. The distribution of goals p(g) is uniform on the state space (uniform block location and uniform end-effector location), and the agent starts with the block and end-effector both in the bottom-left corner of the play area.
Lunar Lander (Brockman et al.) 2016) This environment requires a rocket to land in a specified region. The state space includes the normalized position of the rocket, the angle of the rocket, whether the legs of the rocket are touching the ground, and velocity information. Goals are sampled uniformly along the landing region, either touching the ground or hovering slightly above, with zero velocity.
Door Opening: (Nair et al. 2018) This environment requires a Sawyer manipulator to open a small cabinet door, initially shut closed, sitting on table specified angle. The state space consists of the Cartesian coordinates of the Sawyer end-effector and the door's angle. As in the Robotic Pushing task, the three-dimensional action space controls the position the end-effector. The distribution of goals p(g) is uniform on door angles from 0 (completely closed) to 0.83 radians.
Claw Manipulation: (Ahn et al., 2019) A 9-DOF "claw"-like robot is required to turn a valve to various positions... The state space includes the positions of each joint of each claw (3 joints on claws) and embeds the current angle of the valve in Cartesian coordinate (0 + (sin 0, cos 0)). The robot is controlled via joint angle control. The goal space consists only of the claw angle, which is sampled uniformly from the unit circle.
A.4 ABLATIONS
In Section 5.3 we analyzed the performance of the following variants of GCSL (Figure
1. Limited relabeling- Thhis model relabels only states and goals that are most three steps apart: {(S,, at St4h, h) : 0. h < 3}
2. On-Policy Only the most recent 10000 transitions stored and trained on.
3. Fixed Data Collection Data collected according uniform policy over actions.
4. Time- Varye- Policy Policies are conditioned on the remaining horizon. Alongside the state and goal, the policy gets reverse temperature encoding of the horizon as input.
14

---

Published as a conference paper at ICLR 2021
Four Roome Addifors
Saryer Tuaing Adsson
Laver Lander Addiiom
Dour Opeding Abialces
Teallyyng Play
20c4
2004
Roaret
Seaye Puating Averga Scseitte tee
200
Figure 7: Performance across variations of GCSL (Section 5.3 for all experimental domains.
In addition, we also performed an ablation to measure how the performance of GCSL depends on the ratio of gradient steps to environment steps being collected. A larger number of gradient updates indicate higher levels of data re-use. Our results indicate that GCSL is robust to this hyperparameter.
Rooma Adlidora
Lander Adidoo
Openitg Abilices
3110
60
15
16
22
Figure 8: Policy update frequency: Performance when varying the ratio of policy update steps environment steps. GCSL performs well even when significantly more gradient steps are taken on the replay buffer data.
Finally, we compared the performance of our RL comparisons, TD3-HER and PPO, when optimizing for the final- mnal-timestep objective in in Equation 4 To faithfully optimize the final-timestep objective, the policy and value networks for TD3-HER and PPO also take the remaining horizon as input. Our results indicate that TD3-HER and PPO learn slower (and sometimes not at all) when optimizing the final-timestep objective than the discounted return objective. Therefore, for the most fair evaluation, we compare the performance of GCSL to the RL methods that optimize the discounted return objective.
optimizing the discounted return objective
Seuyer Pasting Avange FaalD
Lander
Oper by Avenge fine Dibrrs
ent the
12-
Ender
Figure 9: Optimizing discounted return vs final-timestep objective: Performance of TD3-HER and PPO when optimizing for the final time-step objective and the discounted return objective, as measured by median final distance to the goal. Both RI. methods perform better with the discounted return objective uniformly across environments, main paper.
use the discounted-return comparisons in the
A.S INITIALIZING WITH DEMONSTRATIONS
We train expert policy for robotic pushing using TRPO with shaped dense reward function, and collect a dataset of 200 trajectories, each corresponding to a different goal. To train GCSL using these demonstrations, of training, and optimize the GCSL objective using these trajectories to wtrm-start the algorithm. Initializing value function method using demonstrates requires significantly more attention:
simply populate the replay buffer with these trajectories at the beginning
15

---

Published as a conference paper at ICLR 2021
perform the following procedure. First, perform goal-conditioned behavior cloning to learn initial policy TBC- Next, we collect 200 new trajectories in the environment using a uniform data collection scheme. Using this dataset of 400 trajectories, perform policy evaluation to learn @ using policy evaluation via bootstrapping. Having trained such an estimate of the Q-function, we initialize the policy and Q-function to these estimates, and the appropriate value function RL algorithm.
BC
A.6 HINDSIGHT POLICY GRADIENTS
We compared GCSL, to hindsight policy gradients (HPG) (Rauber et al.] 2017) the Fetch-Reach task from Andrychowicz et al. (2017). Whereas GCSL uses re- labelling conjunction with supervised imitation learning, HPG performs reward relabeling with a policy gradient algorithm, introducing importance weights correct for the distribution shift. As shown in Fig. 10 GCSL solves the task faster than HPG and converges gets closer to the goal. HPG required more than mil- lion samples run reliably on Fetch-Reach, task far easier to solve than any of our benchmark environments (Andrychowicz al. 2017
Fetch Reach
3.-
Diolance
3.-
0.3
GCSL
0.00
822
0.0
0.55
1.00
1.75
2.00
Timeslapa
policy that
Figure 10: GCSL. converges faster and
learns more accurate goal reaching policy than HPG (Rauber et al. 2017)
A.7 SIMPLE EXPLORATION STRATEGIES
Since GCSL learns via self-imitation like many other self- imitation methods, it faces challenges of poor exploration and converging to sub-optimal points (Oh et al. 2018). However, since GCSL can learn with off-policy data, we can deal with exploration challenges by using a directed exploration strategy or by injecting noise into the data collection process.
08-
Sucorss Rate
Delent Explonstio
As simple demonstration. we combined GCSL with an exploration strategy akin to Go-Explore (Ecoffet et al. 2020). This exploration strategy takes greedy actions with the policy for the first 90% of the trajectory, and then takes random actions for the remainder. Intuitively, this can seen as a crude implementation the fringe states, and then randomly exploring beyond the fringe. The main difference with this intuition that we do not actually adjust the goal sampling strategy only reach goals within the fringe. We tested this exploration strategy on 2x larger version of the four rooms domain, where greedy exploration is insufficient to reach all regions in the space. GCSL with the default exploration strategy is unable to reach all the goals, but when combined with this new exploration strategy, is able to learn faster in this environment and reach 15% more goals.
02
Exploration Stnllgg
Timesteps
Figure 11: Exploration strategies: When
combined with the exploration strategy from Ecoffet al. (2020), GCSL explores more and learns a more accurate goal-reaching pol- icy when navigating larger four-rooms domain.
first getting to
16

---

Published as a conference paper at ICLR 2021
Four Rooms
Sawyer Pushing
Lunar Lander
Final
Distancs
0.5
0.15
0.06
0.0
0.5
1,0
1.5
2.0
0.0
0.5
1.0
0.0
0.5
1,0
1.5
2.0
1e5
1e5
166
Opening
Final Dislance
0.4
1.5
GCSL
PPO
TD3-HER
0.5
TD3-HER (Discrete)
1.0
Timesteps
Timesteps
Four Rooms
Sawyer Pushing
Lunar Lander
Minimum
Distance
0.10
0.
0.0
0.05
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1e5
1eB
165
Opening
Claw Manipulstion
Minimum Distacee
0.4
0.0
0.0
0.5
1,0
0.0
0.5 Timesteps
1.0
Timesteps
1e6
168
Four Rooms
Sawyer Pushing
Lunar Lander
Success Rate
1.0
Firal
0.5
0.5
0.25
0.0
0.0
0.5
1.0
Door Opening
Clow Manipulstion
Final Surcess Rate
1.0
0.5
0.5
0.0
0.5
1.0
Timestops
Timesteps
Four Rooms
Sawyer Pushing
Lunar Lander
Rate
Anytime
0.5
0.50
0.5
0.25
0.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
0.0
0.5
1.0
1.5
2.0
165
1e6
165
Door Opening
Clow Manipulsiio
1.0
Anytme
Succoss
0.5
0.0
0.5
1.0
0.0
0.5
1,0
Timesteps
1e8
Timestaps
1e6
Here present plots reporting (1) median distance goal at the final timestep, (2) median minimum distance to goal within a trajectory (3) proportion of trajectories that were at the desired goal at the final timestep (3) proportion of trajectories that were at the desired goal at any timestep. As mentioned in Appendix A.2 we additionally provide comparisons version of TD3-HES HER that uses discretized actions. See Table for time goal metrics
Figure 12: Alternative Evaluation Metrics:
17

---

Published as a conference paper at ICLR 2021
B THEORETICAL ANALYSIS
B.1 PROOF OF THEOREM 4.1
We will assume discrete state space in this proof, and denote trajectory {50, do...., ST, ar). Let the notation (T)) = sT denote the final state of a trajectory, which represents the goal that the trajectory reached. As there can be multiple paths to goal, we let Ty = (T G(T)= = g} denote the set of trajectories that reach a particular goal g. We abbreviate a policy's 's trajectory distribution (r(g)) p(so) IT- (T ( T) az). The target goal- reaching objective we wish to optimize is the probability of reaching a commanded goal, when goals sampled from pre-specified distribution p(g).
J(x) = gg-r()))r=<((to [1[ =
GCSL optimizes the following objective, where the log- pikeliildoood of the actions conditioned on the goals actually reached by the policy, (rr). The distribution of trajectories used to optimize the objective is collected through a different policy, Fold- We write Taa(TT = Egrp(n) (Tta(T())) to concisely represent the marginalized distribution of trajectories from Told-
T log xa, / (ryls, ((T))
(oC = (E
Σ
==
To analyze how this objective relates to JIT, first analyze the relationship between ((r) and surrogate objective, given by
Jsnn(T) Egg-ppl)))) [L(G(T) = g] log (T(gg)]
Theorem from Schulman et al. (2015) states that
4ye
J ( (T)
(1 (1-7)) 7-
where y is a discount factor, is the maximum advantage over all states and actions, and is the total variation distance between и and oold is straightforward to show that the bound can rewritten in the finite-horizon undiscounted case in terms of the horizon T, following Kakade & Langford (002)): Ross et al. (2011), to obtain the bound
((T) 2 Jsur(() - (T 1)ea?.
where T is the horizon of the task. In the setting where data is collected from multiple policies, for example with a replay buffer, the bound cannot rely on the distance between policies at each state, but rather more generally the total variation distance between the trajectory distributions,
(5) Since our reward function is 1[g(T) = g]. the return for any trajectory is bounded between 0 and 1, allowing us to bound above by 1. This leaves a, which is the total variation divergence between and Tata- This divergence may be high if the data collection policy is very far from the current policy, but is low if the data was collected via recent policy.
J(x)) 2 ( (Drv(T(T)) Tota(T)).
We can now bound the surrogate objective with the GCSL objective via the following: (ogppl).r=xaaatte (T) g) log (T(9))
Jaum TT)
Ep(o) Σ (ota(Tlg) log (T =g]
logg(7(9(TT) Ep(o)aaa(T10))1111 Dee
= gl
Σ (lgg((1(((7)) Σ p(o) Σ log ( G(T)) Σ (1))) +
Σ logg(r19((7) Σ ((o)
Ba (log m (77)))
18

---

Published as a conference paper at ICLR 2021
The final line is our goal-relabeling objective: train the policy reach goals reached. The inequality holds since log T()) is always negative. The inequality is loose by a term related to the probability of not reaching the commanded goal, which analyze in the section below.
Since the initial state and transition probabilities do not depend on the policy, we can simplify log (T)) as (by absorbing non r-dependent terms into C2):
E (xy log 5 (r[9(T))] E (ar
logp(so)
Σ log o maals, log (8
T
log log (ag((())))(())) + C2
-ocss + C2.
Combining this result with the bound on the expected return completes the proof:
((x) 2 (aCSL(#) + C1 + C2 4T(T 1)a²
Note that in order for J(x and (GCSL(() to be vacuously zero, the probability of reaching goal under wold must be non-zero. This assumption reasonable, and matches the assumptions on "exploratory data-collection" and full-support policies that required by Q-learning and policy gradient convergence guarantees.
B.2 QUANTIFYING THE QUALITY OF THE APPROXIMATION
The tightness of the bound presented above is controlled from two locations: the off off-policyness of Told with respect to π and the bound introduced by the lower bound in the thcorem. The first is well-studied in policy gradient methods; in particular, when the data is on-policy, the gap between constant. We seek to better understand the gap introduced by Equation Gōlin the analysis above.
Jsurr(m (Tr) and J(x) ) is known to be a
We define (S(T) * g) to be the probability of failure under Tolds and additionally define Porrnn (7) and Prigh((7 (T) be the conditional distribution of trajectories under wold given that it did not reach and did the commanded goal respectively.
In the following section, show that the gap introduced by Equation Saciatioonoaann Gaan be controlled by the probability of making mistake, Prote (G(T) * g), and Drv (Pwrong (T), (T)) measure of the difference between the distribution trajectories that must be relabeled and those not.
We rewrite Equation 6 as follows:
Jsurr (T) Σ log (T](((T) Σ ((g))aaaTTl)) Σ Elog+o (ogg((((((7)) Σ (l #
-E (log ग ((tT(((((())))))))))))))) - Paets (G(T) # 9)) B.npponcettt Dog (T(9((T)
Define D be the Radon-Nikodyae derivative of Pwrong (T) wrt Tota(T)
=Brwxate(t)logg (T]](TT)) Paess (G(T) + g)) BrNmata(t) [DlogT(r(G(7))
= =(1- (1-Prxe Prass (g(T) * g) ET Olo (TggTTTGT(T)))
+ Prota (G(T) & 9)) E, (F) [(1 D) log π (T()(T))
Relevant Gap
The first term affine with respect to the GCSL loss, so the second term is the error we seek to understand. |Relevant Gap
= PRote (9(T) # g) (Br~Taa(T)) [[1 - D) log (T((()))
< Prace (g(7) # g) BT B (log π (T)
2P (9(T) # Drv (ogl Parong (7)) Bra (T][(()))]
Proia (G(T) * g] Praze (G(T) # DrV (Priht(T), Pwrong (T)) Boor
19

---

Published as a conference paper at ICLR 2021
The inequality is maintained because of the nonpositivity of log (T). and the final step holds because (T)a(T) is a mixture of Pwrong (T) and (Tigh(T). This derivation shows that the gap between Jeurr and JocsI (up to affine consideration) can be controlled by (1) the probability of reaching the wrong goal and (2) the divergence between the conditional distribution trajectories which did reach the commanded goal (do not need to be relabeled) and those which did not reach the commanded goal (must be relabeled). As either term goes to O, this bound becomes tight.
B.3 PROOF OF THEOREM 3.2
In this section, we now prove that sufficiently optimizing the GCSL objective over the full state space causes the probability of reaching the wrong goal to be bounded close to 0, and thus bounds the gap close to O.
Suppose we collect trajectories from a policy mota- Following the notation from before, we define (Tid(T) = E (Tdata(Tlg)) For convenience, we define x² (ay (tlss g) x S-ya, Tdata (T) = (T) sr) to be the conditional distribution of actions for given state given that the goal is reached at the end of the trajectory. If this conditional distribution is not defined, we let nT (arlst g) be uniform, that '² (aa St. 9) is well-defined for all states, goals, and timesteps. The notation for x" is suggestive: in fact, it can be easily shown that under the assumptions the theorem, full data coverage and deterministic dynamics, the induced policy n² is in fact the optimal policy for maximizing the probability of reaching the goal.
To show that the GCSL policy also incurs low error, we provide coupling argument, sim- ilar to Schulman et al.] (2015); Kakade & Langford (2002); Ross et al. (2011). Because Drv (T( St, g), (as St, 9)) S E, we can define (I c)-uuleeed policy pair o*), which take differing actions with probability e. By union bound over all timesteps, the probability that and π* take any different actions throughout the trajectory is bounded by eT' and because of the assumptions of deterministic dynamics, take the same trajectory with probability 1- - ET. Now, since the two policies take different trajectories with probability at most éT, simple bound shows that the probability that TGCSI. reaches the goal is at most cT less than т*, leading to our result that the performance gap (xx) (x) < el', In environments in which every state is reachable from every other state in the desired horizon, this provides global performance bound indicating that the optimal GCSL policy will reach the goal with probability at least eT'
20

---

Published as a conference paper at ICLR 2021
C DIRECTNESS OF POLICIES LEARNED BY GCSL
C.I QUANTITATIVE ANALYSIS
We measure how direct the policies learned by GCSL are. To do so, evaluate the following objective:
(7) In other words, the policy reaches the goal in t timesteps, then the performance is 7², and the agent does not, then the performance is (t oo). We choose this metric, compared to exactly measuring the time taken to reach the goal, because more cleanly handles the situation when the goal is not reached by the agent. We use 7 0.99.
ShoetestTime(() = pp)) [8, [ [( T(T,g,e) = inf e N: d(s < e)
We additionally train an oracle agent using PPO to maximize a shaped reward 3 million timesteps. While the performance of this oracle agent and GCSL aren't directly comparable, since the oracle agent has privileged distance information, serves as an rough upper bound on how fast agent reach the goal.
d(s,)) g) for
may seem surprising that the GCSL policy learns to reach the goal so quickly, especially given that it optimizes the final-timestep optimality objective, and the shortest-path objective. This is due two factors. First, all the environments support the "reach the goal and stay at goal" behavior - if were not possible to remain at the goal, then GCSL would not exhibit this type of behavior. Second, the policy we train ignores the horizon, and at state, takes the same action at any timestep. This inductive bias on the policy prevents it from learning hehaviors that remain still for long period of time, and then go to the goal right before the time horizon is reached.
Environment Name
Oracle
GCSL
PPO
TD3-HER
Four Rooms
0.81
0.74
0.68
0.40
Sawyer Pushing
0.53
0.61
0.20
0.33
Lunar Lander
0.65
0,52
0.66
0,30
Door Opening
0.93
0.58
0.95
0.21
Claw Manipulation
0.70
0.68
0.45
0.55
Table 1: Shortest-Time Objective (higher is better): Values Shortes(Timme(1) for the final trained policy for each algorithm. ShortestTimee(1 is smooth measure of how long it takes the agent to first reach the goal. GCSL reaches the goal nearly as quickly as the oracle policy on all the environments but the door opening task. The oracle expert policy is trained dense reward signal with PPO for significantly longer.
C.2 QUALITATIVE ANALYSIS
Figure 13 below shows parts of the state along trajectories produced by GCSL. In Lunar I Lander, this state is captured by the rocket's position, and in 2D Room Navigation it the agent's position. While these trajectories do not always take the shortest path to the goal, they do often take fairly direct paths to the goal from the initial position avoiding very roundabout trajectories.
M
Figure 13: Examples of trajectories generated by GCSL for the Lunar Lander and 2D Room environments. Stars indicate the goal state.
21


Reviews:
[{"review_id": "phW78C6hPN7", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\n\nThe authors propose “GCSL” (goal-conditioned supervised learning), an algorithm that bridges the gap between reinforcement learning and imitation learning. Specifically, the authors are motivated by the limitations that RL is brittle when used with sparse rewards, and that IL requires expert demonstrations. Their technique performs learning without expert demonstrations, and does not require a learned value function or reward function. The authors formulate GCSL as iterative trajectory collection, goal relabeling, and policy refinement through behavioral cloning, and formally derive performance bounds on this technique. In addition, the authors demonstrate strong goal-reaching performance and robustness improvements over current RL techniques.\n\n### Strengths\n\n- The authors’ proposal is strongly motivated, in that RL techniques are clearly sensitive to hyperparameters and face stability challenges, and that using demonstrations is more robust. The paper is clearly written and the authors very cleanly explain their key ideas and insights. \n- The authors’ proposed algorithm is based on a simple idea, in a very good way. Their technique is stable, requires no value functions or reward function engineering (as is common with traditional RL techniques), and their technique of relabeling generates a large number of samples, resulting in better data efficiency.\n- The authors’ main theoretical insight that iterating on data from sub-optimal agents leads to optimal behavior is both non-trivial as the authors claim, and can likely be used to inspire other similar ideas or areas of study.\n- The authors clearly showcase the experimental use cases of their technique by demonstrating its benefits in terms of stability to hyperparameters and in leveraging expert demonstrations. Apart from this strong set of experimental results, the authors also present detailed ablations in the supplementary pages that are very convincing.\n\n---\n\n### Weaknesses\n\n- The theory section does seem a bit contrived. Specifically, optimality emerges when using the objective proposed by the authors, even if this is not commonly used. Further, the proof assumes that trajectories are collected from a single policy and relabeling is only performed on the last timestep, whereas in practice these conditions are not met. The performance guarantee also only holds with deterministic transitions. Finally, the guarantee on the convergence behavior is a good property, but is maybe less meaningful if in practice GCSL is difficult to converge to (if the optimization is challenging). \n- Notably, the authors do not evaluate the generalization ability of their technique, and instead evaluate with the same train and test environment. Although this is common in RL, it would be interesting to see how GCSL performs in environments that require more sample efficiency and test generalization, like Procgen.\n- The authors quantify performance as the distance of the agent to the goal at the last timestep. I’m not convinced this evaluation metric makes the most sense, especially when success ratio is the main metric we care about.\n- On door opening, do the authors have an idea for why TD3+HER performs so much worse than PPO or GCSL? Similarly, on Sawyer pushing, do the authors have a hypothesis for why PPO and TD3+HER do not learn at all? Is there potentially a qualitative analysis of this unexpected behavior that can be performed? What about GCSL makes it so much more successful on Sawyer pushing?\n- I’m surprised that GCSL is not any more sample efficient, since PPO and broadly speaking other policy gradient techniques are generally well-understood to be greedy sample-wise. \n- The authors earlier in the paper hypothesized that an optimal policy would be non-Markovian, but that GCSL with a Markovian policy would outperform a time-varying one. This seems rather counter-intuitive to me, why do the authors suspect this is the case? Is the model overly exploiting instead of exploring when conditioned on the remaining horizon? Does this behavior change depending on the number of timesteps in an episode?\n- In Figure 4, do the on-policy methods that converge slower do so with an empirically visible better convergence guarantee? For instance, even if it takes longer, the technique is guaranteed in the long-run to arrive at an optima?\n- In Figure 4, the authors compare against limited-horizon relabeling as in prior work and show this method has drawbacks. Are the drawbacks due to the lack of multi-horizon relabeling, or due to having much fewer trajectories after relabeling?\n- The authors compare robustness to hyperparameters against TD3, but do not compare against PPO. Because PPO is generally well-understood to not require much hyperparameter tuning empirically, I would recommend including this comparison as well. \n- The authors claim qualitatively (in Appendix C) that despite the different objective, the learned trajectories generally take a direct path to the goal. Is there a way to quantify this quantitatively against an oracle that does take the shortest path; for example by comparing the time taken to get to the goal, or the time spent at the goal waiting for the episode to complete?\n\n---\n\n### Recommendation\n\nOverall, I vote for accepting. I think this current submission is already relatively convincing as an accept, as it is clearly written, has well explained motivations, strong experimental results, and extensive ablations in the supplementary pages. The authors’ idea is conceptually simple, in a good way. My main gripe is that the theoretical results are only weakly held, or less relevant in practical settings, but that is rather typical and the strong experimental results do speak for themselves. I do have a few clarifying questions on the experimental results, but am regardless confident that this paper meets the ICLR acceptance criteria.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "mWlByX4JVtl", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In the paper \"Learning to Reach Goals via Iterated Supervised Learning\", the authors propose a new approach to build conditional policies for reaching tasks that can reuse the previous failed attempts as new examples on how to reach the state that was actually reached during the failed execution. This approach is similar to the approach introduced in HER, but is based on Behavioural Cloning algorithm (thus the name self-imitation) instead of a value function. \n\nThe paper is overall well written, clearly illustrated and appropriately structured. \n\nIn my opinion, the proposed method shares significant similarities with the \"First return then explore\" paper (Ecoffet et al. 2020), which is a new version of the GO-Explore algorithm that builds a similar conditioned policy to \"return\" to a state and then explore from this state using a random action, which is then used to extend and improve the learned policy. Given the strong links between these two papers, I am surprised to not see this discussed related work section and compared in the experimental section. \n\nMy main concern about the algorithm is the risk of a lack of exploration. An extreme example is a degenerated policy that outputs 0 (i.e., no movement) regardless of its inputs. The relabelling will just reinforce this behaviour and the algorithms will quickly converge in a local optimum (which is actually far from being optimal). Obviously, this is an extreme example which is quite unlikely, but the same can happen if some states/goals are very unlikely to be observed given the policy. This can happen for instance, if the policy has a strong bias. \nIn GO-Explore this is avoided by performing a random action at the end of a rollout so that the exploration is not conditioned by the current policy. However, the combinatorial generation of goals/labels might be more challenging in this case and thus leading to fewer data per rollout. Ideally, this should be evaluated and compared in the experimental results. However, I have to say that the experimental evaluation is already quite elaborated and provide interesting insights into the performance of the algorithm. \n\nOverall, this is a nice paper with an interesting algorithm, which could be made better by more discussion on some related work and potential limitations. \n\n\n----------- \nUpdate after rebuttal:\nI am very pleased by the answers of the authors, in particular, with the additional experiment showing that the algorithm could be extended with more advanced exploration strategies. I reviewed my rating accordingly. \n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "OOB6dOiEH3", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a new RL algorithms dedicated to learning goal-oriented policies. In the described setting, a policy has to reach a particular goal in T steps such that s_T = goal i.e the objective is not to reach the goal as soon as possible but in a particular number of steps. The proposed algorithm is very simple: it is an iterative algorithm where, at each iteration, i) a policy is used to collect trajectories. ii) Then, trajectories are relabelled based on the reached states -- reached states are becoming goals in the relabelling, iii) a new policy is learned by using behavioral cloning. In addition to the algorithm, the authors provide some theoretical insight about how and in which conditions one can prove that the algorithm is working well. At last, experiments are made on different environments and compared to some baselines showing that it allows to solve complex problems in less learning iterations. Moreover, an ablation study is provided to allow one to understand the effect of the different aspects of the learning technique. \n\nFirst, the proposed technique is very simple, sharing some similarities with existing techniques (hindsight relabelling), but simplifying these techniques by just using behavioral cloning to discover a new policy instead of other and more complex value-based techniques. This idea may have two advantages: i) it is simple and ii) it may benefit of the robustness of supervised learning techniques, and particularly the fact they supervised learning may need less hyperparameters than other RL approaches. This is an interesting point and a good aspect of the article. \n\nBut the consequence seems to be that the algorithm is optimizing a criterion which is not the classical criterion used when learning goal-oriented policies. Indeed, here, the objective is to find a policy that reach a particular goal in T steps, and not to reach a goal as soon as possible. I think that this aspect is opening different questions: \n\nA) the first one is the interest of learning such policies. The article does not really justify why and in which applications discovering such policies may be interesting. Indeed, reaching a goal in T steps may appear more difficult than reaching a particular goal in less steps, or as fast as possible. Moreover, in a stochastic environment, reaching a goal in T steps may be impossible due to the stochasticity of the environment while reaching the goal in less than T steps may be easier. I think that a better discussion on that point would improve the quality of the paper, while right now, it is not clear to me when such an objective function is interesting to study. \n\nB) The second problem is related and in the experimental section, when comparing to classical RL techniques. Indeed, the reward function defined in this paper is very specific (i.e the agent get one if it reaches the goal at the last state of the trajectory), and classical RL techniques are using a discount factor lower than one which thus encourages the models to discover policies reaching goals as soon as possible. Again, a discussion explaining how exactly the comparison is made, and what is exactly compared would be interesting. It gives me the impression that RL techniques are used in a particularly difficult setting, why they may perform well on different but interesting reward functions encouraging for instance to reach the goal as soon as possible, or to reach the goal once during the T timesteps (and not only at the last state). \n\nC) Also, some aspects are confusing: in section 5.1, it is written \"we parameterize the policy as a neural network that takes in state, goal, and horizon as input\" while in appendix A.1, it is written \"We parameterize a time-invariant policy using a neural network which takes as input state and goal (not the horizon),\" which is completely different. This is a crucial aspect since I do not understand how a policy that is not using the horizon as input is able to reach a goal at a particular timestep. \n\nOther  comments: \nTheorem 3.2 focused on deterministic environments is not really interesting since it considers a very particular case which is not a classical RL setting. \nThe way T is chosen is not clear to me. For instance, in many environments, in order to be able to reach interesting goals, T may have to be very large, and is unknown until a good policy is  known. \nThe extension about using demonstrations is an interesting aspect that would be interesting to develop more in the paper\n\nSummary: \n The paper proposes an interesting and simple model that optimizes a particular and not classical objective. It is not clear if this objective is really interesting, and in which applications such a method can be used. The comparison with other techniques is fair, but considering this very particular objective while other objectives may be easier to learn and more interesting. Some aspects are not clear in the paper and the experimental section has to be improved.\n\n\nRegarding the answers provided by the authors, I increase my score ", "extra_scores": {}, "extra_text": {}}, {"review_id": "2jLzhocAx4m", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "the authors of the paper propose a new way to learn goal-reaching policies by utilizing the previously collected trajectories in an iterative manner. Their novel approach, called goal-conditioned supervised learning (GCSL), learns to reach goals from a target distribution by running the policy and collecting suboptimal trajectories, and then relabeling these collected trajectories during training to perform supervised learning on them to update the policy. This behavioral cloning is done iteratively until convergence.\n\nRelatively ample evaluations and experiments were conducted in multiple settings (5 control environments). GCSL outperforms some RL algorithms in these tasks and is shown to be more robust to hyperparameters.\n\nThis work is very close to hindsight relabelling methods [Schaul et al., 2015; Andrychowicz et al., 2017; Rauber et al. 2017], but authors state that their method is more stable and does not estimate a value function.\n\nPros:\nThe paper is well written and structured.\nThe idea of using previous rollouts and relabelling them to use as training data iteratively is very interesting.\nThe analysis and ablation of learned behaviors with different data collection and relabelling settings is well done.\n\nCons:\nWhile many ablation studies have been done, the impact of the frequency of performing supervised learning via behavioral cloning on the performance of GCSL is not clear. How many trajectories are collected and relabelled before every policy update via behavioral cloning? and did you do any ablation studies for this?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper leverages concepts coming from hindsight relabelling methods to define a novel "iterated" supervised learning procedure to learn policies to reach different goals. The algorithmic solution is well supported in terms of intuition, preliminary theoretical guarantees, as well as strong empirical validation. 

There is a general consensus among the reviewers that this is a strong submission and the rebuttal helped in clarifying some aspects of the paper (e.g., the comparison with Go-Explore) and reinforced the empirical analysis. This is a clear accept.
Decision: Accept (Oral)