Autonomous Reinforcement Learning via Subgoal Curricula
Authors: Archit Sharma
Abhishek Gupta
Sergey Levine
Karol Hausman
Chelsea Finn
Venue: NeurIPS 2021 Poster (2021)
Keywords: reinforcement learning
curriculum
autonomous learning
reset-free reinforcement learning

Abstract: Reinforcement learning (RL) promises to enable autonomous acquisition of complex behaviors for diverse agents. However, the success of current reinforcement learning algorithms is predicated on an often under-emphasised requirement -- each trial needs to start from a fixed initial state distribution. Unfortunately, resetting the environment to its initial state after each trial requires substantial amount of human supervision and extensive instrumentation of the environment which defeats the goal of autonomous acquisition of complex behaviors. In this work, we propose Value-accelerated Persistent Reinforcement Learning (VaPRL), which generates a curriculum of initial states such that the agent can bootstrap on the success of easier tasks to efficiently learn harder tasks. The agent also learns to reach the initial states proposed by the curriculum, minimizing the reliance on human interventions into the learning. We observe that VaPRL reduces the interventions required by three orders of magnitude compared to episodic RL while outperforming prior state-of-the art methods for reset-free RL both in terms of sample efficiency and asymptotic performance on a variety of simulated robotics problems.

Autonomous Reinforcement Learning via Subgoal Curricula
Archit Sharmatt, Abhishek Gupta $ Sergey 1evine"1, Karol Hausman 1 Chelsea Finn
Stanford University, I Google Brain, UC Berkeley
(architah, cbtiinm)stanoordd.ee.
fabhishekunique, slevine, hoa. karollaaussai@@gggle..ccm
Abstract
Reinforcement learning (RI) promises to enable autonomous acquisition of com- plex behaviors for diverse agents. However, the success of current reinforcement learning algorithms predicated on an often under-emphasised requirement each trial needs to start from fixed initial state distribution. Unfortunately, resetting the environment to its initial state after each trial requires substantial amount of human supervision and extensive instrumentation of the environment which defeats the goal of autonomous acquisition complex behaviors. this work, we propose halue-accelerated Persistent Reinforcement Learning (VaPRL), which generates curriculum of initial states such that the agent can bootstrap on the success easier tasks to efficiently learn harder tasks. The agent also learns to reach the initial states proposed by the curriculum, minimizing the reliance on human interventions into the learning. We observe that VaPRL reduces the interventions required by three orders of magnitude compared to episodic RL while outperforming prior state-of-He the methods for reset-free RL both in terms of sample efficiency and asymptotic performance on a variety of simulated robotics problems
Introduction
Reinforcement learning (RL) offers an appealing opportunity to enable autonomous acquisition of complex behaviors for interactive agents. Despite recent RI. successes on robots [26
34 25 28
35
23 14,, several challenges exist that inhibit wider adoption of reinforcement learning for robotics 48). One of the major challenges
22
32,
the autonomy of current reinforcement learning algorithms, particularly in robotics, is the assumption that each trial starts from an initial state drawn from specific state distribution in the environment. Conventionally, reinforcement learning algorithms assume the ability to arbitrarily sample and reset to states drawn from this distribution, making such algorithms impractical for most real- world setups.
Many prior examples of reinforcement learning on real robots have relied on extensive instrumentation of the robotic setup and human supervision to enable environment resets to this initial state distribution. This is accomplished through human providing the environment reset themselves throughout the
training [8] 12 4
scripted behaviors for the robot to reset the environment [28] 39], an additional
robot executing scripted behavior to reset the environment [32], or engineered mechanical contraptions 23). The additional instrumentation of the environment and creating scripted behaviors are both time-intensive and often require additional resources such as sensors or even robots. The scripted reset behaviors narrow in application, often designed for just single task environment, and their brittleness mandates human oversight of the learning process. Eliminating or minimizing the algorithmic reliance on the reset mechanisms enable more autonomous learning, and in turn will allow agents to scale to broader and harder set of tasks.
Code and supplementary videos are available at
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Training
Evaluation
Episode
Episode
Human resets to initial distribution
Standard
Episodic
ros0.
1050
Persistent RL
Noveaul
Nonual
Figure 1: Comparison of the persistent RL setting with the episodic RL setting. Interventions (human or otherwise orchestrated) reset the environment to the initial state distribution after every episode in episodic RL, while the state of the environment persists through the training in persistent RL. The learned policy is tested starting from the initial state distribution for both the settings.
To address these challenges, some recent works have developed reinforcement learning algorithms that can effectively learn with minimal resets to the initial distribution [9 formal problem definition that encapsulates and sheds light on the general setting addressed by these prior methods, which we refer to as the persistent reinforcement learning in this work. this problem setting, we disentangle the training and the test time settings such that the test-time objective matches that of the conventional RL setting but the train-time setting restricts access to the initial state distribution by giving low frequency periodic reset. In this setting, the agent must persistently learn and interact with the environment with minimal human interventions, as shown in Figure Conventional episodic RL algorithms often fail to solve the task entirely in this setting, shown by Zhu et al. [48] and Figure 2 This is because these methods rely on the ability to sample the initial state distribution arbitrarily. One solution to this problem is to additionally learn reset policy allowing the agent to repeatedly alternate between practicing the task and practicing the reverse. Unfortunately, not only can solving the task directly from the initial state distribution be hard from an exploration standpoint, but (attempting to) return to be sample inefficient. In this paper, we propose to instead have the
6 48
14. We provide
that recovers the initial state distribution 19 6
the initial state repeatedly agent reset itself to and attempt the task from different initial states along the path to the goal state. In particular, the agent can learn to solve the task from easier starting states that are closer to the goal and bootstrap on these to solve the task from harder states farther away from the goal.
The main contribution of this work is Reinforcement Learning (VaPRL), a coal-conditioned RL method that creates an adaptive curriculum of starting states for the agent to efficiently improve test-time performance while substantially re- ducing the reliance on extrinsic reset mechanisms. Additionally, we provide a formal description of the persistent RL problem setting to conceptualize our work and prior methods. We benchmark VaPRL on several robotic control tasks in the persistent RL setting against state-of-the-art methods, which either simulate the initial state dis- tribution by learning a reset controller, or incrementally grow the ssate- from which the given task can be solved. Our experi- ments indicate that using a tailored curriculum generated by VaPRL can be up to 30% more sample-efficient acquiring task behaviors compared to these prior methods. For the most challenging dexterous manipulation problem, VaPRI. provides a 2.5x gain performance compared to the next best performing method.
Persistent
Evslantion Successs
Steps in Training Environment
The performance of episodic RL algorithms substan- tially deteriorates when environ- ment resets are not available.
Figure 2:
2 Related Work
Robot learning. Prior works using reinforcement learning have relied on manually design controllers or human supervision to enable episodic environmental resets, is required by the current algorithms. This can be through human orchestrated resets g I3 12 16), which requires high frequency human intervention in robot training. In some cases, it is possible to execute a scripted behavior to
reset the environment (28 32
46 []. However, programming such behaviors time-intensive
for the practitioner, and robot training still requires human oversight as the scripted behaviors are often brittle. Some prior works have designed the environment 35 22 bypass the need for
having reset mechanism. This is not generally applicable and can require extensive environment

---

design. Some recent works leverage multi-task RL to bypass the need for extrinsic reset mechanisms 14.
Typically, a task-graph uses the current state decides the next task for the agent, such that only minimal intervention is required during training. However, these task-graphs are specific to problem and require additional engineering to appropriately decide the next task.
Reset-free reinforcement learning. Constraining the access to these orchestrated reset mechanisms severely impedes policy learning when using current RL algorithms [48.. Recent works have proposed several algorithms to reduce reliance on extrinsic reset mechanisms by learning a reset controller to
retrieve the initial state distribution [9 6 by learning a perturbation controller [88]. . or by learning reset skills in adversarial games 43 These works implicitly define a target state distribution for
reset controller: Han et al. 19), Eysenbach al. [6] target a fixed initial state distribution; Zhu et al. [47] target uniform distribution over the state space as a consequence of novelty-seeking behavior of the reset controller; and Xu et 4X target an adversarial form of initial state distribution to produce more robust policy. In contrast, our proposed algorithm VaPRI. generates curriculum of starting states tailored to the task and agent's performance. Our experiments demonstrate that VaPRI outperforms these prior methods in both sample efficiency and absolute performance. Other recent work like [29] has considered combining model-based RL with unsupervised skill discovery to solve reset-free learning problems, but largely focus on avoiding sink states rather than attempting tasks repeatedly with a curriculum like VaPRL.
Curriculum generation for reinforcement learning, Curriculum generation is crucial aspect of sample-effiiient learning in VaPRL. Prior works have shown that using a curriculum can enable faster learning and improve performance [9
27]. Task-tailored curriculum can simplify the exploration as it is easier to solve the task from certain states 21. enabling faster progress on the downstream task. In addition to proposing novel method for curriculum generation, we design it for the persistent RL setting without requiring the ability to reset the environment to arbitrary states as assumed by prior work.
10
40
30
33
Persistent vs. lifelong reinforcement learning. Prior reinforcement learning algorithms that reduce the need for oracle resets have relied on the problem setting of lifelong or continual reinforcement learning [41 (24], when the objective in practice is to learn episodic behaviors. Both the persistent RL and the lifelong learning frameworks do transcend the episodic setting for training, promoting more autonomy in reinforcement learning. However, persistent reinforcement learning distinguishes between the training and evaluation objectives, where the evaluation objective matches that of the episodic reinforcement learning. While the assumptions of episodic reinforcement learning are hard to realize for real-world training, real-world deployment of policies is often episodic. This is commonly true for robotics, where the assigned tasks are expected to be repetitive but is hard to orchestrate resets in the training environment. This makes persistent reinforcement learning a suitable framework for modelling robotic learning tasks.
3 Persistent Reinforcement Learning
In this section, we formalize the persistent reinforcement learning as an optimization problem. The key insight is to separate the evaluation and training objectives such that the evaluation objective measures the performance of the desired behavior while the training objective enables us to acquire those behaviors, while recognizing that frequent invocation of a reset mechanism is untenable. We first provide general formulation, and then adapt persistent RL to the goal-condition
setting.
(S,A,,,,,,, HE) [77. Here, S denotes the state space, A denotes the action space, p:S x A x S Rxo denotes the transition dynamies, r: S x A + IR denotes the reward function, p S Rz0 denotes the ini- tial state distribution, 7 E [0, 1] denotes the discount factor, and HE denotes the episode hori- zon. Our objective is to learn policy that maximizes (=(x)= = E 7'r(s;, at),, where 50 ~ p()), a ~ T- st) and st-1 ~ p(- st, at), the episodic expected sum of discounted rewards.
Definition. Consider Markov decision process (MDP) ME
However, generating samples from the initial state distribution P invokes reset mechanism, which is hard realize in the real world. We want to construct MDP MM corresponding to our training environment which reduces invocations of the reset mechanism. To reduce such interventions, we consider a training environment MT = AS, Ft, P, 7, Hr) with episode horizon Hy >> HF. Näively optimizing can substantially deteriorate the performance of episodic RL algorithms, as shown in Figure where we compare the evaluation performance with HE = 200 when training in environments with HT 200 (with resets) versus Hr 200, 000 (without resets). Therefore.
3

---

it becomes beneficial to consider a surrogate reward function F rather than just optimizing for controller which altemates between
naively. As motivating example, consider a solving the task corresponding to and recovering the initial state distribution p. The surrogate reward function corresponding for this approach can be written as:
(e( at) = [r(st,a)) (s)
1 = [1, Hr), + +1,3Hr
t = [E + 1,
Here, f altermates between the task-reward r for HE steps and rg (which encourages initial state distribution recovery) for Hg steps" also illustrated in Figure (right). This surrogate reward function allows the agent to repeatedly practice the task, thus using the autonomous interaction more judiciously as compared to the näive approach. Note, this i loosely recovers the objectives used in some prior works [19] [ For general time-dependent surrogate reward function iz, we define the
training objective as
(x)
Σ I'r (st, a:)]
(2)
1=1
where ý is the initial state distribution at training time (which does not need to match the evaluation- time initial state distribution p). The persistent RL optimization objective is to maximize Jr (x) efficiently under the constraint that Jg(arg max, (T)) = max, Jε()). Intuitively, the objec- tive encourages construction of a training environment that can recover the optimal policy for the evaluation environment. The primary design choice is Ft. which as shown above leads to different algorithms. Another design choice م. which may or may not match p. Importantly, we do not assume is any easier to sample compared to p.
Finally, we note that the formulation discussed here is suitable only for reversible environments. Reversible environments guarantee that the agent can continue to make progress on the task and not get "stuck" (for example, if the object goes out of reach of the robot's s arm). A large class of practical tasks can be considered reversible (door opening. cloth folding, and so on) or the environment can be constructed enforce reversibility (add bounding walls so the object does not go out of reach). A formal definition for reversible environments is provided in Appendix [A] In this work, we will restrict ourselves to reversible environments, and defer full discussion of persistent RL. for environments with irreversible states to future work.
Coal-conditioned persistent reinforcement learning. We adapt the general formulation above to goal-conditioned (20 388 instantiation of persistent RL. Consider a goal-conditioned MDP ME. = (8, A. G,P,r, P, y, HE), where G C 8 denotes the goal space. For goal distribution Pg G + R20.
the evaluation objective is JE(T)) = Egen,() (a HE ('r(s,)) for क : x A g + Rya- The training objective is then stated as:
Hr
(T) E
(ss,as) Σ r'r(Se, G(S),Pg))).
(3)
where we assume that F remains goal reaching objective, but where algorithms instead use goal generator G to generate a curriculum of goals to practice throughout training"] The intuition is that, since Hr >> Hg. the algorithm can repeatedly practice reaching various task-goals. However, the objective is to learn a policy that can reach task-goals from Pg in the test environment, i.e., starting from the initial state distribution p. This implies the goal generator G should expand the goal space beyond the task-goals to improve the policy æ for the test environment, For example, the goal generator could alternate task-goals (g P.)) and the initial state distribution (s ~ p), which again loosely recovers prior works [1966. This instantiation transforms the problem of finding the right reward function to the right curriculum of goals using G.
4 Value-Accelerate Persistent Reinforcement Learning
To address the conditondi persistent RL problem, now describe our proposed algorithm, VaPRL. The key idea in VaPRL is that the agent does not need to return to the initial state distribution between every attempt at the task, and can instead choose to practice from states that facilitate efficient learning. Section 4.1 discusses how to generate this curriculum of initial states. Using
We assume that the state includes information indicating the reward function being optimized so that agent can take appropriate actions, for example, ooe-hot task indicators as is common multi-task RI..
"Th goal generator may use additional memory which is not explicitly represented here.

---

VaPRL Training
Forward Backward RL
Fred
Eecal SEAP
Figure 3: An overview of the VaPRL algorithm (left) compared to forward backward RL (right). For VaPRL, the value function gives us a set of states from where the agent can solve task with some confidence (shaded in green), and the VaPRL chooses the state closest to the initial state distribution among them (purple square). In each iteration, the agent can hootstrap on the knowledge of solving the task from a future state (hold green) which simplifies the exploration from its current state (broken green linc). As the performance of the agent improves, the states commanded by VaPRL move closer the initial state distribution. This in contrast to the forward-backwer controller that alternates between the test-goals and the initial state distribution.
gondicondd RL within VaPRL allows us to use the same policy to solve the task and reach the initial states suggested by the curriculum, in contrast to prior work that learns separate reset and task policy. Section 4.2 describes how careful goal relabeling can be leveraged to efficiently learn this unified goal-reaching policy. We also discuss how VaPRL can effectively use prior data, which often becomes crucial for efficiently solving hard sparse-reward tasks.
4.1 Generating a Curriculum Using the Value Function
Consider the problem of reaching a goal g Pg in the MDP MMg. Learning how reach the goal g is easier starting from state E S that is close to especially when the rewards are sparse. Knowing how reach the goal g from state s in turn makes it easier reach the goal from states in the neighborhood of s, enabling us to incrementally move farther away from the goal g. Bootstrapping on the success of an easier problem to solve a harder problems motivates the use of curriculum in reinforcement learning, also illustrated in Figure 3
Following the intuition above, we aim to define increasingly-difie -ifficcllt curriculum such that the policy is eventually able to reach the goal starting from the initial state distribution p. Our simple scheme is to sample task goal g Pg. run the policy with subgoal C(g) as input, and then the policy with the task goal g as input. The main question now becomes: given goal g. how do we select the subgoal C(g) to attempt the goal g from? We propose to set up C(g) as follows:
C(g)= arg min (s) s.t. V (s 2
(4)
where X, is a user-specified distance function between the state s and the initial state distribution p. (*((sg) = E|EHs 7"r(s,g) $1 = s) denotes the value function of the policy reaching the goal g from the state s, and E some fixed threshold. Here, the value function represents the ability of the policy to reach the goal g from the state To see that, consider the case when discount factor ! and 0 otherwise, the value function V' exactly represents the probability of reaching goal y from state when following the policy The intuition carries over t E [0, 1) too, where the environment can go into a terminal state with probability 1 - 7 at every transition. For general goal-reaching reward functions, state with higher value under VT g) would still represent greater ability to reach the goal g for the policy π.
= and r(x,g) g) =1 I when
Revisiting Equation 4 with this understanding of the value function, the objective C(g) chooses the state closest the initial state distribution for which the value function V" (s. crosses the threshold This encourages the curriculum to be closer to the goal state in the early stages of the training as the policy would less effective at reaching the goal. As the policy improves, larger number of states satisfy the constraint and the curriculum progressively moves closer to the initial state distribution. Eventually, the curriculum converges to the initial state distribution leading to a policy æ that would optimize the evaluation objective in the MDP Mg. Following this intuition,
write the goal
5

---

generator G(st, Pg) as:
G(sl: Ps) g s.t. grask ~ Par g f C(gtosk)
if switch(st,, g) subgoal
(5) elif switch(st, 9) = task goal where the switch( st. Ieur) is true if the yeur has been reached or your has been in place for HE steps. For every new goal Btask ~ Pg, we first attempt to reach the curriculum subgoal ((Itask) (that is switch(s,, g) = subgoal), and then we attempt to reach goal Jrask (that is switch(s,, (s,chh((,,55) 9) = task goal). This cycles repeats until the environment resets after Hr steps.
9 5 Stask
Computing the Curriculum Generator C(g). Equation involves a minimization over the state space S, which is intractable in general. While it is possible to come up with general solutions by constructing a generative model p(s) and taking a minimum over the samples generated by it, we opt for a simpler solution: C(g) over a randomly sampled subset of it by enumeration. If an offline dataset or demonstrations available, we can also minimize C(g) on this data exclusively. The constrained minimization can similarly be approximated by considering the subset of the data which satisfies the constraint VT (s.g) 2 c and choosing the state from this subset which minimizes X,- If no state satisfies the constraint, C())) returns the state with the maximum V (8,9).
use the data collected by the policy during the training and minimize
Measuring the Initial State Distribution Distance. An important component of curriculum gener- ation is choosing the distance function (s). which should reflect the distance to the state from the initial state distribution under the environment dynamics. To factor in the dynamics,
can use the learned goal-conditioned value function as measure of shortest distance between the state and the goal 200 36.. In particular, we use (s)s) - E (T sp). While this choice is convenient as we are already estimating (s, g). there is an even simpler choice for (s)s) when offline demon- stration data is available. Assuming that the trajectories in the provided dataset start from the initial state distribution p. we can use the timestep index of the state as the distance from the initial state distribution, that is (s(s)
arge (s, D) where D denotes the offline demonstration set. The step index distance function defined here encodes the intuition that states which require more steps by the policy are farther away. The function naturally accommodates for the dynamics of the environment. Finally, since we are minimizing (s) in Equation 4 if there are multiple trajectories to the same state or suboptimal loops within single trajectory, we use the shortest distance to that state.
4.2 Relabeling Goals
Not only does our policy need learn how to reach ~ Pg. but it also needs to learn how to reach all the
goals generated by C(g) over the course of training, caus- ing the effective goal-space to grow substantially. How- ever, there is a lot of shared structure in reaching goals, especially those generated by the curriculum. The knowl- edge of how to reach a goal gi also conveys meaningful information about how to reach goal 92- This structure can be leveraged by using techniques from goal relabeling 22. In particular, we relabel every trajectory collected in the training environment with N goals sampled randomly from the set of goals that may be a part of the curriculum. If we do not have any prior data, we randomly sample the replay buffer for relabeling goals. If we are given some prior data D, this reduces sampling to
Cartodaes 50
02
Dos- ih the fooe nodi
Figure 4: An illustration of goal relabeling in VaPRL. Every transition in trajectory relabeled with a randomly sampled subset curriculum goals, yielding large set of relabeled tuples that are added to the replay buffer. This ensures efficient data reuse.
D U (g' ~ } for relabeling.
There is a subtle difference between hindsight experience replay (HER) and the goal relabeling strategy we employ. While HER chooses future states from within an episode as goals for relabeling, exclusively choose states that may be used goal states in the curriculum, which may not occur in the collected trajectory at all. Since our policy will only be tasked with reaching goals generated by the goal generator G, it is advantageous to extract signal specifically for these goals. To summarize, while the goal-space has grown, goal relabeling enables us to generate data for the algorithm commensurately to improve sample-efficiency.
Algorithm Summary. The outline for VaPRL is given in Algorithm I At a high level, VaPRL takes a set of demonstrations as input and adds it to the replay buffer R. These demonstrations are relabelled to generate additional trajectories such that every intermediate state is used as a goal. Next, VaPRL starts collecting data in the training MDP Mr. At every step, VaPRL samples the goal

---

Algorithm 1: Value-Accelerated Persistent Reinforcement Learning (VaPRL)
Input:
initial state(s) p,, N/ N:: number of goals for relabeling
Optional: Demos D;
Initialize replay buffer B, ι(a s,g), (" a, g):
// IIII demos. add them to replay bufler and relabel <- B D:
celabel_demos((0
while not done do
8 ~ p; sample initial state
for Hr steps do
g < ((8, (Eq
a ~ x[- 8,g), pl- 8.
B B [(s,,,, (.r(('s
f 1,-, do
Figure 5: Continuous control environments for goal- conditioned persistent RL. (top left) table-top rear- rangement task, where gripper tasked with moving the mug to four potential goal positions, (top right) sawyer robot learss how to close the door and (bot- tom) nigh-ddmeesiooal dexterous hand attached to sawyer robot tasked to pick up aaree-pronged object.
g ~~ D Pu: /f = 0, sample replay buffer
B - U {(s, as s'
update n. Q":
+ s
generator G to get the current goal and collects the next transition using the current policy æ. This transition is added replay buffer R along with relabelled transitions, as described in Sec Sec The policy π and the critic Q" are updated every step, using any off- off-policy reinforcement learning algorithm. This loop repeated for Hy steps till an extrinsic intervention resets the environment to state p. Note, it isn't 't necessary to initialize the agent close to the goal. Additional details pertaining the algorithm can found in the Appendix
5 Experiments
In this section, we study the performance of VaPRL on continuous control environments for goal- conditioned persistent RL and provide ablations and visualization to isolate the effect of the curricu- lum. In particular, we aim to answer the following questions:
1. Does VaPRL allow efficient reinforcement learning with minimal episodic resets?
2. How does the scheme for generating a curriculum in VaPRL perform compared other methods for persistent reinforcement learning?
3. Does VaPRL scale to high dimensional state and action spaces?
4. What does the generated curriculum look like? Is the curriculum effective?
We next describe the specific choices environments, evaluation metrics and comparisons order to answer the questions above.
Environments. For our experimental evaluation, consider three continuous control environments, shown in Figure 5] The table-top rearrangement is a simplified manipulation environment, where point mass which attach to the object if it is close it) is tasked with
gripper (modelled taking the mug to one of the 4 potential goal squares. The evaluation horizon is HE = 200 steps and the training horizon is Hy 200, 000 steps. This task involves challenging exploration problem in navigating to objects, picking them up, and dropping them at the right location. The sawyer door closing environment involves using sawyer robot arm close door particular target angle (55 For this environment, we set the horizon for evaluation to be H = 400 and Hy = 200, 000 steps for training. Since environment resets are freely available, repeatedly practicing the task implicitly requires the agent to also learn how to open the door. The hand manipulation environment, introduced in [4], involves dexterous hand attached to a sawyer robot. This environment entails a 16 DoF hand that is mounted onto 6 DoF arm, with the goal of manipulating 3 pronged object as seen Figure [5] In particular, the task involves picking up the object from random positions on table and lifting it to a goal position above the table. This task is particularly challenging since it involves complex contact dynamics with high dimensional state and action spaces. Additionally, the robot has to learn how to reconfigure the object diverse locations to simulate the test-time conditions where the agent is expected to pickup the object from random locations. For this environment, we set the horizon for evaluation Hg 400 and for training Hy 400, 000 steps.
7

---

Table top reatangeneet
mantpulstion
Evahaatinn Su00ss
04
0.4
04
82
02
1.00 1.50
1i5e
25na
LE00 1.0 Stops Unvtronment
Environment
Steps Treining Rnvironment
VaPRL
FBRL
Daive
R3L
oracle RL
Figure 6: Performance of each method on (left) the table-top rearrangement environment, (center) the sawyer door closing environment, and (right) the hand manipulation environment. Plots show learning curves with mean and standard error over 5 random seeds. VaPRL is more sample-efficient and outperforms prior methods.
Environment Setup. For table-top rearrangement and sawyer door closing, we consider a sparse reward function r(s, g) = [(s, g),, which is 1 when the state s is close to the goal position g. and 0 otherwise. Since the hand manipulation environment is substantially more challenging problem. we consider dense reward function that rewards the the hand and the object to be close to the goal position. To aid exploration in table-top rearrangement and sawyer door closing, we provide all the algorithms with small set of trajectories (6 per goal, 3 going from initial state to the goal and the other 3 going in reverse) for each environment, though we do not assume that the trajectories take the optimal path (for example, the trajectories could come from teleoperation in practice). For the hand manipulation environment, we provide the agent with 10 trajectories demonstrating the pickup task from random positions the table and 20 trajectories showing how to reposition the object to different locations on the table. For all environments, we report results by evaluating the number of times the policy successfully reaches the goal out of 10 trials in the evaluation environment ME g (by resetting to a state from the initial state distribution p and sampling an appropriate goal from the goal distribution Pg). performing intermittent evaluations as the training progresses. Note, the training agent does not receive the evaluation experience and it is only used to measure the performance on the evaluation environment. Further details about problem setup, demonstrations, implementation, hyperparameters and evaluation metrics can be found in the Appendix.
Comparisons. We compare VaPRL to four approaches: (a) A standard off-policy RL algorithm that only trains to reach the goal distribution, such that a new goal g ~ (s)s) is sampled every Hg steps (labelled naïve RL), (b) A forward-backwart controller [19 6 which alternates between
g ~ (s)s) and ~ p(s) for Hg steps each, as described in Section 3 (labelled FBRL). (c) A
perturbation controller [48] that alternates between optimizing controller to maximize task reward and a controller to maximally perturb the state via task agnostic exploration (labelled R3L), and (d) RL directly on the evaluation environment, resetting to the initial state distribution after every HE steps (labelled oracle RL). This oracle is an expected upper bound on the performance of VaPRI., since it has access episodic resets. We use soft actor-critic [17] as the base RL algorithm for all methods to ensure fair comparison, although any value-based method would be equally applicable. To emphasize, all the algorithms are provided the same set of demonstrations. Further implementation details can be found in the Appendix.
5.1 Persistent RL Results
The performance of each of the algorithms on the three evaluation domains are shown in Figure 6 We that VaPRL outperforms naïve RL, FBRL and R3L, providing substantial improvements in terms of sample efficiency. For our most challenging domain of hand manipulation, the sample efficiency enables reach a much better performance within the training budget. The primary difference between the methods is that VaPRI. uses a curriculum of starting states progressing from easier to harder states. In contrast, FBRL always attempts reach the initial state distribution and R3L uses a perturbation controller to reach novel states in attempt to cover the entire state space uniformly. In the table-top rearrangement environment, the agent starts close to the goal and then gradually brings it back to the initial state distribution, trying different intermediate states in the process (discussed in Section [.3). In the sawyer door closing environment, the agent learns to close the door from intermediate angles, incrementally improving the performance. For the hand manipulation domain, the agent focuses on picking up the object from a particular location, and then incrementally grow the locations from which it can complete pickup. In contrast, FBRL chooses attempts the pickup from random states from the initial state distribution p and R3L attempts to find new states to pickup the object from (even though it might not be succeeding to pickup the object from previous locations).
8

---

Goal State
C(g)
Initial State
Stags in Tratning Destroenmnnt
Figure 8: Visualization of curricula generated by VaPRL on the table-top rearrangement environment. We plot the step index distance between the initial state and the curriculum goals generated by C(g) (blue) and the evaluation performance (orange) as the training progresses. The distance is normalized to be on the same scale as the success metric, such that value of corresponds to the test-goal distribution and 0 corresponds to the initial distribution. We visualize curriculum gradually progresses from goal states to initial states with a correlated improvement evaluation performance.
of the commanded goals C(g) during the training, observing that the
Compared to oracle RL with resets, VaPRL learns to solve the task while requiring 500x fewer environment resets in the door closing environment, 1000 fewer environment resets in the table-top rearrangement euvironment and dexterous hand manipulation. This amounts to less than 20 total interventions for VaPRL, indicating the substantial autonomy with which the algorithm can run.
Surpisingly, in the domains with sparse reward function (that is, sawyer door closing and table-top reaarrangement), VaPRL matches or even outperforms the oracle RL method. In the table-top rearrangement environment, oracle RL does substantially worse than VaPRL. It has been noted in prior work that multi-goal RL problems can converge suboptimally due to issues arising in the optimization [44). We hypothesize that an appropriate initial state distribution can ameliorate some of these issues. In particular, moving beyond deterministic initial distributions may lead to better downstream performance (also noted in (47). For the door opening environment, VaPRI. matches the performance of oracle RL. To emphasize, oracle RL is training on the evaluation environment directly, that is Hr Hg with the environment resetting to a state so p. In contrast, VaPRL also learns how to reverse the task is solving and thus only spends half its training samples collecting the data for the evaluation task (for example, VaPRL learns how to open the door and close it).
5.2 Isolating the Role of the Initial State Distribution
We construct an experiment to isolate the effect of the starting state distribution on learning efficiency and downstream perfor- mance. In this experiment, the environment resets directly to the state C(g) for VaPRI,, such that the policy only has to learn reaching the goals ~ Pg (labelled VaPRI. + reset). Analo- gously, for FBRL, the environment resets to the state so ~ p. which is identical to the oracle RL method (labelled oracle RL /BBRL+ + reset). For R3L, the environment resets a state uni- formly sampled from the state space (labelled uniform / R3L + reset). We run this ablation on the table-top rearrangement environment, where the episode horizon for all the algorithms H 200. The results Figure 7 indicate that the starting state distribution induced by the VaPRI. curriculum improves the performance, translating into improved performance the persistent RL setting.
table-top rearrangement
1.0
Success
0.0
Evaluation
0.2
reset undoom R3L reset onscle RL FORLL + reset
5.0e3
Steps Training Eavironment
Figure 7: Ablation isolating the effect of curriculum generated by VaPRL.
5.3 Visualizations of Generated Curricula
To better understand the curriculum generated by VaPRL, we visualize the sequence of states chosen by Equation 4 as the training progresses on the table-top rearrangement environment, shown in Figure As we can observe, initially the curriculum chooses states which are farther away from the initial state and closer to the goal distribution. As training progresses, the curriculum moves towards the initial state distribution. Correspondingly, the evaluation performance starts to improve

---

move closer to the initial state distribution. Thus, VaPRL can generates an adaptive curriculum for the agent to efficiently improve the performance on the evaluation setting.
6 Conclusion
In this work, we propose VaPRL, an algorithm that can efficiently solve reinforcement learning problems with minimal episodic resets by generating curriculum of starting states. VaPRL is able to reduce amount of human intervention required in the learning process by a factor of 1000 compared to episodic RL, while outperforming prior methods. In the process, we also formalize the problem setting of persistent RL to understand current algorithms and aid the development of future ones.
There number of interesting avenues for future work that VaPRL does not currently address. A natural extension is to environments with irreversible states. This setting can likely be addressed by Another extension is to work with visual state spaces, allowing the algorithm to be more broadly applicable in the real world. These two extensions would be significant step towards enabling autonomous agents in the real world that minimize the reliance on human interventions.
leveraging ideas from the literature in safe reinforcement learning [1] 42 5
7 Disclosure of Funding
This work was supported in part by Schmidt Futures and ONR grant N00014-21-1-2685.
References
[1] Learning to poke by poking: Experiential learning of intuitive physics. In D. D. Lee, M. Sugiyama, U. von Luxburg, I. Guyon, and R. Garnett, Niitors,Neuriln 2016.
[2] M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong. P. Welinder, B. McGrew, J. Tobin,
P. Abbeel, and W. Zaremba. Hindsight experience replay. arXiv preprint voXiv:177.01499 2017.
[3] Berkenkamp, M. Turchetta, A. P. Schoellig, and A. Krause. Safe model- based reinforcement learning with stability guarantees. arXiv preprint arXiv: 2017.
[4] Y. Chebotar, K. Hausman, M. Zhang, G. Sukhatme, S. Schaal, and Levine. Combining model- based and model updates for trajectory-centrr reinforcement learning. In International conference on machine learning, pages 703-711. PMLR, 2017.
[5] Y. Chow, O. Nachum, E. muenez-Guzman and M. Ghavamzadeh. A lyapunov- approach to safe reinforcement learning. arXiv preprint arXiv: 1rXivv1805.0777 000. 2018.
[6] B. Eysenbach, Gu, Ibarz, and S. Levine. Leave no trace: Learning reset for safe and autonomous reinforcement learning. arXiv preprint 17Xvv:711.006822 2017
[7] C. Finn and S. Levine. Deep visual foresight for planning robot motion. In 2017 IEEE International Conference on Roboties and Automation (ICRA), pages 2786-2792) IEEE, 2017.
[8] C. Finn, X. Y. Tan, Y. Duan, T. Darrell, S. Levine, and P. Abbeel. Deep spatial autoencoders for visuomotor learning. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pages 512-519. IEEE, 2016.
[9] C. Florensa, D. Held, M. Wulfmeier, M. Zhang, and P. Abbeel. Reverse curriculum generation for reinforcement learning. In Conference on robot learning, pages 482-495. PMLR, 2017.
(10) C. Florensa, D. Held, X. Geng, and P. Abbeel. Automatic goal generation for reinforcement learning agents. In International conference on machine learning, pages 1515-1528. PMLR, 2018.
[11] J. Garcia and F. Fernández. A comprehensive survey safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437-1480, 2015.
[12] A. Ghadirzadeh, A. Maki, D. Kragic, and M. Björkman. Deep predictive policy training using reinforcement learning. In 20/7 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2351-2358 IEEE, 2017.
10

---

[13] S. Gu, E. Holly, T. Lillicrap, and S. Levine. Deep reinforcement learning for robotic manipula- tion with asynchronous off-policy updates. In 2017 IEEE international conference on roboties and automation (ICRA), pages 138993396. IEEE, 2017.
[14] A. Gupta, J. Yu, T. Zhao, V. Kumar, A. Rovinsky, K. Xu, T. Devlin, and S. Levine. Reset-free reinforcement learning via multi-task learning: Learning dexterous manipulation behaviors without human intervention. ArXiv, abs/2104.1 102000044.20,,, 2021.
[15] S. Ha, P. Xu, Z. Tan, S. Levine, and J. Tan. Learning to walk in the real world with minimal human effort, 2020.
[16] T. Haarnoja, S. Ha, A. Zhou, J. Tan, G. Tucker, and S. Levine. Learning to walk via deep reinforcement learning. arXiv preprint arXiv:1812.11100 2018.
[17] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, pages 1861-1870. PMLR, 2018.
[18] D. Hafner, J. Davidson, and V. Vanhoucke. Tensorflow agents: Efficient batched reinforcement learning in tensorflow. arXiv preprint arXiv: I 700.02287, 2017.
[19] W. Han, S. Levine, and P. Abbeel. Learning compound multi-step controllers under unknown dynamics. In 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 6435-6442. IEEE, 2015.
[20] L. P. Kaelbling, Learning to achieve goals. In IJCAI, pages 1094-1099. Citeseer, 1993.
[21] S. Kakade and J. Langford. Approximately optimal approximate reinforcement learning. In In Proc. 19th International Conference on Machine Learning. Citeseer, 2002.
[22] D. Kalashnikov, A. Irpan, Pastor, Ibarz, A. Herzog, E. Jang, D. Quillen, Holly, M. Kalakr- ishnan, V. Vanhoucke, et al. Qt-opt: Scalable deep reinforcement learning for iision-based robotic manipulation. arXiv preprini arXiv: 10293, 2018.
[23] D. Kalashnikov, J. Varley, Y. Chebotar, B. Swanson, R. Jonschkowski, C. Finn, S. Levine, and K. Hausman. Mt-opt: Continuous multi-task robotic reinforcement learning at scale, arXiv preprint arXiv:2 082 12, 2021.
[24] K. Khetarpal, M. Riemer, Rish, and D. Precup. Towards continual reinforcement learning: A review and perspectives. arXiv preprint arXiv:2012.13499. 2020.
[25] J. Kober, J. A. Bagnell, and J. Peters. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 32(1 2013.
[26] N. Kohl and P. Stone. Policy gradient reinforcement learning for fast quadrupedal locomotion. In IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA '04. 2004, volume 3, pages 2619-2624. IEEE, 2004.
[27] G. Konidaris and A. Barto. Skill discovery in continuous reinforcement learning domains using skill chaining. Advances in neural information processing systems, 12:1015- 2009.
[28] S. Levine, C. Finn, T. Darrell, and P. Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1): 1334-1333,, 2016.
[29] K. Lu, A. Grover, P. Abbeel, and I. Mordatch. Reset-free lifelong learning with skill-space planning, arXiv preprint 2=XXi:2012.035455 2020.
[30] Matiisen, A. Oliver, T. Cohen, and Schulman. Teacher student curriculum learning. IEEE transactions on neural networks and learning systems, 31(9):3732-31) -7400, 2019.
[31] T. M. Moldovan and P. Abbeel. Safe exploration in markov decision processes. arXiv preprint ar Xrxiv: /205. 4810, 2012.
[32] A. Nagabandi, K. Konolige, S. Levine, and V. Kumar. Deep dynamics models for learning dexterous manipulation. In Conference Robot Learning, pages 1101-1112, PMLR, 2020.
11

---

[33] S. Narvekar and P. Stone. Learning curriculum policies for reinforcement learning. arXiv preprint arXiv: 1812.00888 2018.
[34] A. Y. Ng. H. J. Kim, M. Jordan, S. Sastry, and S. Ballianda. Autonomous helicopter flight via reinforcement learning. In NIPS, volume 16. Citeseer, 2003.
[35] L. Pinto and A. Gupta. Supersizing self-supervision: Learning grasp from 50k tries and 700 robot hours. In 2016 IEEE international conference on robotics and automation (ICRA), pages 1406-3415 IEEE, 2016.
[36] V. Pong. S. Gu, M. Dalal, and S. Levine. Temporal difference models: Model-free deep rl for model-based control. arXiv preprint arXiv:/802.09000 2018.
[37] M. L. Puterman. Markov decision processes. Handbooks in operations research and manage- ment science. 2:331-4344 1990.
[38] Schaul, D. Horgan, Gregor, and D. Silver. Universal value function approximators. In International conference muchine learning, pages 1312-1320. PMLR, 2015.
[39] A. Sharma, M. Ahn, S. Levine, V. Kumar, K. Hausman, and S. Gu. Emergent real-world robotic skills via unsupervised off-policy reinforcement learning. arXiv preprint arXiv:2004. 2020.
[40] S. Sukhbaatar, Z. Lin, I. Kostrikov, G. Synnaeve, A. Szlam, and R. Fergus. Intrinsic motivation and automatic curricula via asymmetric self-play. arXiv preprint arXiv: 1703.05477 2017.
[41] R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT press, 2018.
[42] B. Thananjeyan, A. Balakrishna, S. Nair, M. Luo, K. Srinivasan, M. Hwang, J. E. Gonzalez, J. Ibarz, Finn, and K. Goldberg. Recovery rl: Safe reinforcement learning with learned recovery zones, arXiv preprint arXiv:2010. 15920, 2020.
[43] K. Xu, S. Verma, C. Finn, and S. Levine. Continual learning of control primitives: Skill discovery via reset-games. ArXiv, abs/201 1. 1.05286, 2020.
[44] Yu, S. Kumar, A. Gupta, S. Levine, K. Hausman, and C. Finn. Gradient surgery for multi-task learning, arXiv preprint 200.::00000007822 2020.
[45] T. Yu, D. Quillen, Z. He. R. Julian, K. Hausman, C. Finn, and S. Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning, pages 1094- 1 100. PMLR, 2020.
[46] A. Zeng, S. Song. J. Lee, A. Rodriguez, and T. Funkhouser. Tossingbot: Learning throw arbitrary objects with residual physics. JEEE Transactions on Robotics, 36(4):1307-1319, 2020.
[47] H. Zhu, A. Gupta, A. Rajeswaran, S. Levine, and V. Kumar. Dexterous manipulation with deep reinforcement learning: Efficient, general. and low-cost. In 2019 International Conference on Robotics and Automation (ICRA), pages 3651-3657. IEEE, 2019.
[48] H. Zhu, J. Yu, A. Gupta, D. Shah, K. Hartikainen, A. Singh, V. Kumar, and S. Levine. The ingredients of real-world robotic reinforcement learning. arXiv preprint arXiv:2004.15 12570. 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See Section
(b) Did you describe the limitations of your work? [Yes] See Section 3 and Section 6
(c) Did you discuss any potential negative societal impacts of your work? [No] this work inherits the potential negative societal impacts of reinforcement learning and its applications to roboties. We do not anticipate any additional negative impacts that are unique to this work.
12

---

(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [No]. we will release the code and the enviromments upon publication.
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes]. See Section and Appendix.
and
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [Yes], See Section
(d) Did you include the total amount compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Section
(b) Did you mention the license of the assets? [Yes]. See Appendix.
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you' 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "h3cj_wLFdcJ", "reviewer": "Reviewer_HS6a", "summary": "This paper introduces the definition of Persistent Reinforcement Learning (PRL), and an algorithm to solve a goal conditioning version of PRL - Value-Accelerated PRL (VaPRL) - based on two ideas: separate the training and evaluation MDPs; and build a curriculum of (easier to solve) MDPs. \n\nThe main idea behind the paper is to solve episodic RL tasks by minimizing the number of resets to the initial state, which is particularly important in real-work RL. This is achieved by using a generator of surrogate goals instead of solving directly for the initial task goal, and by using an initial state distribution different from the evaluation MDP. This initial state distribution is at the beginning of training closer to the goal, and during training it is expanded such that it approaches the initial state distribution of the evaluation MDP.\n\nThe experimental results empirically show the claimed benefits, and are compared with other algorithms that do not use any curriculum or have access to resetting to the initial state distribution.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The motivation for this work is very important for real-world reinforcement learning, especially in robotic tasks where resetting to an initial state distribution includes some form of human intervention, either by manually resetting the system, or by engineering a solution to do it automatically.\n\nThe paper is well written, organized, easy to follow and read.\n\nAlthough I like this work, my opinion is that it lacks a more theoretical treatment of the problem and that some claims can be difficult to place in practice in a real system. Nevertheless, I see a good potential for this idea, but I think it is better suited to a robotics conference.\n\nThe following are a couple of comments and questions.\n\nMethod\n----------\nA central idea of the paper is to target the sparse reward episodic MDP setting for real scenarios, where the solution of the problem is partly known to humans, e.g., picking up an object. To solve such tasks, the authors claim that placing the agent near the goal is beneficial. While I agree with this statement, there are two concerns. First, for many tasks it is difficult to define what “close to the goal” really means. Second, there are tasks where placing the robot close to the goal is extremely difficult. Take the example of a robot hitting a hockey puck. The reward in this task is sparse and equal to placing the puck in a radius of a given location. One can place the puck near the goal, but if the hockey stick cannot reach it, then it will not be able to solve the task. \n“The objective of this goal generator is to recover the optimal policy with respect to J_E efficiently training in an environment with H_T >> H_E”.\nIf the training environment is different from the evaluation environment, how do you guarantee that the return is maximized in the evaluation environment? From my understanding, the authors define a proxy based on the curriculum training, but unfortunately there are no theoretical guarantees that training on the training MDP will lead to an optimal performance in the evaluation MDP.\nBy optimizing the policy locally, i.e., from the initial state given by the curriculum to a generated goal, is the trajectory generated by the global policy optimal? Or is it piecewise optimal? \nA big focus of the paper is on constructing the curriculum of starting states. Even though it is a good idea, it is in my opinion done in a very heuristic way, and without any guarantees that following this curriculum improves the returns in the evaluation environment.\nIn the main paper, for the constraint V(s,g) >= epsilon, the authors defined it as belonging to the [0, 1] interval, but in the experiment for hand manipulation, epsilon is out of this range (-300). Could you comment on this?  How to choose epsilon? Is it just a hyperparameter? Can it happen that a “bad” epsilon leads to an underperforming policy?\nGoal relabeling is important to reduce the space of possible curriculum goals, and it is solved by sampling N goals at random. My question is again if this choice ensures that we obtain an optimal policy with respect to the evaluation MDP?\n\n\nExperiments\n-----------------\nThe experiments are well constructed and show the potential of the algorithm in sparse reward environments.\nWhile I understand that NeurIPS is not a robotics conference, it would be good to see how this method applies in reality, especially how easy it is to place the agent near the goal state for a more intricate task.\n\n\nCitations\n------------\nThe authors seem to be citing a reasonable amount of work from a specific group.\nEven though this work is not set in the hierarchical RL framework, it has also some connections to the idea of skill chaining, which although not directly matching this paper it could also be taken into consideration - Konidaris, G. et al, Skill Discovery in Continuous Reinforcement Learning Domains using Skill Chaining, NIPS, 2009\n\nTypos\nEq. (4), the state s is missing under the argmin\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Not applicable.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "PntfEiwlDHG", "reviewer": "Reviewer_Ta14", "summary": "The paper proposes a curriculum approach for persistent goal conditioned reinforcement learning, that is, for learning how to train continuously and then evaluate in episodic fashion. The main idea is, in each iteration, to try to keep the value of moving from a selected state to the goal above a threshold while selecting the state (where the agent tries to move before moving to the goal) as close as possible to the initial state distribution. The proposed approach outperforms comparison methods in three different simulated robotic benchmark tasks.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "According to my knowledge persistent learning is a not so well researched problem setting and thus the proposed approach is novel. The robotics motivation is definitely valid. Related work appears to be adequately cited.\n\nThe submission appears technically sound. The experimental results show that in three different simulated robotic goal-conditioned tasks the proposed algorithm outperforms baselines.\n\nThe proposed algorithm is based on generating a state that is as close as possible to the initial state distribution while maintaining predicted value over a threshold. The goal generator then uses that state as the current goal until reaching that state and going towards the goal of the actual task. Distance is measured using the number of time steps from the state to the initial state from existing data.\n\nAn interesting question is how the discount factor influences the behavior of the algorithm. Compared to continuous learning, since the agent always goes to the same goal and the same task repeats, here long term planning and thus the discount factor are not as important?\n\nThe paper makes the assumption of a reversible environment but this is adequately discussed.\n\nComputation time should be reported in the paper since the approach does computations that the comparison methods do not.\n\nThe paper is well written and organized. There are some typos that should be corrected. Examples:\n\nLine 79: \"a task-graph uses the current state to decides\" -> \"a task-graph uses the current state to decide\"\nLine 292: \"the the\" -> \"the\"\nLine 370: \"Thus, VaPRL can generates\" -> \"Thus, VaPRL generates\"\n\nThe results are important for real robotic reinforcement learning. Training without resets is critical in many robotic tasks. The proposed approach appears efficient which is validated in three benchmark tasks.\n\nUpdate:\n\nI was satisfied with the rebuttal and will keep my score.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "H2YdGjhkfcV", "reviewer": "Reviewer_yCgQ", "summary": "The authors present a novel RL method that learns an adaptive curriculum to minimize the number of resets needed in episodic RL. This is evaluated against competing algorithms on manipulation tasks.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Minimizing the number of resets needed could be useful in many real-world RL applications. The approach is interesting and the experimental results look very promising compared to competing approaches.\n\nMy main concern is with the presentation. The paper is mostly well-written, but there appears to be many complicated design choices in there with subtle theoretical implications. I'm only superficially familiar with the related work in this niche of RL, but the general idea and mathematical formalization could be clearer. I had to read it several times.\n\nSome comments:\n- If the focus is *reducing* resets (see main contribution on L54), it was a bit confusing why the main baseline wasn't the number of resets needed vs. episodic RL? Instead you compare against \"episodic\" RL without resets, which is also a bit counterintuitive.\n- It is not obvious to me why RL without resets performs so poorly in Fig.2. You claim \"This is because these methods rely on the ability to sample the initial state distribution arbitrarily\", but this does not seem entirely true as RL can be non-episodic (and many \"episodic\" applications just use approximate infinite horizons via e.g. GAE). Could you comment on why this happens? Is it perhaps because your learned reversion to the initial state distribution yields better exploration somehow (e.g. if items otherwise get pushed out to the corners with time), or could it just be that you are using HER?\n- Likewise in the formal problem definition in (3), it's a bit confusing to use a finite horizon formulation when considering an infinite horizon setting in Fig.2 and calling it \"persistent\" RL. It really seems to me that this is an inifinite horizon problem. It's not that you are minimizing resets, you want to maximize reward over an infinite horizon without resets at all, right? It would help to at least clarify this.\n- The abstract and intro also seems to equate all RL with episodic RL. Again, careful with terminology and notation.\n\nMinor details:\n\nL66: \"can be up to 30% more sample-efficient\" - needlessly vague, were on average x% more sample-efficient in our experiments perhaps?\n\nL70: manually design*ed*\n\nL246 & 252: Conenct reference [2] to HER.\t\n\nL252: \"... and *the* goal ...\"\n\nL292: \"the the\"\n\n----\n\n\nAfter rebuttal:\nI thank the authors for their clear response. I share some of the other reviewers' concerns that this paper could benefit from a more formal treatment, but the authors' clarifications were helpful. I will maintain my score of 6.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Superficial but adequate.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "6-B9yy0_KOx", "reviewer": "Reviewer_2Rka", "summary": "This paper focuses on ‘reset-free’ RL, i.e., the aim to eliminate the need for RL agents to be reset to one of a number of states from a start state distribution. The proposed a curriculum inspired approach, where the agent resets to progressively harder starting states, that are further away from the goal. They first formally define the persistent reinforcement learning problem. Then they specify a new method to solve this problem (VaPRL), which indeed outperforms baselines in three simulation experiments. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strong:\n* Good formal definition of the type of problem. Although it is close to continual learning, I do agree with the authors that the setting is slightly different, and it is useful to identify this.\n* The method is clear and works better than the baselines. \n\nComments:\n* Does the epsilon specification scale with funtion approximation? Then, it may become really hard to trust the epsilon estimate.\n* It is intuitively a bit strange that you generate the curriculum by moving start states backwards from the goal, but to find out which state is a good next candidate as start state (using the distance function), you already use V(s,s_0), i.e., the value to get there from the start state. This is kind of the thing you wanted to eliminate with the curriculum in the first place, i.e., only focus on the easier last part first. \t\n* Sec 4.2: I had slight issues understanding what’s going on here. First, how do you know all the elements in the goal space (distribution), are they explicitly given? Second, you start the section about C(g), but then you only relabel the goals right, not the start states? (and why not?)\n* Algorithm 1: I would comment again that the switching happens within G(s,g_p), this confused me at first.\n* I have some issues with the fact that you give demonstration trajectories to guide initial exploration (L289-305). You wanted to eliminate manual intervention, yet now you start from good trajectories. It is hard for me to judge how optimal these trajectories are, and thereby judge to what extend you are only inpainting/working backwards along an already optimal path with your method.\n* There is no discussion at all, which I think is a problem. You should reflect a bit on your method, identify its potential problems and pitfalls, what did you encounter during experimentation. \n\nConclusion:\nThis is a decent paper. The problem is relevant, notation and methods are clear, and results are good. My main issue is the use of expert demonstration trajectories to warm-start the algorithm, which I find really hard to judge because I do not know the quality of these demonstration trajectories. They may have a major impact, but the authors do not report on this ablation. A bit more discussion would be helpful al well, but in general this is a good paper with clear contribution, good structure and writing, and good results. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors state there are no explicit negative potential impacts of this work (beyond the general ones of RL), I agree", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: After reading each other's reviews and the authors' feedback, the reviewers discussed the merits and flaws of the paper. 
The reviewers did not reach a consensus. In particular, the lack of a theoretical analysis together with some doubts about its applicability to realistic scenarios have been considered as significant limitations.
Nonetheless, the majority of the reviewers were satisfied with the answers provided by the authors and think that the approach proposed in the paper is interesting and promising.
Overall, I think that, despite its limits, the paper provides a nice contribution and I propose to accept it.
I want to congratulate the authors and invite them to modify their paper following the reviewers' suggestions.
Decision: Accept (Poster)