Autonomous Reinforcement Learning via Subgoal Curricula

1 Introduction . Reinforcement learning ( RL ) offers an appealing opportunity to enable autonomous acquisition of complex behaviors for interactive agents . Despite recent RL successes on robots [ 26 , 34 , 25 , 28 , 35 , 22 , 32 , 23 , 14 ] , several challenges exist that inhibit wider adoption of reinforcement learning for robotics [ 48 ] . One of the major challenges to the autonomy of current reinforcement learning algorithms , particularly in robotics , is the assumption that each trial starts from an initial state drawn from a specific state distribution in the environment . Conventionally , reinforcement learning algorithms assume the ability to arbitrarily sample and reset to states drawn from this distribution , making such algorithms impractical for most real-world setups . Many prior examples of reinforcement learning on real robots have relied on extensive instrumentation of the robotic setup and human supervision to enable environment resets to this initial state distribution . This is accomplished through a human providing the environment reset themselves throughout the training [ 8 , 12 , 4 ] , scripted behaviors for the robot to reset the environment [ 28 , 39 ] , an additional robot executing scripted behavior to reset the environment [ 32 ] , or engineered mechanical contraptions [ 46 , 23 ] . The additional instrumentation of the environment and creating scripted behaviors are both time-intensive and often require additional resources such as sensors or even robots . The scripted reset behaviors are narrow in application , often designed for just a single task or environment , and their brittleness mandates human oversight of the learning process . Eliminating or minimizing the algorithmic reliance on the reset mechanisms can enable more autonomous learning , and in turn it will allow agents to scale to broader and harder set of tasks . 1Code and supplementary videos are available at https : //sites.google.com/view/vaprl/home 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . To address these challenges , some recent works have developed reinforcement learning algorithms that can effectively learn with minimal resets to the initial distribution [ 19 , 6 , 48 , 43 , 14 ] . We provide a formal problem definition that encapsulates and sheds light on the general setting addressed by these prior methods , which we refer to as the persistent reinforcement learning in this work . In this problem setting , we disentangle the training and the test time settings such that the test-time objective matches that of the conventional RL setting but the train-time setting restricts access to the initial state distribution by giving a low frequency periodic reset . In this setting , the agent must persistently learn and interact with the environment with minimal human interventions , as shown in Figure 1 . Conventional episodic RL algorithms often fail to solve the task entirely in this setting , as shown by Zhu et al . [ 48 ] and Figure 2 . This is because these methods rely on the ability to sample the initial state distribution arbitrarily . One solution to this problem is to additionally learn a reset policy that recovers the initial state distribution [ 19 , 6 ] allowing the agent to repeatedly alternate between practicing the task and practicing the reverse . Unfortunately , not only can solving the task directly from the initial state distribution be hard from an exploration standpoint , but ( attempting to ) return to the initial state repeatedly can be sample inefficient . In this paper , we propose to instead have the agent reset itself to and attempt the task from different initial states along the path to the goal state . In particular , the agent can learn to solve the task from easier starting states that are closer to the goal and bootstrap on these to solve the task from harder states farther away from the goal . The main contribution of this work is Value-accelerated Persistent Reinforcement Learning ( VaPRL ) , a goal-conditioned RL method that creates an adaptive curriculum of starting states for the agent to efficiently improve test-time performance while substantially reducing the reliance on extrinsic reset mechanisms . Additionally , we provide a formal description of the persistent RL problem setting to conceptualize our work and prior methods . We benchmark VaPRL on several robotic control tasks in the persistent RL setting against state-of-the-art methods , which either simulate the initial state distribution by learning a reset controller , or incrementally grow the state-space from which the given task can be solved . Our experiments indicate that using a tailored curriculum generated by VaPRL can be up to 30 % more sample-efficient in acquiring task behaviors compared to these prior methods . For the most challenging dexterous manipulation problem , VaPRL provides a 2.5× gain in performance compared to the next best performing method . 2 Related Work . Robot learning . Prior works using reinforcement learning have relied on manually design controllers or human supervision to enable episodic environmental resets , as is required by the current algorithms . This can be through human orchestrated resets [ 8 , 13 , 12 , 4 , 16 ] , which requires high frequency human intervention in robot training . In some cases , it is possible to execute a scripted behavior to reset the environment [ 28 , 32 , 47 , 39 , 46 , 1 ] . However , programming such behaviors is time-intensive for the practitioner , and robot training still requires human oversight as the scripted behaviors are often brittle . Some prior works have designed the environment [ 35 , 7 , 22 ] to bypass the need for having a reset mechanism . This is not generally applicable and can require extensive environment design . Some recent works leverage multi-task RL to bypass the need for extrinsic reset mechanisms [ 15 , 14 ] . Typically , a task-graph uses the current state to decides the next task for the agent , such that only minimal intervention is required during training . However , these task-graphs are specific to a problem and require additional engineering to appropriately decide the next task . Reset-free reinforcement learning . Constraining the access to these orchestrated reset mechanisms severely impedes policy learning when using current RL algorithms [ 48 ] . Recent works have proposed several algorithms to reduce reliance on extrinsic reset mechanisms by learning a reset controller to retrieve the initial state distribution [ 19 , 6 ] , by learning a perturbation controller [ 48 ] , or by learning reset skills in adversarial games [ 43 ] . These works implicitly define a target state distribution for a reset controller : Han et al . [ 19 ] , Eysenbach et al . [ 6 ] target a fixed initial state distribution ; Zhu et al . [ 47 ] target a uniform distribution over the state space as a consequence of novelty-seeking behavior of the reset controller ; and Xu et al . [ 43 ] target an adversarial form of initial state distribution to produce a more robust policy . In contrast , our proposed algorithm VaPRL generates a curriculum of starting states tailored to the task and agent ’ s performance . Our experiments demonstrate that VaPRL outperforms these prior methods in both sample efficiency and absolute performance . Other recent work like [ 29 ] has considered combining model-based RL with unsupervised skill discovery to solve reset-free learning problems , but largely focus on avoiding sink states rather than attempting tasks repeatedly with a curriculum like VaPRL . Curriculum generation for reinforcement learning . Curriculum generation is a crucial aspect of sample-efficient learning in VaPRL . Prior works have shown that using a curriculum can enable faster learning and improve performance [ 9 , 10 , 40 , 30 , 33 , 27 ] . Task-tailored curriculum can simplify the exploration as it is easier to solve the task from certain states [ 21 , 9 ] enabling faster progress on the downstream task . In addition to proposing a novel method for curriculum generation , we design it for the persistent RL setting without requiring the ability to reset the environment to arbitrary states as assumed by prior work . Persistent vs. lifelong reinforcement learning . Prior reinforcement learning algorithms that reduce the need for oracle resets have relied on the problem setting of lifelong or continual reinforcement learning [ 41 , 24 ] , when the objective in practice is to learn episodic behaviors . Both the persistent RL and the lifelong learning frameworks do transcend the episodic setting for training , promoting more autonomy in reinforcement learning . However , persistent reinforcement learning distinguishes between the training and evaluation objectives , where the evaluation objective matches that of the episodic reinforcement learning . While the assumptions of episodic reinforcement learning are hard to realize for real-world training , real-world deployment of policies is often episodic . This is commonly true for robotics , where the assigned tasks are expected to be repetitive but it is hard to orchestrate resets in the training environment . This makes persistent reinforcement learning a suitable framework for modelling robotic learning tasks . 3 Persistent Reinforcement Learning . In this section , we formalize the persistent reinforcement learning as an optimization problem . The key insight is to separate the evaluation and training objectives such that the evaluation objective measures the performance of the desired behavior while the training objective enables us to acquire those behaviors , while recognizing that frequent invocation of a reset mechanism is untenable . We first provide a general formulation , and then adapt persistent RL to the goal-conditioned setting . Definition . Consider a Markov decision process ( MDP ) ME ≡ ( S , A , p , r , ρ , γ , HE ) [ 37 ] . Here , S denotes the state space , A denotes the action space , p : S ×A× S 7→ R≥0 denotes the transition dynamics , r : S ×A 7→ R denotes the reward function , ρ : S 7→ R≥0 denotes the initial state distribution , γ ∈ [ 0 , 1 ] denotes the discount factor , and HE denotes the episode horizon . Our objective is to learn a policy π that maximizes JE ( π ) = E [ ∑HE t=1 γ tr ( st , at ) ] , where s0 ∼ ρ ( · ) , at ∼ π ( · | st ) and st+1 ∼ p ( · | st , at ) , the episodic expected sum of discounted rewards . However , generating samples from the initial state distribution ρ invokes a reset mechanism , which is hard to realize in the real world . We want to construct a MDPMT corresponding to our training environment which reduces invocations of the reset mechanism . To reduce such interventions , we consider a training environmentMT ≡ ( S , A , p , r̃t , ρ̃ , γ , HT ) with episode horizon HT HE . Näively optimizing r can substantially deteriorate the performance of episodic RL algorithms , as shown in Figure 2 where we compare the evaluation performance with HE = 200 when training in environments with HT = 200 ( with resets ) versus HT = 200 , 000 ( without resets ) . Therefore , it becomes beneficial to consider a surrogate reward function r̃t rather than just optimizing for r naively . As a motivating example , consider a forward-backward controller which alternates between solving the task corresponding to r and recovering the initial state distribution ρ . The surrogate reward function corresponding for this approach can be written as : r̃t ( st , at ) = { r ( st , at ) t = [ 1 , HE ] , [ 2HE + 1 , 3HE ] , . . . rρ ( st , at ) t = [ HE + 1 , 2HE ] , . . . ( 1 ) Here , r̃ alternates between the task-reward r for HE steps and rρ ( which encourages initial state distribution recovery ) for HE steps2 , also illustrated in Figure 3 ( right ) . This surrogate reward function allows the agent to repeatedly practice the task , thus using the autonomous interaction more judiciously as compared to the näive approach . Note , this r̃ loosely recovers the objectives used in some prior works [ 19 , 6 ] . For a general time-dependent surrogate reward function r̃t , we define the training objective as JT ( π ) = Es0∼ρ̃ , at∼π ( ·|st ) , st+1∼p ( ·|st , at ) [ HT∑ t=1 γtr̃t ( st , at ) ] ( 2 ) where ρ̃ is the initial state distribution at training time ( which does not need to match the evaluationtime initial state distribution ρ ) . The persistent RL optimization objective is to maximize JT ( π ) efficiently under the constraint that JE ( argmaxπ JT ( π ) ) = maxπ JE ( π ) . Intuitively , the objective encourages construction of a training environment that can recover the optimal policy for the evaluation environment . The primary design choice is r̃t , which as shown above leads to different algorithms . Another design choice is ρ̃ , which may or may not match ρ . Importantly , we do not assume ρ̃ is any easier to sample compared to ρ . Finally , we note that the formulation discussed here is suitable only for reversible environments . Reversible environments guarantee that the agent can continue to make progress on the task and not get “ stuck ” ( for example , if the object goes out of reach of the robot ’ s arm ) . A large class of practical tasks can be considered reversible ( door opening , cloth folding , and so on ) or the environment can be constructed to enforce reversibility ( add bounding walls so the object does not go out of reach ) . A formal definition for reversible environments is provided in Appendix A . In this work , we will restrict ourselves to reversible environments , and defer a full discussion of persistent RL for environments with irreversible states to future work . Goal-conditioned persistent reinforcement learning . We adapt the general formulation above to a goal-conditioned [ 20 , 38 ] instantiation of persistent RL . Consider a goal-conditioned MDPME ≡ ( S , A , G , p , r , ρ , γ , HE ) , where G ⊆ S denotes the goal space . For a goal distribution pg : G 7→ R≥0 , the evaluation objective is JE ( π ) = Eg∼pg ( · ) Eπ ( ·|s , g ) [ ∑HE t=1 γ tr ( s , g ) ] for π : S ×A× G 7→ R≥0 . The training objective is then stated as : JT ( π ) = Es0∼ρ̃ , at∼π ( ·|st , G ( st , pg ) ) , st+1∼p ( ·|st , at ) [ HT∑ t=1 γtr ( st , G ( st , pg ) ) ] , ( 3 ) where we assume that r̃ = r remains as a goal reaching objective , but where algorithms instead use a goal generator G to generate a curriculum of goals to practice throughout training3 . The intuition is that , since HT HE , the algorithm can repeatedly practice reaching various task-goals . However , the objective is to learn a policy that can reach task-goals from pg in the test environment , i.e. , starting from the initial state distribution ρ . This implies the goal generator G should expand the goal space beyond the task-goals to improve the policy π for the test environment . For example , the goal generator could alternate task-goals ( g ∼ pg ) and the initial state distribution ( s ∼ ρ ) , which again loosely recovers prior works [ 19 , 6 ] . This instantiation transforms the problem of finding the right reward function r̃ to the right curriculum of goals using G .

Summary: This paper introduces the definition of Persistent Reinforcement Learning (PRL), and an algorithm to solve a goal conditioning version of PRL - Value-Accelerated PRL (VaPRL) - based on two ideas: separate the training and evaluation MDPs; and build a curriculum of (easier to solve) MDPs.   The main idea behind the paper is to solve episodic RL tasks by minimizing the number of resets to the initial state, which is particularly important in real-work RL. This is achieved by using a generator of surrogate goals instead of solving directly for the initial task goal, and by using an initial state distribution different from the evaluation MDP. This initial state distribution is at the beginning of training closer to the goal, and during training it is expanded such that it approaches the initial state distribution of the evaluation MDP.  The experimental results empirically show the claimed benefits, and are compared with other algorithms that do not use any curriculum or have access to resetting to the initial state distribution.