Autonomous Reinforcement Learning: Formalism and Benchmarking
Authors: Archit Sharma
Kelvin Xu
Nikhil Sardana
Abhishek Gupta
Karol Hausman
Sergey Levine
Chelsea Finn
Venue: ICLR 2022 Poster (2022)
Keywords: reinforcement learning
autonomous
reset-free reinforcement learning
continual reinforcement learning

Abstract: Reinforcement learning (RL) provides a naturalistic framing for learning through trial and error, which is appealing both because of its simplicity and effectiveness and because of its resemblance to how humans and animals acquire skills through experience. However, real-world embodied learning, such as that performed by humans and animals, is situated in a continual, non-episodic world, whereas common benchmark tasks in RL are episodic, with the environment resetting between trials to provide the agent with multiple attempts. This discrepancy presents a major challenge when we attempt to take RL algorithms developed for episodic simulated environments and run  them on real-world platforms, such as robots. In this paper, we aim to address this discrepancy by laying out a framework for Autonomous Reinforcement Learning (ARL): reinforcement learning where the agent not only learns through its own experience, but also contends with lack of human supervision to reset between trials. We introduce a simulated benchmark EARL based on this framework, containing a set of diverse and challenging simulated tasks reflective of the hurdles introduced to learning when only a minimal reliance on extrinsic intervention can be assumed. We show that standard approaches to episodic RL and existing approaches struggle as interventions are minimized, underscoring the need for developing new algorithms for reinforcement learning with a greater focus on autonomy.

Published as a conference paper at ICLR 2022
AUTONOMOUS REINFORCEMENT LEARNING: FORMALISM AND BENCHMARKING
Archit Sharma'
Kelvin Xu
Nikhil Sardana' Sergey Levine
Abhishek Gupta
Karol Hausman'
Chelsea Finn' 1
I Stanford University
University of California, Berkeley 3 MIT
4 Google Brain
ABSTRACT
Reinforcement learning (RL) provides a naturalistic framing for learning through trial and error, which is appealing both because of its simplicity and effectiveness and because of its resemblance to how humans and animals acquire skills through experience. However, real-world embodied learning, such as that performed by humans and animals, is situated in continual, non-episodic world, whereas com- mon benchmark tasks in RL are episodic, with the environment resetting between trials to provide the agent with multiple attempts. This discrepancy presents a major challenge when attempting to take RL algorithms developed for episodic simulated environments and run them on real-world platforms, such robots. In this paper, we aim to address this discrepancy by laying out a framework for Autonomouy Reinforcement Learning (ARL): reinforcement learning where the agent not only learns through its own experience, but also contends with lack of human supervision to reset between trials. We introduce simulated bench- mark EARL around this framework, containing a set of diverse and challenging simulated tasks reflective of the hurdles introduced to learning when only a min- imal reliance on extrinsic intervention can be assumed. We show that standard approaches to episodic RL and existing approaches struggle as interventions are minimized, underscoring the need for developing new algorithms for reinforce- ment learning with a greater focus on autonomy.
INTRODUCTION
One of the appeals of reinforcement learning that it provides naturalistic account for how com- plex behavior could emerge autonomously from trial and error, similar to how humans and animals can acquire skills through experience in the real world. Real world learning however, is situated in a continual, non-episodic world, whereas commonly benchmarks in RL often assume access to an oracle reset mechanism that provides agents the ability to make multiple attempts. This presents major challenge with attempting to deploy RL algorithms in environments where such an as- sumption would necessitate costly human intervention or manual engineering. Consider an example of a robot learning to clean and organize home. We would idcally like the robot to be able to autonomously explore the house, understand cleaning implements. identify good strategies on its own throughout this process, and adapt when the home changes. This vision of robotic learning in real world, continual, non-episodic manner is stark contrast to typical experimentation in rein- forcement learning in the literature (Levine et al., 2016; Chebotar al., 2017: Yahya et al., 2017; Ghadirzadeh et al., 2017; Zeng et al., 2020), where agents must be consistently reset to a set of initial conditions through human effort or engineering so that they may try again. Autonomy, especially for data hungry approaches such as reinforcement learning, is an enabler of broad applicability, but rarely an explicit consideration. In this work, we propose to bridge this specific gap by providing a formalism and set of benchmark tasks that considers the challenges faced by agents situated in non-epi non-episodic environment, rather than treating them as being abstracted away by an oracle reset.
"Equal contribution. Corresponding authors: architsh stanford.cduu kelvinxu @erinxuu@hert herkeley.cdu Code and related information for EARL can he found architsharma97. gwthht/...ihhho/aaaaaat benchmark/

---

Published as a conference paper at ICLR 2022
Our specific aim is to place greater focus developing algorithms under assumptions that more closely resemble autonomous real world robotic deployment. While the episodic setting naturally captures the notion of completing "task", it hides the costs of assuming oracle resets, which when removed can cause algorithms developed in the episodic setting to perform poorly (Sec. 6.1). More- over, while prior work has examined settings such as RL without resets (Eysenbach et al., 2017; Xu et at. Zhu et al., 2020; Gupta et al., 2021), I), ecological RL (Co-Reyes et al., 2020), or RL amidst non-stationarity (Xie et al., 2020) in isolated scenarios, these settings are not well-represented in ex- isting benchmarks. As a result, there is not consistent formal framework for evaluating autonomy in reinforcement learning and there is limited work in this direction compared to the vast literature on reinforcement learning. By establishing this framework and benchmark, we aim to solidify the importance of algorithms that operate with greater autonomy in RL..
Before we can formulate set of benchmarks for this type of autonomous reinforcement learning, we first formally define the problem we are solving. As we will discuss in Section 4, we can formulate two distinct problem settings where autonomous learning presents a major challenge. The first is a setting where the agent first trains in a non-episodic environment, and is then "deployed" into an episodic test environment. In this setting, which is most commonly studied in prior works "reset- free" learning (Han et al., 2015; Zhu et al., 2020; Sharma et al., 2021), the goal is to learn the best possible episodic policy after a period of non-episodic training. For instance, in the case of home cleaning robot, this would correspond to evaluating its ability to clean a messy home. The second setting is a continued learning setting: like the first setting, the goal to learn in a non-episodic environment, but there is no distinct "deployment" phase, and instead the agent must minimize regret over the duration of the training process. In the previous setting of the home cleaning robot, this would evaluate the persistent cleanliness of the home. We discuss in our autonomous RL problem definition how these settings present a number of unique challenges, such as challenging exploration.
The main contributions of our work consist of a benchmark for autonomous RL (ARL), as well as formal definitions of two distinct ARI. settings. Our benchmarks combine components from pre- viously proposed environments (Coumans & Bai, 2016; Gupta et al., 2019; Yu et al., 2020; Gupta et al., 2021; Sharma et al., 2021), but reformulate the learning tasks to reflect ARL constraints, such as the absence of explicitly available resets. Our formalization of ARL relates it to the standard RL problem statement, provides a concrete and general definition, and provides a number of instan- tiations that describe how common ingredients of ARL, such irreversible states, interventions, and other components, fit into the general framework. We additionally evaluate a range of previ- ously proposed algorithms on our benchmark, focusing on methods that explicitly tackle reset-free learning and other related scenarios. We find that both standard RL methods and methods designed for reset-free learning struggle to solve the problems in the benchmark and often get stuck in parts of the state space, underscoring the need for algorithms that can learn with greater autonomy and suggesting a path towards the development of such methods.
2 RELATED WORK
Prior work has proposed number of benchmarks for reinforcement learning, which are often either explicitly episodic (Todorov et al., 2012; Beattic et al., 2016; Chevalier-Boisvee et al., 2018), or consist of games that are implicitly episodic after the player dies completes the game (Bellemare et al., 2013; Silver et al., 2016). In addition, RL benchmarks have been proposed in the episodic set- ting for studying a number of orthogonal questions, such multi-task learning (Bellemare al., 2013; Yu et al., 2020), sequential task learning (Wolczyk et al., 2021), generalization (Cobbe et al., 2020), and multi-agent learning (Samvelyan et al., 2019; Wang et al., 2020). These benchmarks differ from our own in that we propose to study the challenge of autonomy. Among recent benchmarks. the closest to our own is Jelly Bean World (Platanios et al., 2020), which consists of set of procedural generated gridworld tasks. While this benchmark also considers the non-episodic setting, our work is inspired by the challenge of autonomous learning robotics, and hence considers an array of manipulation and locomotion tasks. In addition, our work aims to establish conceptual framework for evaluating prior algorithms in light of the requirement for persistent autonomy.
Enabling embodied agents to learn continually with minimal interventions is a motivation shared by several subtopics of reinforcement learning research. The setting we study in our work shares con- ceptual similarities with prior work in continual and lifelong learning (Schmidhuber, 1987; Thrun & Mitchell, 1995; Parisi et al., 2019; Hadsell et al., 2020). In context of reinforcement learning, this work has studied the problem of episodic learning in sequential MDPs (Khetarpal et al., 2020; Rusu
2

---

Published as a conference paper at ICLR 2022
0) Autonomous dothes hanging pradtioe
1) Depicyad policy evalustion on new basket of clothes
Rosett
2) policy evaluation home
Two evaluation schemes autonomous RL. First, deployment setting (top row, (1)), where we are interested in obtaining policy during training phase, æ, that performs well when deployed from a &p ~ p. Second, the continuing setting (bottom row, (2)), where a floor cleaning robot is tasked with keeping floor clean and is only evaluated on its cumulative performance (Fq. 2) over the agent's ' lifetime.
Figure I:
et al., 2016; Kirkpatrick al., 2017; Fernando et al., 2017; Schwarz et al., 2018; Mendez et al., 2020), where the main objective is forward/hackward transfer or learning in non-stationary dynam- ics (Chandak et al., 2020; Xie et al., 2020; Lu et al., 2020). In contrast, the emphasis of our work is learning in non-episodic settings while literature in continual RL assumes episodic setting. As we will discuss, learning autonomously without access to oracle resets a hard problem even when the task dask-iistibuution and dynamics stationary. In a similar vein, unsupervised RL (Gregor et al., 2016; Pong et al., 2019; Eysenbach et al., 2018; Sharma et al., 2019; Campos et al., 2020) also enables skill acquisition in the absence of rewards, reducing human intervention required for designing reward functions. These works are complimentary to our proposal and form interesting future work.
Reset-free RL has been studied by previous works with a focus on safety (Eysenbach et al., 2017), automated and unattended learning in the real world (Han al., 2015; Zhu al., 2020; Gupta et al., 2021), skill discovery (Xu et al., 2020; Lu et al., 2020), and providing a curriculum (Sharma al., 2021). Strategies to learn frse behavior include directly learning a backward reset con- troller (Eysenbach et al., 2017), learning set of auxillary tasks that can serve as approximate reset (Ha et al., 2020; Gupta al., 2021), or using a novelty seeking reset controller (Zhu et al., 2020). Complementary to this literature, we aim to develop a set of benchmarks and framework that allows for this class of algorithms to be studied in a unified way. Instead of proposing new algorithms, our work is focused on developing set of unified tasks that emphasize and allow to study algorithms through the lens of autonomy.
3 PRELIMINARIES
Consider a Markov Decision Process (MDP) = (S, 7) (Sutton & Barto, 2018). Here, denotes the state space, denotes the action space, p S x A S + R20 denotes the transition dynamics, r S x A ++ IR denotes the reward function, p S - Rx0 denotes the initial state distribution and 7 E [0, denotes the discount factor. The objective in reinforce- ment learning is maximize (mT)
E IIEEo ? with respect the policy π, where 80 p(-), ag ~ T(- s:) and 8;-1 ~ p(- St, az). Importantly, the RL. framework assumes the abil- P arbitrarily. Typical implementations of reinforcement learning algorithms carry out thousands or millions of these trials, implicitly requiring the environment to provide a mechanism to be "reset" to a state 8q ~ p for every trial.
E
ity sample
AUTONOMOUS REINFORCEMENT LEARNING
In this section, we develop a framework for autonomous reinforcement learning (ARL) that formal- izes reinforcement learning in settings without extrinsic interventions, We first define non-episodic training environment where the agent can autonomously interact with its environment in Section 4.1, building on the formalism of standard reinforcement learning. We introduce two distinct evaluation settings as visualized in Figure I: Section 4.1.1 discusses the deployment setting where the agent will be deployed in test environment after training, and the goal is to maximize this "deployed" performance. Section 4.1.2 discusses the continuing setting, where the agent has no separate de- ployment phase and aims to maximize the reward accumulated over its lifetime. In its most general form, the latter corresponds closely to standard RL, while the former can interpreted as a kind of transfer learning. As will discuss in Section 4.2, this general framework can be instantiated such
3

---

Published as a conference paper at ICLR 2022
that, for different choices of the underlying MDP, model different realistic autonomous RL scenarios such as settings where a robot must learn to reset itself between trials or settings with non- reversible dynamics. Finally, Section 4.3 considers algorithm design for autonomous RL, discussing the challenges in autonomous operation while also contrasting the evaluation protocols.
4.1 GENERAL SETUP
Our goal is to formalize a problem setting for autonomous reinforcement learning that encapsu- lates realistic autonomous learning scenarios. We define the setup in terms of training MDP My = (S, A. p. p), where the environment initializes to so ~ P. and then the agent interacts with the environment autonomously from then on. Note, this lack of episodic resets in our setup departs not only from the standard RL setting, but from other continual reinforcement learning set- tings e.g. Wotezyk et al. (2021), where resets are provided between tasks. Symbols retain their meaning from Section 3. In this setting, a learning algorithm A can be defined as a function A : {8; 0z ra)1 + (at, me), which maps the transitions collected in the environment until the time & (e.g., a replay buffer), to (potentially exploratory) action E AA applied in the en- vironment and its best guess at the optimal policy कe : S x A + I220 used for evaluation at time t. We note that the assumption of reward function implicitly requires human engineering, but in principle could relaxed by methods that learn reward functions from data. addition, we note that a does not need to come from Tr, which is already implicit in most reinforcement learning algorithms: Q-learnng (Sutton & Barto, 2018) methods such as DQN (Mnih et al., 2015), DDPG (Lillicrap et al., 2015) an e-greedy policy as exploration policy top of the greedy policy for evaluation. However, our setting necessitates more concerted exploration, and the exploratory action may come from an entirely different policy. Note that the initial state distribution is sampled (so ~ p) exactly once to begin training, and then the algorithm A is run until t + oo, generating the sequence Su, Gg S1, 01. ................................ in the MDP Mr. This the primary difference compared to the episodic setting described in Section 3, which can sample the initial state distribution repeatedly.
4.1.1 ARL DEPLOYMENT SETTING
Consider the problem where a robot has to leamn how to close a door. Traditional reinforcement learning algorithms require several trials, repeatedly requiring interventions open the door be- tween trials. The desire is that the robot autonomously interacts with the door, learning to open it if it is required to practice closing the door. The output policy of the training procedure is evaluated in its deployment setting, in this case on its ability to close the door. Formally, the evaluation objective JD for a policy π in the deployment setting can be written as:
(T( = P.o-p..ayr--lbl (atrr
Σ r'((s,, az)]. 1=0
(I)
Definition 1 (Deployed Policy Evaluation). For algorithm A {sa, ai a, 1/2 + (at, TT), de- ployed policy evaluation D(A) is given by D(A) = Ei= (Jn(TT) Jn(T))). where (T) is defined in Eq and x² E arg max, (T)
The evaluation objective (T) is identical to the one defined in Section MDP M (deployment environment). The policy evaluation is "hypothetical", the environment rollouts used for evaluating policies are not used in training. Even though the evaluation trajectories are rolled out from the initial state, there are no interventions in training. Concretely, the algorithmic goal in this setting can stated as min, D(A). In essence, the policy outputs ne from the autonomous algorithm A should match the oracle deployment performance, i.c. JD(x"), as quickly possible. Note that JD( (x)) is a constant that can be ignored when comparing two algorithms, i.c. we only need to know p (TT) for a given algorithm in practice.
4.1.2 ARL CONTINUING SETTING
For some applications, the agent's experience cannot be separated into training and deployment phases. Agents may have to learn and improve in the environment that they are "deployed" into, and thus these algorithms need to be evaluated example, a robot tasked with keeping the home clean learns and improves on the job as it adapts to the home in which it is deployed. To this end, consider the following definition:
their performance during the agent's lifetime. For
4

---

Published as a conference paper at ICLR 2022
Definition 2 (Continuing Policy Evaluation). For algorithm A {st, a, $4111=00 + (ar, mi),
continuing policy evaluation C(A) can be defined as:
h
C(A) = lim E (0 12-d) pl 10000 h
Σ 1=
r(st, at)]
(2)
Here, is the action taken by the algorithm A based on the transitions collected in the environment until time I, measuring the performance under reward r. The optimization objective can be stated as max,. C(A). Note that is not used in computing C(A). In practice, this amounts to measuring the reward collected by the agent in the MDP MM during its lifetime".
4.2 How SPECIFIC ARL PROBLEMS FIT INTO OUR FRAMEWORK
The framework can easily be adapted to model possible autonomous reinforcement learning scenar- ios that may be encountered:
Intermittent interventions. By default, the agent collects experience in the environment with fully autonomous interaction in the MDP Mr. However, we can model the occasional intervention with transition dynamies defined as p0( 8, a) = (1 ')p( 8, s,a) + <p(.), where the next state is sam- pled with 1 - probability from the environment dynamics or with probability from the initial state distribution via intervention for some E E [0, 1]. A low e represents very occasional interven- tions through the training in MDP MMr. In fact, the framework described in Section 3, which is predominantly assumed by reinforcement learning algorithms, can be understood to have a large c. To contextualize the agent should expect get an intervention after 1/c steps in the environment. Current episodic settings typically provide an environment reset every 100 to 1000 steps, corre- sponding (1e-3, le-2) and an autonomous operation time of typically few seconds few minutes depending on the environment. While full autonomy would be desirable (that is, 0), intervening every few hours to few days may be reasonable to arrange, which corresponds to envi- ronment resets every 100,000 to 1,000,000 steps or E (1e-6, le-5). We evaluate the dependence of algorithms designed for episodic reinforcement learning on the reset frequency in Section 6.1.
Irreversible states. An important consideration for developing autonomous algorithms is the "re- versibility" of the underlying MDP M. Informally, if the agent can reverse any transition in the environment, the agent is guaranteed to not get stuck in the environment. As an example, a static robot arm can be setup such that there always exists an action sequence to open or close the door. However, the robot arm can push the object out of its reach such that no action sequence can retrieve it. Formally, we require MDPs to be ergodic for them to be considered reversible (Moldovan & Abbeel, 2012). In the case of non-ergodic MDPs, we adapt the ARL framework to enable the agent to request extrinsic interventions, which we discuss in Appendix A.
4.3 DISCUSSION OF THE ARL FORMALISM
The ARL framework provides two evaluation protocols for autonomous RL. algorithms. Algorithms can typically optimize only one of the two evaluation metries. Which evaluation protocol should the designer optimize for? In a sense, the need for two evaluation protocols arises from task spe- cific constraints, which themselves
be sometimes relaxed depending the specific trade-off between the cost of real world training and the cost of intervention. The continuing policy evalua- tion represents the oracle metric one should strive to optimize when continually operational agents are deployed into dynamic environments. The need for deployment policy evaluation arises from two implicit practical constraints: (a) requirement of large number of trials to solve desired tasks and (b) absence of interventions to enable those trials. If either of these can be easily relaxed, then one could consider optimizing for continuing policy evaluation. For example, if the agent can learn in few trials because it was meta-trained for quick adaptation (Finn et al., 2017), providing a few interventions for those trials may be reasonable. Similarly, if the interventions are easy to obtain during deployment without incurring significant human cost, perhaps through scripted behaviors or enabled by the deployment setting (for example, sorting trash in a facility), the agent can repeatedly try the task and learn while deployed. However, if these constraints cannot be relaxed at deploy- ment, one should consider optimizing for the deployment policy evaluation since this incentivizes the agents to learn targeted behaviors by setting up its own practice problems.
We can also consider the more commonly used setting of expected discounted sum of rewards as the objective. To ensure that future rewards are relevant, the discount factors would need to be much larger than values typically used.
5

---

Published as a conference paper at ICLR 2022
5 EARL: ENVIRONMENTS FOR AUTONOMOUS REINFORCEMENT LEARNING
In this section, we introduce the of environments in our proposed benchmark, Environments for Autonomous Reinforcement Learning (EARL). We first discuss the factors in our design criteria and provide a description of how each environment fits into our overall benchmark philosophy, before presenting the results and analysis. For detailed descriptions of each environment, see Appendix A.
5.1 BENCIMARK DESIGN FACTORS
Representative Autonomous Settings. We include broad array of tasks that reflect the types of autonomous learning scenarios agents may encounter in the real world. This includes different problems manipulation and locomotion, and tasks with multiple object interactions for which would be challenging to instrument resets. We also ensure that both the continuing and deployment evaluation protocols of ARL are realistic representative evaluations.
Directed Exploration. In the autonomous setting, the necessity to practice task again, potentially from different initial states, gives rise the need for agents learn rich reset behaviors. For example, in the instance of a robot learning to interact with multiple objects in a kitchen, the robot must also learn to implicitly or explicitly compose different reset behaviors.
Rewards and Demonstrations. One final design aspect for our benchmark is the choice of reward functions, Dense rewards are a natural choice in certain domains (e.g., locomotion), but designing and providing dense rewards in real world manipulation domains can be exceptionally challenging. Sparse rewards are easier to specify in such scenarios, but this often makes exploration impractical. As result, prior work has often leveraged demonstrations (e.g., (Gupta et al., 2019)), especially in real world experimentation. To reflect practical usage of RI real world manipulation settings, we include small number of demonstrations for the sparse-reward manipulation tasks.
5.2 ENVIRONMENT DESCRIPTIONS
Tabletop-Organization (TO). The agnostic object manipulation environment proposed by Sharma et al. (2021). The agent consists of a gripper modeled as pointmass, which can grasp objects that are close to it. The agent's goal is to bring mug to four different locations designated by goal coaster. The agent's reward function is sparse indicator function when the mug is placed at the goal location. Limited demonstrations are provided to the agent.
task a di-
Sawyer-Door (SD). The Sawyer-Door task, from the Meta World bench- mark (Yu et al., 2020) consists of a Sawyer robot arm who's goal is close the door whenever it is in an open position. The task reward is sparse indicator function based on the angle of the door. Repeatedly practicing this task implicitly requires the agent to learn to open the door. Limited demonstrations for opening and closing the door are provided.
Sawyer-Peg (SP). The Sawyer-Peg task (Yu et al., 2020) consists of a Sawyer robot required to insert a peg into a designed goal location. The task reward is a sparse indicator function for when the peg is in the goal location. In the deployment setting, the agent must Iearn to insert the peg starting the table. Limited demonstrations for inserting and removing the peg are provided.
Tranka-Kitchen (FK). The Franka-Kitchin (Gupta et al., 2019) is a do- main where 9-DoF robot, situated in kitchen environment, is required to solve tasks consisting of compound object interactions. The environment consists of microwave, hinged cabinet, burner, and slide cabinet. One example task is to open the microwave, door and burner. This domain presents number of distinct challenges for ARL. First, the compound nature of each task results in a challenging long horizon problem, which introduces exploration and credit assign- ment challenges. Second, while generalization is important solving the environment, combining resef behaviors are equally important given the compositional nature of the task.

---

Published as a conference paper at ICLR 2022
DHand-LightBulb (DL). The DHand-Lightbull environment consists of a 22- DoF 4 fingered hand, mounted on 6 DoF Sawyer robot. The environment based originally proposed by Gupta et al. (2021). The task in this do- main is for the robot to grasp pickup a lightbulb to specific location. The high- dimensional action space makes the task extremely challenging. In the deploy- ment setting, the bulb can be initialized anywhere on the table, testing the agent on wide initial state distribution.
Minitaur-Pen (MP). Finally, the Minitaur-Pen task consists of an 8-DoF Minitaur robot (Coumans & Bai, 2016) confined to a pen environment. The goal of the agent is to navigate is designed to mimic the setup of leaving a robot to learn to navigate within enclosed setting in autonomous fashion. This task is different from the other tasks given it is a locomotion task, as opposed to the other tasks being manipulation tasks.
set of goal locations in the pen. The task
6 BENCHMARKING AND ANALYSIS
The aim of this section understand the challenges in autonomous reinforcement learning and to evaluate the performance and shortcomings of current autonomous RL algorithms. In Section 6.1, we first evaluate standard episodic RL algorithms in ARL settings as they are required to operate with increased autonomy, underscoring the need for greater focus on autonomy in RL algorithms. We then evaluate prior autonomous learning algorithms on EARL in Section 6.2. While these al- gorithms do improve upon episodic RL methods, they fail to make progress on more challenging tasks compared to methods provided with oracle resets leaving a large gap for improvement. Lastly, in Section 6.3, investigate the learning of existing algorithms, providing hypothesis for their inadequate performance. We also find that when autonomous RL does succeed, tends to find more robust policies, suggesting an intriguing connection between autonomy and robustness.
6.1 GOING FROM STANDARD RL TO AUTONOMOUS RL
Cheotah (Uroken TD Chemn)
16.3
60-2
782
260
100
X80
0.00
K00
See
Se
360
166
Environment
Steps
Enviromment Steps
Enviroument Steps
steps
2000 stape
5000 steps
10000 stops
50000 steps
100000 steps
200000 staps
Figure 2: Performance of standard RI. at with varying levels of autonomy, ranging from resets provided every 1000 to 200000 steps. Performance degrades substantially environment resets hecome infrequent.
In this section, evaluate standard RL methods to understand how their performance changes when they are applied naïvely to ARL problems. To create a continuum, we will vary the level of "autonomy" (i.e., frequency of resets), corresponding experiments only, we use the simple cheetah and fish environments from the DeepMind Con- trol Suite (Tassa al., 2018). We use soft actor-critic (SAC) (Haarnoja et al., 2018a) as rep- resentative standard RL algorithm. We consider different training environments with increasing number of steps between resets, ranging from 1000 to 200, 000 steps. Figure 2 shows the perfor- mance of the learned policy as the training progresses, where the return is measured by running the policy for 1000 steps. The cheetah environment an infiitt-hhrizon running environment, so changing the training horizon should not affect the performance theoretically. However, we find that the performance degrades drastically as the training horizon is increased, as shown in Fig (left). We attribute this problem to a combination of function approximation and temporal differ- ence learning. Increasing the episode length destabilizes the learning as the effective bootstrapping length increases: the Q-value function (s(s ao) bootstraps bootstraps on Q* a2) and so on till Q (2 000) To break this chain, we consider a biased TD update: Q" (8t, ar) r(st, a+) + 7Q" (8t+1, at+1) 1) if & is not multiple of 1000, else
defined in Section 4.2. For these
the value of Q" (s((s,,00 a1). which
7

---

Published as a conference paper at ICLR 2022
Method
TO
SD
FK
DL
MP
nolve
0.32 (0.17)
0.00 (0.00)
(0.00)
-2705.21 166
30 (8.85)
-0041. (44.58)
FBRL
0,94 (0.04)
1.00 (0.00)
(0.00)
-2733.15 15 (324, 10)
-242.38 (8.84)
-986.34 (67.95)
R3L
0.96 (0.04)
0.54 (0.18)
0.00 (0.00)
-2639.28 (233.28)
728.54 (122. 86)
-186.30 (34.79)
VaPRL
0.98 (0.02)
0.94 (0.05)
0.02 (9.02)
onaele RT.
0.80 (0.1
1.00 (0.00)
1.00 (0.00)
120388 (203.86)
2028.75 (35.95)
-11.50 (3.40)
Table 1: Average return of the final deployed policy. Performance is averaged over 5 random seeds. The mean and and the standard error reported, with the best performing entry in bold. For sparse reward domains (TO, SD, SP), 1.0 indicates the maximum performance and 0.0 indicates minimum performance.
Method
TO
SD
FK
DL
MP
naive RL
0.012 (0.004)
0,373 (0.073)
< 0.001
4,944 (0.440)
-0.734 (0.024)
-18.193 (2.375)
FBRI.
(0.001)
(0.080)
0.003 (0.00.)
(0.405)
-0.747 (0.014)
-22.087 (1.285)
R3L
0.001 (0.000)
0.369 (0.016)
< 0.001
-6.577 (0.309)
0,091 (0.026)
-1,093 (0.020)
VaPRL
0.009 (0.0000
0.574 (0.085)
< 0.001
Table 2: Average reward accumulated over the training lifetime in accordance to continuing policy evaluation. Performance is averaged over 5 random seeds. The mean and the standard error (brackets) reported, with the best performing entry in hold.
(st; ar) r(st, az). This is inspired by practical implementations of SAC (Haarnoja et al., 2018b), where the Q-value function regresses to rs,, a) for terminal transitions to stabilize the train- ing. This effectively fixes the problem for cheetah, this solution does not translate in general, as can be seen observed in the fish environment, where the performance continues to degrade with increasing training horizon, shown in Fig (right). The primary difference between cheetah and f fish is that the latter is a goal-reaching domain. The cheetah can continue improving its gait on the infinite plane without resets, whereas the fish needs to undo the task to practice goal reaching again, creating a non-trivial exploration problem.
shown in Figure 2 (middle). However,
6.2 EVALUATION: SETUP, METRICS, BASELINES, AND RESULTS
Training Setup. Every algorithm A receives set of initial states so p, and a set of goals from the goal-distribution ~ p(g). Demonstrations, if available, are also provided to A. We consider the intermittent interventions scenario deseribed in Section 4.2 for the benehmark. In practice, the agent after a fixed number of steps HT. The value of Hy is fixed for every environment, ranging between 100, 000 400, 000 steps. Every algorithm A is run for fixed number of steps mmaz after which the training is terminated. Environment-speccin details are in Appendix A.4.
reset
Evaluation Metrics. For deployed policy evaluation, we compute IT(A) = -- -E Jp(m), ignoring Jn(x") as is a constant for all algorithms. Policy evaluation (Te) is carried out every 10000 training steps, where (m,) = EEo r(st: 0z) is the average return accumulated over an episode of Hg steps by running the policy Te 10 times, starting from so ~ p for every trial. These roll-outs are only used for evaluation, and are not provided to the algorithm. In practice, we plot (n,) versus time such that minimizing ((A) can be understood as maximizing the area under the learning curve, which we find more interpretable. Given finite training budget Hmaz, the policy ne may be quite suboptimal compared to x². Thus, we also report the performance of the final policy, that is ( in Table 1. For continuing policy evaluation C(A), we compute the average reward as F(h) Dt r(s,, a;)/h. We plot 7(h) versus h. while we report F(Hmmz) in Table 2. The evaluation curves corresponding to continuing and deployed policy evaluation are in Appendix A.6.
Baselines. We evaluate baakwrdd RL (FBRL) (Han et al., 2015; Eysenbach et al., 2017), a perturbation controller (R3L) (Zhu al., 2020), accel-accel persistent RL (VaPRL) (Sharma et al., 2021), comparison to simply running the base RI. algorithm with the biased TD update discussed in Section 6.1 (naïve RL), and finally an oracle (oracle RL) where resets are provided are provided every HE steps (HT is typically three orders of magnitude larger than Hg.. We benchmark VaPRL only when demonstrations are available, in accordance to the proposed algorithm in Sharma al. (2021). We average the performance of all algorithms across 5 random seeds. More details pertaining to these algorithms can be found in Appendix A.3, A.5.
Overall, see in Table that based the deployed performance of the final policy, tonomous RL algorithms substantially underperform oracle RL and fail to make any progress on sawyer-peg and franka- - -iennhhn. Notable exceptions are the performances of VaPRI on
8

---

Published as a conference paper at ICLR 2022
tabletop-orrgaizaatee and R3L minitaur-pen, outperforming oracle RL. Amongst the autonomous RL algorithms, VaPRL does best when the demonstrations are given and R3L does well domains when demonstrations are provided. Nonetheless, this leaves substantial room for improvement for future works evaluating on this benchmark. More detailed learning curves are shown in Section A.6. In the continuing setting, we find that naïve RL. performs well on certain domains (best on 2 out of 6 domains). This is unsurprising, as naive RL is incentivized to oc- cupy the final "goal" position, and continue accumulate over the course of its lifetime, whereas other algorithms are explicitly incentivized to explore. Perhaps surprisingly, we find that VaPRL on sawyer-doo
and R3L on chand--ightbull and minitaur does better than naive RL, suggesting that optimizing for deployed performance can also improve the continuing performance. Overall, we find that performance in the continuing setting does not necessarily translate im- proved performance in the deployed policy evaluation, emphasizing the differences between these two evaluation schemes.
6.3 ANALYZING AUTONOMOUS RL ALGORIHHMS
tablet.ppoorgnsaaatico
sawyer-paa
minitaur-pon
Goal State
Gripper Inirial State
Goal State
Mug
Initial
Initial State
State
Figure 3: Comparing the distribution of states visited with resets (blue) and without resets (brown). Heatmaps visualize the difference between state visitations for oracle RL and FBRL, thresholded to highlight states with large differences. Resets enable the agent to stay around the initial state distribution and the goal distribution, whereas the agents operating autonomously skew farther away, posing an exploration challenge.
Robusiness to initial state distribution
A hypothesis to account for the relative underperfor- mance of autonomous RL algorithms compared to ora- cle RL is that environment resets constrain the state dis- tribution visited by the agent close to the initial and the goal states. When operating autonomously for long peri- ods of time, the agent can skew far away from the goal states, creating hard exploration challenge. To test this hypothesis, we compare the state distribution when using oracle RL (in blue) versus FBRL (in brown) in Figure 3. We visualize the (x,w) positions visited by the gripper for tabletoo--orgaizaat of the peg for sawyer-peg and the x, y positions of the minitaur for minitaur-pen. As seen in the figure, autonomous operation skews the grip- per towards the corners in tabl etop-organ' zat on, the peg is stuck around the goal box and minitaur can completely go away from the goal distribution. However, when autonomous algorithms able to solve the task, the learned policies can be more robust as they are faced with a tougher exploration challenge during training. We visualize this in Figure 4, where we test the final policies learned by oracle RL, FBRL and VaPRL on tabletop-organi zat ion starting from uniform state distribution instead of the default ones. We observe that the policies learned by VaPRL and FBRL depreciate by 2% and 14.3% respectively, which is much smaller than the 37.4% depreciation of the policy learned by oracle RI, suggesting that autonomous RI. can lead to more robust policies.
Evaluation Performance
sebvat
oracle
V&PRL
Figure 4: Evaluating policies starting from uniform initial state distribution. Poli- cies learned via autonomous RL (FBRL and VaPRL) are more robust to initial state dis- tribution than policies learned in oracle RI.
the (xc y) positions
CONCLUSION
We proposed a formalism and benchmark for autonomous reinforcement learning, including an evaluation of prior state-of-the algorithms with explicit emphasis autonomy. We present two distinct evaluation settings, which represent different practical use cases for autonomous learning. The main conclusion from our experiments is that existing algorithms generally do not perform well in scenarios that demand autonomy during learning. We also find that exploration challenges, while present in the episodic setting, greatly exacerbated in the autonomous setting.

---

Published as a conference paper at ICLR 2022
While work focuses predominantly autonomous settings, there may be task-specific trade- offs between learning speed and the cost of human interventions, and it may indeed be beneficial to provide some human supervision to curtail total training time. How to best provide this supervi- sion (rewards and goal setting, demonstrations, resets etc) while minimizing human cost provides a number of interesting directions for future work. However, believe that there is lot of room to improve the autonomous learning algorithms and our work attempts to highlight the importance and challenge of doing so.
REFERENCES
Charles Beattie, Joel Z. Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq. Simon Green, Victor Valdés, Amir Sadik, Julian Schrittwieser, Keith Ander- son, Sarah York, Max Cant, Adam Cain, Adrian Bolton, Stephen Gaffney, Helen King, Demis Hassabis, Shane Legg, and Stig. Petersen. Deepmind lab. arXiv: [cs.Al]. 2016.
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Víctor Campos. Alexander Trott, Caiming Xiong, Richard Socher, Xavier Giró-i Nieto, and Jordi Torres. Explore, discover and learn: Unsupervised discovery state-covering skills, Interna- tional Conference on Machine Learning, pp. 1317-1327. PMLR, 2020.
Yash Chandak, Georgios Theocharous, Shiv Shankar, Martha White, Sridhar Mahadevan, and Philip Thomas. Optimizing for the future in non-stationary mdps. In International Conference on Ma- chine Learning, pp. 1414-1425. PMLR, 2020.
Yevgen Chebotar, Karol Hausman, Marvin Zhang, Gaurav Sukhatme, Stefan Schaal, and Sergey Levine. Combining model-based and model-free updates for trajectory-centrr reinforcement learning. International conference on machine learning, pp. 703-711. PMLR, 2017.
Maxime Chevalier-Boisvert, Lucas Willems, and Suman Pal. Minimalistic gridworld environment for openai gym. https: : / / thub. com ma x x www./ gym-mimm-in..mmmmmmm grid, 2018.
John Co-Reyes, Suvansh Sanjeev, Glen Berseth, Abhishek Gupta, and Sergey Levine. Ecological reinforcement learning. arXiv preprint arXiv:2006.. 12478, 2020.
Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In International conference machine learning, pp. 2048- 2056. PMLR, 2020.
Erwin Coumans and Yunfei Bai. Pybullet, a python module for physics simulation for games, robotics and machine learning. http:/ http:://ybbulet.oorg 2016.
Benjamin Eysenbach, Shixiang Gu, Julian Ibarz, and Sergey Levine. Leave no trace: Learning to reset for safe and autonomous reinforcement learning. arXiv preprint 1rXiv:1711.06785 2017.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity all you need: Learning skills without a reward function. arXiv preprint arXiv: /802.06070 2018.
Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. arXiv preprint arXiv:1701.08734 2017.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta- -eaanleanning for fast adaptation of deep networks. In International conference machine learning. PMLR, 2017.
Ali Ghadirzadeh, Atsuto Maki, Danica Kragic, and Märten Björkman. Deep predictive policy train- ing using reinforcement learning. In 2017 IEEE/RS.J International Conference on Intelligent Robots and Systems (IROS), pp. 2351-2355 IEEE, 2017.
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv preprint arXiv: 075507 2016.
10

---

Published as a conference paper at ICLR 2022
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving ong-horizon tasks via imitation and reinforcement learning. arXiv preprint arXiv:1910.11956. 2019.
Abhishek Gupta, Justin Yu, Tony Z Zhao, Vikash Kumar, Aaron Rovinsky, Kelvin Xu, Thomas Devlin, and Sergey Levine. Reset-free reinforcement learning via multi-task learning: Learning dexterous manipulation behaviors without human intervention. arXiv preprint arXiv:2 11003 2021.
Schoon Ha, Peng Xu, Zhenyu Tan, Sergey Levine, and Jie Tan. Learning walk in the real world with minimal human effort. arXiv preprint arXiv:2002.085550 2020.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learning. pp. 1861-1870. PMLR, 2018a.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and appli- cations. arXiv preprint arXiv:1812.05900 2018b.
Raia Hadsell, Dushyant Rao, Andrei A Rusu, and Razvan Pascanu. Embracing change: Continual learning in deep neural networks. Trends Cognilive Sciences, 2020.
Weiqiao Han, Sergey Levine, and Pieter Abbeel. Learning compound multi-step controllers un- der unknown dynamics. In 2015 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 6435 6442. IEEE, 2015.
Leslie Pack Kaelbling. Learning achieve goals. In ICCAI, pp.
Citeseer, 1993.
Khimya Khetarpal, Matthew Riemer, Irina Rish, and Doina Precup. Towards continual reinforce- ment learning: A review and perspectives. arXiv preprint arXiv:2012.134990 2020.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Trabska-Barwiiska. et al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521-355 -3222616,,,, 2017.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuo- motor policies. The Journal of Machine Learning Research, 17(1):1334-137,, 2016.
Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv 2015.
preprint
Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mordatch. Reset-free lifelong learning with skill- space planning. arXiv preprint arXiv:2012.03544 2020.
Jorge Mendez, Boyu Wang, and Eric Faton. Lifelong policy gradient learning of factored policies for faster training without forgetting. Advances in Neural Information Processing Systems, 33, 2020.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 51 5187754)) 2015.
Teodor Mihai Moldovan and Pieter Abbeel. Safe exploration in markov decision processes. arXiv preprint arXiv:1205. 48000 2012.
Genman I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 113:54-71, 2019.
Emmanouil Antonios Platanios, Abulhair Saparov, and Tom Mitchell. Jelly bean world: A testbed for never-ending learning. arXiv preprint 0rX1v:2000.066366 2020.

---

Published as a conference paper at ICLR 2022
Vitchyr H Pong. Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine. Skew- fit: State-covering self-supervised reinforcement learning. arXiv preprint arXiv:1903.0300 2019.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv:1606.04677, 2016.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder De Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. arXiv preprint (KKi::992.0444
2019.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approxima- tors. In International conference on machine learning, pp. 1312 1320. PMLR, 2015.
Jurgen Schmidhuber: Evolutionary principles in self-referential learning- learning now to learn: The meta-mmtta-meta..hhoo... Diploma thesis, Technische Universitat Munchen, Germany, 14 May 1987. URL hwwp:httt/tt //tp::///wwwww/ iisii.ii....... wtlreuueernnndppppepe httpp..mm.mmm....
Jonathan Schwarz, Wojeiech Czarnecki, Jelena Luketina, Agnieszka Bariskkaaaaiiksk Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable framework for contin- ual learning. In International Conference on Machine Learning, pp. 4528 4437. PMLR, 2018.
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-aware unsupervised discovery of skills. arXiv preprint
2019.
Archit Sharma, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Persistent reinforcement learning via subgoal curricula. arXiv preprint arXiv:2107. 12931, 2021.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schritwieser, Ioannis Antonoglou, Veda Panneershelvam, Mare Lanctot, al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484 489, 2016.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmalcki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.0066 2018.
Sebastian Thrun and Tom M Mitchell. Lifelong robot learning. Robotics and autonomous systems, 15(1-2):25-46 1995.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-500 5033. IEEE, 2012.
Rose E Wang, Sarah A Wu, James A Evans, Joshua B Tenenbaum, David C Parkes, and Max Kleiman-Weine Too many cooks: Coordinating multi-agent collaboration through inverse plan- ning. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems, pp. 2032-203-- 2020.
Maciej Wolczyk, Michat Zajc, Razvan Pascanu, Lukasz Kucinski, and Piotr Mitoš. Continual world: A robotic benchmark for continual reinforcement learning. arXiv preprint arXiv:2105.10919, 2021.
Annie Xie, James Harrison, and Chelsea Finn. Deep reinforcement learning amidst lifelong non- stationarity. arXiv preprint arXiv:2006. 10701, 2020.
Kelvin Xu, Siddharth Verma, Chelsea Finn, and Sergey Levine. Continual learning of control prim- itives: Skill discovery via reset-games. arXiv preprint arXiv:2011.05286 2020.
Ali Yahya, Adrian Li, Mrinal Kalakrishnan, Yevgen Chebotar, and Sergey Levine. Collective robot reinforcement learning with distributed asynchronous guided policy search. In 2017 JEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 79-86. IEFF, 2017.
12

---

Published as a conference paper at ICLR 2022
Tianhe Yu. Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference Robot Learning, pp. 1094-1100. PMLR, 2020.
Andy Zeng, Shuran Song, Johnny Lee, Alberto Rodriguez, and Thomas Funkhouser. Tossingbot: Learning to throw arbitrary objects with residual physics. IEEE Transactions on Robotics, 36(4): 1307-1319, 2020.
Henry Zhu, Justin Yu, Abhishek Gupta, Dhruv Shah, Kristian Hartikainen, Avi Singh, Vikash Ku- mar, and Sergey Levine. The ingredients of real world robotic reinforcement learning. In Inter- national Conference on Learning Representations, 2020.
13

---

Published as a conference paper at ICLR 2022
A APPENDIX
A.1
GOAL-CONDITIONED ARL
This framework can be readily extended to goal-conditioned reinforcement learning (Kaelbling, 1993; Schaul et al., 2015), which is also an area of study covered in some prior works on reset- free reinforcement learning (Sharma et al., 2021). Assuming goal space G and task-distribution Pg 9 + >> assume that the algorithm A : {s,, ag $4+1 1 1
H Te), where de A and
Te S A + Rzo- Equation can be redefined follows:
JD(T) = Egrpa ( (t.1))))))))))))pppp)))) (eee)))))
7 rr((s, at,9)]
(3)
1=
Additionally, we will assume that the algorithm has access to a set of samples 9i ~ p(g) from the goal distribution. The definitions for deployed policy evaluation and continuing policy evaluation remains the same.
A.2 REVERSIBILITY AND NON-ERGODIC MDPS
We expand on the discussion of reversibility in 4.2. We also discuss how we can deal with non- ergodic MDPs by augmenting the action space MDP Mr with calls for extrinsic interventions.
Definition 3 (Ergodic MDPs). A MDP considered ergodic if for all states Sa;
8 86 E S, F. such that π ((6))= > 0, where T(*) = 111-)) -) Ei= 7'p(s; s 8o Sa; .T) denotes discounted state dis- tribution induced by the policy æ starting from the state 80 = s. (Moldovan & Abbeel, 2012). A policy that assigns a non-zero probability to all actions on its support A ensures that all states in S are visited in the limit for ergodic MDPs, satisfying the condition above.
We adapt the ARL framework to develop learning algorithms for non-ergodic MDPs. In particular, introduce mechanism below for the agent to call for an intervention in the MDP Mr. Our goal here is to show that our described ARL framework is general enough to include cases where a human is asked for help in irreversible states. Consider augmented state space S+ S U Rz0 and an augmented action space A = A U An. where Ar denotes the action of asking for help via human intervention. A state s+ E S can be written as a state s E S. and h E R20 which denotes the remaining budget for interventions into the training. The budget h is initialized to Amaz when the training begins. The intervention can be requested by the agent itself using an action a E Ad, or it can enforced by the environment (for example, if the agent reaches certain detectable irreversible states the environment and requests a human to bring back into the reversible set of states). For an action E An. the environment transitions from (s, h) + (8', h c(s, a)). where the next state s' depends on the nature of the requested intervention a and c(s,a) S An + I2 denotes the cost of intervention. A similar transition occurs when the environment enforces the human intervention. When the cost exceeds the remaining budget, that is h - c(s, a) < O, the environment transitions into an absorbing state Sg. and the training is terminated. We retain the definitions introduced in Sec 4.1.1 and 4.1.2 for evaluation protocols. In this way, we see that we can actually encompass the setting of non-ergodic MDPs well, with some reparameterizztioo of the MDP
A.3 RELATING PRIOR WORKS TO ARL
In this section, we connect the settings in prior works to our proposed autonomous reinforcement learning formalism. First, consider the typical reinforcement learning approach: Algorithm A as- signs the rewards to transitions using r(s, a). learns a policy π and outputs n, = π and ~ ı (possibly adding noise the action). The algorithm exclusively optimizes the reward function throughout training.
Reconsider the door closing example: the agent needs practice closing the door repeatedly, which requires opening the door repeatedly. However, if we optimize π to maximize r over its lifetime, it will never be incentivized ergodic MDP and that the exploratory actions have support over all actions. the agent will open the door given enough time, and the agent will practice closing it again. However, in practice, this can be quite an inefficient strategy to rely on and thus, prior reset-free reinforcement learning algorithms consider other strategies for exploration. To understand current work, we introduce the notion of
open the door to practice closing it again. In theory, assuming an
Thhe action space An is not available in M, only in Mr. This discrepancy should be considered when parameterizing. ा..
14

---

Published as a conference paper at ICLR 2022
surrogate reward function Pr S A + R. At every time step t, A outputs ds Te and Te for evaluation, where Te optimizes ft over the transitions seen till time t- 1 reset-free reinforcement learning can be encapsulated within this framework as different choices for surrogate reward function it- Some pertinent examples: Assuming rp is some reward function designed that shifts the agent's state distribution towards initial state distribution p. alternating F. reinforcement
Prior works on
and F rp for a fixed number of environment steps recovers the learning algorithms proposed by Han al. (2015); Eysenbach al. (2017). Similarly, R3L (Zhu et 2020) can be understood as alternating between a perturbation controller optimizing a state novelty reward and the forward controller optimizing the task reward r. Recent work on using multi-task learning for reset-free reinforcement learning (Gupta et al., 2021) can be understood as choosing (5 04) Σk= k= (s a4 [s{+ Sk] such that SI.... SS is partition of the state space S and only reward function rı is active in the subset Sz. Assuming the goal-conditioned autonomous reinforcement learning framework, the recently proposed algorithm VaPRL (Sharma et al., 2021) can be understood as creating a curriculum of goals Ir such that at every step, the action az (- St; 9t). The curriculum simplifies the task for the agent, bootstrapping on the success of easier tasks to efficiently improve (T).
A.4 ENVIRONMENT DESCRIPTIONS AND REWARD FUNCTIONS
Paaletop-Organizzation The Tablecop-Organization task is a diagnostics object manipulation task proposed by Sharma al. (2021). The observation space is 12 dimensional vector consisting of the object position, gripper position, gripper state, and the current goal. The action space is 3-D action that consists of 2-D position delta and an automated gripper that can attach the mug if the gripper is close enough to the object. The reward function is a sparse indicator function, r(s, g) = ([|s جااو S 0.2). The agent is provided with 12 forward and 12 backward demonstrations, 3 for each of the 4 goal positions.
Sawyer-Door. The Sawyer Door environment consists of a Sawyer robot with 12 dimension observation space consisting of 3-D end effector position, 3-D door position, gripper state and desired goal. The action space is a 4-D action space that consisting of a 3-D end effector con- trol and normalized gripper torque. Let sa be dimensions of the observation corresponding to the door state, and g be the corresponding goal. The reward function is a sparse indicator function r(s,g) = [[]|sa 9 I 0.08). The agent is provided with 5 forward and 5 backward demos.
Sawyer-Peg. The Sawyer- Pegyer-Peggare environment shares observation and action space as the Sowyer-Door environment. Let sp be the state of the peg and be the corresponding goal. The reward function is a sparse indicator function r(s, g) = 1 ([I gl|2 S 0.05). The agent is provided with 10 forward and 10 backward demonstrations for this task.
Franka-Kitchen. The Franka- Kitchen environment consists of a 9-DoF Franka robot with an array of objects (microwave, two distinct burner, door) represented by a 14 dimensional vector. The reward function is composed of a dense reward that is a sum of the euclidean distance between the goal position of the arm and the current state plus shaped reward per object as described in Gupta et al. (2019). No demonstrations are provided for this task.
HHand-LightBulty The DHand is fingered robot (16-DoF) mounted on 6-DoF Sawyer Arm. The observation space of the DHand consists of 30 dimensional observation and corresponding goal state. The observation is composed of 16 dimensional hand position, 7 dimensional arm posi- tion, 3 dimension object position, 3 dimensional euler angle and 6 dimensional vector representing the dimensional wise distance to between objects in the environment. The action space is position delta over the combined 22 DoF of the robot. No demonstrations are provided for these tasks.
The Minitaur pen's observation space is the joint positions of its links, their corre- sponding velocity, current torque, quaternion of its basc position, and goal location in the pen. The action space is 8 dimensional PD target. Let sy be the 2-D position of the agent, and g be the corresponding goal. Let s; be the current torques the agent, and s, be their velocities The reward for the agent is a dense reward r(s, a) = -2. ||s9 - gll + 0.02 || st No demonstrations are provided for these tasks.
Minitaur-Pen.
"This can also be captured in a multi-task reinforcement learning framework, where Te, We are the same policy with different task variables input.
15

---

Published as a conference paper at ICLR 2022
A.5 ALGORITHMS
We use soft actor-critic (Haarnoja et al., 2018b) as the base algorithm for our experiments in this paper. When available, the demonstrations added to the replay buffer at the beginning of training. Further details on the parameters of the environments and algorithms are reported in Tables A.5. The implementations will be open-sourced and implementation details can be found there.
Hyperparameter
Value
Actor-critic architecture
fully connected(256, 256)
Nonlinearity
RcLU
architecture
fully connected(256, 256, 512)
RND Gamma
0.99
Optimizer
Adam
Learning rate
3e-4
Y
0.99
Target update
0.005
Target update period
Batch size
256
Classifier batch size
128
Initial collect steps
10a
Collect steps per iteration
Reward scale
Min log std
-20
Max log
Table 3: Shared algorithm parameters.
Environment
Training Horizon (Hfr)
Evaluation Horizon (Hg)
Replay Buffer Capacity
Tabletop-Ore
OraatttoOrganizatioon
200,000
20,000,000
Sawyer-Don Door
200,000
300
20,000,000
lawyer-Peg.
100,000
200
20,000,000
Franka-Kitchen
100,000
400
10,000,000
Dighd-Lighthulb
400,000
400
10,000,000
Minitaur-Pen 100,000
1000
10,000,000
Table 4: Environment specific parameters, including the training horizon (i.c. how frequently an intervention is provided), the evaluation horizon, and the replay buffer capacity.
A.6 EVALUATION CURVES
In this section, we plot deployed policy evaluation and continuing policy evaluation curves for dif- ferent algorithms and different environments:
16

---

Published as a conference paper at ICLR 2022
tabletop-orrgaizztiion
Deployed Policy Evaluation
Consizunng Dalicy
1-:00
16.40
500
sawyer-door
Deployed Policy Evaluntion
DomLinitag Patroy Evelustion
10:41
06-41
2.6
124 The 3an 7h4
sawyer-peg
Deployed Policy Evaluation
1.
Continding Policy Eveluetivo
kitchen
Deployed Pulicy Evalusticon
Continaing Poltty Ereltusioo
2.50
0.60
1.40
Mee
chand-lightbulb
Deployed Palicy Evaluation
Conimuing Palicy Evalustion 1no
6.60
1.45
1.66
3,68
minitaur-pen
Degloyed Polkey Evaluniuu
Continuing Policy Eveluntion
VaPRL
FBRL
naive
R3L
oracle
Figure 5: Deployed Policy Evaluation and Continuing Policy Evaluation per environment. Results and averaged over seeds. Shaded regions denote 95% confidence bounds.
17


Reviews:
[{"review_id": "jcjoAYOX9C", "reviewer": "Reviewer_AL8i", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Even if I always commend papers that propose benchmarks and evaluation framework, I found this work quite problematic. First of all, the authors use the expression autonomous reinforcement learning, which is per se already quite \"charged\". In fact, the concept of autonomy is rather well defined in our community. It is unclear why the behavior described in this work is considered \"autonomous\". Usually, the concept of autonomy is defined with respect to the concept of human intervention in setting goals, etc. The authors do not consider this type of autonomy. Instead, they differentiate systems according to learning based on episodes or not. It seems to me that the actual classification used by the authors is not really distant from the existing separation between continual and non continual learning.\n\nThe authors also discuss a distinction between systems that are trained using episodic training and then deployed as continual learning systems, but the reviewer believes that it is not really necessary to devise new benchmarks, etc. since existing ones can be considered sufficient to deal with these situations.\n\nThe definition of deployed policy evaluation (Definition 1) is rather unclear. How are you going to derive that? It seems to me that in practical terms, the calculation of such a value is not possible.\n\nDefinition 2 (\"continual policy evaluation\") appears to me as the classic definition of a RL problem in a non-episodic case.\n\nThe reviewer also does not see the need of introducing the concept of \"irreversible states\". At the end of the day, a situation where an agent is stuck in a certain situation might happen. The reviewer is not sure why this case should be considered separately. The reviewer also wonders if there is any specific requirement for this assumption.\n\nThe models used in the benchmarks are not novel (see Section 5), but they are essentially existing ones. The contributions of the authors with respect these existing environments is unclear.\n\nThe evaluation appears to be a sort of standard analysis of the performance of the environments under some specific conditions. The reviewer was not able to see a novel contribution in it.\n\nMinor points:\n\n- The authors say: \"Current episodic settings typically provide an environment reset every 100 to 1000 steps, corresponding to ε ∈ (1e-3, 1e-2) and an autonomous operation time of typically a few seconds to few minutes depending on the environment.\". I am not sure about this claim of the authors. The reset really depends on the actual experiments.\n\n- The authors say: \"In essence, the policy output from the algorithm A trained in the autonomous training setting should do well on the deployment setting as quickly as possible.\" I am not sure about this - I think it really depends on the environment, if it is stationary, etc. Also, the authors are assuming here that learning happens at deployment time as well and this is not always the case.\n", "summary_of_the_paper": "The paper describes a framework for benchmarking episodic and non-episodic algorithms. The authors provide both a mathematical formalisation and a description of an experimental framework composed of existing environments. It is quite difficult to see clear contributions in this paper since the theoretical framework does not really introduce novel concepts and the experimental framework is essentially a collection of existing environments. ", "summary_of_the_review": "The contributions are rather limited in my opinion since the theoretical framework essentially essentially proposes re-definitions of existing concepts. The experimental framework is composed of existing benchmarks so it is difficult to see a contributions there.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "VSQhlduKff4", "reviewer": "Reviewer_omLm", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The strength of this paper is that they clearly argue that the studies on continual learning are important and give a clear notion about ARL.\nThe weakness of this paper is the lack of theoretical novelty and new proposal except for the argument around ARL and its benchmark tasks. \n", "summary_of_the_paper": "This paper describes a new framework called Autonomous Reinforcement Learning (ARL) to facilitate studies on continual real-world embodied learning, such as that performed by humans and animals. Most reinforcement learning (RL) studies use episodic benchmark tasks. The paper argues that conducting studies on non-episodic tasks is crucial. After defining ARL, the authors introduce simulated benchmarks. ", "summary_of_the_review": "I totally agree with the authors' argument about the importance of continual reinforcement learning. I also agree that the RL culture that makes agents' life \"episodic\" tends to prevent us from tackling the problem. I also think the \"non-episodic\" feature is one of the important ones, although the real-world embodied learning, such as that performed by humans and animals, has many other essential features, e.g., the reward function is not unique and environmental dynamics are changing over time even in an episode, \nTherefore, I like the argument of the authors, especially in the Introduction. \n\nHowever, it's questionable if the paper has a sufficient paper as an ICLR conference paper though it has great importance in its aspect of \"position paper.\" \n\n1) Theoretical contribution around ARL. \nTo my understanding, the theoretical notion of ARL is not new. The general and classical framework of RL does not necessarily have the assumption of episodic segmentation. Therefore, the novelty of the authors' proposal about ARL is limited in a theoretical viewpoint though it is quite meaningful in the \"cultural\" viewpoint.\n\n2) Benchmarks.  \nThe authors described some benchmarks. However, to my understanding, even though they showed the pre-existing RL algorithms underperformed in the ARL settings, they are not providing a suitable task for developing ARL algorithms. This is because they only showed the pre-existing methods tend to underperform in the ARL settings. \nSome more constructive proposal is expected. \n\nThe paper is very good from the viewpoint of position/short journal paper. However, I am not sure if the paper is suitable for the ICLR conference paper. \n  \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "R8nWcBwNn94", "reviewer": "Reviewer_C7Cj", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n-------------\n\n- The paper provides a nice formalization of an important problem (that of Autonomous Reinforcement Learning)\n- I like the distinction into \"Deployment Setting\" and the \"Continuing Setting\". Usually people when hearing of ARL, they have in mind the \"Continuing Setting\", but the \"Deployment Setting\" is also very useful. More importantly, they are different problems. In any case, the distinction can make things easier to talk about and compare methods/ideas.\n- Nice benchmark environments capturing a wide variety of tasks/scenarios\n- Thorough analysis providing useful insights (mostly validating our intuition, but this is important!)\n\nWeaknesses\n------------------\n\n- The ARL problem is strongly related to the autonomous skill discovery problem (see for example {1,2,3}). For instance, without proper exploration and skill distillation (either automatic or with some heuristic), ARL is not really a solvable problem without many resets. I would definitely like to see a nice discussion about this and other similarities. This would make the paper complete.\n\n\nReferences\n-----------------\n{1}: Campos, V., Trott, A., Xiong, C., Socher, R., Giró-i-Nieto, X. and Torres, J., 2020, November. Explore, discover and learn: Unsupervised discovery of state-covering skills. In International Conference on Machine Learning (pp. 1317-1327). PMLR.\n\n{2}: Gregor, K., Rezende, D.J. and Wierstra, D., 2016. Variational intrinsic control. arXiv preprint arXiv:1611.07507.\n\n{3}: Pong, V.H., Dalal, M., Lin, S., Nair, A., Bahl, S. and Levine, S., 2020, November. Skew-fit: State-covering self-supervised reinforcement learning. In International Conference on Machine Learning. PMLR.\n\n", "summary_of_the_paper": "The manuscript presents a concrete formalization of the \"Autonomous Reinforcement Learning\" (ARL) problem: this problem refers to a scenario where the agents has no (or very few) resets during learning of the task(s). The paper also unifies previous works into two evaluation scenarios: (a) Deployed Policy Evaluation, and (b) Continuing Policy Evaluation. The manuscript, also, provides a benchmark: \"Environments for Autonomous Reinforcement Learning (EARL)\". This benchmark contains diverse environments from simple manipulation tasks to locomotion and complex manipulation. Using this benchmark the authors provide an analysis of the crucial factors that affect the performance of current state-of-the-art algorithms for the ARL problem.\n\nThe main contributions of the paper are:\n\n- Concrete formalization of the \"Autonomous Reinforcement Learning\" (ARL) problem\n- A novel benchmark for ARL with diverse tasks\n- Novel evaluation settings (Deployment Setting and Continuing Setting) for ARL\n- Insights on the important factors for effective ARL\n", "summary_of_the_review": "The manuscript was a nice and interesting read. The formalism is solid, the benchmark is going to be useful to many researchers and the insights are interesting and although somewhat expected from intuition, it is good to see them result from experiments and written down nicely.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "A1pdAB5w-4i", "reviewer": "Reviewer_sj9u", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\nExcellent example of carefully chosen augmentations of a standard problem formulation in order to factor in hidden costs with substantial practical implications in several experimental settings of interest, such as real-world robotics experiments.\n\nWell designed toy domains which emulate some such difficulties for quick experimentation in a comparable way.\n\n\nCons:\n\nIt is not clear whether the proposed methodology really deals with issues of environment resets in the fullness of time. Will following this methodology result in much fewer resets and superior learning if tasked to achieve the same level of performance as expensive and impractical full episodic reset methodology, at least for tasks of interest with real-world robots? While algorithms developed for ARL may indeed require fewer interventions in a fixed amount of experiment time, this does not mean that convergence to an optimal policy will not take considerably more experimentation time than episodic reset approaches. If total experimentation time is also a factor, then the trade-off of autonomous operation in the real-world vs. cost of human resets seems to be still an open problem in need of task specific trade-offs.\n\nPerhaps there is no choice but to resort to ARL, but it is imho not clear that ARL doesn’t introduce learning issues of its own, since learning from biased data streams is known to be particularly challenging and empirically leads to suboptimal learning, as per the continual learning literature. This view is not necessarily invalidated by experimental evidence presented in the paper, at least as far as I can tell.\n\nMore discussion and standardization may be needed in order to reduce variance of ARL experiments when reporting results. Without the aid of environment resets, ARL data sampling is even more contingent on learned policies and learning dynamics of particular algorithms and function approximators. Hence, standard convergence assumptions and hopes may be even more ephemeral, especially with non-linear function approximation. Also, sensitivity to implementation details may be increased, e.g. hyperparameters, software libraries and their default settings, even model initializations, etc.\n", "summary_of_the_paper": "This paper highlights the important concept of autonomous RL, inspired by real-world scaling constraints of robot learning experiments. Hiding the cost of environmental resets is unrealistic and hinders progress in embodied RL. This incentivises the search for algorithms which score better trade-offs between expensive environment resets and average return of learned policies, using the two definitions given in the paper. Clear conceptual problem formulations are given which reproduce challenges observed during real-world RL experiments with robots, but in tractable settings using simulation.", "summary_of_the_review": "This paper gives two convincing definitions of autonomous reinforcement learning and introduces clear evaluation methodologies accessible for the community. Relevant baseline results are provided, such that the proposed benchmarks are good starting points for future research. While some conceptual challenges linger, this could be resolved by followup works.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper formalizes the setting where an autonomous RL agent operates with zero or very few resets, and provides a novel benchmark for this setting with diverse environments ranging from simple manipulation to complex manipulation/locomotion. The paper then uses this benchmark to analyze current methods and provide insight into those crucial factors that affect performance in this setting. The insights into current methods especially are appreciated. As one reviewer stated, "This paper isolates one problematic assumption in the way of [progress in RL], the environment reset problem, and provides the groundwork for [such] progress, i.e. baselines, clear metrics, etc. I believe the community is much better off with this paper published, since prior works don't seem to have used compatible methodologies."
Decision: Accept (Poster)