The Ingredients of Real World Robotic Reinforcement Learning
Authors: Henry Zhu
Justin Yu
Abhishek Gupta
Dhruv Shah
Kristian Hartikainen
Avi Singh
Vikash Kumar
Sergey Levine
Venue:  (2020)
Keywords: Reinforcement Learning
Robotics

Abstract: The success of reinforcement learning in the real world has been limited to instrumented laboratory scenarios, often requiring arduous human supervision to enable continuous learning. In this work, we discuss the required elements of a robotic system that can continually and autonomously improve with data collected in the real world, and propose a particular instantiation of such a system. Subsequently, we investigate a number of challenges of learning without instrumentation -- including the lack of episodic resets, state estimation, and hand-engineered rewards -- and propose simple, scalable solutions to these challenges. We demonstrate the efficacy of our proposed system on dexterous robotic manipulation tasks in simulation and the real world, and also provide an insightful analysis and ablation study of the challenges associated with this learning paradigm.

Published as a conference paper at ICLR 2020
THE INGREDIENTS OF WORLD ROBOTIC REINFOR LEARNING
Henry Zhu" Justin Yu" Abhishek Gupta"), Dhruy Shah', Kristian Hartikainen', Avi Singh', Vikash Kumar Sergey Levine" University of California, Berkeley
University of Oxford University of Washington
ABSTRACT
The success of reinforcement learning for real world robotics has been, in many cases limited to instrumented laboratory scenarios, often requiring arduous hu- man effort and oversight to enable continuous learning. In this work, we discuss the elements that are needed for robotic learning system that can continually and autonomously improve with data collected in the real world. We propose a particular instantiation of such system, using dexterous manipulation as our case study, Subsequently, we investigate a number of challenges that come up when learning without instrumentation. In such settings, learning must be feasi- ble without manually designed resets, using only on- on-board perception, and with- out hand-engineered reward functions. We propose simple and scalable solutions to these challenges, and then demonstrate the efficacy of our proposed system on a set of dexterous robotic manipulation tasks, providing an in-depth analysis the challenges associated with this learning paradigm. We demonstrate that our complete system can learn without any human intervention, acquiring a variety vision- based skills with real-world three- fingered hand. Results and videos be found at
google. com/ wwww/reawwwrldd-...
INTRODUCTION
Reinforcement learning (RL) can in principle enable autonomous systems, such as robots, to acquire a large repertoire of skills automatically. Perhaps even more importantly, reinforcement learning enable such systems continuously improve the proficiency of their skills from experience. However, realizing this in reality has proven challenging: even with reinforcement learning methods that can acquire complex behaviors from nigh-dimensional low-level observations, such as images, the assumptions of the reinforcement learning problem setting do not fit cleanly into the constraints of the real world. For this reason, most successful robotic learning experiments have employed various kinds of environmental instrumentation in order to define reward functions, reset between trials, and obtain ground truth state (Levine et al. 20166] Haarnoja et al. 20188| Kumar et al. 2016 Andrychowicz et al. 2018, Zhu et al.) 2019) Chebotar et al. 20166; Nagabandi et al., 2019)] Gupta ct al. (0166). In order to practically and scalably deploy autonomous learning systems that improve continuously through real-worls operation, must lift these limitations and design algorithms that can learn under the constraints of real-world environments, as illustrated in Figure
We propose that overcoming these challenges in scalable way requires designing robotic systems that possess three capabilities: they are able to (1) learn from their own raw sensory inputs, (2) assign rewards to their own trials without hand-designed perception systems or instrumentation, and (3) learn continuously in non-episodic settings without requiring human intervention to manually reset the environment. A system with these capabilities can autonomously collect large amounts of real world data- typically crucial for effective generalization without significant instrumentation in each training environment, an example of which is shown in Figure [ If successful, this would lift a major constraint that stands between current reinforcement learning algorithms and the ability to learn scalable, generalizable, and robust real-world behaviors. Such a system would also bring us significantly closer to the goal of embodied learning-based systems that improve continuously through their own real-world experience.
'These authors contributed equally. Correspondence to henryzhu@berkelley.com edu.

---

Published as a conference paper at ICLR 2020
1
2
3
User Provides Goal Images
Unattended Training With R3L
Successful Policy From Any State
Illustration of proposed instrumentation-fiee system requiring minimal human engineering. Human intervention only required in the goal collection phase (1). The robot to train unattended (2) during the learning phase and can be evaluated from arbitrary initial states at the end of training (3). We show sample goal and intermediate images from the training process of a real hardware system
Figure 1:
Having laid out these requirements, propose a practical instantiation of such learning system. While prior works have studied many of these issues in isolation, combining them into a complete real-world learning system presents number of challenges, vide an empirical analysis of these issues, both in simulation and on a real-world robotic system, and propose number simple but effective solutions that together produce complete robotic learning system. This system can autonomously learn from raw sensory inputs, Iearn reward func- tions from easily available supervision, and learn without manually designed show that this system can learn dexterous robotic manipulation tasks in the real world, substantially outperforming ablations and prior work.
discuss in Section We pro-
mechanisms. We
2 THE STRUCTURE OF A
RL SYSTEM
The standard reinforcement learning paradigm assumes that the controlled system is represented as a Markov decision process with state space S, action space A, unknown transition dynamics T, unknown reward function R, and (typically) episodic initial state distribution p. The goal is to learn policy that maximizes the expected sum rewards via interactions with the environment.
Although this formalism is simple and concise, it does not capture all of the complexities of real- world robotic learning problems. If a robotic system is to learn continuously and autonomously in the real world, must ensure that it can learn under the actual conditions that imposed by the real world. To move from the idcalized MDP formulation to the real world, we require a system that has the following properties. First, all of the information necessary for learning must be obtained from the robot's own sensors. This includes information about the state and necessitates that the policy must learned from high-dimensional and low-level sensory observations, such as camera images. Second, the robot must also obtain the reward signal itself from its own sensor readings. This is exceptionally difficult for all but the simplest tasks (e.g., reward functions that depend on interactions with specific objects require perceiving those objects explicitly). Third, we must be able learn without access episodic resets. setup with explicit resets quickly becomes impractical in open-world settings, due to the requirement for significant human engineering of the environment, or direct human intervention during learning- While this list may not exhaustively enumerate all the components necessary for effective real-world learning system, posit that the properties listed here are fundamental to building such systems.
While some of the components discussed above can be tackled in isolation by current algorithms, there are considerable challenges inherent to assembling all these components into a complete learn- ing system for real world robotic learning. In the rest of this section, outline the challenges associated with each component, then discuss the challenges associated with combining these com- ponents in Section 3 and then proceed address these challenges in Section
2

---

Published as a conference paper at ICLR 2020
Prior Approaches Real-World RL
R3L System
Dupti et 2036
The, as al, 2013
Nagaband et el, 2015
Learned Behavior
Camera
Training
Setup
Environment is Instrumented for Resetting and State Reward Estimation
No Instrumentation
We draw comparison between current real world learning systems which rely instrumentation versus system that learns in environment more representative While all three prior works utilize instrumentation for resets, state estimation, and reward, the motion capture system of Gupta et al. (2016), sensor attached to the door in Zhu et al. (2019), and auxiliary rohot which picks up fallen balls in Nagabandi et al. (2019) are good examples of engineered state estimation, reward estimation, and resct mechanisms respectively.
Figure 2:
the real world. free instrumentation.
2.1 LEARNING FROM RAW SENSORY INPUT
To enable learning without complex state estimation systems or environment instrumentation, we require robotic systems to be able to learn from their these sensory observations are raw camera images from a camera mounted on the robot, as well as proprioceptive sensory inputs such the joint angles. These observations do not directly provide the poses of the objects in the scene, which is the typical assumption in simulated robotic environments any such information must be extracted by the learning system.
sensory observations. Typically,
While in principle many RL frameworks can support learning from raw sensory inputs (Levine et al, 2016 Mnih al., 2015) Schulman et al. 2015 Lillicrap et al. 2015). it is important consider the practicalities of this approach. For instance, we can instantiate vision-bases RL with policy gradient algorithms such as TRPO (Schulman al. 2015) and PPO (Schulman et al. (017). but these have high sample complexities which make them unsuited for real world robotic learning (Haarnoja et al.) 2018ap. In our work, we consider adopting the general framework of off-policy actor-critic rein- forcement learning, using a version of the soft actor critic (SAC) algorithm described by (Haarnoja al.) 2018b). This algorithm effectively uses off- off-policy data and has been shown learn some tasks directly from visual inputs. However, while SAC can learn directly from images, we find in our experiments that, as the task complexity increases, the efficiency of direct end-to-end learning (particularly without resets and with learned rewards) still degrades substantially. However, will discuss Section 3 augmenting end-to-end learning with unsupervised representation learning substantially alleviates such challenges.
2.2 REWARD FUNCTIONS WITHOUT REWARD ENGINEERING
Vision-based RL algorithms, such as SAC, rely on reward function being provided the system, which is typically manually programmed by a user. While this can be simple to do in simulation by using ground truth state information, it is significantly harder to implement in uninstrumented real world environments. In the real world, the robot must obtain the reward signal itself from its own sensor readings. A few options for tackling this challenge have been discussed in prior work: design complete computer vision systems to detect objects and extract the reward signals (Devin et al.) 2018 Nagabandi et al. 2019), engineer reward functions that use various task-specific heuristics to obtain rewards from pixels (Schenck & Fox 2017; Kalashnikov al. 2018), instrument every environment (Chebotar et al. | 2017). Many of these solutions are manual and tedious, and a more general approach is needed scale real-world robotic learning gracefully.
3

---

Published as a conference paper at ICLR 2020
2.3 LEARNING WITHOUT RESETS
While the components described Section 2.1 and 2.2 are essential to building continuously learn- ing RL systems in the real world, they have often been implemented with the assumption of episodic learning Fu et al. (2018); Haarnoja et al. (2018b). However, natural open-world settings do not pro- vide any such reset mechanism, and in order enable scalable and autonomous real-world learning we need systems that do not require an episodic formulation of the learning problem.
To devise a system that requires minimal human engineering for providing rewards, we must use algorithms that are able to assign themselves rewards, using learned models that operate on the same raw sensory inputs as the policy. One candidate for user to specify intended behavior beforehand through examples of desired outcomes (i.e., images). The algorithm can then assign itself rewards based on measure of how well it is accomplishing the specified goals, with no additional human supervision. This approach can scale well in principle, since it requires minimal human engineering, and goal images are easy to provide.
In principle, algorithms such SAC do not actually require episodic learning; however, in practice, most instantiations use explicit resets, even in simulation, and removing resets has resulted in failure to solve challenging tasks. In our experiments in Section 3 as well, we see that actor-critic methods applied naïvely to the reset free setting do not learn the intended behaviors. Introducing visual observations and classifier based rewards exacerbates these challenges.
We propose that these three components vision-based RL with actor-critic algorithms, vision- based goal classifier for rewards, and reset-free learning to build a real world robotic learning system. However, when we actually combine the individual components in Sections 3 and 6 we find that learning effective policies is quite challenging. We provide insight into these challenges in Section 3 Based on these insights, we propose simple but important changes in Section to build system, R3L, that can learn effectively and autonomously in the real world without human intervention.
the fundamental pieces that need
THE CHALLENGES OF REAL WORLD RL
The system design outlined in Section 2 principle gives us a com- plete system perform real world reinforcement learning with- out instrumentation. However, when utilized for robotic learning problems, we find this basic design to be largely ineffective. To study this, we present results for simulated robotic manipulation task that requires repositioning a free-fl floating objects with three- fingered robotic hand, shown in Fig 3 We use this task for our investigative analysis and show that the same insights extend to sev- eral other tasks, including real world tasks, in Section 6 The goal in this task to reposition the object to a target pose from any ini- tial pose in the arena. When the system is instantiated with vision- based soft actor-critic, rewards from goal images using VICE, and run without episodic resets, we that the algorithm fails make progress (Fig 4. Although it might appear that this setup fits within the assumptions of all of the components that are used, the complete system is ineffective. Which particular components of this problem make it so difficult?
Our object reposition- ing task. The goal is move the object from any starting config- uration to a particular goal posi- tion and orientation.
Figure 3:
To investigate this issue, we perform experiments investigating the combination of the three main ingredients: varying observation type (visual vs. low-dimensional state), reward structure (VICE vs. hand-defined rewards that utilize ground-truth object state), and the ability to reset (episodic resets vs. reset-free, non-episodic learn- ing). We start by considering the training time reward under each combination of factors as shown in Fig[4 which reveals several trends. First, the results in Fig show that learning with resets achieves high training time reward from both vision and state, while reset-free only achieves high training time reward with low- dimensional state. Second, we find that the policy is able to pass the threshold for training time reward in non-episodic setting when learning from dow-dimensional state, but it
4

---

Published as a conference paper at ICLR 2020
VICE
With Resets
Without Resets
True Reward
Testing
700k
1M
0.6
Training
State
200k
500k
Pose Distance [rad]
X
X
Vision
800k
X
0.2
Figure 4: We report the approximate number of samples needed for policy learned with a prior off- policy RI. algorithm (SAC) to achieve average train- ing performance of less than 0.15 in pose distance (defined in Appendix CC..3) across seeds the positioning task. compare training performance after varying three axes: ground truth rewards vs. learned rewards, with vs. without episodic resets, low-level state images inputs, We observe learning without resets is harder than with resets and much harder when combined with visual inputs.
0.0
100
200 300
400
500
600
700
Samples [x1000 timesteps)
Figure 5: We observe that when training reset free reach single goal, while the pose distance at training time is quite low, the pose emorr obtained at test-time with the Iearned policy are very high. This indicates that while the object is getting close to the goal training time, the policies being learned are still not effective.
not able do the same using image observations. This suggests that combining the reset-free learning problem with visual observations makes it significantly more challenging.
However, the table in Fig paints an incomplete picture. These numbers are related to the per- formance of the policies at training time, not how effective the learned policies are when being evaluated. When we consider the test-time performance (Fig(5) of the policies that are learned under reset free conditions, we obtain a different set of conclusions. While learning from low-dimensional state the reset free setting achieves high reward at training time, the test-time performance of the corresponding learned policies is very poor. This can likely be attributed to the fact that when the agent spends all its time stuck at the goal, and sees very little diversity of data in other parts of the state space, which significantly reduces the efficacy of the actual policies being learned. In sense, the reset encodes some prior information about the task: it tells the policy about what types of states it might be required succeed from at test time. Without this knowledge, performance is substan- tially worse. This makes it very challenging to learn policies with completely reset-free schemes, which has prompted prior work to consider schemes such as learning reset controllers (Eysenbach) et al., 2018). As we discuss in the following section and in our experiments, these schemes are often insufficient for learning effective policies in the real world without any resets.
4 A REAL-WORLD ROBOTIC REINFORCEMENT LEARNING SYSTEM
To address the challenges identified in Section 3 we present two improvements, which we found to be essential for uninstrumented real-world training: randomized perturbation controllers and un- supervised representation learning. Incorporating these components into the system in Section results in a system that can leam successfully in uninstrumented environments, as we will show in Section 6 and attains good performance both at training time and at test time.
4.1 RANDOMIZED PERTURBATION CONTROLLER
Prior works in addressing the reset free setting problem have considered converting the problem into a more standard episodic problem, by learning a "reset controller," which is trained to reset the system particular initial state (Bysenbach al. 2018 Han et al., 2015). This scheme has been thought make the learning problem casier by reducing the variance of the resulting initial state distribution. However, as we will show in our experiments in Section 6 this still results in policies whose success depends heavily methods all reset to a single initial state (Eysenbach et al. 2018 Han et al. 2015).
narrow range of initial states. Indeed, prior reset controller
We take different approach to learning in reset-free setting, Rather than attributing the problem to the variance of the initial state distribution, we hypothesize that a major problem with reset-free learning is that the support of the distribution of states visited by the policy narrow, which
5

---

Published as a conference paper at ICLR 2020
makes the learning problem challenging and doesn't 't allow the agent to learn how to perform the desired task from any state it might find itself in. In this view, the goal should not be to reduce the variance of the initial state distribution, but instead to increase it.
To this end, we utilize what we call random perturbation controllers: controllers that introduce per- turbations intermittently into the system through policy that is trained to explore the environment. The standard actor x <{as)) is executed for H time-steps, following which we executed the perturba- tion controller Ty (als) for H/ steps, and repeat. The policy is trained with the VICE-based rewards for reaching the desired goals, while the perturbation controller Tp is trained only with an intrinsic motivation objective that encourages visiting under-explored states. In our implementation, we use the random network distillation (RND) objective for training the perturbation controller (Burda et al.) 2018), but any effective exploration method can be used for the same. This procedure is described in detail in Appendix [A] and is evaluated on the tasks we consider in 6 The perturbation controller ensures that the support of the training distribution grows and as a result the policies can learn the desired behavior much more effectively, as shown in Fig 7
4.2 GOAL CLASSIFIER
To design a system that requires minimal human engineering for providing reward, we use a data- driven reward specification framework called variational inverse control with events (VICE) intro- duced by Fu et al. (2018). VICE learns rewards in a task-agnostic way: we provide the algorithm with success examples in the form of images where the task is accomplished, and learn discrimi- nator that is capable of distinguishing successes from failures. This discriminator can then be used to provide learning signal to nudge the reinforcement learning agent towards success. This algo- rithm has been previously considered in the context of learning tasks from raw sensory observations in the real world by (Singh et al.] (019) but we show that it presents unique challenges when used in conjunction with learning without episodic resets. Details and specifics of the algorithms being considered are described in Appendix aadd also discussed by SSnggh et al. (2019).
4.3 UNSUPERVISED REPRESENTATION LEARNING
The perturbation controller discussed above allows us to learn policies that can succeed at the task from variety of starting states. However, learning from visual observations still present a chal- lenge. Our experiments in Fig 4 show that learning without resets from low-dimensional states is comparatively casier. We therefore aim to convert the vision-based learning problem into one that more closely resembles state-based learning, by training variational autoencoder (VAE, Kingma & Welling (2013)) and sharing the latent-variable representation across the actor and critic networks (refer to Appendix B for more details). Note that we use a VAE as an instantiation of representa- tion learning techniques that works well in the domains we considered, but other more sophisticated density models proposed in prior work may also be substituted in place of the VAE (Lee al. 2019
Hjelm et al. 2019; Anand et al. 2019).
While several works have also sought to incorporate unsupervised learning into reinforcement learn- ing to make learning from images easier (Nair et al.) 2018 Lee et al., 2019), we note that this be- comes especially critical in the vision-based, reset-free setting, as motivated by the experiments in Section 3 which indicate that it is precisely this combination of factors vision and no resets that presents the most difficult learning problem. Therefore, although the particular solution we use in our system has been studied in prior work, it is brought to bare to address challenge that arises in real-world learning that we believe has not been explored in prior studies.
These two improvements - the perturbation controller and unsupervised learning combined with the general system described above, give us a complete practical system for real world reinforcement learning. The overall method uses soft-actor critic for learning with visual observations and classifier based rewards with VICE, introduces auxiliary reconstruction objectives or pretrains encoders for unsupervised representation learning, and uses perturbation controller during training to ensure that the support of visited states grows sufficiently. We term this full system for real-world robotic reinforcement learning R3I.. Further implementation details can be found in Appendix A

---

Published as a conference paper at ICLR 2020
5 RELATED WORK
The primary contribution of this work is to propose a paradigm for continual instrumentation -ree real world robotic learning and a practical instantiation of such system. A number of prior works have studied reinforcement learning in the real world for acquiring robotic skills (Levine et al.) 2016
Kumar et al. 2016 Gu et al. 2017 Kalashnikov et al. 2018: Haarnoja et al. 201 8b
Finn & Levine,
2016 et al. 2019) Nagabandi et al. 2019). Much of the focus in prior work has been on improv- ing the efficiency and generalization of RL algorithms to make real- world training feasible, or else
on utilizing simulation and transferring policies into the real world (Tzeng et al. 2015 Tobin et al.
2017 Peng al., 2017, Clavera al. 2018 Kang al., 2019). The dimulation-basee methods typically require considerable effort in terms of both simulator design and overcoming the distribu- tional shift between simulated and real-experience, while prior real-world training methods typically require additional instrumentation for either reward function evaluation (Levine & Koltun 2013) or resetting between trials (Gu et al., 2017; Chebotar et al., 2017; Zhu et al.] (019), both. In con- trast, our work is primarily focused on lifting these requirements, rather than devising more efficient RL methods. We show that removing the need for instrumentation (i.e., for reward evaluation and resets) introduces additional challenges, which in turn require careful set of design choices. Our complete R3I. method is able to learn completely autonomously, without manual resets or reward design.
A key component of our system involves learning from raw visual inputs. This has proven to be difficult for policy gradient style algorithms (Pinto et al.) 2017a) due to challenging representation learning problems. This has been made easier in simulated domains by using modified objectives, such as auxiliary losses (Jaderberg al. 2016),, by using more efficient algorithms (Haarnoja jet al. 2018a). We show that reinforcement learning on raw visual input, while possible in standard RL settings, becomes significantly more challenging when considered in conjunction with non- episodic, reset-free scenarios.
Reward function design is crucial for any RL system, and is non-trivial to provide in the real world. Prior works have considered instrumenting the environment with additional sensors to evaluate re- wards (Gu et al., 2017; Chebotar et al., 2017 Zhu et al., 2019), which is highly manual process, using demonstrations, which require manual effort to collect (eecerik et al. 2017 Ng & Russell 2000 Liu et al. 20118) or using interactive supervision from the user (Christiano et al. [017]] In this work, we leverage the algorithm introduced by Fu et al. (2018) to assign rewards based on the like- lihood of a goal classifier. While prior work also applied this method to robotic tasks (Singh et al.) 2019), this was done in setting where manual resets were provided by hand, while we demonstrate that we can use learned rewards in a fully uninstrumented, reset-firee setup.
Learning without resets has been considered in prior works (Eysenbach et al., 2018 Han et al.) 2013), although different contexts safe learning and learning compound controllers, respectively. Eysenbach et al. (2018) provide an algorithm to learn reset controller with the goal of ensuring safe operation, but makes several assumptions that make difficult to use in the real world: assumes access
ground truth reward function, it assumes access an oracle function that can detect if an attempted reset by the reset policy was successful or not, and it assumes the ability to perform manual resets if the reset policy fails a certain number of times. In contrast, we propose an algorithm that allows for fully automated reinforcement learning in the real world. We compare to an ablation of our method that uses reset controller similar [Eysebbach al. (2018), and show that our method performs substantially better. Our perturbation controller also resembles the adversarial RL setup Pinto et al.) (2017b); Sukhbaatar al. (0188.. However, while these prior methods explicitly aim to train policies that are robust to perturbations Pinto et al. (201 or explore effectively Sukhbaatar et al. 20188) we are concerned with learning without access to resets.
While this line of work has connections to developmental robotics (Lungarella et al.) 2003 Asada ct al. 2009) and its subfields, such as continual (Lesort et al. (201)) and lifelong (Thrun & Mitchell, 1995) learning, the goal of our work is to handle the practicalities of enabling reinforcement learn- ing systems to learn in the real world without instrumentation or interruption, even for a single task setting. Though our work does not directly study continual lifelong learning, nor all facets of devel- opmental robotics, it relates to continual learning (Lesort et al.) 2019), intrinsic motivation (Schmid-) huber (006) and sensory-mote development involving proprioceptive manipulation (Stoica 2001)
7

---

Published as a conference paper at ICLR 2020
6 EXPERIMENTS
In our experimental evaluation, study how well the R3L system, described in Sections and can learn under realistic settings visual observations, no hand-specified rewards, and no resets. We consider the following hypotheses:
Can we use R3L to learn complex robotic manipulation tasks without instrumentation? Does this system learn skills in both simulation and the real world?
2. Do the solutions proposed in Section 4
actually enable R3L to perform tasks without in- strumentation that would not have been otherwise possible?
6.1 EXPERIMENTAL SETUP
We consider the task of dexterous manipulation with three fingered robotic hand, the D'Claw (Zhu et al., [219 Ahn et al.] 2019), on number of simulated and real world tasks. These tasks involve complex coordination of three fingers with DoF each in order manipulate objects. Prior works that used this robot utilized explicit resets and low-dimensional true state observations, while we consider settings with visual observations, no hand-specified rewards, and no resets.
goal configurations of the simulated and real world tasks being considered. From left to right we depict valve rotation, bead manipulation and free object repositioning in simulation, as well as valve rotation and bead manipulation manipulation in the real world.
Figure 6: Visualizations
The tasks in our experiments are shown Fig manipulating beads on an abacus row, valve rotation, and free object repositioning. These tasks represent wide class of problems that robots might encounter in the real world. For each task, we consider the problem reaching the depicted goal configuration: moving the abacus beads to a particular position, rotating the valve to a particular angle, and repositioning the free object to a particular goal position. For each task, policies are evaluated from a wide selection of initial configurations. Additional details about the tasks and evaluation procedures are provided in Appendix C Videos and additional details can be found at https://sites.. wwwgle.oooooooooooooooooo wwwpoooll...//vviiooooee wtt.cccw.....t.. wwt.e//wwwwwwww.....e.....
6.2 LEARNING IN SIMULATION WITHOUT INSTRUMENTATION
We compare our entire proposed system implementation (Section 4 with a number of baselines and ablations. Importantly, all methods must operate under the same assumptions: none of the algo- rithms have access to system instrumentation for state estimation, reward specification, or episodic resets. Firstly, we compare the performance of R3L system which uses SAC for vision-based RL from raw pixels, VICE for providing rewards and running reset-free (denoted "VICE"). This cor- responds to the vanilla version of R3L (Section 2 with none the proposed insights and changes. We then compare with prior reset-free RI. algorithms (Eysenbach et al., 2018) that explicitly learn reset controller to alternate goals the state space ("Reset Controller + VAE"). Lastly, we compare algorithm performance with two ablations: running R3L without the perturbation controller ("VICE VAE") and without the unsupervised learning ("R3L w/o VAE"). This highlights the significance of each of the components of R3L.
From the experimental results in Fig 7 it is clear that R3L able to reach the best performance across tasks, while none of the other methods are able to solve all of the tasks. Different prior methods and ablations fail for different reasons: (1) methods without the reconstruction objective struggle at parsing the high-dimensional input and unable to solve the harder task; (2) methods
8

---

Published as a conference paper at ICLR 2020
Beed Mamipulation
Velve Rotation
Repositionnng
Irsa (boo
Oilaana Coal Tradd uil
8.112
Diplanc to 0001
0.005
4015
6:8h
6.82
301
200 (1000 dimestops)
3-6
Epochs [1.00 Eimesteps
Epochs [1000 timostaps)
R3L
Controiler VAE Eysenbach al. 2018)
VICE
VICE
Figure 7: Quantitative evaluation of performance in simulation for bead manipulation, valve rotation and free object repositioning (left to right) run with 10 random sceds. The error bars show 95% bootstrap confidence intervals for average performance. While other variants are sufficient to get good evaluation performance on easier tasks, harder tasks like free object repositioning require random perturbations and unsupervised repre- sentation learning to learn skills reset-free. See Appendix C.I for details of evaluation procedures.
without the perturbation controller are ineffective at learning how to reach the goal from novel initialization positions for the more challenging object repositioning tasks, discussed in Section
We note that an explicit reset controller, which can be viewed as a softer version our perturbation controller with dial-directedness, learns solve the easier tasks due the reset controller encour- aging exploration of the state space. In our experiments for free object repositioning, performance was reported across choices of reset states. The high variance in evaluation performance indicates that the performance of such controller (or a goal conditioned variant of it) is highly sensitive to the choice of reset states, poor choice of reset states, such as two that are very close together, may yield poor exploration leading to performance similar to the single goal VICE baseline. Further- more, the choice of reset states highly task dependent and is often not clear what choice of goals will yield the best performance. On the contrary, our method does not require such task-specific knowledge and uses random perturbations reset while training without any explicit reset states: this allows for robust, instrumentation-ffe controller while also ensuring fast convergence.
6.3 LEARNING IN THE REAL WORLD WITHOUT INSTRUMENTATION
Since the aim of R3L is to enable uninstrumented training in the real world, we next evaluate method on a real-world robotic system, providing evidence that our insights generalize to the real world without any instrumentation. After providing the initial outcome examples for learning the reward function with VICE, we leave the robot unattended, and the algorithm learns the desired behavior through interaction. The experiments are performed the D'Claw robotic hand with an RGB camera as the only sensory input. Intermediate policies are saved at regular intervals, and evaluations of all policies performed after training has completed. For valve rotation, we declare an evaluation rollout success if the final orientation is within within 15e of the goal. For bead manipulation, we declare success if all the beads are within 2cm of the goal state. Fig R compares the performance of our method without supervised learning ("R3L w/o VAE") in the real world against a baseline that uses SAC for vision-based RL from raw pixels, VICE for providing rewards, and running reset-free (denoted as "VICE"). We see that our method learns policies that succeed from nearly all the initial configurations, whereas VICE alone fails from most initial configurations. Fig[d depicts sample evaluation rollouts of the policies learned using our method. For further details about real world experiments see Appendix C.2
7 DISCUSSION
We presented the design and instantiation of a system for real world reinforcement learning. We identify and investigate the various ingredients required for such a system to scale gracefully with minimal human engineering and supervision. We show that this system must be able to learn from raw sensory observations, leam from very easily specified reward functions without reward engi- neering, and learn without any episodic resets. We describe the basic elements that required

---

Published as a conference paper at ICLR 2020
Sead Manioulation Hardwara Evaluations
Fvallsations
R3L w/o
VICE (Siggh ot, 2018)
Succass S. Jout of evaituations)
VICE (Singh al. 2019)
Time
Figure 8: Quantitative evaluation of performance in real-world for valve rotation and bead manipu- lation. Policies trained with perturbation controllers have effectively learned behaviors after 17 and hours of training, respectively. For more fine- fine-grained reporting of results see Figs 13
Initial State
Time
Goal State
Bead Manipulation
Valve Rotation
Figure 9: Evaluation rollouts of R3L the real world tasks for policies trained without instrumentation. Successful evaluation rollouts bead manipulation (top) and valve rotation (bottom) are shown here.
construct such system, and identify unexpected learning challenges that arise from interplay of these elements. We propose simple and scalable fixes to these challenges through introducing unsupervised representation learning and a randomized perturbation controller.
The ability to train robots directly in the real world with minimal instrumentation opens a number of exciting avenues for future research. Robots that can learn unattended, without resets hand- designed reward functions, can in principle collect very large amounts of experience autonomously, which may enable very broad generalization in the future. However, there are also number of ad- ditional challenges, including sample complexity, optimization and exploration difficulties complex tasks, safe operation, communication latency, sensing and actuation noise, and so forth, all of which would need to be addressed in future work in order to enable truly scalable real-world robotic learning.
more
ACKNOWLEDGMENTS
This research is supported by the Office of Naval Research, the National Science Foundation under IIS- 165 16-1668433 and IIS-170069 1700696, Google, NVIDIA, and Amazon. The authors would like to thank Ignasi Clavera, Gregory Kahn, Coline Devin, Benjamin Eysenbach, Aviral Kumar, Anusha Naga- bandi, Marvin Zhang, Ashyin Nair and several others in the RAIL Lab for helpful discussions and feedback.
10

---

Published as a conference paper at ICLR 2020
REFERENCES
Michael Ahn, Henry Zhu, Kristian Hartikainen, Hugo Ponte, Abhishek Gupta, Sergey Levine, and Vikash Kumar. ROBEL: RObotics BEnchmarks for Learning with low-cost robots. Conference on Robot Learning (CoRL), 2019.
Ankesh Anand, Evan Racah, Sherjil Ozair, Yoshua Bengio, Mare-Alexandre Côté, and R. Devon Hjelm. Unsupervised state representation learning atari. arXiv, 2019.
Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pa- chocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation. arXiv preprint 1808.v88000001111 2018.
Minoru Asada, Koh Hosoda, Yasuo Kumiyoshi, Hiroshi Ishiguro. Toshio Inui, Yuichiro Yoshikawa, Masaki Ogino, and Chisato Yoshida. Cognitive developmental robotics: A survey. IEEE Trans. Autonomous Mental Development, 1(1): 34, 2009. doi: 109/TAMDD20099.20000 2021702. URL
hittps://dol wttp://...r./..111///111110.. TAMD. 2009 2...90002221 2021702
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint
2018.
Yevgen Chebotar, Mrinal Kalakrishnan, Ali Yahya, Adrian Li, Stefan Schaal, and Sergey Levine. Path integral guided policy search. CoRR, abs/1610.00529, 2016.
Yevgen Chebotar, Karol Hausman, Marvin Zhang, Gaurav Sukhatme, Stefan Schaal, and Sergey Levine. Combining model-based and model-free updates for trajectory-centric reinforcement learning. In Proceedings of the 34th International Conference Machine Learning- Volume 70, pp. 703-711. JMLR. org, 2017.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, pp. 4299-4307 2017.
Ignasi Clavera, Jonas Rothfuss, John Schulman, Yasuhiro Fujita, Tamim Asfour, and Pieter Abbeel. Model-Based Reinforcement Learning via Meta-Policy Optimization. arXiv e-prints, art. 1oXiv:1809.05215 Sep 2018.
Coline Devin, Pieter Abbeel, Trevor Darrell, and Sergey Levine. Deep object-centric representa- tions for generalizable robot learning. In 2018 IEEE International Conference Robotics and Automation (ICRA), pp. 7111-7118. IEEE, 2018
Benjamin Eysenbach, Shixiang Gu, Julian Ibarz, and Sergey Levine. Leave no trace: Learning to reset for safe and autonomous reinforcement learning. In International Conference on Learning Representations (ICLR), 2018
Chelsea Finn and Sergey Levine. Deep visual foresight for planning robot motion. CORR, abs/1610.00666, 2016. httt:t/ httttp://axxv...oo org..v..rrr/aaa. haarg//bs//1111 /1c. 00696
Justin Fu, Avi Singh, Dibya Ghosh, Larry Yang. and Sergey Levine. Variational inverse control with events: A general framework for data-driven reward definition. In Advances in Neural Information Processing Systems, 2018.
Shixiang Gu, Ethan Holly. Timothy Lillicrap, and Sergey Levine. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In 2017 IEFF international confer- ence on robotics and automation (ICRA), pp. 3389-3396. IEEE, 2017.
Abhishek Gupta, Clemens Eppner, Sergey Levine, and Pieter Abbeel. Learning dexterous manipula- tion for soft robotic hand from human demonstrations, 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3786-3793, 2016.
Tuomas Haarojja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference Machine Learning, 2018a.

---

Published as a conference paper at ICLR 2020
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algo- rithms and applications. arXiv preprint arXiv: 2018b.
Weiqiao Han, Sergey Levine, and Pieter Abbeel. Learning compound multi-step controllers un- der unknown dynamics. In 2015 JEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 6435-6442. IEEE, 2015.
R Devon Hjelm, Alex Fedorov, Samuel Maavoie-Marchildeee Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estima- tion and maximization. In International Conference Learning Representations, 2019. URL htttps://openrevie
net/ htt///oormmmmmkmkkmm JOcKXX
Max Jaderberg, Volodymyr Mnih, Wojeiech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint /61 1.003997 2016.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Iharz, Alexander Herzog, Eric Jang. Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, al. Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation. arXiv preprint arXiv: 1806. 10293, 2018.
Katie Kang, Suneel Belkhale, Gregory Kahn, Pieter Abbeel, and Sergey Levine. Generalization through Simulation: Integrating Simulated and Real Data into Deep Reinforcement Learning for Vision-Based Autonomous Flight. arXiv e-prints, art. 1eXiv:1902.0370, Feb 2019.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. CoRR, abs/1312.6114, 2013.
Vikash Kumar, Emanuel Todorov, and Sergey Levine. Optimal control with leamed local models: Application to dexterous manipulation. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 378-383. IEBE, 2016.
Alex X. Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic latent actor-critic: Deep reinforcement learning with latent variable model. CoRR, abs/1907.00953, 2019.
Timothée Lesort, Vincenzo Lomonaco, Andrei Stoian, Davide Maltoni, David Filliat, and Na- talia Díaz Rodríguez. Continual learning for robotics. CoRR, 0bs//997.00012,, 2019. URL
http://arxiv, www.oorgbsbb... 0018
Sergey Levine and Vladlen Koltun. Guided policy search. In Proceedings The The 30th International
Conference on Machine Learning, 2013.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuo- motor policies. J. Mach. Learn. Res., 17(1), 2016. ISSN 1532-4435.
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: NXXi:/509000977 2015.
YuXuan Liu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Imitation from observation: Learn- ing to imitate behaviors from raw video via context translation. In 2018 IEEE International Con- ference on Robotics and Automation (ICRA), pp. 1118-111111111 1125. IEEE, 2018.
Max Lungarella, Giorgio Metta, Rolf Pfeifer, and Giulio Sandini. Developmental robotics: survey. Connection Science, 1544):151-190, 2003. doi: 10.00000054000031000015511
Volodymyr Mnib, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wier- stra, Shane Legg, and Demis Hassabis. Human-level. control through deep reinforcement learning. Nature, 518(7540), 2015.
12

---

Published as a conference paper at ICLR 2020
Anusha Nagabandi, Kurt Konoglie, Sergey Levine, and Vikash Kumar. Deep Dynamics Models for Learning Dexterous Manipulation. In Conference on Robot Learning (CoRL), 2019.
Ashvin Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. CoRR, abs/1807.04744 2018.
Andrew Y. Ng and Stuart J. Russell. Algorithms for inverse reinforcement learning. In Proceedings of the Seventeenth International Conference on Machine Learning, ICML '00, 2000. ISBN 1- 55860-707-2.
Xue Bin Peng, Marcin Andrychowiez, Wojeiech Zaremba, and Picter Abbeel. Sim-to-Real Transfer of Robotic Control with Dynamics Randomization. arXiv e-prints, art. arXiv:1710006537, Oct 2017.
Lerrel Pinto, Marcin Andrychowicz, Peter Welinder, Wojeiech Zaremba, and Pieter Abbeel, Asym- metric actor critic for image-based robot learning. arXiv preprint arXivv 2017a.
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforce- ment learning. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference Machine Learning, volume 70 of Proceedings of Machine Leurning Research, pp. 2817-2826, International Convention Centre, Sydney, Australia, 06-10 Aug 2017b. PMLR. URL html
http://ppet http://proceecinng .ww...r..............m.... wttppppsssss..pppppppeepp
Connor Schenck and Dieter Fox. Visual closed-loop control for pouring liquids. In 2017 IEEE International Conference on Robolics and Automation (ICRA), pp. 2629-2636. IEEE, 2017.
Jürgen Schmidhuber. Developmental robotics, optimal artificial curiosity, creativity, music, and the fine arts. Connect. Sci., 18(2):173-187, 2006. doi: 10. 10.00000950000000066666666666360 URL https://ddii.rr/100
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region
policy optimization. In International conference on machine learning, pp. 1889-1897, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: (707.067070647 2017.
Avi Singh, Larry Yang, Kristian Hartikainen, Chelsea Finn, and Sergey Levine End-to-end robotic reinforcement learning without reward engineering. Robotics: Science and Systems (RSS), 2019.
Adrian Stoica. Robot fostering techniques for monor development of humanoid robots. Robotics and Autonomous Systems, 37(2-3):127-143, 2001. doi: 10.1016/S0921- -08000211-880000)
00154-3. URL/attps://co: org 111110.111110000 /tt/://...000/0100000/222222-2-2 8890 (C1) 00154-3
Sainbayar Sukhbaatar, Zeming Lin, Ilya Kostrikov, Gabriel Synnaeve, Arthur Szlam, and Rob Fer- gus. Intrinsic motivation and automatic curricula via asymmetric self-play. In 6th International Conference on Learning Representations, ICLR 2018, 2018.
Sebastian Thrun and Tom M. Mitchell. Lifelong robot learning. Robotics and Autonomous Systmss, 15(1-2):25-46, 1995. doi: 10. 101106600221-889099))00000000000 1016/0921- 8890 (95) 00004-Y
URL httpp:://dii.zo////
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Do- main Randomization for Transferring Deep Neural Networks from Simulation to the Real World. arXiv e-prints, art.
Mar 2017.
Eric Tzeng, Coline Devin, Judy Hoffman, Chelsea Finn, Pieter Abbeel, Sergey Levine, Kate Saenko, and Trevor Darrell. Adapting Deep Visuomotor Representations with Weak Pairwise Constraints. arXiv e-prints, art. arXiv: 151.0111, Nov 2015.
Matej Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nico- las Heess, Thomas Rothörl, Thomas Lampe, and Martin A. Riedmiller. Leveraging demon- strations for deep reinforcement learning on robotics problems with sparse rewards. CoRR, abs/1707.0881 2017. httph//tp////rrttoooo/a//////// 08817
13

---

Published as a conference paper at ICLR 2020
Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empiri- cal risk minimization. In International Conference on Learning Representations, 2018.
Henry Zhu, Abhishek Gupta, Aravind Rajeswaran, Sergey Levine, and Vikash Kumar. Dexterous manipulation with deep reinforcement learning: Efficient, general, and low-cost. In 2019 Inter- national Conference on Robotics and Automation (ICRA), pp. 3651-3651 IEEE, 2019.
14

---

Published as a conference paper at ICLR 2020
A ALGORITHM DETAILS
Algorithm Real-World Robotic Reinforcement Learning (R3L)
procedure R3I.
2: N + number of training epochs
3: H + trajectory length (horizon)
4: TVICE number of VICE classifier training iterations per epoch
5: Initialize forward and perturbing policies To, T1
6:
Obtain goal states of and initialize as goal pool g
7:
Initialize RND target and predictor networks (s),, f()
8:
Initialize VICE reward classifier rvICE(s)
9:
Initialize replay buffer
10:
Collect initial exploration data and add to D
11:
for = 1 to 2.V do
12:
k < % for t to H do
13:
14:
Sample 0, Tx (s:) Sample St+1 p(s141l5(,a)
15:
1t 0 then
17:
(st) = OVICE (vICH((:) + CRND * (RND(SE)
else if then
19:
(s,) = (8,)
20:
end if
21:
Sample batch from D
22:
Update Fe with batch according to SAC
23:
Update RND predictor network with batch
24:
Update running estimate of standard deviations of classifier and RND reward end for
25:
26:
Add experience to the replay buffer with D + DU
27:
Sample equal number of goal examples from G and negative examples from D for \ to 7lyICE do
28:
Train the VICE classifier on this batch with binary labels
30:
end for
end for
32:
end procedure
15

---

Published as a conference paper at ICLR 2020
B TRAINING DETAILS
B.0.1 HYPERPARAMETERS
General
Standard deviation up-
0.99
date coefficient
Image Sizes
[(16, 16, 3), (32, 32, 3), (64, 64, 3)]
SAC
Learning Rate
3e-4
Y
0.99
Batch Size
256
Convnet Filters
[(64, 64, 64), (16, 32, 64)]
Stride
(2, 2)
Kernel Sizes
(3.3)
Pooling
[MaxPool2D, None]
Actor/Critic FC Lay-
(512, 512), (256, 256, 256))
VICE
7IVICE
[1, 10]
Batch Size
128
Learning Rate
le-4 Uniform(0, 1)
Mixup a
Convnet Filters
[(64, 64, 64), (16, 32, 64)]
Stride
(2 2)
Kernel Sizes
(3
Pooling
MaxxPol22D None] (512, 512), (256. 256, 256)
FC Layers
RND
Learning Rate
3e-4
Batch Size
256
Convnet Filters
(16, 32, 64)
Stride
(2, 2)
Kernel Sizes
(3,
Pooling
[MaxPoo12D, None]
FC Layers
(512, 512), (256, 256, 256)]
VAE
Learning Rate
le-4
Batch Size
256
Encoder
(Convnet)
(64, 64, 32)
Filters
Latent Dimension
[8, 16, 32, 64] [le-3, 0.1, 0.5, 10] (2, 2)
Stride
Kernel Sizes
(3, 3) [MaxPool2D, None]
Pooling
The ranges of values listed above represent the hyperparameters we searched over, and the bolded values what we use in the Section 6 experiments.
B.0.2 VICE
We use variant of VICE which defines the reward as the logits of the classifier, notably omitting the -log(=(a(s)) term. We also regularize our classifier with mixup (Zhang et al.) 2018). train all of our experiments using 200 goal images, which takes under hour to collect the real world for each task.
B.0.3 RANDOM NETWORK DISTILLATION (RND)
We found it important to normalize the predictor errors, just (Burda et al. 2018) did.
16

---

Published as a conference paper at ICLR 2020
B.0.4 VAE
We train a standard Vet to maximize the evidence lower bound, given by:
(zlag(z(a) [po(=]))] - 3Dxx(90(21x) po(2))
To collect training data, we sampled random states in the observation space. In the real world, this sampling can be replaced with training an exploratory policy (i.e. using the RND reward as the policy's s only objective). The learned weights of the encoder of the VAE are frozen, and the latent input is used to train the policy for reset-free RI.
C TASK DETAILS
C.I SIMULATED TASKS
We evaluated our system across three tasks in simulation: bead manipulation, valve rotation, free object repositioning.
C.I.I BEAD MANIPULATION
The bead manipulation task involves an abacus rod with four beads that can slide freely. The goal is position beads on each end from any initial configuration of beads. This can take the form of sliding one bead over (if three beads start on one side), two beads over (if all four beads start on one side), splitting beads apart (all four beads start in the middle), or some intermediate combination of those. The true reward is defined as the mean goal distance of all four beads. Policies are evaluated starting from the initial configurations depicted in Fig T0 Evaluation performance reported in Section 6 for this task is defined as the final reward averaged across the 8 evaluation rollouts.
Figure 10:
These are the 8 initial positions used for evaluating the performance of the bead manipulation policy. The goal configuration (which is also initial evaluation position) highlighted in yellow.
C.1.2 VALVE ROTATION
The claw is positioned above a three pronged valve (15 em in diameter). The objective turn the valve given orientation from any initial orientation. The "true reward" is log(10 ( (goat)). Policies are evaluated starting from the initial configurations depicted in Fig II Evaluation performance reported in Section 6 for this task is defined as the final orientation distance averaged across the 8 evaluation rollouts.
C.1.3 FREE OBJECT REPOSITIONING
The claw is positioned atop a free (6 DoF) three pronged object (15em in diameter), which can translate and rotate within 30cmr33em box. The goal is specified by xy-position well z- zaaglee where the xy-plane is the plane the arena. The true reward is defined as the weighted sum of the angular and translational distances, -2 log sstates Ustate] [ggatt, Mgoat]]
17

---

Published as a conference paper at ICLR 2020
Figure 11:
These are the initial positions used for evaluating the performance of the valve rotation policy. The goal configuration (which is also initial evaluation position) is highlighted in yellow.
log(1osttt: 0goatl). In our experiments, 1r,
(0,0, -) where the origin is centered the arena. Policies are evaluated starting from the initial configurations depicted Figure Evaluation performance reported in Section 6 for this task is defined as the final pose distance [ -g rad
(ffinal-Minnnl |Spoat-Mzoailll
averaged across the evaluation rollouts.
0.25
In our reset controller experiments, we averaged evaluation performance over three different choices of reset states, where the first reset state always the goal:
1. (x, V, 0)rsset, = (xr, y, 0) gout; (x, 0)rsset, (0.05, 0.05, )
2. (Ir, y, 0resset, = (xr, y, 0) goat, (x, 0) 0essett = (0, 0, -
3. (xr, y, 0) 0)esser1 = (xx, M, gooa (x, 0) resset, (-0.04,-0.44 -
Figure 12: These are the 15 initial positions used for evaluating the performance of the free object reposi- tioning policy. The goal configuration (Ir, y, 0) which is also initial evaluation position highlighted in yellow.
18

---

Published as a conference paper at ICLR 2020
C.2 REAL WORLD TASKS
For each setup we use an RGB camera to get images. We execute actions on the DClaw at 10Hz. In order operate at such high frequency while also training from images we sample and train asynchronously, but limit training to not exceed two gradient steps per transition sampled in the real world. Since direct performance metrics cannot be measured during training due the lack of object instrumentation, evaluations of performance are done post-training.
C.2.1 VALVE ROTATION
The task identical to the one in simulation. Evaluations were done post-training. An evaluation trajectory was defined success if at the last step, the valve was within 15 degrees of the goal. Each policy was evaluated over 8 rollouts, with initial configurations evenly spaced out between at increments of 45 degrees. Results are reported in Figures 14
C.2.2 BEAD MANIPULATION
The rod is 22em in length, and each bead measures 3.5em in diameter. Evaluations done post-training, using the following procedure: the environment was manually reset to each of the specified configurations shown in Figures [1 [ (which cover a full range of the state space) at the start of each evaluation rollout. An evaluation trajectory was defined as success if at the last time step, all beads were within 2cm of their goal positions. Performance 20 hours, at which point the policy achieved greater than 80% success on the 10 evaluation rollouts random policy achieved a success of 10%). Results are reported in Figs [5] 16
evaluated around
RND VICE
Evaluation Initial Positions
Time [hr]
Success
0.36
0.71
25
1.07
1.43
50
1.78
25
2.14
2.49
25
2.85
3.21
75
3.56
75
3.92
4.28
62.5
4.99
87.5
5.34
87.5
5.70
62.5
Figure 13: These are the results of the evaluation rollouts the valve rotation task in the real world using method (without the VAE). Trained policies row a different policy, each column an evaluation rollout from different initial configuration. The goal highlighted in yellow. Our method is able to achieve high success rates after hours training.
saved regular intervals and evaluated post-training. Each
19

---

Published as a conference paper at ICLR 2020
VICE
Evaluation Initial Positions
Time [hr]
Success %
0.29
25
25
0.87
12.5
1.16
50
1.45
25
1.74
50
62.5
2.32
50
2.61
12.5
12.5
3.19
50
3.48
37.5
3.77
37.5
4,06
37.5
37.5
4.64
62.5
4.93
50
50
5.51
50
Figure 14: These are results of evaluation rollouts on the valve rotation task the real world using VICE single goal baseline, policies fail to evaluate well, especially from initial positions from the goal position.
RND VICE
Evaluation Initial Positions
Time [hr]
Success
2.93
50
5.86
37.5
8.79
37.5
11.73
87.5
14.66
75
17.59
100
20.52
87.5
Figure 15: These are the results of the evaluation rollouts on the valve rotation task in the real world using our method (without the VAE). Trained policies were saved at regular intervals and evaluated post-training. Each row a different policy, and each column an evaluation rollout from different initial configuration. The goal is highlighted in yellow. Our method is able to achieve high success rates after 17 hours of training.
20

---

Published as a conference paper at ICLR 2020
VICE
Evaluation Initial Positions
Time [hr]
Success %
2.68
25
5.36
12.5
8.03
25
10.71
12.5
13.39
25
16.07
37.5
18.74
12.5
21.42
12.5
Figure 16: These are results evaluation rollouts on the valve rotation task the real world using VICE single goal baseline. The policies fail to evaluate consistently, except when the initial configuration matches the goal configuration.
21


Reviews:
[{"review_id": "r1l7Uvpn_r", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper takes seriously the question of having a robotic system learning continuously without manual reset nor state or reward engineering. The authors propose a first approach using vison-based SAC, shown visual goals and VICE, and show that it does not provide a satisfactory solution. Then they add a random pertubation controller which brings the robot or simulated system away from the goal and a VAE to encode a compressed state, and show that it works better.\n\nThe paper is a nice read, it contains useful messages thus I'm slightly in favor of accepting it, but I may easily change my mind as it suffers from serious weaknesses.\n\nFirst, and most importantly, the experimental study is very short, the authors have chosen to spend much more space on careful writing of the problem they are investigating.\n\nTo mention a few experimental weaknesses, in Section 6.2 the authors could have performed much more detailed ablation studies and stress in more details the impact of using the VAE alone versus using the random pertubation controller alone, they could say more about the goals they show to the system, etc. There is some information in Figure 7, but this information is not exploited in a detailed way. Furthermore, Figure 7 is far to small, it is hard to say from the legend which system is which.\n\nAbout Fig.8, we just have a qualitative description, the authors claim that without instrumenting they cannot provide a quantitative study, which I don't find convincing: you may instrument for the sake of science (to measure the value of what you are doing, even if the real-world system won't use this instrumentation).\n\nSo the authors have chosen to spend more space on the positionning than on the empirical study, which may speak in favor of sending this paper to a journal or magazine rather than a technical conference. But there is an issue about the positionning too: the authors fail to mention a huge body of literature trying to address very close or just similar questions. Namely, their concern is one the central leitmotives of Developmental Robotics and some of its \"subfields\", such as Lifelong learning, Open-ended learning, Continual learning etc.  The merit of the paper in this respect is to focus on a specific question and provide concrete results on this question, but this work should be positionned with respect to the broader approaches mentioned above. The authors will easily find plenty of references in these domains, I don't want to give my favorite selection here.\n\nKnowing more about the literature mentioned above, the authors could reconsider their framework from a multitask learning perspective: instead of a random perturbation controller, the agent may learn various controllers to bring the system into various goal states (using e.g. goal-conditioned policies), and switching from goal to goal to prevent the system fro keeping stuck close to some goal.\n\nMore local points:\n\nIn the middle of page 5, it is said that the system does not learn properly just because it is stuck at the goal. This information comes late, and makes the global message weaker.\n\nin Fig. 4, I would like to know what is the threshold for success.\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "r1gseP0nKH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "  *Synopsis*:\n  This paper focuses on current limitations of deploying RL approaches onto real world robotic systems. They focus on three main points: the need to use raw sensory data collected by the robot, the difficulty of handcrafted reward functions without external feedback, the lack of algorithms which are robust outside of episodic learning. They propose a complete system which addresses these concerns, combining approaches from the literature and novel improvements. They then provide an empirical evaluation and ablation testing of their approach and other popular systems, and show a demonstration on a real robotic system.\n \n  Main Contributions:\n  - A discussion of the current limitations of RL on real robotic systems\n  - A framework for doing real world robotic RL without extra instrumentation (outside of the robot).\n\n  *Review*: \n  Overall, I think the paper is well written and provides some nice analysis of the current state of RL and robotics. I am not as familiar with the RL for robotics literature, but from some minor snooping around I believe these ideas to be novel and useful for the community. I have a few suggestions for the authors, and a few critical pieces I would like added to the main text.\n\n  Critical additions:\n  1. I would like some more details on your simulation experiments. Specifically:\n    - How many runs were your experiments? \n    - What are the error bars on your plots?\n    - What ranges of hyper-parameters did you test for tuning?\n\n  2. I would quite like the discussion of the real world tasks from the appendix to appear in the main text. Specifically, giving the evaluation metrics you mentioned in the appendix. \n\n  Suggestions/Questions:\n\n  S1: It is not clear if a VAE is the best choice for unsupervised representation learning for RL agents. Although a reasonable choice, Yashua Bengio recently released a look at several unsupervised techniques for representation learning in Atari which you may want to look at: https://arxiv.org/pdf/1906.08226.pdf. \n\n  Q1: Did you try any of the other approaches on the real robotics system? Or was there no way to deploy these algorithms to your specific setup without instrumentation?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Hyl29EPXtr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents approaches to handle three aspects of real-world RL on robotics: (1)learning from raw sensory inputs (2) minimal reward design effort (3) no manual resetting. Key components:(1) learn a perturbation policy that allows the main policy to explore a wide variety of state. (2) learn a variational autoencoder to transform images to low dimensional space.\n\nExperiments in simulation on the physical robots are performed to demonstrate the effectiveness of these components. Close related work is also used for comparison. The only concern I have is that the tasks considered involve robots that can automatically reset themselves pretty easily. I doubt that this will scale to unstable robots such as biped/quadruped, where once they fail, the recovering/resetting tasks will be as much or more difficult than the main locomotion tasks. But I understand this is too much to address in one paper and limitation is also briefly discussed in the final section.\n\nOverall I think this is a good paper and valuable to the community.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a very interesting paper which discusses practical issues and solutions around deploying RL on real physical robotic systems, specifically involving questions on the use of raw sensory data, crafting reward functions, and not having resets at the end of episodes.

Many of the issues raised in the reviews and discussion were concerned with experimental details and settings, as well as relation to different areas of related work. These were all sufficiently handled in the rebuttal, and all reviewers were in favour of acceptance.
Decision: Accept (Spotlight)