The Ingredients of Real World Robotic Reinforcement Learning

1 INTRODUCTION . Reinforcement learning ( RL ) can in principle enable autonomous systems , such as robots , to acquire a large repertoire of skills automatically . Perhaps even more importantly , reinforcement learning can enable such systems to continuously improve the proficiency of their skills from experience . However , realizing this in reality has proven challenging : even with reinforcement learning methods that can acquire complex behaviors from high-dimensional low-level observations , such as images , the assumptions of the reinforcement learning problem setting do not fit cleanly into the constraints of the real world . For this reason , most successful robotic learning experiments have employed various kinds of environmental instrumentation in order to define reward functions , reset between trials , and obtain ground truth state ( Levine et al. , 2016 ; Haarnoja et al. , 2018a ; Kumar et al. , 2016 ; Andrychowicz et al. , 2018 ; Zhu et al. , 2019 ; Chebotar et al. , 2016 ; Nagabandi et al. , 2019 ; Gupta et al. , 2016 ) . In order to practically and scalably deploy autonomous learning systems that improve continuously through real-world operation , we must lift these limitations and design algorithms that can learn under the constraints of real-world environments , as illustrated in Figure 2 . We propose that overcoming these challenges in a scalable way requires designing robotic systems that possess three capabilities : they are able to ( 1 ) learn from their own raw sensory inputs , ( 2 ) assign rewards to their own trials without hand-designed perception systems or instrumentation , and ( 3 ) learn continuously in non-episodic settings without requiring human intervention to manually reset the environment . A system with these capabilities can autonomously collect large amounts of real world data – typically crucial for effective generalization – without significant instrumentation in each training environment , an example of which is shown in Figure 1 . If successful , this would lift a major constraint that stands between current reinforcement learning algorithms and the ability to learn scalable , generalizable , and robust real-world behaviors . Such a system would also bring us significantly closer to the goal of embodied learning-based systems that improve continuously through their own real-world experience . ∗These authors contributed equally . Correspondence to henryzhu @ berkeley.edu . Having laid out these requirements , we propose a practical instantiation of such a learning system . While prior works have studied many of these issues in isolation , combining them into a complete real-world learning system presents a number of challenges , as we discuss in Section 3 . We provide an empirical analysis of these issues , both in simulation and on a real-world robotic system , and propose a number of simple but effective solutions that together produce a complete robotic learning system . This system can autonomously learn from raw sensory inputs , learn reward functions from easily available supervision , and learn without manually designed reset mechanisms . We show that this system can learn dexterous robotic manipulation tasks in the real world , substantially outperforming ablations and prior work . 2 THE STRUCTURE OF A REAL-WORLD RL SYSTEM . The standard reinforcement learning paradigm assumes that the controlled system is represented as a Markov decision process with a state space S , action space A , unknown transition dynamics T , unknown reward function R , and a ( typically ) episodic initial state distribution ρ . The goal is to learn a policy that maximizes the expected sum of rewards via interactions with the environment . Although this formalism is simple and concise , it does not capture all of the complexities of realworld robotic learning problems . If a robotic system is to learn continuously and autonomously in the real world , we must ensure that it can learn under the actual conditions that are imposed by the real world . To move from the idealized MDP formulation to the real world , we require a system that has the following properties . First , all of the information necessary for learning must be obtained from the robot ’ s own sensors . This includes information about the state and necessitates that the policy must be learned from high-dimensional and low-level sensory observations , such as camera images . Second , the robot must also obtain the reward signal itself from its own sensor readings . This is exceptionally difficult for all but the simplest tasks ( e.g. , reward functions that depend on interactions with specific objects require perceiving those objects explicitly ) . Third , we must be able to learn without access to episodic resets . A setup with explicit resets quickly becomes impractical in open-world settings , due to the requirement for significant human engineering of the environment , or direct human intervention during learning . While this list may not exhaustively enumerate all the components necessary for an effective real-world learning system , we posit that the properties listed here are fundamental to building such systems . While some of the components discussed above can be tackled in isolation by current algorithms , there are considerable challenges inherent to assembling all these components into a complete learning system for real world robotic learning . In the rest of this section , we outline the challenges associated with each component , then discuss the challenges associated with combining these components in Section 3 , and then proceed to address these challenges in Section 4 . 2.1 LEARNING FROM RAW SENSORY INPUT . To enable learning without complex state estimation systems or environment instrumentation , we require our robotic systems to be able to learn from their own raw sensory observations . Typically , these sensory observations are raw camera images from a camera mounted on the robot , as well as proprioceptive sensory inputs such as the joint angles . These observations do not directly provide the poses of the objects in the scene , which is the typical assumption in simulated robotic environments – any such information must be extracted by the learning system . While in principle many RL frameworks can support learning from raw sensory inputs ( Levine et al. , 2016 ; Mnih et al. , 2015 ; Schulman et al. , 2015 ; Lillicrap et al. , 2015 ) , it is important to consider the practicalities of this approach . For instance , we can instantiate vision-based RL with policy gradient algorithms such as TRPO ( Schulman et al. , 2015 ) and PPO ( Schulman et al. , 2017 ) , but these have high sample complexities which make them unsuited for real world robotic learning ( Haarnoja et al. , 2018a ) . In our work , we consider adopting the general framework of off-policy actor-critic reinforcement learning , using a version of the soft actor critic ( SAC ) algorithm described by ( Haarnoja et al. , 2018b ) . This algorithm effectively uses off-policy data and has been shown to learn some tasks directly from visual inputs . However , while SAC can learn directly from images , we find in our experiments that , as the task complexity increases , the efficiency of direct end-to-end learning ( particularly without resets and with learned rewards ) still degrades substantially . However , as we will discuss in Section 3 , augmenting end-to-end learning with unsupervised representation learning substantially alleviates such challenges . 2.2 REWARD FUNCTIONS WITHOUT REWARD ENGINEERING . Vision-based RL algorithms , such as SAC , rely on a reward function being provided to the system , which is typically manually programmed by a user . While this can be simple to do in simulation by using ground truth state information , it is significantly harder to implement in uninstrumented real world environments . In the real world , the robot must obtain the reward signal itself from its own sensor readings . A few options for tackling this challenge have been discussed in prior work : design complete computer vision systems to detect objects and extract the reward signals ( Devin et al. , 2018 ; Nagabandi et al. , 2019 ) , engineer reward functions that use various task-specific heuristics to obtain rewards from pixels ( Schenck & Fox , 2017 ; Kalashnikov et al. , 2018 ) , or instrument every environment ( Chebotar et al. , 2017 ) . Many of these solutions are manual and tedious , and a more general approach is needed to scale real-world robotic learning gracefully . 2.3 LEARNING WITHOUT RESETS . While the components described in Section 2.1 and 2.2 are essential to building continuously learning RL systems in the real world , they have often been implemented with the assumption of episodic learning Fu et al . ( 2018 ) ; Haarnoja et al . ( 2018b ) . However , natural open-world settings do not provide any such reset mechanism , and in order to enable scalable and autonomous real-world learning we need systems that do not require an episodic formulation of the learning problem . To devise a system that requires minimal human engineering for providing rewards , we must use algorithms that are able to assign themselves rewards , using learned models that operate on the same raw sensory inputs as the policy . One candidate is for a user to specify intended behavior beforehand through examples of desired outcomes ( i.e. , images ) . The algorithm can then assign itself rewards based on a measure of how well it is accomplishing the specified goals , with no additional human supervision . This approach can scale well in principle , since it requires minimal human engineering , and goal images are easy to provide . In principle , algorithms such as SAC do not actually require episodic learning ; however , in practice , most instantiations use explicit resets , even in simulation , and removing resets has resulted in failure to solve challenging tasks . In our experiments in Section 3 as well , we see that actor-critic methods applied naı̈vely to the reset free setting do not learn the intended behaviors . Introducing visual observations and classifier based rewards exacerbates these challenges . We propose that these three components – vision-based RL with actor-critic algorithms , visionbased goal classifier for rewards , and reset-free learning – are the fundamental pieces that we need to build a real world robotic learning system . However , when we actually combine the individual components in Sections 3 and 6 , we find that learning effective policies is quite challenging . We provide insight into these challenges in Section 3 . Based on these insights , we propose simple but important changes in Section 4 to build a system , R3L , that can learn effectively and autonomously in the real world without human intervention . 3 THE CHALLENGES OF REAL WORLD RL The system design outlined in Section 2 in principle gives us a complete system to perform real world reinforcement learning without instrumentation . However , when utilized for robotic learning problems , we find this basic design to be largely ineffective . To study this , we present results for a simulated robotic manipulation task that requires repositioning a free-floating objects with a threefingered robotic hand , shown in Fig 3 . We use this task for our investigative analysis and show that the same insights extend to several other tasks , including real world tasks , in Section 6 . The goal in this task is to reposition the object to a target pose from any initial pose in the arena . When the system is instantiated with visionbased soft actor-critic , rewards from goal images using VICE , and run without episodic resets , we see that the algorithm fails to make progress ( Fig 4 ) . Although it might appear that this setup fits within the assumptions of all of the components that are used , the complete system is ineffective . Which particular components of this problem make it so difficult ? To investigate this issue , we perform experiments investigating the combination of the three main ingredients : varying observation type ( visual vs. low-dimensional state ) , reward structure ( VICE vs. hand-defined rewards that utilize ground-truth object state ) , and the ability to reset ( episodic resets vs. reset-free , non-episodic learning ) . We start by considering the training time reward under each combination of factors as shown in Fig 4 , which reveals several trends . First , the results in Fig 4 show that learning with resets achieves high training time reward from both vision and state , while reset-free only achieves high training time reward with low-dimensional state . Second , we find that the policy is able to pass the threshold for training time reward in a non-episodic setting when learning from low-dimensional state , but it is not able to do the same using image observations . This suggests that combining the reset-free learning problem with visual observations makes it significantly more challenging . However , the table in Fig 4 paints an incomplete picture . These numbers are related to the performance of the policies at training time , not how effective the learned policies are when being evaluated . When we consider the test-time performance ( Fig 5 ) of the policies that are learned under reset free conditions , we obtain a different set of conclusions . While learning from low-dimensional state in the reset free setting achieves high reward at training time , the test-time performance of the corresponding learned policies is very poor . This can likely be attributed to the fact that when the agent spends all its time stuck at the goal , and sees very little diversity of data in other parts of the state space , which significantly reduces the efficacy of the actual policies being learned . In a sense , the reset encodes some prior information about the task : it tells the policy about what types of states it might be required to succeed from at test time . Without this knowledge , performance is substantially worse . This makes it very challenging to learn policies with completely reset-free schemes , which has prompted prior work to consider schemes such as learning reset controllers ( Eysenbach et al. , 2018 ) . As we discuss in the following section and in our experiments , these schemes are often insufficient for learning effective policies in the real world without any resets .

Summary: This paper presents approaches to handle three aspects of real-world RL on robotics: (1)learning from raw sensory inputs (2) minimal reward design effort (3) no manual resetting. Key components:(1) learn a perturbation policy that allows the main policy to explore a wide variety of state. (2) learn a variational autoencoder to transform images to low dimensional space.