The Ingredients of Real World Robotic Reinforcement Learning

1 INTRODUCTION . Reinforcement learning ( RL ) can in principle enable autonomous systems , such as robots , to acquire a large repertoire of skills automatically . Perhaps even more importantly , reinforcement learning can enable such systems to continuously improve the proficiency of their skills from experience . However , realizing this in reality has proven challenging : even with reinforcement learning methods that can acquire complex behaviors from high-dimensional low-level observations , such as images , the assumptions of the reinforcement learning problem setting do not fit cleanly into the constraints of the real world . For this reason , most successful robotic learning experiments have employed various kinds of environmental instrumentation in order to define reward functions , reset between trials , and obtain ground truth state ( Levine et al. , 2016 ; Haarnoja et al. , 2018a ; Kumar et al. , 2016 ; Andrychowicz et al. , 2018 ; Zhu et al. , 2019 ; Chebotar et al. , 2016 ; Nagabandi et al. , 2019 ; Gupta et al. , 2016 ) . In order to practically and scalably deploy autonomous learning systems that improve continuously through real-world operation , we must lift these limitations and design algorithms that can learn under the constraints of real-world environments , as illustrated in Figure 2 . We propose that overcoming these challenges in a scalable way requires designing robotic systems that possess three capabilities : they are able to ( 1 ) learn from their own raw sensory inputs , ( 2 ) assign rewards to their own trials without hand-designed perception systems or instrumentation , and ( 3 ) learn continuously in non-episodic settings without requiring human intervention to manually reset the environment . A system with these capabilities can autonomously collect large amounts of real world data – typically crucial for effective generalization – without significant instrumentation in each training environment , an example of which is shown in Figure 1 . If successful , this would lift a major constraint that stands between current reinforcement learning algorithms and the ability to learn scalable , generalizable , and robust real-world behaviors . Such a system would also bring us significantly closer to the goal of embodied learning-based systems that improve continuously through their own real-world experience . ∗These authors contributed equally . Correspondence to henryzhu @ berkeley.edu . Having laid out these requirements , we propose a practical instantiation of such a learning system . While prior works have studied many of these issues in isolation , combining them into a complete real-world learning system presents a number of challenges , as we discuss in Section 3 . We provide an empirical analysis of these issues , both in simulation and on a real-world robotic system , and propose a number of simple but effective solutions that together produce a complete robotic learning system . This system can autonomously learn from raw sensory inputs , learn reward functions from easily available supervision , and learn without manually designed reset mechanisms . We show that this system can learn dexterous robotic manipulation tasks in the real world , substantially outperforming ablations and prior work . 2 THE STRUCTURE OF A REAL-WORLD RL SYSTEM . The standard reinforcement learning paradigm assumes that the controlled system is represented as a Markov decision process with a state space S , action space A , unknown transition dynamics T , unknown reward function R , and a ( typically ) episodic initial state distribution ρ . The goal is to learn a policy that maximizes the expected sum of rewards via interactions with the environment . Although this formalism is simple and concise , it does not capture all of the complexities of realworld robotic learning problems . If a robotic system is to learn continuously and autonomously in the real world , we must ensure that it can learn under the actual conditions that are imposed by the real world . To move from the idealized MDP formulation to the real world , we require a system that has the following properties . First , all of the information necessary for learning must be obtained from the robot ’ s own sensors . This includes information about the state and necessitates that the policy must be learned from high-dimensional and low-level sensory observations , such as camera images . Second , the robot must also obtain the reward signal itself from its own sensor readings . This is exceptionally difficult for all but the simplest tasks ( e.g. , reward functions that depend on interactions with specific objects require perceiving those objects explicitly ) . Third , we must be able to learn without access to episodic resets . A setup with explicit resets quickly becomes impractical in open-world settings , due to the requirement for significant human engineering of the environment , or direct human intervention during learning . While this list may not exhaustively enumerate all the components necessary for an effective real-world learning system , we posit that the properties listed here are fundamental to building such systems . While some of the components discussed above can be tackled in isolation by current algorithms , there are considerable challenges inherent to assembling all these components into a complete learning system for real world robotic learning . In the rest of this section , we outline the challenges associated with each component , then discuss the challenges associated with combining these components in Section 3 , and then proceed to address these challenges in Section 4 . 2.1 LEARNING FROM RAW SENSORY INPUT . To enable learning without complex state estimation systems or environment instrumentation , we require our robotic systems to be able to learn from their own raw sensory observations . Typically , these sensory observations are raw camera images from a camera mounted on the robot , as well as proprioceptive sensory inputs such as the joint angles . These observations do not directly provide the poses of the objects in the scene , which is the typical assumption in simulated robotic environments – any such information must be extracted by the learning system . While in principle many RL frameworks can support learning from raw sensory inputs ( Levine et al. , 2016 ; Mnih et al. , 2015 ; Schulman et al. , 2015 ; Lillicrap et al. , 2015 ) , it is important to consider the practicalities of this approach . For instance , we can instantiate vision-based RL with policy gradient algorithms such as TRPO ( Schulman et al. , 2015 ) and PPO ( Schulman et al. , 2017 ) , but these have high sample complexities which make them unsuited for real world robotic learning ( Haarnoja et al. , 2018a ) . In our work , we consider adopting the general framework of off-policy actor-critic reinforcement learning , using a version of the soft actor critic ( SAC ) algorithm described by ( Haarnoja et al. , 2018b ) . This algorithm effectively uses off-policy data and has been shown to learn some tasks directly from visual inputs . However , while SAC can learn directly from images , we find in our experiments that , as the task complexity increases , the efficiency of direct end-to-end learning ( particularly without resets and with learned rewards ) still degrades substantially . However , as we will discuss in Section 3 , augmenting end-to-end learning with unsupervised representation learning substantially alleviates such challenges . 2.2 REWARD FUNCTIONS WITHOUT REWARD ENGINEERING . Vision-based RL algorithms , such as SAC , rely on a reward function being provided to the system , which is typically manually programmed by a user . While this can be simple to do in simulation by using ground truth state information , it is significantly harder to implement in uninstrumented real world environments . In the real world , the robot must obtain the reward signal itself from its own sensor readings . A few options for tackling this challenge have been discussed in prior work : design complete computer vision systems to detect objects and extract the reward signals ( Devin et al. , 2018 ; Nagabandi et al. , 2019 ) , engineer reward functions that use various task-specific heuristics to obtain rewards from pixels ( Schenck & Fox , 2017 ; Kalashnikov et al. , 2018 ) , or instrument every environment ( Chebotar et al. , 2017 ) . Many of these solutions are manual and tedious , and a more general approach is needed to scale real-world robotic learning gracefully . 2.3 LEARNING WITHOUT RESETS . While the components described in Section 2.1 and 2.2 are essential to building continuously learning RL systems in the real world , they have often been implemented with the assumption of episodic learning Fu et al . ( 2018 ) ; Haarnoja et al . ( 2018b ) . However , natural open-world settings do not provide any such reset mechanism , and in order to enable scalable and autonomous real-world learning we need systems that do not require an episodic formulation of the learning problem . To devise a system that requires minimal human engineering for providing rewards , we must use algorithms that are able to assign themselves rewards , using learned models that operate on the same raw sensory inputs as the policy . One candidate is for a user to specify intended behavior beforehand through examples of desired outcomes ( i.e. , images ) . The algorithm can then assign itself rewards based on a measure of how well it is accomplishing the specified goals , with no additional human supervision . This approach can scale well in principle , since it requires minimal human engineering , and goal images are easy to provide . In principle , algorithms such as SAC do not actually require episodic learning ; however , in practice , most instantiations use explicit resets , even in simulation , and removing resets has resulted in failure to solve challenging tasks . In our experiments in Section 3 as well , we see that actor-critic methods applied naı̈vely to the reset free setting do not learn the intended behaviors . Introducing visual observations and classifier based rewards exacerbates these challenges . We propose that these three components – vision-based RL with actor-critic algorithms , visionbased goal classifier for rewards , and reset-free learning – are the fundamental pieces that we need to build a real world robotic learning system . However , when we actually combine the individual components in Sections 3 and 6 , we find that learning effective policies is quite challenging . We provide insight into these challenges in Section 3 . Based on these insights , we propose simple but important changes in Section 4 to build a system , R3L , that can learn effectively and autonomously in the real world without human intervention . 3 THE CHALLENGES OF REAL WORLD RL The system design outlined in Section 2 in principle gives us a complete system to perform real world reinforcement learning without instrumentation . However , when utilized for robotic learning problems , we find this basic design to be largely ineffective . To study this , we present results for a simulated robotic manipulation task that requires repositioning a free-floating objects with a threefingered robotic hand , shown in Fig 3 . We use this task for our investigative analysis and show that the same insights extend to several other tasks , including real world tasks , in Section 6 . The goal in this task is to reposition the object to a target pose from any initial pose in the arena . When the system is instantiated with visionbased soft actor-critic , rewards from goal images using VICE , and run without episodic resets , we see that the algorithm fails to make progress ( Fig 4 ) . Although it might appear that this setup fits within the assumptions of all of the components that are used , the complete system is ineffective . Which particular components of this problem make it so difficult ? To investigate this issue , we perform experiments investigating the combination of the three main ingredients : varying observation type ( visual vs. low-dimensional state ) , reward structure ( VICE vs. hand-defined rewards that utilize ground-truth object state ) , and the ability to reset ( episodic resets vs. reset-free , non-episodic learning ) . We start by considering the training time reward under each combination of factors as shown in Fig 4 , which reveals several trends . First , the results in Fig 4 show that learning with resets achieves high training time reward from both vision and state , while reset-free only achieves high training time reward with low-dimensional state . Second , we find that the policy is able to pass the threshold for training time reward in a non-episodic setting when learning from low-dimensional state , but it is not able to do the same using image observations . This suggests that combining the reset-free learning problem with visual observations makes it significantly more challenging . However , the table in Fig 4 paints an incomplete picture . These numbers are related to the performance of the policies at training time , not how effective the learned policies are when being evaluated . When we consider the test-time performance ( Fig 5 ) of the policies that are learned under reset free conditions , we obtain a different set of conclusions . While learning from low-dimensional state in the reset free setting achieves high reward at training time , the test-time performance of the corresponding learned policies is very poor . This can likely be attributed to the fact that when the agent spends all its time stuck at the goal , and sees very little diversity of data in other parts of the state space , which significantly reduces the efficacy of the actual policies being learned . In a sense , the reset encodes some prior information about the task : it tells the policy about what types of states it might be required to succeed from at test time . Without this knowledge , performance is substantially worse . This makes it very challenging to learn policies with completely reset-free schemes , which has prompted prior work to consider schemes such as learning reset controllers ( Eysenbach et al. , 2018 ) . As we discuss in the following section and in our experiments , these schemes are often insufficient for learning effective policies in the real world without any resets .

Summary:   This paper focuses on current limitations of deploying RL approaches onto real world robotic systems. They focus on three main points: the need to use raw sensory data collected by the robot, the difficulty of handcrafted reward functions without external feedback, the lack of algorithms which are robust outside of episodic learning. They propose a complete system which addresses these concerns, combining approaches from the literature and novel improvements. They then provide an empirical evaluation and ablation testing of their approach and other popular systems, and show a demonstration on a real robotic system.