Improving Exploration of Deep Reinforcement Learning using Planning for Policy Search

Most Deep Reinforcement Learning methods perform local search and therefore are prone to get stuck on non-optimal solutions . Furthermore , in simulation based training , such as domain-randomized simulation training , the availability of a simulation model is not exploited , which potentially decreases efficiency . To overcome issues of local search and exploit access to simulation models , we propose the use of kinodynamic planning methods as part of a model-based reinforcement learning method and to learn in an off-policy fashion from solved planning instances . We show that , even on a simple toy domain , D-RL methods ( DDPG , PPO , SAC ) are not immune to local optima and require additional exploration mechanisms . We show that our planning method exhibits a better state space coverage , collects data that allows for better policies than D-RL methods without additional exploration mechanisms and that starting from the planner data and performing additional training results in as good as or better policies than vanilla D-RL methods , while also creating data that is more fit for re-use in modified tasks . 1 INTRODUCTION . Robots in human-centric environments are confronted with less structured , more varied and more quickly changing situations than in typical automated manufacturing environments . Research in autonomous robots adresses these challenges using modern machine learning methods . However , learning and trying out actions directly on a real robot is time-consuming and potentially dangerous to the environment as well as to the robot . In contrast , physically-based simulation provides the benefit of faster , cheaper , and safer ways for robot learning . If simulation models are available , they can be used by sampling-based planning methods that are able to directly plan robot behaviour using these models . However , the time required to perform planning can make this intractable for execution . Finding policies that directly map from the current state to the next applicable action eliminates the need for planning . While Deep-Reinforcement Learning ( D-RL ) has shown promising results , for example those by OpenAI et al . ( 2018 ) , D-RL training can be tedious and resource demanding . Plappert et al . ( 2017 ) report problems on the HalfCheetah environment where the algorithms converge to a local optimum corresponding to the cheetah wiggling on its back . They alleviated this problem by a different exploration scheme . In preliminary experiments ( not included in this paper ) we found similar problems : D-RL algorithms were not able to learn a pushing task with a simulated 7-DoF robot arm . The algorithms we used were Deep Deterministic Policy Gradient ( DDPG ) Lillicrap et al . ( 2015 ) and Proximal Policy Gradient ( PPO ) Schulman et al . ( 2017 ) ( from OpenAI Baselines by Dhariwal et al . ( 2017 ) ) . The algorithms were also not reaching relevant parts of the state space . Consequently , and in line with the findings of Plappert et al . ( 2017 ) we assume that part of the problem of failing to learn good policies is related to insufficient exploration . To remedy this problem , one might increase search time while keeping exploration noise high , or use more principled exploration . While increasing search time will in the limit also yield acceptable solutions , directed exploration appears more promising to find good solutions more reliably and in less time . We thus focus on the latter approach , as covering a more diverse area of the state space increases the chances of finding an optimal solution , and moving away from random or exhaustive search reduces the number of samples required to learn a good policy . Model-based methods can use their models of the task in an efficient way to plan over multiple steps and explore the state space in a more directed way . Given an accurate model , optimal policies can be produced without interacting with the world and thus with fewer samples ( Hester & Stone , 2012 ) . In particular , Rapidly Exploring Random Tree ( RRT ) are planning methods that focus on maximizing state-space exploration . We propose to take advantage of the benefits the aforementioned planning methods provide while tackling the problem of planning time by synthesizing the planning results into a policy . This essentially makes the proposed method a model-based method ( Sutton & Barto , 2018 ) . We will refer to this method as Planning for Policy Search ( PPS ) . This is of particular interest in domain-randomized training , where simulation models are always available , to increase the data efficiency of exploration . Here we investigate a preliminary version of this method that combines planning and policy search but does not perform randomizations yet . In particular , we investigate the following questions : Q1 How do the data generated by RRT compare to those from D-RL methods ? Do they cover a larger area of the state space ? Do the reward distributions differ ? Q2 Are PPS methods less susceptible to local optima than D-RL methods ? Q3 Can the data collected by PPS be reused more easily ? The experimental setup used to investigate these questions is described in Figure 1 . In a simulated environment , the planner and reinforcement learning agent are run – each separately – to generate environment interactions . In the case of the reinforcement learning agent a policy is learned , and its return is evaluated ( Q2 , Sec . 4.2 ) . In both cases , the collected data are stored as a dataset . In a second step , these datasets are analyzed with respect to their state-space coverage ( Q1 , Sec . 4.1 ) . Then the datasets are used to train an RL agent in an off-policy fashion . The returns of this agent ’ s policy are again evaluated ( Q1 , Sec . 4.1 ) . In a further experiment an agent is trained partially from these datasets and partially from experience it generates ( Q3 , Sec . 4.3 ) . 2 RELATED WORK . Using physically-based simulations for learning is limited by the necessity to approximate physical phenomena , causing discrepancies between simulated and real world results . This difference is called the reality gap and is a well-known problem in various fields of robotics . An important approach to cross the gap is Domain randomization ( Tobin et al. , 2017 ; Sadeghi & Levine , 2017 ; James et al. , 2017 ) : instead of one simulated environment , learning is done using a distribution of models with varying properties – such as for example mass , friction , shape , position , force/torque noise , etc . The idea is to make the behavior policies learned by the reinforcement learning process more robust to the differences within this distribution , thereby increasing robustness against the difference between the training distribution and the target domain , i.e . against the reality gap . The work from OpenAI has shown a successful use of domain randomization for learning in-handmanipulation , however the number of required training steps is raised by a factor of 33 ( OpenAI et al. , 2018 ) when domain randomizations are introduced . This increases the number of training steps to the magnitude of about 3.9·1010 from a magnitude of 1.2·109 – classical deep reinforcement learning approaches typically require 105 to 109 iterations of simulation steps – many algorithms are being tested on 106 timesteps , depending on the environment . The required amount of training data can make this method prohibitively expensive and typically the availability of a simulation model is not exploited . Improving the efficiency of domain randomization is an active topic of research , for example by using adversarial randomizations ( Mandlekar et al. , 2017 ) or limiting the training to stop before overfitting to idiosyncrasies of the simulation ( Muratore et al. , 2018 ) . There are also reinforcement learning methods that are more sample-efficient such as guided policy search ( Levine & Koltun , 2013 ) which is a model-based deep reinforcement learning method . In Guided Policy Search ( GPS ) , rollouts from a deep neural network controller are optimised by an optimal control method such as the iterative Linear Quadratic Regulator ( iLQR ) ( Todorov & Weiwei Li , 2005 ; Tassa et al. , 2012 ) method . However , guided policy search is , depending on the task , usually initialized from demonstrations since the exploration capabilities of the underlying optimization method ( iLQR ) are limited . Furthermore , the optimization method requires an applicable , engineered cost function which is able to guide the search procedure towards relevant solutions . The benefits from combining a model-based method with model-free reinforcement learning has been highlighted in Renaudo et al . ( 2014 ) . However their work focuses on discrete problems and the model-based method and the model-free algorithm are controlling the agent together whereas we address continuous RL problems where the planning method produces data for the policy learner . Affine Quadratic Regulator ( AQR ) -RRT ( Glassman & Tedrake , 2010 ) or LQR-RRT ( Perez et al. , 2012 ) are examples of RRT methods , which use a dynamics-based cost metric to guide the tree extension , making this methods able to deal with kinodynamic planning problems . The problem of performance is also recognized in planning and work is being undertaken to make RRT faster , for example by Wolfslag et al . ( 2018 ) . 3 METHOD . The evaluations are done on a simple , one-dimensional double integrator task where the goal is to move a point mass to a goal position . The environment is illustrated and described in more detail in Table 1 . The reason for this choice is that this environment is easy to visualize and problems present in a simple task are likely to manifest as well in a more complex setting . The environment contains two distinct goal locations . The agent receives a reward based on the distance to the goal points , where the reward is dominated by the distance to the closer one of the two goal locations . The placement of the goal locations is chosen such that the agent starts at a position where the gradient of the goal with the smaller reward is nonzero . This implies that simply maximizing the reward from the starting position will lead to a suboptimal policy and a smaller overall return . The goals are located at position −2.5 and 6.0 , both at 0.0 velocity . The placement is chosen such that a random action selection is more likely to stumble upon the lower reward solution ( −2.5 ) rather than the higher reward solution ( 6.0 ) . Figure 2 illustrates both the state space coverage in the case of uniformly-random actions without exploring starts and the reward received at each point in the state space . Although simple , this task is still relevant for robotics where the situation of a closer goal of less interest than a farther one is quite common ( e.g . two charging batteries where the closest charges the robot more slowly than the furthest ) . Simply following the gradient of the reward might Table 1 : Description of the 1D double-integrator test environment : a point mass M can be moved in a one-dimensional space X = ( position , velocity ) by applying a continuous-valued force . Reward is received based on the distance to two possible goal locations ( G1 , G2 ) . Dynamics X = [ x ẋ ] ẋ = Ax+Bu position velocity G1 Position wrapping d1 G2 d2 M A = [ 0 1 0 0 ] B = [ 0 1 ] Reward max ( ( 1− tanh |X −G∗1| ) , 2 ( 1− tanh |X −G∗2| ) ) G1 = [ −2.5 0.0 ] G2 = [ 6.0 0.0 ] Limits u ∈ [ −1 ; 1 ] x ∈ [ −10 ; 10 ] ẋ ∈ [ −2.5 ; 2.5 ] also lead the robot to get stuck at obstacles and thus prevent it from successfully completing the task . To have a broader baseline we included an exploring-starts variant of the environment where the initial state of the system is sampled uniformly from the state space . This is a sound algorithmic variant and easy to implement in this toy environment . However , in more complicated settings , such as a robot arm performing a pushing task , it may bring the agent into unreachable ( disconnected ) parts of the state space . Such unreachable parts could , for example , be locations that the robot can not reach from its initial position , or position-velocity configurations that would be damaging to the robot . Furthermore , this would also imply randomizing the state of the objects the robot interacts with , which requires additional engineering effort . This is contrary to what we want to achieve by learning – which is why we assume that in many applications exploring starts are undesirable or impractical . Unless otherwise noted , the algorithms are run for 105 environment steps ; the D-RL algorithms use 100-step episodes .

Summary: This paper suggested that conventional deep-reinforcement learning (D-RL) methods struggle to find global optima in toy problem when two local optima exist.  The authors proposed to tackle this problem using planning method (Rapidly Exploring Random Tree, RRT) to expand the search area. Since the collected data are not correlated with reward, it is more likely to find the global optima in toy problem with two local optima . As to the planning time problem,  they proposed to synthesize the planning results into a policy.