Reinforcement Learning with Probabilistically Complete Exploration

1 INTRODUCTION . Reinforcement Learning ( RL ) studies how agents can learn a desired behaviour by simply using interactions with an environment and a reinforcement signal . Central to RL is the long-standing problem of balancing exploration and exploitation . Agents must first sufficiently explore the environment to identify high-reward behaviours , before this knowledge can be exploited and refined to maximize long-term rewards . Many recent RL successes have been obtained by relying on well-formed reward signals , that provide rich gradient information to guide policy learning . However , designing such informative rewards is challenging , and rewards are often highly specific to the particular task being solved . Sparse rewards , which carry little or no information besides binary success or failure , are much easier to design . This simplicity comes at a cost ; most rewards are identical , so that there is little gradient information to guide policy learning . In this setting , the sample complexity of simple exploration strategies was shown to grow exponentially with state dimension in some cases ( Osband et al. , 2016b ) . Intuition behind this phenomenon can be gained by inspecting Figure 1a : exploration in regions where the return surface is flat leads to a random walk type search . This inefficient search continues until non-zero gradients are found , which can then be followed to a local optimum . Planning algorithms can achieve much better exploration performance than random walk by taking search history into account ( Lavalle , 1998 ) . These techniques are also often guaranteed to find a solution in finite time if one exists ( Karaman & Frazzoli , 2011 ) . In order to leverage the advantages of these methods , we formulate RL exploration as a planning problem in the state space . Solutions found by search algorithms are then used as demonstrations for RL algorithms , initializing them in regions of policy parameter space where the return surface is not flat . Figure 1b shows the importance of such good initialization ; surface gradients can be followed , which greatly facilitates learning . This paper brings the following contributions . We first formulate RL exploration as a planning problem . This yields a simple and effective method for automatically generating demonstrations without the need for an external expert , solving the planning problem by adapting the classic Rapidly-exploring Random Tree algorithm ( RRT ) ( Kuffner & LaValle , 2000 ) . The demonstrations are then used to initialize an RL policy , which can be refined with a classic RL method such as TRPO ( Schulman et al. , 2015 ) . We call the proposed method Rapidly Randomly-exploring Reinforcement Learning ( R3L ) 1 , provide theoretical guarantees for finding successful solutions 1Code will be made available on Github . and derive bounds for its sampling complexity . Experimentally , we demonstrate R3L improves exploration and outperforms classic and recent exploration techniques , and requires only a fraction of the samples while achieving better asymptotic performance . Lastly , we show that R3L lowers the variance of policy gradient methods such as TRPO , and verify that initializing policies in regions with rich gradient information makes them less sensitive to initial conditions and random seed . The paper is structured as follows : Section 2 analyzes the limitations of classic RL exploration . Section 3 describes R3L and provides theoretical exploration guarantees . Related work is discussed in Section 4 , followed by experimental results and comments in Section 5 . Finally , Section 6 concludes and gives directions for future work . 2 SPARSE-REWARD RL AS RANDOM WALK . Many recent RL methods are based on a policy gradient optimization scheme . This approach optimizes policy parameters θ with gradient descent , using a loss function L ( θ ) ( e.g . expected return ) and gradient g ( θ ) ≡ ∇θL ( θ ) . Since computing L ( θ ) exactly is intractable , it is common to use unbiased empirical estimators L̂ ( θ ) and ĝ ( θ ) , estimated from samples acquired by executing the policy . Optimization of θ then follows the common stochastic gradient descent ( SGD ) update-rule ( Bottou , 2010 ; Robbins & Monro , 1951 ) : θn+1 = θn − ĝ ( θn ) , where is the learning rate . The SGD update rule defines a discrete-time stochastic process ( Mandt et al. , 2017 ) . Note that ĝ is the mean of nmb i.i.d . samples . Following the central limit theorem , the distribution over ĝ is approximately ĝ ( θ ) ∼ N ( g ( θ ) , 1nmbC ( θ ) ) , meaning ĝ is an unbiased estimator of g with covariance 1 nmb C ( θ ) . Consequently , the update rule can be rewritten as ( Mandt et al. , 2017 ) : θn+1 = θn − g ( θn ) + nmb B∆W , ∆W ∼ N ( 0 , I ) . ( 1 ) Here we assume that C ( θ ) = C , i.e . approximately constant w.r.t . θ , and factorizes as C = BBT . SGD is efficient for high-dimensional problems as it offers almost dimension independent convergence rates ( Nesterov , 2018 ) . However , SGD requires non-zero gradients to guide the search towards the optimum θ∗ , i.e . |g ( θ ) | > g , ∀θ 6= θ∗ , g ∈ R. In the case of sparse-reward RL problems , such as in Figure 1 , much of the loss surface is flat . This leads to inefficient exploration of parameter space Θ , as the drift component in Eq . ( 1 ) g ≈ 0 , turning the SGD to a random walk in Θ ; ∆θ = nmbB∆W . Random walk is guaranteed to wander to infinity when dimensionality dΘ ≥ 3 ( Pólya , 1921 ; Kakutani , 1944 ) . However , the probability of it reaching a desired region in Θ , e.g . where g 6= 0 , depends heavily on problem specific parameters . The probability of θn ever reaching a sphere of radius r centered at C such that ‖C − θ0‖ = R > r is ( Dvoretzky & Erdős , 1951 ) : Pr { ‖θn − C‖ < r , for some n > 0 } = ( r R ) dΘ−2 < 1 , ∀dΘ ≥ 3 . ( 2 ) In sparse RL problems r < R , thus the probability of reaching a desired region by random walk is smaller than 1 , i.e . there are no guarantees of finding any solution , even in infinite time . This is in stark contrast with the R3L exploration paradigm , as discussed in Section 3.5 . 3 R3L : RAPIDLY AND RANDOMLY-EXPLORING REINFORCEMENT LEARNING . R3L adapts RRT to the RL setting by formulating exploration as a planning problem in state space . Unlike random walk , RRT encourages uniform coverage of the search space and is probabilistically complete , i.e . guaranteed to find a solution ( Kleinbort et al. , 2019 ) . R3L is decomposed into three main steps : ( i ) exploration is first achieved using RRT to generate a data-set of successful trajectories , described in Sections 3.2 and 3.3 , ( ii ) successful trajectories are converted to a policy using learning from demonstrations ( Section 3.4 ) , and ( iii ) the policy is refined using classic RL methods . 3.1 DEFINITIONS . This work is based on the Markov Decision Process ( MDP ) framework , defined as a tuple < S , A , T , R , γ > . S andA are spaces of states s and actions a respectively . T : S×A×S → [ 0 , 1 ] is a transition probability distribution so that T ( st , at , st+1 ) = p ( st+1|st , at ) , where the subscript t indicates the tth discrete timestep . R : S×A×S → R is a reward function defining rewards rt associated with transitions ( st , at , st+1 ) . γ ∈ [ 0 , 1 ) is a discount factor . Solving a MDP is equivalent to finding the optimal policy π∗ maximizing the expected return J ( π∗ ) = ET , π∗ [ ∑H t=0 γ tR ( st , at , st+1 ) ] for some time horizon H , where actions are chosen according to at = π∗ ( st ) . Lastly , let S be a Euclidean space , equipped with the standard Euclidean distance metric with an associated norm denoted by ‖·‖ . The space of valid states is denoted by F ⊆ S . 3.2 EXPLORATION AS A PLANNING PROBLEM WITH RRT . The RRT algorithm ( LaValle & Kuffner , 2001 ) provides a principled approach for planning in problems that can not be solved directly ( e.g . using inverse kinematics ) , but where it is possible to sample transitions . RRT builds a tree of valid transitions between states in F , grown from a root s0 . As such , the tree T maintains information over valid trajectories . The exploration problem is defined by the pair ( F , s0 ) . In RL environments with a known goal set Fgoal ⊆ F ( e.g . MountainCar ) , the exploration problem is defined by ( F , s0 , Fgoal ) . The RRT algorithm starts by sampling a random state srand ∈ S , used to encourage exploration in a specific direction in the current iteration . This necessitates the first of two assumptions . Assumption 1 . Random states can be sampled uniformly from the MDP state space S. Sampled states are not required to be valid , thus sampling a random state is typically equivalent to trivially sampling a hyper-rectangle . Then , the vertex snear = arg mins∈T‖srand − snear‖ is found . RRT attempts to expand the tree T from snear toward srand by sampling an action a ∈ A according to a steering function Υ : S × S → A . In many planning scenarios , Υ samples randomly from A . A forward step snew = f ( snear , a ) is then simulated from snear using action a , where f is defined by the transition dynamics . Being able to expand the tree from arbitrary snear relies on another assumption . Assumption 2 . The environment state can be set to a previously visited state s ∈ T. Although this assumption largely limits the algorithm to simulators , it has previously been used in Florensa et al . ( 2017 ) ; Nair et al . ( 2018 ) ; Ecoffet et al . ( 2019 ) for example ; see discussion in Section 6 on overcoming the limitation to simulated environments . snew is added as a new vertex of T , alongside an edge ( snear , snew ) with edge information a . This process repeats until a sampling budget k is exhausted or the goal set is reached ( i.e . snew ∈ Fgoal ) . Definition 1 . A valid trajectory is a sequence τ = [ s0 , a0 , s1 , . . . , stτ−1 , atτ−1 , stτ ] such that ( st , at , st+1 ) is a valid transition and st ∈ F , ∀t ∈ { 0 , 1 , . . . , tτ } . Whenever a goal set is defined , a successful valid trajectory end state satisfies stτ ∈ Fgoal . Algorithm 1 : R3L exploration Input : s0 , k : sampling budget Once planning is finished , a successful valid trajectory can easily be generated from T by retrieving all nodes between the leaf sleaf ∈ Fgoal and the root . Because tree T is grown from valid transitions between states in F , these trajectories are valid by construction .

Summary: This paper proposes a method, R3L, for exploration in reinforcement learning. R3L performs an exploration procedure before policy optimization. In R3L exploration, it considers the task as a planning problem, and applies RRT to find feasible solutions (trajectories). Then it applies a warm start procedure for policy optimization, where the feasible solutions found by RRT are used to initialize the policy by supervised learning. The paper provides theoretical guarantees of R3L exploration finding feasible solutions. Empirically, the algorithmic contribution is demonstrated by comparing with information theoretical exploration methods in benchmark control domains.