Generative Planning for Temporally Coordinated Exploration in Reinforcement Learning

1 INTRODUCTION . Reinforcement learning ( RL ) has received great success recently ( Mnih et al. , 2015 ; Lillicrap et al. , 2016 ; Gu et al. , 2017 ; Haarnoja et al. , 2018 ) , reaching or even surpassing human-level performances on some tasks ( e.g . Mnih et al. , 2015 ; Silver et al. , 2016 ) . However , the number of samples required for training is still quite large , rendering sample efficiency improvement a critical problem . Standard model-free RL methods operate at the granularity of individual time step . Although flexible , one prominent issue is inefficient exploration , as the consecutive one-step actions are not necessarily temporally consistent , which makes the probability of consistent movement decays exponentially with the exploration steps , rendering many of the exploration efforts wasted ( Dabney et al. , 2021 ) . Extensive efforts have been made on improving the temporal consistency of exploration from different perspectives with different approaches , including parameter space perturbation ( e.g . Plappert et al. , 2018b ; Fortunato et al. , 2018 ; Xie et al. , 2019 ) , enhancing correlation between policy steps ( e.g . Lillicrap et al. , 2016 ; Korenkevych et al. , 2019 ; Amin et al. , 2021 ; Zhang & Van Hoof , 2021 ) and skill/option-based exploration ( e.g . Agarwal et al. , 2018 ; Tirumala et al. , 2020 ; Pertsch et al. , 2020 ) . Among them , simple action repeat-based temporal persistency has been shown to be very successful ( e.g . Mnih et al. , 2015 ; Sharma et al. , 2017 ) . Action repeat refers to the strategy of repeating a previous action for a fixed or variable number of time steps . This simple strategy demonstrates strong performance on a broad range of tasks and has been actively investigated recently ( Neunert et al. , 2019 ; Dabney et al. , 2021 ; Biedenkapp et al. , 2021 ; Yu et al. , 2021 ) . Planning is another effective approach for obtaining a control policy alternative to RL ( e.g . Chua et al. , 2018 ; Srinivas et al. , 2018 ; Hafner et al. , 2019 ; Eysenbach et al. , 2019 ) . Typically , it is used to optimize a sequence of actions by by online minimization of a cost function computed together with a model ( Chua et al. , 2018 ; Srinivas et al. , 2018 ; Hafner et al. , 2019 ) . The control loop is closed with model-predictive control ( MPC ) , i.e . by executing the first action only and re-run the optimization again in the next time step ( Srinivas et al. , 2018 ; Chua et al. , 2018 ; Hafner et al. , 2019 ) . Planning excels at reasoning over long horizons and has a high sample efficiency , but is more computationally demanding and requires access to a model ( Levine & Koltun , 2013a ; b ; Lowrey et al. , 2019 ; Srinivas et al. , 2018 ; Chua et al. , 2018 ; Hafner et al. , 2019 ) . A large body of work in the literature has explored how to leverage the notable complementary strengths of RL and planning from many perspectives , e.g. , guiding policy learning with planning ( Levine & Koltun , 2013a ; b ) , improving planning with a learned policy ( Schrittwieser et al. , 2019 ; Wang & Ba , 2020 ) and generating intrinsic reward for exploration ( Sekar et al. , 2020 ) . This work is motivated by recognizing both the apparent complementary strengths and less apparent connections between these two streams of seemingly different approaches , i.e. , simple but successful action repeat strategy and flexible but more demanding planning-based control . We note that they can be connected from the perspective of planning for exploration . On the one hand , we can go beyond simply repeating the same actions by using planning to induce more flexible forms of action sequences for temporally coordinated exploration . On the other hand , the empirical success of action repeat suggests that the form of planning that is useful for exploration could be possibly done in a way that is lightweight and retains the overall model-free nature of the whole algorithm . Motivated by these observations , we propose a Generative Planing Method ( GPM ) as an attempt towards the goal of model-free RL with temporally coordinated , planning-like behavior for exploration . The overall framework is shown in Figure 1 . At the beginning of an episode , a plan τ is generated . The plan is a temporally extended sequence of actions and the one-step action for the current time step of execution is derived from it as ω ( τ ) . Upon entering the next time step , the old plan is adjusted by “ time-forwarding ” one step for alignment as ρ ( τ ) . The old plan is committed to until certain conditions are met when compared to a new one generated at each time step . By doing this , we have enabled both flexible intentional action sequences and temporally extended exploration . We refer this type of temporally extended and intentional exploration as temporally coordinated exploration . The contributions of this work can be summarized as follows : • we provide a natural perspective on extending standard model-free RL algorithms to generate multi-step action sequences for intentional exploration via planning by leveraging a connection between planning and temporally extended exploration . • we propose a generative planning method as a concrete instantiation and initial attempt in this direction ; • we verify the effectiveness of the proposed method by comparing it with a number of baseline methods ; • we show that the proposed method has additionally improved interpretability on several aspects , including both the evolvement of its exploration behavior and final policy behavior . 2 BACKGROUND . We will focus on continuous control tasks in this work . We first review some basics on model-free RL for continuous control , temporally extended exploration and model-based planning . 2.1 MODEL-FREE RL FOR CONTINUOUS CONTROL . Standard RL methods like SAC ( Haarnoja et al. , 2018 ) learn a policy function which maps the current state s to action a as π ( s ) → a . The policy is typically modelled with a neural network with learnable parameters θ and the optimization of the policy parameters is achieved by maximizing the following function w.r.t . θ : πθ = arg max θ Es∼DEa∼πθ [ Q ( s , a ) − α log πθ ( a|s ) ] . ( 1 ) where D denotes replay buffer and α is a temperature parameter . Q ( s , a ) denotes the soft state-action value function and is defined as : Q ( s , a ) = r̄ ( s , a ) + γEs′∼T ( s , a ) , a′∼πθ ( s′ ) [ Q ( s′ , a′ ) ] , where T ( s , a ) denotes the dynamics function and r̄ ( s , a ) the policy entropy augmented reward function ( Haarnoja et al. , 2018 ) . γ ∈ ( 0 , 1 ) is a discount factor . By definition , Q ( s , a ) represents the accumulated discounted future entropy augmented reward starting from s , taking action a , and then following policy πθ thereafter . 2.2 TEMPORALLY EXTENDED EXPLORATION MEETS MODEL-BASED PLANNING . The policy of the form in Eqn . ( 1 ) generates a one-step action at a time and the exploration is typically achieved by incorporating noise at single step level ( Haarnoja et al. , 2018 ) . While this is simple and flexible in representing the control policy , it might not be effective during exploration as the consecutive one-step actions are not necessarily temporally consistent , limiting its ability to escape local optima thus wasting many of the exploration efforts ( Dabney et al. , 2021 ) . Instead , a directed form of exploration can potentially mitigate this issue by generating consistent actions for a certain period of time ( Sutton et al. , 1999 ; Mnih et al. , 2015 ; Sharma et al. , 2017 ; Dabney et al. , 2021 ) . Build upon this philosophy , action repeat-based temporally extended exploration , despite of its simplicity , has been proven to be very effective and has been actively explored recently ( Sharma et al. , 2017 ; Dabney et al. , 2021 ; Biedenkapp et al. , 2021 ; Yu et al. , 2021 ) . Apart from directly improving the temporal property of the exploration behavior , another set of approaches guide policy optimization with model-based planning ( e.g . Levine & Koltun , 2013a ; b ; Lowrey et al. , 2019 ; Hoppe et al. , 2019 ) . In model-based planning , the optimized action for a particular time step is obtained by online minimization of a cost function . To better leverage the temporal structure of the cost function , planning is typically done for a span of time period by optimizing a sequence of actions τ = [ a0 , a1 , · · · ] ( e.g . Williams et al. , 2017 ; Srinivas et al. , 2018 ; Chua et al. , 2018 ; Hafner et al. , 2019 ; Schrittwieser et al. , 2019 ) : [ a0 , a1 , · · · ] = arg min τ= [ a0 , a1 , ··· ] C ( s , [ a0 , a1 , · · · ] ) . ( 2 ) This type of approaches has the potential to generate more intentional and flexible action sequences than repeating actions , as they are optimized at the sequence level . The generated action sequences is typically used with MPC by executing only the first action and discarding the rest . Also , the prerequisite of an additional model for calculating C and the online optimization of it indicate a large gap between this type of approaches and model-free RL . This makes the task of how to leverage planning for exploration in model-free RL more challenging . 3 GENERATIVE PLANNING METHOD FOR MODEL-FREE RL . Model-based planning can generate flexible plans , but requires online planning with a model which is computationally intensive . On the other hand , action-repeat is more efficient , but is less adaptive in its form of plans . This motivates us to present GPM , which supports flexible plans beyond repeating for exploration without online optimization and resembles typical model-free RL algorithms . We define a plan τ as a sequence of decisions that spans a number of time steps in the corresponding space A . For example , if A is the space of primitive actions , τ is a sequence of primitive actions , i.e . τ = [ a0 , a1 , · · · ] . If A is the spatial coordinate space as common in robotics control and driving , then τ is a sequence of spatial positions and is essentially a spatial-temporal trajectory . The framework of GPM is illustrated in Figure 1 . At the core of GPM is a plan generator Πθ ( s ) 7→ τ , which is a generative model that takes state s as input and outputs a plan τ ( c.f . Section 3.1 ) . In essence , it is a form of policy that outputs multi-step actions . ( therefore termed as generative planning ) . The plan generator is trained by maximizing plan values , which shapes it to generate plans that are more adaptive along the training process ( c.f . Section 3.2 ) . During rollout , at each time step t , give the observation st , the plan generator Π generates a plan τnew = [ at , at+1 , at+2 , · · · ] . Also , the old plan ( if available ) from the previous step is “ time-forwarded ” one step to generate τold = ρ ( τ ′ actual ) , where τ ′ actual denotes the actual plan from the previous time step . The replanning mechanism is based on the values of both plans τold and τnew to determine which one is used as the actual plan τactual ( c.f . Section 3.3 ) .

Summary: This paper proposes a method called Generative Planning (GPM), which aims to improve exploration for model-free RL. GPM learns a recurrent model to generate short term plans at each time step, and only decides to switch to the new plan if it is a lot better than the old plan. This encourages temporally extended explorations, and also does it in an adaptive manner. Experiments on a set of continuous control benchmarks show that GPM is able to converge faster than prior approaches, explore more effectively, and generate interpretable short-term plans.