Generative Planning for Temporally Coordinated Exploration in Reinforcement Learning

1 INTRODUCTION . Reinforcement learning ( RL ) has received great success recently ( Mnih et al. , 2015 ; Lillicrap et al. , 2016 ; Gu et al. , 2017 ; Haarnoja et al. , 2018 ) , reaching or even surpassing human-level performances on some tasks ( e.g . Mnih et al. , 2015 ; Silver et al. , 2016 ) . However , the number of samples required for training is still quite large , rendering sample efficiency improvement a critical problem . Standard model-free RL methods operate at the granularity of individual time step . Although flexible , one prominent issue is inefficient exploration , as the consecutive one-step actions are not necessarily temporally consistent , which makes the probability of consistent movement decays exponentially with the exploration steps , rendering many of the exploration efforts wasted ( Dabney et al. , 2021 ) . Extensive efforts have been made on improving the temporal consistency of exploration from different perspectives with different approaches , including parameter space perturbation ( e.g . Plappert et al. , 2018b ; Fortunato et al. , 2018 ; Xie et al. , 2019 ) , enhancing correlation between policy steps ( e.g . Lillicrap et al. , 2016 ; Korenkevych et al. , 2019 ; Amin et al. , 2021 ; Zhang & Van Hoof , 2021 ) and skill/option-based exploration ( e.g . Agarwal et al. , 2018 ; Tirumala et al. , 2020 ; Pertsch et al. , 2020 ) . Among them , simple action repeat-based temporal persistency has been shown to be very successful ( e.g . Mnih et al. , 2015 ; Sharma et al. , 2017 ) . Action repeat refers to the strategy of repeating a previous action for a fixed or variable number of time steps . This simple strategy demonstrates strong performance on a broad range of tasks and has been actively investigated recently ( Neunert et al. , 2019 ; Dabney et al. , 2021 ; Biedenkapp et al. , 2021 ; Yu et al. , 2021 ) . Planning is another effective approach for obtaining a control policy alternative to RL ( e.g . Chua et al. , 2018 ; Srinivas et al. , 2018 ; Hafner et al. , 2019 ; Eysenbach et al. , 2019 ) . Typically , it is used to optimize a sequence of actions by by online minimization of a cost function computed together with a model ( Chua et al. , 2018 ; Srinivas et al. , 2018 ; Hafner et al. , 2019 ) . The control loop is closed with model-predictive control ( MPC ) , i.e . by executing the first action only and re-run the optimization again in the next time step ( Srinivas et al. , 2018 ; Chua et al. , 2018 ; Hafner et al. , 2019 ) . Planning excels at reasoning over long horizons and has a high sample efficiency , but is more computationally demanding and requires access to a model ( Levine & Koltun , 2013a ; b ; Lowrey et al. , 2019 ; Srinivas et al. , 2018 ; Chua et al. , 2018 ; Hafner et al. , 2019 ) . A large body of work in the literature has explored how to leverage the notable complementary strengths of RL and planning from many perspectives , e.g. , guiding policy learning with planning ( Levine & Koltun , 2013a ; b ) , improving planning with a learned policy ( Schrittwieser et al. , 2019 ; Wang & Ba , 2020 ) and generating intrinsic reward for exploration ( Sekar et al. , 2020 ) . This work is motivated by recognizing both the apparent complementary strengths and less apparent connections between these two streams of seemingly different approaches , i.e. , simple but successful action repeat strategy and flexible but more demanding planning-based control . We note that they can be connected from the perspective of planning for exploration . On the one hand , we can go beyond simply repeating the same actions by using planning to induce more flexible forms of action sequences for temporally coordinated exploration . On the other hand , the empirical success of action repeat suggests that the form of planning that is useful for exploration could be possibly done in a way that is lightweight and retains the overall model-free nature of the whole algorithm . Motivated by these observations , we propose a Generative Planing Method ( GPM ) as an attempt towards the goal of model-free RL with temporally coordinated , planning-like behavior for exploration . The overall framework is shown in Figure 1 . At the beginning of an episode , a plan τ is generated . The plan is a temporally extended sequence of actions and the one-step action for the current time step of execution is derived from it as ω ( τ ) . Upon entering the next time step , the old plan is adjusted by “ time-forwarding ” one step for alignment as ρ ( τ ) . The old plan is committed to until certain conditions are met when compared to a new one generated at each time step . By doing this , we have enabled both flexible intentional action sequences and temporally extended exploration . We refer this type of temporally extended and intentional exploration as temporally coordinated exploration . The contributions of this work can be summarized as follows : • we provide a natural perspective on extending standard model-free RL algorithms to generate multi-step action sequences for intentional exploration via planning by leveraging a connection between planning and temporally extended exploration . • we propose a generative planning method as a concrete instantiation and initial attempt in this direction ; • we verify the effectiveness of the proposed method by comparing it with a number of baseline methods ; • we show that the proposed method has additionally improved interpretability on several aspects , including both the evolvement of its exploration behavior and final policy behavior . 2 BACKGROUND . We will focus on continuous control tasks in this work . We first review some basics on model-free RL for continuous control , temporally extended exploration and model-based planning . 2.1 MODEL-FREE RL FOR CONTINUOUS CONTROL . Standard RL methods like SAC ( Haarnoja et al. , 2018 ) learn a policy function which maps the current state s to action a as π ( s ) → a . The policy is typically modelled with a neural network with learnable parameters θ and the optimization of the policy parameters is achieved by maximizing the following function w.r.t . θ : πθ = arg max θ Es∼DEa∼πθ [ Q ( s , a ) − α log πθ ( a|s ) ] . ( 1 ) where D denotes replay buffer and α is a temperature parameter . Q ( s , a ) denotes the soft state-action value function and is defined as : Q ( s , a ) = r̄ ( s , a ) + γEs′∼T ( s , a ) , a′∼πθ ( s′ ) [ Q ( s′ , a′ ) ] , where T ( s , a ) denotes the dynamics function and r̄ ( s , a ) the policy entropy augmented reward function ( Haarnoja et al. , 2018 ) . γ ∈ ( 0 , 1 ) is a discount factor . By definition , Q ( s , a ) represents the accumulated discounted future entropy augmented reward starting from s , taking action a , and then following policy πθ thereafter . 2.2 TEMPORALLY EXTENDED EXPLORATION MEETS MODEL-BASED PLANNING . The policy of the form in Eqn . ( 1 ) generates a one-step action at a time and the exploration is typically achieved by incorporating noise at single step level ( Haarnoja et al. , 2018 ) . While this is simple and flexible in representing the control policy , it might not be effective during exploration as the consecutive one-step actions are not necessarily temporally consistent , limiting its ability to escape local optima thus wasting many of the exploration efforts ( Dabney et al. , 2021 ) . Instead , a directed form of exploration can potentially mitigate this issue by generating consistent actions for a certain period of time ( Sutton et al. , 1999 ; Mnih et al. , 2015 ; Sharma et al. , 2017 ; Dabney et al. , 2021 ) . Build upon this philosophy , action repeat-based temporally extended exploration , despite of its simplicity , has been proven to be very effective and has been actively explored recently ( Sharma et al. , 2017 ; Dabney et al. , 2021 ; Biedenkapp et al. , 2021 ; Yu et al. , 2021 ) . Apart from directly improving the temporal property of the exploration behavior , another set of approaches guide policy optimization with model-based planning ( e.g . Levine & Koltun , 2013a ; b ; Lowrey et al. , 2019 ; Hoppe et al. , 2019 ) . In model-based planning , the optimized action for a particular time step is obtained by online minimization of a cost function . To better leverage the temporal structure of the cost function , planning is typically done for a span of time period by optimizing a sequence of actions τ = [ a0 , a1 , · · · ] ( e.g . Williams et al. , 2017 ; Srinivas et al. , 2018 ; Chua et al. , 2018 ; Hafner et al. , 2019 ; Schrittwieser et al. , 2019 ) : [ a0 , a1 , · · · ] = arg min τ= [ a0 , a1 , ··· ] C ( s , [ a0 , a1 , · · · ] ) . ( 2 ) This type of approaches has the potential to generate more intentional and flexible action sequences than repeating actions , as they are optimized at the sequence level . The generated action sequences is typically used with MPC by executing only the first action and discarding the rest . Also , the prerequisite of an additional model for calculating C and the online optimization of it indicate a large gap between this type of approaches and model-free RL . This makes the task of how to leverage planning for exploration in model-free RL more challenging . 3 GENERATIVE PLANNING METHOD FOR MODEL-FREE RL . Model-based planning can generate flexible plans , but requires online planning with a model which is computationally intensive . On the other hand , action-repeat is more efficient , but is less adaptive in its form of plans . This motivates us to present GPM , which supports flexible plans beyond repeating for exploration without online optimization and resembles typical model-free RL algorithms . We define a plan τ as a sequence of decisions that spans a number of time steps in the corresponding space A . For example , if A is the space of primitive actions , τ is a sequence of primitive actions , i.e . τ = [ a0 , a1 , · · · ] . If A is the spatial coordinate space as common in robotics control and driving , then τ is a sequence of spatial positions and is essentially a spatial-temporal trajectory . The framework of GPM is illustrated in Figure 1 . At the core of GPM is a plan generator Πθ ( s ) 7→ τ , which is a generative model that takes state s as input and outputs a plan τ ( c.f . Section 3.1 ) . In essence , it is a form of policy that outputs multi-step actions . ( therefore termed as generative planning ) . The plan generator is trained by maximizing plan values , which shapes it to generate plans that are more adaptive along the training process ( c.f . Section 3.2 ) . During rollout , at each time step t , give the observation st , the plan generator Π generates a plan τnew = [ at , at+1 , at+2 , · · · ] . Also , the old plan ( if available ) from the previous step is “ time-forwarded ” one step to generate τold = ρ ( τ ′ actual ) , where τ ′ actual denotes the actual plan from the previous time step . The replanning mechanism is based on the values of both plans τold and τnew to determine which one is used as the actual plan τactual ( c.f . Section 3.3 ) .

Summary: This paper presents a method called generative planning method (GPM) to improve exploration in RL. GPM performs exploration by learning a planner (a map from state to a sequence of action, aka a “plan”)) and performing MPC with a special rule for whether or not to use the latest plan or keep the old plan. The planner is an auto-regressive model (specifically, a stochastic RNN) and is trained to maximize an auto-regressive Q-function. Each time step, the plan from the previous time step is shifted forward by one time step and compared to the newly generated plan. The plans are compared using their predicted Q-values, and the policy switches plans with some probability that increases monotonically with Q(new plan) - Q(old plan). The exact likelihood is determined by a hyperparameter, l_commit_target, that, intuitively, sets a soft target for how long a plan is typically kept. The authors compare GPM to a variety of action-repeat-based exploration methods, from epsilon greedy policies to policies with learned action repeat counts (DAR & TAAC) and find that it is competitive with or outperforms these methods on various low-dimensional robot domains, as well as the image-based CARLA environment. The authors also visualize the trajectories and qualitatively show that GPM improves exploration.