Year: 2021

Abstract:
Standard model-free reinforcement learning algorithms optimize a policy that generates the action to be taken in the current time step in order to maximize expected future return. While flexible, it faces difficulties arising from the inefficient exploration due to its single step nature. In this work, we present Generative Planning method (GPM), which can generate actions not only for the current step, but also for a number of future steps (thus termed as generative planning). This brings several benefits to GPM. Firstly, since GPM is trained by maximizing value, the plans generated from it can be regarded as intentional action sequences for reaching high value regions. GPM can therefore leverage its generated multi-step plans for temporally coordinated exploration towards high value regions, which is potentially more effective than a sequence of actions generated by perturbing each action at single step level, whose consistent movement decays exponentially with the number of exploration steps. Secondly, starting from a crude initial plan generator, GPM can refine it to be adaptive to the task, which, in return, benefits future explorations. This is potentially more effective than commonly used actionrepeat strategy, which is non-adaptive in its form of plans. Additionally, since the multi-step plan can be interpreted as the intent of the agent from now to a span of time period into the future, it offers a more informative and intuitive signal for interpretation. Experiments are conducted on several benchmark environments and the results demonstrated its effectiveness compared with several baseline methods.

heading: 1 INTRODUCTION
text: Reinforcement learning (RL) has received great success recently (Mnih et al., 2015; Lillicrap et al., 2016; Gu et al., 2017; Haarnoja et al., 2018), reaching or even surpassing human-level performances on some tasks (e.g. Mnih et al., 2015; Silver et al., 2016). However, the number of samples required for training is still quite large, rendering sample efficiency improvement a critical problem. Standard model-free RL methods operate at the granularity of individual time step. Although flexible, one prominent issue is inefficient exploration, as the consecutive one-step actions are not necessarily temporally consistent, which makes the probability of consistent movement decays exponentially with the exploration steps, rendering many of the exploration efforts wasted (Dabney et al., 2021).
Extensive efforts have been made on improving the temporal consistency of exploration from different perspectives with different approaches, including parameter space perturbation (e.g. Plappert et al., 2018b; Fortunato et al., 2018; Xie et al., 2019), enhancing correlation between policy steps (e.g. Lillicrap et al., 2016; Korenkevych et al., 2019; Amin et al., 2021; Zhang & Van Hoof, 2021) and skill/option-based exploration (e.g. Agarwal et al., 2018; Tirumala et al., 2020; Pertsch et al., 2020). Among them, simple action repeat-based temporal persistency has been shown to be very successful (e.g. Mnih et al., 2015; Sharma et al., 2017). Action repeat refers to the strategy of repeating a previous action for a fixed or variable number of time steps. This simple strategy demonstrates strong performance on a broad range of tasks and has been actively investigated recently (Neunert et al., 2019; Dabney et al., 2021; Biedenkapp et al., 2021; Yu et al., 2021).
Planning is another effective approach for obtaining a control policy alternative to RL (e.g. Chua et al., 2018; Srinivas et al., 2018; Hafner et al., 2019; Eysenbach et al., 2019). Typically, it is used to optimize a sequence of actions by by online minimization of a cost function computed together with a model (Chua et al., 2018; Srinivas et al., 2018; Hafner et al., 2019). The control loop is closed with
model-predictive control (MPC), i.e. by executing the first action only and re-run the optimization again in the next time step (Srinivas et al., 2018; Chua et al., 2018; Hafner et al., 2019). Planning excels at reasoning over long horizons and has a high sample efficiency, but is more computationally demanding and requires access to a model (Levine & Koltun, 2013a;b; Lowrey et al., 2019; Srinivas et al., 2018; Chua et al., 2018; Hafner et al., 2019). A large body of work in the literature has explored how to leverage the notable complementary strengths of RL and planning from many perspectives, e.g., guiding policy learning with planning (Levine & Koltun, 2013a;b), improving planning with a learned policy (Schrittwieser et al., 2019; Wang & Ba, 2020) and generating intrinsic reward for exploration (Sekar et al., 2020).
This work is motivated by recognizing both the apparent complementary strengths and less apparent connections between these two streams of seemingly different approaches, i.e., simple but successful action repeat strategy and flexible but more demanding planning-based control. We note that they can be connected from the perspective of planning for exploration. On the one hand, we can go beyond simply repeating the same actions by using planning to induce more flexible forms of action sequences for temporally coordinated exploration. On the other hand, the empirical success of action repeat suggests that the form of planning that is useful for exploration could be possibly done in a way that is lightweight and retains the overall model-free nature of the whole algorithm.
Motivated by these observations, we propose a Generative Planing Method (GPM) as an attempt towards the goal of model-free RL with temporally coordinated, planning-like behavior for exploration. The overall framework is shown in Figure 1. At the beginning of an episode, a plan τ is generated. The plan is a temporally extended sequence of actions and the one-step action for the current time step of execution is derived from it as ω(τ). Upon entering the next time step, the old plan is adjusted by “time-forwarding” one step for alignment as ρ(τ). The old plan is committed to until certain conditions are met when compared to a new one generated at each time step. By doing this, we have enabled both flexible intentional action sequences and temporally extended exploration. We refer this type of temporally extended and intentional exploration as temporally coordinated exploration. The contributions of this work can be summarized as follows:
• we provide a natural perspective on extending standard model-free RL algorithms to generate multi-step action sequences for intentional exploration via planning by leveraging a connection between planning and temporally extended exploration.
• we propose a generative planning method as a concrete instantiation and initial attempt in this direction; • we verify the effectiveness of the proposed method by comparing it with a number of
baseline methods; • we show that the proposed method has additionally improved interpretability on several
aspects, including both the evolvement of its exploration behavior and final policy behavior.

heading: 2 BACKGROUND
text: We will focus on continuous control tasks in this work. We first review some basics on model-free RL for continuous control, temporally extended exploration and model-based planning.

heading: 2.1 MODEL-FREE RL FOR CONTINUOUS CONTROL
text: Standard RL methods like SAC (Haarnoja et al., 2018) learn a policy function which maps the current state s to action a as π(s)→ a. The policy is typically modelled with a neural network with learnable parameters θ and the optimization of the policy parameters is achieved by maximizing the following function w.r.t. θ:
πθ = arg max θ Es∼DEa∼πθ [Q(s, a)− α log πθ(a|s)]. (1) where D denotes replay buffer and α is a temperature parameter. Q(s, a) denotes the soft state-action value function and is defined as:
Q(s, a) = r̄(s, a) + γEs′∼T (s,a),a′∼πθ(s′) [ Q(s′, a′) ] ,
where T (s, a) denotes the dynamics function and r̄(s, a) the policy entropy augmented reward function (Haarnoja et al., 2018). γ ∈ (0, 1) is a discount factor. By definition, Q(s, a) represents the accumulated discounted future entropy augmented reward starting from s, taking action a, and then following policy πθ thereafter.

heading: 2.2 TEMPORALLY EXTENDED EXPLORATION MEETS MODEL-BASED PLANNING
text: The policy of the form in Eqn.(1) generates a one-step action at a time and the exploration is typically achieved by incorporating noise at single step level (Haarnoja et al., 2018). While this is simple and flexible in representing the control policy, it might not be effective during exploration as the consecutive one-step actions are not necessarily temporally consistent, limiting its ability to escape local optima thus wasting many of the exploration efforts (Dabney et al., 2021).
Instead, a directed form of exploration can potentially mitigate this issue by generating consistent actions for a certain period of time (Sutton et al., 1999; Mnih et al., 2015; Sharma et al., 2017; Dabney et al., 2021). Build upon this philosophy, action repeat-based temporally extended exploration, despite of its simplicity, has been proven to be very effective and has been actively explored recently (Sharma et al., 2017; Dabney et al., 2021; Biedenkapp et al., 2021; Yu et al., 2021).
Apart from directly improving the temporal property of the exploration behavior, another set of approaches guide policy optimization with model-based planning (e.g. Levine & Koltun, 2013a;b; Lowrey et al., 2019; Hoppe et al., 2019). In model-based planning, the optimized action for a particular time step is obtained by online minimization of a cost function. To better leverage the temporal structure of the cost function, planning is typically done for a span of time period by optimizing a sequence of actions τ = [a0, a1, · · · ] (e.g. Williams et al., 2017; Srinivas et al., 2018; Chua et al., 2018; Hafner et al., 2019; Schrittwieser et al., 2019):
[a0, a1, · · · ] = arg min τ=[a0,a1,··· ] C(s, [a0, a1, · · · ]). (2)
This type of approaches has the potential to generate more intentional and flexible action sequences than repeating actions, as they are optimized at the sequence level. The generated action sequences is typically used with MPC by executing only the first action and discarding the rest. Also, the prerequisite of an additional model for calculating C and the online optimization of it indicate a large gap between this type of approaches and model-free RL. This makes the task of how to leverage planning for exploration in model-free RL more challenging.

heading: 3 GENERATIVE PLANNING METHOD FOR MODEL-FREE RL
text: Model-based planning can generate flexible plans, but requires online planning with a model which is computationally intensive. On the other hand, action-repeat is more efficient, but is less adaptive in its form of plans. This motivates us to present GPM, which supports flexible plans beyond repeating for exploration without online optimization and resembles typical model-free RL algorithms.
We define a plan τ as a sequence of decisions that spans a number of time steps in the corresponding space A. For example, if A is the space of primitive actions, τ is a sequence of primitive actions, i.e. τ = [a0, a1, · · · ]. If A is the spatial coordinate space as common in robotics control and driving, then τ is a sequence of spatial positions and is essentially a spatial-temporal trajectory.
The framework of GPM is illustrated in Figure 1. At the core of GPM is a plan generator Πθ(s) 7→ τ , which is a generative model that takes state s as input and outputs a plan τ (c.f. Section 3.1). In essence, it is a form of policy that outputs multi-step actions. (therefore termed as generative planning). The plan generator is trained by maximizing plan values, which shapes it to generate plans that are more adaptive along the training process (c.f. Section 3.2). During rollout, at each time step t, give the observation st, the plan generator Π generates a plan τnew = [at, at+1, at+2, · · · ]. Also, the old plan (if available) from the previous step is “time-forwarded” one step to generate τold = ρ(τ ′ actual), where τ ′ actual denotes the actual plan from the previous time step. The replanning mechanism is based on the values of both plans τold and τnew to determine which one is used as the actual plan τactual (c.f. Section 3.3).

heading: 3.1 PLAN GENERATOR MODELING
text: It is natural to model the plan generator Π with an autoregressive structure to capture the inherent temporal relations between consecutive actions. In practice, we use a recurrent neural network (RNN) to model Π, with its structure shown in Figure 2 (a). More concretely, the plan generator Πθ(s) is implemented as a stochastic recurrent network as follows:
zt = encoder(st), ht = h(zt)
at = µmean(zt) + σstd(zt) · n n ∼ N (0, 1) at+i = f(ht+i, at+i−1), ht+i = RNN(ht+i−1, at+i−1) i ∈ [1, n− 1].
(3)
z denotes the feature extracted from s. h(z) denotes a network for initializing the state for RNN. In the special case when τ only contains a single action, Eqn.(3) reduces to the standard stochastic actor as used in Haarnoja et al. (2018), with µmean and σstd the network for predicting mean and standard derivation of the action distribution for the current time step. f denotes the action decoder for future time steps, which can be implemented with some prior incorporated when available. For example, we can implement f with a skip-connection to leverage the temporal structure: at+i = f(ht+i, at+i−1) = at+i−1 + g(ht+i, at+i−1) i ∈ [1, n− 1] (4) where g is implemented with neural networks. In this case, if we initialize g to generate small values, then we actually incorporated an action repeat prior to start with for exploration and learning. The
degrees of stochastic of Πθ(s) can be adjusted by matching a specified a target entropy in a way similar to SAC (Haarnoja et al., 2018), contributing a term J(α) into the total cost (c.f. Appendix A.1).

heading: 3.2 PLAN VALUE LEARNING AND PLAN GENERATOR TRAINING
text: We propose to optimize the plan generator by solving the following optimization problem:
Πθ ← arg min θ JΠ(θ) = arg min θ Es∼DEτ∼Πθ,l∼U(1,L)[−Q(s, τl)] (5)
where τl denotes sub-plan of length l extracted from τ , starting from its beginning. U(1, L) denotes a discrete uniform distribution and L denotes the maximum plan length. By taking expectation w.r.t. l ∼ U(1, L), we essentially train the plan generator by maximizing not only the value of the full plan, but also the value of the sub-plans. The plan value Q(st, τl) is defined as:
Q(st, τl)=Q(st, at, at+1, · · · , at+l−1)=rt+1 + γrt+2 + · · ·+ γlEs′∼T (s,τl),τ ′∼πθ(s′) [ Q(s′, τ ′) ] .
Here st denotes the current state, rt+1 = R(st, at), rt+2 = Est+1∼P (st,at)R(st+1, at+1), with R the reward function. P (st, at) and T (s, τ) denote the action-level and plan-level transition functions respectively. This formulation has a number of potential benefits: i) it can generate flexible and adaptive form of plans to be used for temporally extended exploration; ii) the generative approach used for plan generation retains the resemblance to typical model-free RL algorithms while avoids expensive online planning; iii) it gains additional interpretability because of its multi-step plan.
To train the plan generator, we use the re-parametrization trick as in SAC (Kingma & Welling, 2014; Schulman et al., 2015; Haarnoja et al., 2018), and also leverage the differentiability of the plan generator for gradient calculation: ∆θ = E(s,τ)∼D,l∼U(1,L) [ dτl dθ dQ(s,τl) dτl ] . Q needs to be learned in order to calculate ∆θ. In practice, we parametrize it with neural networks as Qφ, whose structure shown in Figure 2(b). More concretely, the plan value function is implemented as a recurrent network, taking the one-step action as input at each step, and outputting the plan value up to the current step. To leverage the temporal structure, we compute the plan value as the sum of the plan value from the previous step and the output from the decoder of the current step. Since there is no previous step value for the first step of plan, the decoder of the first step is not shared with the rest of plan steps. To train the plan value function Qφ, we use TD-learning as commonly used for value learning in RL algorithms (e.g. Haarnoja et al., 2018; Fujimoto et al., 2018)
Qφ ← arg minφ JQ(φ) = arg minφ E(s,τ)∼D,l∼U(1,L)‖Qφ(s, τl)− T(s, τl, r)‖22 (6) where T(s, τl, r) = rt+1+γrt+2+· · ·+γlEs′∼T (s,τl),τ ′∼πθ(s′) [ Qφ′(s′, τ ′) ] denotes the target value computed with a target critic network Qφ′ , with φ′ denoting the parameter values periodically copied from φ softly (Haarnoja et al., 2018). By taking expectation w.r.t. l ∼ U(1, L), we learn not only the value of the full plan, but also the value of the sub-plans, which is used in replanning. In addition, an entropy term (computed on the first step of the plan generator) is also added to the target value following Haarnoja et al. (2018). This form of learning has the following potential benefits: i) it can be used to train the plan generator as in Eqn.(5); ii) it can learn from trajectories sampled from a replay buffer, which is not necessarily aligned with the plans used during rollout (c.f. Appendix A.6.1); iii) it supports the evaluation of sub-plans, which is necessary for adaptive replanning, as detailed below.

heading: 3.3 TEMPORAL PLAN UPDATE AND REPLANNING
text: To encourage temporally coordinated exploration, we propose to commit to the current plan by default and switch if a new plan is better according to certain criteria. More concretely, at the current step, we have a plan candidate τold obtained from the plan τ ′actual of the previous step as τold = ρ(τ ′actual). In the typical case where the plan is consisted of a sequence of actions, The action to be executed at the current step is ω(τ) = τ [0], and ρ is simply a shifting operator extracting the rest of the plan: ρ(τ)=τ [1 :], serving as a plan candidate for the current time step. At the same time, we generate another new plan τnew ∼ Π(st). Now we can make a decision on which of the two candidates we shall use. If the length of τold (denoted as lold = |τold|) is zero, meaning the previous plan has been fully executed, then we just need to switch to τnew. Otherwise, we compare their plan values for determining a switch. Note that since τold is always shorter than τnew, the plan values need to be computed up to lold. Denoting the plan values under this setting as Qlold(s, τold) and Qlold(s, τnew), we then construct a categorical distribution for generating replanning signals as replan ∼ Categorical(z), where z denotes the logits and is defined as z , [ ,Qlold(s, τnew)−Qlold(s, τold)]. If replan=1, we switch to τnew; otherwise, we commit to
τold at the current time step. Intuitively, if the value of the new plan is much higher than that of the old one, a plan switch is preferred. Otherwise, it prefers to commit to the old plan. To adjust , we use an objective similar to the entropy-based temperature adjustment in SAC (Haarnoja et al., 2018):
J( ) = · (lcommit − lcommit_target), (7) where ≥ 0, lcommit_target is a target commit length and lcommit is the actual commitment length computed as the exponential moving average of commitment length during rollout. By doing this, we can better leverage the temporally coordinated plans from the plan generator for exploration, while allowing to switch plans if necessary. The overall algorithm is summarized in Appendix A.1.

heading: 4 RELATED WORK
text: Exploration with Temporally Correlated Action Sequences. Our work mainly lies in this category. Methods in this category focus on improving the correlations of consecutive actions to improve over simple but widely used exploration methods like -greedy (Dabney et al., 2021). Some methods focus on making the stochastic process underlying the exploration policy more temporally correlated (e.g. Lillicrap et al., 2016; Korenkevych et al., 2019). Action repeat strategy leads to a family of successful methods in this category by fully discarding action space randomness for the repeating steps after the first one (e.g. Mnih et al., 2015; Sharma et al., 2017; Neunert et al., 2019; Dabney et al., 2021; Biedenkapp et al., 2021; Yu et al., 2021). The proposed method is connected with action repeat in the sense that the source of stochasticity is also introduced in the first step of the action sequence, but it supports more flexible form of action sequences beyond repeating, and achieves this through plan generator training, rather than relying on certain type of specifically designed stochastic process as in Lillicrap et al. (2016); Korenkevych et al. (2019). This category of methods mostly retain the flat RL structure, in contrast to the more complicated hierarchical RL structures as reviewed below. Hierarchical RL. There is a large body of work on improving exploration and learning efficiency by operating at skill level with a higher level meta-controller (e.g. Dayan & Hinton, 1993; Sutton et al., 1999; Randlov, 1999; Vezhnevets et al., 2016; Bacon et al., 2017; Florensa et al., 2017; Co-Reyes et al., 2018; Konidaris et al., 2010; Pertsch et al., 2020). However, most hierarchical RL methods are nontrivial to adapt because of some task-specific aspects, e.g. the proper design of skill space, its form of representation and its way of acquisition (Dayan & Hinton, 1993; Sutton et al., 1999; Florensa et al., 2017; Co-Reyes et al., 2018; Konidaris et al., 2010; Pertsch et al., 2020). In contrast, the proposed method is a light-weight design that resembles standard model-free RL algorithms to a great extend, thus lies in a space that is mostly complementary to hierarchical RL methods. Goal-conditional RL. Goal conditional RL methods learn a goal-conditional policy and address the efficiency of policy learning by using techniques such hindsight experience replay. There are some recent efforts on generating goals beyond simply relabeling the experienced states (e.g. Ren et al., 2019; Pong et al., 2020; Pitis et al., 2020). In this setting, it resembles the hierarchical RL structure, where the goal generator plays the role of a high-level meta-controller and the goal-conditional policy functions as a low-level worker. Differently, we focus on how to improve action sequence based exploration for plain model-free RL algorithms without resorting to goal-oriented designs. Therefore, we view the proposed approach as orthogonal to goal-conditional RL and they could potentially be integrated together to bring further improvements. Model-based RL and Plan-to-Explore. Model-based RL is another large family of methods that has been explored from many perspectives (e.g. Sutton, 1991; Silver et al., 2016; Srinivas et al., 2018; Chua et al., 2018; Agarwal et al., 2018; Lowrey et al., 2019; Hoppe et al., 2019). i) Learning with latent dynamics. While some works may require a given model (Silver et al., 2016; Lowrey et al., 2019), many recent methods work with a latent dynamics model (e.g. Oh et al., 2017; Silver et al., 2017; Schrittwieser et al., 2019; Hafner et al., 2020). Our work is related to Oh et al. (2017); Silver et al. (2017); Schrittwieser et al. (2019) in the sense that our plan value function is trained by predicting future values, and related to Hafner et al. (2020) in the way of policy training by maximizing values. However, these approaches do not leverage their implicit models for temporally extended exploration, which is the focus of our work, thus are orthogonal to our work. Also, our value function has a dedicated latent dynamics model that is not shared with the plan generator, which offers more flexibilities in structural design (e.g. the usage of a structural prior Eqn.(4) in plan generator) and learning (each is responsible for its own objective). Therefore, the overall algorithmic structure resemble standard model-free RL methods, making the transition between them more natural. ii) Integrating model into model-free RL. There are many possible ways to integrate model into model-free RL methods (e.g. Sutton, 1991; Levine & Koltun, 2013a;b; Racanière et al., 2017;
Srinivas et al., 2018). Imagination-augmented agent (Racanière et al., 2017) uses the predictions from a model as additional contexts for a model-free policy. Srinivas et al. (2018) embeds differentiable planning within a goal-directed policy and optimizes the action plan through gradient descent. Our work is different in that it uses a plan generator to directly generate action sequences conditioned on the current state, much like a model-free policy, without using additional context from a model-based prediction (Racanière et al., 2017) or requiring expensive gradient based plan generation (Srinivas et al., 2018). iii) Plan-to-explore. The general idea of plan-to-explore is broadly applicable and has appeared in many categories including hierarchical (e.g. Konidaris et al., 2010; Pertsch et al., 2020), goal-conditional (e.g. Ren et al., 2019; Pong et al., 2020; Pitis et al., 2020) as well as model-based RL (e.g. Pathak et al., 2017; Henaff, 2019; Shyam et al., 2019; Sekar et al., 2020). In this work, we focus on improving standard model-free RL methods by generating coordinated action sequence for temporally extended exploration, which has a different focus and complementary to these approaches.

heading: 5 EXPERIMENTS
text: Baselines Details. We compare the performance of the proposed method with seven baseline methods: i) SAC: the Soft Actor Critic (Haarnoja et al., 2018) algorithm which is a representative model-free RL algorithm; ii) EZ: temporally-extended -Greedy exploration ( z-Greedy) (Dabney et al., 2021), which repeat the sampled action for a random duration during exploration; iii) FAR: Fixed Action Repeat (Mnih et al., 2015), which repeats the policy action for a fixed number of steps; iv) DAR: Dynamic Action Repeat with TempoRL (Biedenkapp et al., 2021), which improves over FAR with a action-conditional policy for predicting a duration to repeat, outperforming non-conditional alternatives (Sharma et al., 2017); v) TAAC: Temporally Abstract Actor Critic (Yu et al., 2021), which uses a switching policy for determining whether to repeat the previous action, instead of generating a duration to repeat beforehand as in DAR (Biedenkapp et al., 2021); vi) ICM: Intrinsic Curiosity Module (Pathak et al., 2017), which is a model-based method for reward bonus based exploration. vii) Dreamer: a model-based RL method which trains the policy by back-propagation through the learned latent dynamics (Hafner et al., 2020). SAC is used as the backbone algorithm for ii)∼vi).

heading: 5.1 EXPERIMENTS ON CONTINUOUS CONTROL
text: Benchmark Results. We conduct experiments on 12 benchmark tasks and the results are shown in Figure 3 (image-based CARLA results are deferred to Figure 4). It is observed that vanilla SAC can achieve reasonable performance on most of the tasks, albeit with a large performance variance and high sample complexity. FAR shows better performance than SAC on some tasks (e.g.
0.0 0.2 0.4 0.6 0.8 1.0 Environment Steps 1e7
BipedalWalker and FetchPush), but no clear improvements (e.g. InvertedPendulum) or even worse performance on some other tasks (e.g. CartpoleSwingUp). This is potentially because of the relatively more restrictive policy with a fixed repetition scheme. EZ outperforms SAC on some tasks (e.g. MountainCarContinuous and FetchPickAndPlace) and retain mostly comparable performance to SAC on others, demonstrating the contribution of incorporating temporal consistency to the exploration policy. ICM and Dreamer improves over SAC on some tasks but not always. Both DAR and TAAC show consistent improvements over SAC due to their flexibility on leveraging action repeat to the extend where it is useful. The proposed GPM demonstrates further improvements in terms of sample efficiency on some of the tasks, with final performance matching or outperforming those of the baseline methods. Apart from state-based control, GPM also delivers strong performance in a more challenging image-based control task (c.f. Figure 4), as detailed in Section 5.2. State Trajectory Visualization. To understand the behavior of different algorithms, we further visualize the state visitation distribution and its evolution during training (Figure 12). It shows that during the initial phase of training, the state coverage of the methods with extended temporal exploration is much higher than that of SAC, which are consistent with the results reported in (Dabney et al., 2021). Furthermore, as training progresses, GPM is able to find the high rewarding state more quickly, visually depicting its improved exploration performance and sample efficiency.

heading: 5.2 END-TO-END AUTONOMOUS DRIVING WITH EMERGED INTERPRETABLE PLANS
text: Autonomous Driving (AD) is an important application for learning-based algorithms. Although the end-to-end approaches have showed encouraging results (e.g. Bojarski et al., 2016; Toromanoff et al., 2020), one great limitation is their limited interpretability. Although GPM is also an end-to-end approach, it has improved interpretability, in addition to its better performance. It is important to note that this ability of generating interpretable plans is emerged from pure reinforcement learning, without direct supervisions. This is in clear contrast to the methods from the decomposed paradigm, which requires additional labels as direct supervision for training (Phan-Minh et al., 2020; Rhinehart et al., 2020). We use CARLA (Dosovitskiy et al., 2017) to setup a challenging image-based driving task. We set the plan length as L= 20 to enable more perceivable results when visualized. More details including observations, actions, rewards and network structure are presented in Appendix A.5. Behavior and Performance Comparison. The exploration behavior comparison of GPM with SAC and EZ are shown in Figure 4 (a). It can be observed that the effective range of exploration of a SAC agent increases slowly and is quite limited, demonstrating its inefficiency in exploration. EZ
shows enhanced exploration ability over SAC, as depicted by the enlarged range of explored areas. GPM shows a further improved exploration ability. More specifically, GPM shows some directional movement ability during the initial phase of training. It further shows reasonable turning behavior after about 100 training episodes and more complex behavior with multiple turns after about 150 training episodes. In terms of performance, it can be observerd from Figure 4 (b) that GPM learns faster than other baseline methods, with final performance better than or competitive to other methods. Emerged and Evolved Planning. The evolution of the plans from GPM is visually shown in Figure 5. It shows the emergence of meaningful plans and the evolution of them along the training process. This naturally depicts several advantages of GPM over standard end-to-end AD systems: i) it has more interpretable plans, which can serve as a media for communicating the intent of the algorithm; ii) the evolution of the algorithm behavior during training is more perceivable; iii) the plans, with grounded meaning of spatial-temporal trajectories, offer the opportunity of further adjustment according downstream requirements in the AD pipeline, which could potentially narrow down the gap between end-to-end and decomposed approaches and provide an opportunity to integrate. The plans from GPM together with the visual context images are provided in Figure 6. It can be observed that GPM plans are interpretable and meaningful. Note that this a type of emerged planning ability as there is no direct supervision on it but purely acquired through interactions with the environment.

heading: 6 CONCLUSION AND FUTURE WORK
text: We explore a perspective on using planning for inducing temporally extended, flexible and intentional action sequence for exploration in a model-free fashion. We present a Generative Planning method as an attempt towards this goal. The proposed approach features an inherent mechanism for generating temporally coordinated plans for exploration, while retaining the major features and advantages of typical model-free RL algorithms. We empirically verify the effectiveness of the proposed approach on a number of benchmark tasks. The proposed approach also shows improved exploration behavior and an acquired ability for generating interpretable plans. Results suggest that this is a promising direction that is worthy of pursuit further. We hope this work can serve as an initial encouraging step that can inspire more developments in this direction. For the proposed method, currently the plan generator is trained by a cost function (plan value function) that is unknown and estimated by purely interacting with the environment. Although this leads to reasonable and interpretable plans, the quality of the plans can potentially be further improved by learning a more refined cost with the help of additional data (Ziebart et al., 2009; Wulfmeier et al., 2016; Choi et al., 2019; Rosbach et al., 2019). While we have mainly focused on continuous control tasks in this work, it is possible to generalize the current model to handle discrete actions (Tennenholtz & Mannor, 2019). The network architecture also provides another axis for potential improvements. We leave the exploration of these interesting directions as our future work.

heading: A APPENDIX

heading: A.1 GPM ALGORITHM
text: The overall procedure of the proposed method is presented in Algorithm 1.
Algorithm 1 Generative Planning Method (GPM) Initialize: θ, φ φ′ ← φ, D ← ∅ for each iteration do
for each environment step do τold = ∅ if |τold| is 0 else τold ← ρ(τt−1) . time-forward the old plan τnew ∼ Π(st) . generate a new plan m = 1 if lold = |τold| is 0 else m∼Categorical([ ,Qlold(s, τnew))−Qlold(s, τold)]) τt = (1−m) · τold +m · τnew . select the actual plan from old and new plans at ← ω(τt) . extract the primitive action for the current step st+1 ∼ p(st+1|st, at) D ← D ∪ {(st, at, r(st, at), st+1)} . save transition to replay buffer end for for each train step do φ← φ− λ∇φJQ(φ) . plan value update θ ← θ − λ∇θJΠ(θ) . plan generator update α← α− λ∇αJ(α) . temperature update ← − λ∇ J( ) . replan threshold update φ′ ← ηφ+ (1− η)φ′ . target parameter soft-copy
end for end for
J(α) = Eat=ω(τ),τ∼Π [ −α log π0(at|st, α)− αH̄ ] , where π0 denotes the policy corresponding to the first step of the plan, which is similar to the one used in Haarnoja et al. (2018). H̄ is the target entropy set in the same way as SAC (Haarnoja et al., 2018) for controlling the degree of stochasticity of the policy by adjusting the temperature α.
A.2 DETAILS ON ρ AND ω OPERATORS
Here we summarize in Table 1 the meaning and implementations of some entities as well as operators that are used in this work to further help understanding.
In Table 1, T ego2world is an operator for transforming the plan from ego to world coordinate and T world2ego the reverse operator.

heading: A.3 NETWORK STRUCTURE AND DETAILS
text: Network details for different tasks are summarized in Table 2, where we use [K]×L to denote an MLP with L FC layers and each with the output size of K. For baseline methods, they use the network structure as shown in Table 2 for both actor and critic networks. For GPM, we implement the plan generator and plan value network as in Figure 2. The plan generator first use an MLP that has the same structure as baseline methods as shown in Table 2 for encoding the observation s to a feature vector z. z is used as the input of the Normal Layer, which uses FCs for generating mean and std vectors of a diagonal Gaussian distribution (Haarnoja et al., 2018). z is passed through a FC layer to the initial state of a GRU network (plan RNN), with its hidden size the same as the single layer size in Table 2. Action is detached before inputting to the GRU to to enforce the training of GRU and improve training stability. A shared FC layer is used for generating the actions after the first step, taking the output from plan RNN as input. The plan value network is implemented with LSTM (value RNN) with hidden size the same as the single layer size in Table 2. An MLP decoder with the same structure as those used in other baseline methods (Table 2) are used for plan value
estimation, taking the output from value RNN as decoder input. The first step uses one such decoder with parameters not shared with others. The rest of the steps uses a shared one. The reason for this is that the conditioning state for the first step is different from the rest. ReLU activation is used for hidden layers. Following standard practice, the minimum over the outputs of two value networks is used to mitigate value overestimation (Fujimoto et al., 2018; Haarnoja et al., 2018).

heading: A.4 GYM TASK DETAILS
text: The Gym tasks used in this paper and their corresponding Gym environment names are summarized in Table 3. Fetch tasks are from Plappert et al. (2018a).

heading: A.5 CARLA TASK DETAILS
text: The CARLA driving task is implemented based on the open source Car Learning to Act (CARLA) simulator (Dosovitskiy et al., 2017). More details on sensor/observation/action/reward, network structure etc. are presented in the sequel.

heading: A.5.1 SENSORS AND OBSERVATIONS
text: The following types of sensors are used in CARLA driving task (c.f. Figure 7):
• Camera: frontal camera capturing RGB image of size 64×64×3; • IMU sensor: 7D vector containing acceleration (3D), gyroscope (3D), compass (1D),
providing some measurements on the status of the current vehicle;
• Velocity: 3D vector containing velocity relative to ego-coordinate in m/s; • Navigation: the collection of the positions of 8 waypoints d meters away from the ego
vehicle along the route, with d ∈ {1, 3, 10, 30, 100, 300, 1000, 3000} meters. The positions of the waypoints are represented in 3D ego-coordinate. The sensory reading is a 24D vector;
1https://github.com/angelolovatto/gym-cartpole-swingup
• Goal: 3D position in ego-coordinate representing the target position the ego car should navigate to and stop at;
• Previous action: 2D vector representing the previous action. More details on the action space are provided in Appendix A.5.3.
The delta time between two time steps is ∆t = 0.05s, i.e., the perception and control rate is 20 Hz.

heading: A.5.2 CARLA NETWORK STRUCTURE
text: The network structure used in CARLA task is shown in Figure 7. This is the common structured shared by all the methods including GPM. Given the observation containing multi-modality signals, an observation encoder is first used to fuse them into a low-dimensional feature vector. This is achieved by processing each modality of signals with a dedicated encoder with the same dimensionality for the outputs. More specifically, the visual encoder is a ResNet-18 based CNN. Each of other modalities are encoded by a fully connected layer. The outputs from each of the encoder are fused together with a sum operator followed by a FC layer to produce the feature vector as input to downstream networks. The observation encoder is shared by both the actor and critic network. During training, the the gradient from the actor is stopped and the observation encoder is only trained by value learning.

heading: A.5.3 CARLA CONTROLLER AND ACTION SPACE
text: The CARLA environment with its controller interface is illustrated in Figure 8. It takes as input a setpoint in ego-coordinate, specifying an immediate position to go to for the current step. For the driving case, the setpoint can be represented in the 2D spatial ego-coordinate. Therefore there are two dimensions (x: lateral, y: longitudinal) in the action space, with a = (x, y) ∈ [−20, 20]× [−20, 20]. The actor controls the movement of the car by generating 2D action as the setpoint. Target direction and speed will be computed based on the setpoint and ego-position (origin), which will be passed to a PID-controller to produce control signals including throttle, steer, brake and reverse. These signals will be used by the simulator to generate the next state of the car. As part of the environment, a shrinkage is applied to the setpoint before computing the control signals:
ã = a · max(‖a‖ − δ, 0)‖a‖ . (8)
This is used to balance the portion of the braking behavior w.r.t. others in the control space. δ = 3 is used in this work.

heading: A.5.4 REWARD
text: The reward in CARLA environment is consisted of the following components:
• Route distance reward: this reward is computed as the distance traveled along route from the previous to the current time step.
• Collision reward: this is computed as min(collision_penalty, 0.5 ∗max(0, episode_reward)), where collision_penalty=20, episode_reward denotes the accumulated rewards up to now within the current episode. Issued only once for a contiguous collision event.
• Red-light reward: this is computed as min(red_light_penalty, 0.3 ∗max(0., episode_reward)), where red_light_penalty=20. Issued only once for a contiguous red light violation event.
• Success reward: a success reward of 100 will be received by the agent (ego-car) if its position is within 5 meters to the goal position and its speed is less than 10−3 m/s.

heading: A.5.5 EPISODE TERMINATION CONDITIONS
text: The episode terminates if either of the following conditions are satisfied:
• success: the car’s position is within 5 meters to the goal position and its speed is less than 10−3 m/s.
• the maximum number of allowed time steps K is reached: this number is computed as K = route_lengthmin_velocity×∆t , where the min_velocity = 5 m/s and ∆t = 0.05s. The state is marked as a non-terminal state, allowing bootstrapping for value learning at this state;
• stuck at collision: if the car is involved in the same collision event and does not move more than 1 meter after 100 time steps. The state is marked as a non-terminal state, allowing bootstrapping for value learning at this state.

heading: A.5.6 MAP AND EXAMPLE ROUTES
text: For CARLA driving task, we use Town01, which is a town with representative layout consisting a number of T junctions and intersections. The layout of Town01 is shown in Figure 9 (a). The size of the town is about 410m×344m. We also spawn 20 other vehicles controlled by autopilot and 20 pedestrians to simulate typical driving conditions involving not only static objects but also dynamic ones.
The goal location is randomly selected from the set of waypoints. The agent is spawned to a random valid waypoint location at the beginning of each episode. A waypoint is valid if it is on the road and is at least 10 meters away from any other actor in the world (vehicles and pedestrians). Some examples
routes are shown in Figure 9(b). As can be observed from the figure, the routes constructed in this way have a reasonable amount of diversity and cover representative driving scenarios. For example, different routes could include different number of turns, directions for turning, static objects (as the routes are different) and also different dynamic objects (e.g. other vehicles) at different space-time locations. The task of the agent is to navigate to the goal position following the route and stay still after arrived at the goal location.

heading: A.6 MORE IMPLEMENTATION DETAILS

heading: A.6.1 PLAN SAVING, SAMPLING AND GENERATION
text: In practice, for plan value learning, for each training iteration, we sample a batch of (state, plan, rewards, next-plan-state) tuple {sit, τ it , rit, sin}Ni=1 from the replay buffer. More concretely, for each state sit randomly sampled from the replay buffer, we sample an associated plan τ i t as the actions starting from the current step of sit to a variable number of future steps uniformly sampled in the range of 1 and maximum plan length L, i.e. τ it = [a i t, · · · , ait+l−1] where l ∼ uniform(1, L). The reward vector is consisted of the rewards along the plan rit = [r i t+1, · · · , rit+l]. The plan terminal state sin is the state at the time step after the last step of τ i t , i.e. s i n = s i t+l. Therefore, in effect, we use sampled plan for plan value learning. By constructing plans in this way, we have maximal flexibility of leveraging the action sequences from replay buffer with minimal assumption on the plan distribution. During training, lcommit_target is set to 0.5L.
Note that in the case of driving (and other similar scenarios), since the plan is a spatial trajectory in the ego-coordinate, and the plan update operator in this case is ρ(τ) = T world2ego ◦ T shift ◦ T ego2world(τ) = T world2ego◦T shift(τ ′) (c.f. Table 1). To enable the construction of plans from action sequences sampled from the replay buffer (not necessarily aligned with the plan boundary during rollout), we need to save the a′ = ω(τ ′) into replay buffer, i.e. D ← D ∪ {(st, a′t, r(st, at), st+1)}. a′ = ω(τ ′) actually denotes the setpoint in world coordinate. In this case, since the actions saved in replay buffer are in world coordinate, we can construct plans in world coordinate by by sampling sequences from the replay buffer, in the same way as described above, i.e. τ ′t = [at, · · · , a′t+l−1] where l ∼ uniform(1, L). Then the plans in ego-coordinate can be obtained as T world2ego(τ ′t). For the standard Gym tasks, since both T world2ego and T world2ego are identity operators, we have τ ′ ≡ τ , meaning we can simply save the current primitive action a = ω(τ) into replay buffer.
A scaling factor can be used for the replanning distribution Categorical(zκ ). We simply use κ=1. For testing the plan generator after training, the mode of the policy is used, by taking the mean of the Gaussian policy used in the plan generator and taking the mode of Categorical(z) for replanning.

heading: A.6.2 PLAN GENERATOR IN AUTONOMOUS DRIVING
text: We implement f used in plan generator as follows
at+i = at+i−1 + ∆at+i−1 + g(ht+i, at+i−1,∆at+i−1) i ∈ [1, n− 1] (9) where ∆at+i−1 = (at+i−1− at+i−2). This actually incorporates a linear prior into f , which is useful for tasks with position-based controls. A smoothing cost is also applied to the plan to minimize the maximum absolute value of its third order derivative max |...τ | (Kyriakopoulos & Saridis, 1988). In the most basic form, the recurrent plan generator can run at the scale of time-step in order to generate the full plan. In practice, it can be adjusted according to the actual situation, and running at a coarser scale, which has the advantage of lower-computational cost. For example, we can let the policy only output a few number (k) of anchor points and interpolate between them to get the full plan using a particular form of interpolation methods. Linear interpolation is used in this work.

heading: A.7 MORE RESULTS

heading: A.7.1 ABLATION RESULTS ON PLAN LENGTH AND TARGET COMMITMENT LENGTH
text: Impact of Plan Length. The results with different plan lengths are shown in Figure 10(a). It is observed that the overall the performance is relatively insensitive to plan length, although the performance could be further optimized by tuning this hyper-parameter.
Target Commitment Length. The results with different target commitment lengths are shown in Figure 10(b). The results indicate that the performance of the proposed method is relatively stable across different target commitment lengths, potentially due to its ability in generating plans with adaptive shapes.

heading: A.7.2 OBSERVATIONAL DROPOUT: THE IMPORTANCE OF ADAPTIVE PLANS
text: Here we compare the proposed method and action repeat under the setting where a fixed replanning duration is used. For action repeat, this corresponds to the FAR algorithm. For GPM, we also execute the full plan before switching (denoted as GPM-commit). This actually an instance of observational dropout (Freeman et al., 2019), which requires the algorithm to extrapolate controls generated from available observations to those time steps where observations are unavailable. The results are shown in Figure 11. This results highlight the importance of adaptive form of plans on performance. Fixed form of plan as used in FAR might be applicable to certain types of tasks (e.g. CartPoleSwingUp). However, for other tasks, it might be less appropriate and leads to significant performance decreases with increasing plan length (e.g., LunarLanderContinuous). In contrast, the proposed approach shows stable performance across a wide range of plan lengths. By contrasting the performance of FAR with GPM-commit, we highlight the importance on the adaptiveness of the plans.
A.7.3 MPC V.S. GPM
0.0 0.2 0.4 0.6 0.8 1.0 Environment Steps 1e7
100
0
100
200
300
A ve
ra ge
R et
ur n
GPM-MPC GPM
BipedalWalker
MPC has been typically used together with planning for obtaining the control policy (Lowrey et al., 2019). This strategy is appropriate when a model is given (Lowrey et al., 2019). However, for exploration in RL, MPC is less desirable as it essentially switches the plan at every step, not leveraging the temporal property of the plans, as shown on the right, by comparing with a variant of using MPC together with GPM (GPM-MPC).

heading: A.7.4 EXPLORATION TRAJECTORY VISUALIZATION AND COMPARISON
text: We show more results on exploration trajectories of different algorithms for Pendulum in Figure 12 and for CARLA-based driving task in Figure 13.

Review summaries:
This paper proposes a method for exploration called Generative Planning method (GPM), which generates a multi-step action sequence such that the exploration is more temporally consistent and "intentional" compared to regular single-step action noise exploration. The multi-step action sequence is output by a generator with an RNN structure, and the generator is optimized by maximizing the plan value function. The authors show that their method GPM performs better than some other methods in several continuous control tasks and present some interesting qualitative results (e.g. state trajectory) showing that the exploration is more effective.

This paper presents a method called generative planning method (GPM) to improve exploration in RL. GPM performs exploration by learning a planner (a map from state to a sequence of action, aka a “plan”)) and performing MPC with a special rule for whether or not to use the latest plan or keep the old plan. The planner is an auto-regressive model (specifically, a stochastic RNN) and is trained to maximize an auto-regressive Q-function. Each time step, the plan from the previous time step is shifted forward by one time step and compared to the newly generated plan. The plans are compared using their predicted Q-values, and the policy switches plans with some probability that increases monotonically with Q(new plan) - Q(old plan). The exact likelihood is determined by a hyperparameter, l_commit_target, that, intuitively, sets a soft target for how long a plan is typically kept. The authors compare GPM to a variety of action-repeat-based exploration methods, from epsilon greedy policies to policies with learned action repeat counts (DAR & TAAC) and find that it is competitive with or outperforms these methods on various low-dimensional robot domains, as well as the image-based CARLA environment. The authors also visualize the trajectories and qualitatively show that GPM improves exploration.

A generative planning method is proposed, which sits somewhere between model-free and model-based methods. No explicit model is learned. However, an action plan is generated using a recurrent action-plan generator, paired with a similarly recurrent critic.  At each environment step, a new action plan is generated, and  the current action plan can be abandoned in favor of the new action plan if the estimated benefit is large enough. The method builds on SAC.  Overall the benefits of the temporally extended action plans are: (a) temporally-coordinated exploration; (b) more effective than action-repeat; (c) some degree of interpretability given that an action-sequence plan represents then intent of a policy in a given state.

This paper proposes a method called Generative Planning (GPM), which aims to improve exploration for model-free RL. GPM learns a recurrent model to generate short term plans at each time step, and only decides to switch to the new plan if it is a lot better than the old plan. This encourages temporally extended explorations, and also does it in an adaptive manner. Experiments on a set of continuous control benchmarks show that GPM is able to converge faster than prior approaches, explore more effectively, and generate interpretable short-term plans.