Generative Planning for Temporally Coordinated Exploration in Reinforcement Learning
Authors: Haichao Zhang
Wei Xu
Haonan Yu
Venue: ICLR 2022 Spotlight (2022)

Abstract: Standard model-free reinforcement learning algorithms optimize a policy that generates the action to be taken in the current time step in order to maximize expected future return. While flexible, it faces difficulties arising from the inefficient exploration due to its single step nature. In this work, we present Generative Planning method (GPM), which can generate actions not only for the current step, but also for a number of future steps (thus termed as generative planning). This brings several benefits to GPM. Firstly,  since GPM is trained by maximizing value, the plans generated from it can be regarded as intentional action sequences for reaching high value regions. GPM can therefore leverage its generated multi-step plans for temporally coordinated exploration towards high value regions, which is potentially more effective than a sequence of actions generated by perturbing each action at single step level, whose consistent movement decays exponentially with the number of exploration steps. Secondly, starting from a crude initial plan generator, GPM can refine it to be adaptive to the task, which, in return, benefits future explorations. This is potentially more effective than commonly used action-repeat strategy, which is non-adaptive in its form of plans. Additionally, since the multi-step plan can be interpreted as the intent of the agent from now to a span of time period into the future, it offers a more informative and intuitive signal for interpretation. Experiments are conducted on several benchmark environments and the results demonstrated its effectiveness compared with several baseline methods.

Published as a conference paper at ICLR 2022
GENERATIVE PLANNING FOR TEMPORALLY COORDI- NATED EXPLORATION IN REINNTTCEMENT LEARNING
Haichao Zhang Wei Haonan Yu
Horizon Robotics, Cupertino CA 95014
(haichac. zhang, wei.xu xu, haonan. yu) @horizon.ai
ABSTRACT
Standard model-free reinforcement learning algorithms optimize a policy that gen- erates the action to be taken in the current time step in order to maximize expected future return. While flexible, it faces difficulties arising from the inefficient explo- ration due to its single step nature. In this work, we present Generative Planning method (GPM), which can generate actions not only for the current step, but also for number of future steps (thus termed as generative planning). This brings several benefits to GPM. Firstly, since GPM trained by maximizing value, the plans generated from it can be regarded as intentional action sequences for reaching high value regions. GPM can therefore leverage its generated multi-step plans for temporally coordinated exploration towards high value regions, which is potentially more effective than a sequence of actions generated by perturbing each action at single step level, whose consistent movement decays exponentially with the number of exploration steps. Secondly, starting from a crude initial plan generator, GPM can refine it to be adaptive the task, which, in return, benefits future explorations. This is potentially more effective than commonly used action-repeat strategy, which is non-adaptive in its form of plans. Additionally, since the multi-step plan can be interpreted as the intent of the agent from now to a span of time period into the future, it offers a more informative and intuitive signal for interpretation. Experiments are conducted on several benchmark environments and the results demonstrated its effectiveness compared with several baseline methods. Code is available: https: //tptbb.iittt.... com / Ha chao bhhaang/geeerttt iatprattve__llelaaeeeeeee planning
INTRODUCTION
Reinforcement learning (RL) has received great success recently (Mnih et al. 2015 Lillicrap et al. 2016: Gu et al. 2017 Haamojj et al. 2018), reaching or even surpassing human-level performances on some tasks (e.g. Mnih et al. 2015 Silver et al. 2016). However, the number of samples required for training is still quite large, rendering sample efficiency improvement a critical problem. Standard model-free RL methods operate the granularity of individual time step. Although flexible, one prominent issue is inefficient exploration, as the consecutive one-step actions are not necessarily temporally consistent, which makes the probability of consistent movement decays exponentially with the exploration steps, rendering many of the exploration efforts wasted (Dabney et al. 202T).
Extensive efforts have been made improving the temporal consistency of exploration from different perspectives with different approaches, including parameter space perturbation (e.g. Plapperr et al. 2018b Fortunato al. 2018 Xie et al. 2019). enhancing correlation between policy steps (e.g. Lillicrap et al. 2016] Korenkevych et al. 2019 Amin et al. 2021 Zhang & Van Hoof 2021 exploration (e.g. Agarwa et al.] 2018 Tirumala al.
and skill/option-ban
et al.
2020). Among them, simple action repeat- based temporal persistency has been shown to be very successful (e.g. Mnih et al. 2015 Sharma et al. 2017). Action repeat refers to the strategy of repeating previous action for fixed or variable number of time steps. This simple strategy demonstrates strong performance on a broad range of tasks and has been actively investigated recently (Neunert
et al. 2019 Dabney al. 2021 Biedenkapp al. 202 1
Yu et al. (2021).
Planning is another effective approach for obtaining a control policy alternative to RL (e.g. Chua
al. 2018 Srinivas al. 2018 Hafner et al. 2019 Eysenbach al. 2019). Typically, it is used to

---

Published as a conference paper at ICLR 2022
plan
setpoint
Toww
ago posttion
setpoint
Told
Environment
Tactual
Figure 1: Generative Plamning Method. GPM has three main components. i) Plan generation. At each time step, the plan generator II generates plan Tnew based on the state s₁. ii) Plan update. Told is obtained by fiiee-forwarding the previous actual plan p( p(Tagtual) one step. iii) Plan switch. The replanning signal is generated based the values of the previous plan Told and the new Tnew. and only switches if certain criteria is satisfied. The plan enclosed with a black box indicates the one been selected at that time step. the case of driving, the primitive action be executed is derived from the plan as w (Tactual), which is sent to the environment for execution as shown on the right.
optimize sequence of actions by online minimization of cost function computed together with model (Chua et al.] 2018 [rriniaas al., 2018 Hafnnr et al. 2019). The control loop closed with -reee-pree control (MPC), f.e. by executing the first action only and re-run the optimization again the next time step. Planning excels at reasoning over long horizons and has a high sample efficiency, but is more computationally demanding and requires access to model (Levine & Koltun 2013a ( Lowrey et al. 2019) Srinivas et al., 2018 Chua et al., 2018 Hafner et al. 2019). A large body of work in the literature has explored how to leverage the notable complementary strengths of RL. and planning from many perspectives, e.g. guiding policy learning with planning (Levine &
Koltun 2013a b). improving planning with learned policy (Schrittwieser et al. 2019 Wang & Ba 2020) and generating intrinsic reward for exploration (Sekar et 2020)).
This work is motivated by recognizing both the apparent complementary strengths and less apparent connections between these two streams of seemingly different approaches, i.e., simple but successful action repeat strategy and flexible but more demanding planning-based control. We note that they can be connected from the perspective of planning for exploration. On the one hand, we can go beyond simply repeating the same actions by using planning to induce more flexible forms of action sequences for temporally coordinated exploration. On the other hand, the empirical success of action repeat suggests that the form of planning that useful for exploration could be possibly done in way that is lightweight and retains the overall model-free nature of the whole algorithm.
Motivated by these observations, we propose a Generative Planing Method (GPM) as an attempt to- wards the goal of model-free RI with temporally coordinated, planning-like behavior for exploration. The overall framework is shown in Figure [ The plan is temporally extended sequence of actions and the one-step action for the current time step of execution is derived from as w(T). Upon entering the next time step, the old plan is adjusted by "time-foree forwarding" conditions are met when compared to a new one generated at each time step. By doing this, we have enabled both flexible intentional action sequences and temporally extended exploration. We refer this type of temporally extended and intentional exploration as temporally coordinated exploration. The contributions of this work can be summarized follows:
At the beginning an episode, a plan 7 is generated.
step for alignment p(T). The old plan is committed to until certain
we provide natural perspective on extending standard model-free RL. algorithms to gen- erate multi-step action sequences for intentional exploration via planning by leveraging a connection between planning and temporally extended exploration.
we propose generative planning method as a concrete instantiation and initial attempt in this direction;
2

---

Published as a conference paper at ICLR 2022
verify the effectiveness of the proposed method by comparing it with number of baseline methods;
we show that the proposed method has additionally improved interpretability several aspects, including both the evolvement of its exploration behavior and final policy behavior.
2
BACKGROUND
We will focus continuous control tasks in this work. We first review some basics model-free RL for continuous control, temporally extended exploration and model-based planning.
2.1 MODEL-FREE RL FOR CONTINUOUS CONTROL
Standard RL methods like SAC (Haamoja al. 2018) learn policy function which maps the current state & to action a as (Ts)) + a. The policy is typically modelled with neural network with learnable parameters 0 and the optimization of the policy parameters achieved by maximizing the following function w.r.t. 0:
arg max E5-DBa-T [Q(s,a) a) - -a log mo(als))
(1)
where D denotes replay buffer and is temperature parameter. Q(s, a) denotes the soft state-action value function and is defined as:
Q(s, = = + ((.)) [Q(s',
(s)) a')],
where (8,, a) denotes the dynamics function and F(s, a) the policy entropy augmented reward
function (Haarnoja et al. 2018). (0, 1) is discount factor. By definition, Q(, a) represents the accumulated discounted future entropy augmented reward starting from 8, taking action o, and then following policy Te thereafter.
2.2 TEMPORALLY EXTENDED EXPLORATION MEETS MODEL-BASED PLANNING
The policy of the form in Eqn. Q generates one-step action at time and the exploration typically achieved by incorporating noise at single step level (Haarnoja et al. 2018). While this is simple and flexible in representing the control policy, might not effective during exploration as the consccutive one-step actions are not necessarily temporally consistent, limiting its ability to escape local optima thus wasting many of the exploration efforts (Dabney et al. 2021).
Instead, directed form of exploration can potentially mitigate this issue by generating consistent actions for a certain period of time (Sutton et al.) 1999, Mnih et al. 2015 Sharma et al. 2017 Dabney 1021. Build upon this philosophy, action repeat-
et al.
repeat-based based temporally extended exploration, despite
of its simplicity, has been proven to be very effective and has been actively explored recently (Sharma
al. 2017 Dabney al.
2021 [Biedenkapp al. 2021 Yu et al. 2021).
Apart from directly improving the temporal property of the exploration behavior, another set of approaches guide policy optimization with model-based planning (e.g. [evvine Koltun 2013a b)
Lowrey et al.) 2019 Hoppe et al., 2019). In model-based planning, the optimized action for
particular time step is obtained by online minimization of cost function. To better leverage the
temporal structure of the cost function, planning is typically done for a span of time period by [0g, a........... (e.g. Williams et al. 2017 Srinivas al. 2018
optimizing sequence of actions 7
Chua et al. 2018: Hafner et al. 2019 Schrittwieser et al. 2019):
[ag: 01,
arg
min
C(s, [ao, .I).
7= =(00,41.."
This type of approaches has the potential to generate more intentional and flexible action sequences than repeating actions, as they are optimized at the sequence level. The generated action sequences typically used with MPC by executing only the first action and discarding the rest. Also, the prerequisite of an additional model for calculating C' and the online optimization of indicate large gap between this type of approaches and model-free RL. This makes the task of how to leverage planning for exploration in model-free RI. more challenging.
3

---

Published as a conference paper at ICLR 2022
3 GENERATIVE PLANNING METHOD FOR MODEL-FREE RL
Model- badel-based planning can generate flexible plans, but requires online planning with a model which is computationally intensive. On the other hand, action-repeat is more efficient, but is less adaptive in its form of plans. This motivates us to present GPM, which supports flexible plans beyond repeating for exploration without online optimization and resembles typical model-free RL algorithms.
We define plan 7 as a sequence of decisions that spans a number of time steps in the corresponding space A. For example, if A is the space of primitive actions, T is a sequence of primitive actions, i.e. r [ag. 01. ... If is the spatial coordinate space as common in robotics control and driving. then r is a sequence of spatial positions and is essentially a mpatial-temporee trajectory.
The framework of GPM illustrated in Figure D At the core of GPM is plan generator [T)(s) b T. which is generative model that takes state as input and outputs a plan (c.f. Section 3.1. In essence, it is a form of policy that outputs multi-step actions. (therefore termed as generative planning). The plan generator is trained by maximizing plan values, which shapes it to generate plans that are more adaptive along the training process (c.f. Section 3.2). During rollout, at each time step t, give the observation St, the plan generator II generates a plan Tnew (as, 0++11 4x+2. .. Also, the old plan (if available) from the previous step is
one step to generate Told ((Tauual) where Tactual denotes the actual plan from the previous time step. The replanning
mechanism is based on the values of both plans Told and Tnew to determine which one is used as the actual plan Tactual (c.f. Section 3.3
81
ac+1
aj+2
42
0
0442
GC43
(a)
(b)
Q(+, a:)
(( Q6+ Rce))
Figure 2: Network Structures of (a) plan generator (generative actor) and (b) plan value function
(critic). (a) The plan generator generates a sequence of actions in an autoregressive way to form the full plan. For each step start from the second one, a skip-connection is added from the input previous action to the output action. (b) The plan value function takes state and plan as input, and outputs sequence of values along the plan. In this example, the plan value function takes 8 and 7 (ae, a++1, 4x+2, as input and outputs values [Q(s,, ax), Q(st, de, 1(+1), Qsst, Oz, 0x+1, (x+2), .. ... Q az) uses separate decoder different from the rest of steps + 1, + 2.... whose values are computed as the sum of the value from the previous step and the output from the decoder.
3.1 PLAN GENERATOR MODELING
natural to model the plan generator with an autoregressive structure to capture the inherent temporal relations between consecutive actions. In practice, we use recurrent neural network (RNN) to model II, with its structure shown in Figure (a). More concretely, the plan generator 11g (s) is implemented stochastic recurrent network as follows:
21 encoder(s:). h4 h(2))
0: = |mean (2)) + (2t) n
n n~ 1(,,1)
(3)
Ge++i [hetii a1ti-1), he++ RNN((
i [1, 1].
z denotes the feature extracted from s. f(z) denotes a network for initializing the state for RNN. In the special case when only contains single action, Eqn. ( reduces to the standard stochastic actor as used in Haarnoja et al. (2018), with Mmean and aad the network for predicting mean and standard derivation of the action distribution for the current time step. f denotes the action decoder for future time steps, which can be implemented with some prior incorporated when available. For example, we can implement f with skip-connection leverage the temporal structure:
ac+s= = f(he+é, a(+1-1) = 04+1-1 + g g(he+e, a(+s-1) i [1,n-1]
(4)
where g is implemented with neural networks. In this case, if we initialize g to generate small values, then we actually incorporated action repeat prior to start with for exploration and learning. The
4

---

Published as a conference paper at ICLR 2022
degrees of stochastic of [Ig(s) can be adjusted by matching specified target entropy in way similar to SAC (Haarnoja et al. 2018), contributing a term J(a) into the total cost (c.f. Appendix A.11
3.2
PLAN VALUE LEARNING AND PLAN GENERATOR TRAINING
We propose to optimize the plan generator by solving the following optimization problem:
T <- arg min Jn (0) arg min Bew (s,,)))
(5)
0
where 7. denotes sub-plan of length extracted from starting from its beginning. (11, L) denotes a discrete uniform distribution and L denotes the maximum plan length. By taking expectation w.r.t. I ~ (1, L), essentially train the plan generator by maximizing not only the value of the full plan, but also the value of the sub-plans. The plan value Q(s n) is defined as:
(st n) ($1, at, a++1,"
((+1-1)==1p + T2++2 + ... +
[Q(s', ,')].
Here 5, denotes the current state, r++1 R(st, az), Te42 = 1E..,1~(Pa. R($ ((+1), with R the reward function. P(st, ae) and Ts denote the action-level and plan-level transition functions respectively. This formulation has a number of potential benefits: i) it can generate flexible and adaptive form of plans to be used for temporally extended exploration: ii) the generative approach used for plan generation retains the resemblance to typical model-free RL algorithms while avoids expensive online planning; üi) it gains additional interpretability because of its multi-step plan.
To train the plan generator. we use the re- parametrization trick as in SAC (Kingma & Welling 2014 Schulman et al., 2015] Haarnoja et al.] 2018), and also leverage the differentiability of the plan
generator for gradient calculation: A ((..))-D.- dr, dQ(a.))
Q needs to be learned in dr
order to calculate A0. In practice, we parametrize it with neural networks as Qg. whose structure shown in Figure bb). More concretely, the plan value function is implemented as a recurrent network, taking the one-step action as input at each step, and outputting the plan value up to the current step. To leverage the temporal structure, we compute the plan value as the sum of the plan value from the previous step and the output from the decoder of the current step. Since there is no previous step value for the first step of plan, the decoder of the first step is not shared with the rest of plan steps. To train the plan value function Qa. we use TD-learnig as commonly used for value learning in RI. algorithms (e.g. Haarnoja et al. 2018: Fujimoto et al. 2018)
f arg min, (Ø) = arg min, ((arr) Qe 7)) T(s, n1 r
(6)
Qar (8', [')] denotes the target value computed with a target critic network Qa., with ø' denoting the parameter values periodically copied from & softly (Haarnoj et al. 2018). By taking expectation w.r.t. I U11, L), we learn not only the value of the full plan, but also the value of the sub-plans, which is used in replanning. In addition, an entropy term (computed on the first step of the plan generator) is also added to the target value following Haarnoja et al. (2018). This form of learning has the following potential benefits: i) it can be used to train the plan generator as in Eqn. 5): ii) it can learn from trajectories sampled from a replay buffer, which is not necessarily aligned with the plans used during rollout (c.f. Appendix[A.o iii) it supports the evaluation of sub-plans, which is necessary for adaptive replanning, as detailed below.
where T(s,, m, r) = r++1
3.3 TEMPORAL PLAN UPDATE AND REPLANNING
To encourage temporally coordinated exploration, we propose to commit to the current plan by default and switch if a new plan is better according certain criteria. More concretely, at the current step, we have a plan candidate Told obtained from the plan /' actual of the previous step as Told p(T In the typical case where the plan is consisted of sequence of actions, The action to be executed at the current step is w(7) = -[0]. and p is simply a shifting operator extracting the rest of the plan: p(r)=r(1 :). serving as plan candidate for the current time step. At the same time, we generate another new plan Tuew ~ I[(s)). Now we can make decision on which of the two candidates we shall use. If the length of Vold (denoted as lold = |Toia)) is zero, meaning the previous plan has been fully executed, then we just need to switch to Tnew- Otherwise, we compare their plan values for determining a switch. Note that since Fold is always shorter than Tnew, the plan values need to be computed up to lola. Denoting the plan values under this setting as Quea (s, Tola) and Quaa (s, Fnew), we then construct a categorical distribution for generating replanning signals replan Categorical(z)) z), where denotes the logits and is defined
as z e (e, Quea (8, knew) - Quaa (8, vola)]. If replan=1, switch to Tnew; otherwise, we commit to
5

---

Published as a conference paper at ICLR 2022
Told the current time step. Intuitively, if the value of the new plan is much higher than that of the old one, a plan switch is preferred. Otherwise, it prefers to commit to the old plan. To adjust e, we use an objective similar to the entropy-based temperature adjustment in SAC (Haarnoja et al. 2018):
J(c) where e 2 O, leommit target is a target commit length and loommir. is the actual commitment length computed as the exponential moving average of commitment length during rollout. By doing this, we can better leverage the temporally coordinated plans from the plan generator for exploration, while allowing to switch plans if necessary. The overall algorithm is summarized in Appendix A.I
(Icommit lccomitt_taraaae
RELATED WORK
Exploration with Temporally Correlated Action Sequences. Our work mainly lies in this category. Methods in this category focus on improving the correlations of consecutive actions to improve over simple but widely used exploration methods like e-greedy (Dabney al.. 2021) Some methods focus on making the stochastic process underlying the exploration policy more temporally correlated (e.g.
Lillicrp et al. 20166] Korenkevych et al. 2019). Action repeat strategy leads to family of successful methods in this category by fully discarding action space randomness for the repeating steps after the first one (e.g. Mnih al., [2015] Shharma al., 2017 Neunert et al., 2019 Dabney al., [202T] Biedenkapp et al. [021] Yu et al. (021)). The proposed method is connected with action repeat in the sense that the source of stochasticity is also introduced in the first step of the action sequence, but supports more flexible form of action sequences beyond repeating, and achieves this through plan generator training, rather than relying on certain type of specifically designed stochastic process as in inLiilicrap et al. (2 Korenkevych et al. (2019). This category of methods mostly retain the flat RL. structure, contrast to the more complicated hierarchical RI. structures as reviewed below.
Hierarchical RL. There is large body of work improving exploration and learning efficiency by
operating at skill level with a higher level meta-controller (e.g. Dayan & Hinton 1993; SSttonn et al.,
1999 Randlov] 1999 Vezhnevets al., [2066] [Bacon et 2017 Florenaa al., |
ct al. 2018; Konidaris et al. [2100] Pertsch et al. (2020). However, most hicrarchical RL methods are nontrivial to adapt because of some task-specific aspects, e.g. the proper design of skill space. its form of representation and its way of acquisition (Dayan & Hinton 1993) Sutton et al. 1999]
Florensa al., 2017 Co-Reyes et al., 2018 Konidaris et al. 2010] et al. 2020). In contrast, the proposed method is wiight design that resembles standard model free RL algorithms to a
great extend, thus lies a space that mostly complementary to hierarchical RL methods.
Goal-conditional RL. Goal conditional RL methods learn a goal-conditional policy and address the efficiency of policy learning by using techniques such hindsight experience replay. There are some recent efforts on generating goals beyond simply relabeling the experienced states (e.g. |Ren et al.. 2019 Pong et al. 2020 Pitis et al. 2020). In this setting, it resembles the hierarchical RL. structure, where the goal generator plays the role of high-level meta-controller and the goal-conditional policy functions as a low-level worker. Differently, we focus on how to improve action sequence based exploration for plain model-free RL algorithms without resorting goal-oriented designs. Therefore, we view the proposed approach as orthogonal to goal-conditional RL and they could potentially be integrated together to bring further improvements.
Model-based RL and Plan-to-Explore Model-based RL is another large family of methods that
has been explored from many perspectives (e.g. |Sutton 1991 Silver et al.] 2016 Srinivas et al.,
2018 Chua et al., 2018 Aggawal et al.) 2018 Lowrey et al. 2019 Hoppe et al. [009)) i) Learning
with latent dynamics. While some works may require a given model (Silver el al.] 2016 Lowrey
al. 2019), many recent methods work with a latent dynamics model (e.g. Oh al. 2011 Silver
ct al. 2017 Schrittwieser et al. 2019; Hafner et al. 2020). Our work is related to Oh et al. (2017); Silver et al. (
al. (2019) the sense that our plan value function is trained
by predicting future values, and related to Hafner et al. (2020) in the way of policy training by maximizing values. However, these approaches do not leverage their implicit models for temporally extended exploration, which is the focus of our work, thus are orthogonal to our work. Also, our value function has a dedicated latent dynamics model that is not shared with the plan generator, which offers more flexibilities in structural design (e.g. the usage of a structural prior Eqn.(4) in plan generator) and learning (each is responsible for its own objective). Therefore, the overall algorithmic structure resemble standard model-free RL methods, making the transition between them more natural. ii) Integrating model into model-free RL. There are many possible ways to integrate model into model-free RL methods (e.g. Sutton 1991 Levine & Koltun 2013a b Racanière et al. 2017

---

Published as a conference paper at ICLR 2022
Pendulum
MeuumainnCanConnimuuus
InvertooPoonuulum
InveriodDouzidoorddllumm
50080
2000
600
Cinices Stepp
Dinverent Shene
305
Drwiremet Shace
145
Eraronmant Stape
SaripcleSwingUp
BipedalWalker
FetchFeech
Aonsa Thare
Wessage Brom
200
-20
-20
Snvruwserr.oo Sarps
Envrrnment Stace
Endronmare Stape
Ensirommen Stape
FetchPush
FetohSide
FetchPiidknnPPace
-10
SAC
ICM
12
FAR
TAAC
Dreamer
Drweroment Dhagg
347
Envermmen Steps
Figure 3: Return Curves on benchmark continuous control tasks for different methods.
Srinivas et al. 2018). imagination-augmeeee agent (Racanière et al. (017) uses the predictions from model as additional contexts for a model-free policy. Srrinvass et al. (2018) embeds differentiable planning within a goal-directed policy and optimizes the action plan through gradient descent. Our work is different in that it uses a plan generator to directly generate action sequences conditioned on the current state, much like a model-free policy, without using additional context from a model- based prediction (Racanière al., (017) or requiring expensive gradient based plan generation (Srinivas et al. 2018). ii) Plan-to-explore: The general idea of plan-to-explore is broadly applicable and has appeared in many categories including hierarchical (e.g. Konidaris al. 2010 Pertsch et al. 2020). goal-conditional (e.g. Ren et al., 2019 Pong et al., 20200 Pitis et al. 2020) as well as model- based RL (e.g. Pathak al., 2017] 2019, (Syam et al. 2019] Sekar et al. 2020). In this work, we focus on improving standard model-free RL methods by generating coordinated action sequence for temporally extended exploration, which has a different focus and complementary to these approaches.
EXPERIMENTS
Baselines Details. We implement and compare the performance of the proposed method with seven baseline methods (Xu et al.).: : i) SAC: the Soft Actor Critic (Haarnoja et al., (018)) algorithm which representative modell-free RL algorithm; ii) EZ: lemporally-extended e-Greedy exploration (ez-Greedy) (Dabney et al. 2021). which repeat the sampled action for random duration during exploration; ii) FAR: Fixed Action Repeat (Mnih et al. 2015), which repeats the policy action for a fixed number of steps; iv) DAR: Dynamic Action Repeat with TempoRI. (Biedenkapp et al.) 202I), which improves over FAR with a action-conditional policy for predicting a duration to repeat, outperforming non-conditional alternatives (Sharma et al. 2017); TAAC: Temporally Abstract Actor Critic (Yu et al. 202T), which uses a switching policy for determining whether to repeat the previous action, instead of generating duration to repeat beforehand in DAR (Biedenkapp et al. 202T); vi) ICM: Intrinsic Curiosity Module (Pathak et al. 2017), which is a model-based method for reward bonus based exploration. vii Dreamer: model-based RL. method which trains the policy by pro-ppropagttion through the learned latent dynamics (Hafner et al.) 2020). SAC is used as the backbone algorithm for ti)>vi). More results and resources are available on the project page.
tttpa://siten http://aaaa......o..e..ecoecatte

---

Published as a conference paper at ICLR 2022
50 99 episodes
100-149 apisodes
150-198 opisades
CARLA
SAD
SAC
SAC
0-
123
GPM
GPM
GPM
GPM
02-
94
05
LG
(8)
200
(b)
Figure 4: Exploration Trajectory Visualization and Return Curves. (a) Each figure shows collec- tively the 50 rollout trajectories during training for the range depicted above. Trajectories are plotted relative its start position for easier visualization (unit:meter). The chronological order of positions in the trajectory is encoded in the same way F Visually, an effective agent should push the position of the red points as far as possible along its route. All plots including thumbnails are rendered at the same scale relative their axes. (b) Return curves of different methods.
5.1 EXPERIMENTS ON CONTINUOUS CONTROL
Benchmark Results. We conduct experiments on 12 benchmark tasks and the results are shown in Figure 3 (image-based CARLA results are deferred to Figure 4 It is observed that vanilla SAC can achieve reasonable performance on most of the tasks, albeit with large performance variance and high sample complexity. FAR shows better performance than SAC on some tasks (e.g. Bi Bipedalwar walkel and chtush))),, but no clear improvements (e.g. InvertedPendulun um) or even worse performance on some other tasks (e.g. the relatively more restrictive policy with a fixed repetition scheme. EZ outperforms SAC on some tasks (e.g. Mounta i ICaa rCont nuous and FetchP ck AndP1acc)) and retain mostly comparable performance to SAC on others, demonstrating the contribution of incorporating temporal consistency to the exploration policy. ICM and Dreamer improves over SAC on some tasks but not always. Both DAR and TAAC show consistent improvements over SAC due to their flexibility on leveraging action repeat to the extend where it is useful. The proposed GPM demonstrates further improvements in terms of sample efficiency on some of the tasks, with final performance matching or outperforming those of the baseline methods. Apart from state-based control, GPM also delivers strong performance in more challenging image-based control task (c.f. Figure as detailed in Section [.2
This is potentially because of
State Trajectory Visualization. To understand the behavior of different algorithms, we further visualize the state visitation distribution and its evolution during training (Figure [2). It shows
that during the initial phase of training, the state coverage of the methods with extended temporal exploration is much higher than that of SAC, which are consistent with the results reported in (Dabney itt al. 202T). Furthermore, as training progresses, GPM is able to find the high rewarding state more quickly, visually depicting its improved exploration performance and sample efficiency.
5.2 END-TO-END AUTONOMOUS DRIVING WITH EMERGED INTERPRETABLE PLANS
Autonomous Driving (AD) an important application for learning- based algorithms. Although the end-to-end approaches have showed encouraging results (e.g. Bojarski et al. 2016 Toromanoff ct al. 2020), one great limitation is their limited interpretability. Although GPM is also an end-to-end approach, it has improved interpretability, in addition to its better performance. It is important to note that this ability of generating interpretable plans is emerged from pure reinforcement learning, without direct supervisions. This is in clear contrast to the methods from the decomposed paradigm, which requires additional labels as direct supervision for training (Phan-Minh et al.) 2020 Rhinehart et al. 2020). We use CARLA (Dosovitskiy et al.) 2017) to setup challenging image- based driving task. We set the plan length as L= 20 to enable more perceivable results when visualized. More details including observations, actions, rewards and network structure presented in Appendix A.5 Behavior and Performance Comparison. The exploration behavior comparison of GPM with SAC and EZ are shown in Figure 4 (a). It can be observed that the effective range of exploration of a SAC agent increases slowly and is quite limited, demonstrating its inefficiency in exploration. EZ shows enhanced exploration ability over SAC, as depicted by the enlarged range of explored areas. GPM shows further improved exploration ability. More specifically, GPM shows some directional movement ability during the initial phase of training. It further shows reasonable turning behavior after about 100 training episodes and more complex behavior with multiple turns after about 150 training episodes. In terms of performance, it can be observerd from Figure (b) that GPM learns faster than other baseline methods, with final performance better than competitive other methods.
8

---

Published as a conference paper at ICLR 2022
Longtuoinal
Lateral
30
48
(b)
(C)
(d)
(e)
Figure 5: Evolution of GPM Plans: (a) initially the plans are nearly uniformly distributed across
all the directions; (b) forward-moving plans appear to take a larger portion afterwards; (c) further
training shows the emergence of plans with more adjustments plans; (d) later, more curved turning plans and plans with increased length appear; (e) at the end of training, the agent drive smoothly, therefore forward moving and turning plans take the majority.
speed, indicated by shorter length
front-vehicle
epo-vehicle
Figure Plan Visualization within an Episode. Note that the agent is trained with a frontal camera capturing 64x64 RGB images. The top-down view higher resolution is only used for visualization. The plan is color-coded as in Figure Frames switching to new plans are enclosed with a black box. At tt, a roughly straight plan is generated, and is committed next time step t3, the agent switches to new plan, passing a car in the front. This plan is committed by the agent (43 -> (4) and 4s, another new plan curved towards the turning direction is adopted. la and 17 show two frames while committing to the (s-plan for right turn. After that, the old plan is replaced by a new plan at (g, executed till ig and replaced by another one 100-
multiple time steps till t2. At the
Emerged and Evolved Planning. The evolution of the plans from GsM visually shown in Figure 5 shows the emergence of meaningful plans and the evolution them along the training process. This naturally depicts several advantages of GPM over standard end-to-end AD systems: i) it has more interpretable plans, which can serve as a media for communicating the intent of the algorithm; ii) the evolution of the algorithm behavior during training is more perceivable; fii) the plans, with grounded meaning of spatial-temporal trajectories, offer the opportunity of further adjustment according downstream requirements in the AD pipeline, which could potentially narrow down the gap between end-to-end and decomposed approaches and provide an opportunity to integrate. The plans from GPM together with the visual context images are provided in Figure It can observed that GPM plans are interpretable and meaningful. Note that this a type of emerged planning ability as there is no direct supervision on it but purely acquired through interactions with the environment.
CONCLUSION AND FUTURE WORK
We explore a perspective on using planning for inducing temporally extended, flexible and intentional action sequence for exploration a generative fashion. We present Generative Planning method as an attempt towards this goal. The proposed approach features an inherent mechanism for generating temporally coordinated plans for exploration, while retaining the major features and advantages of typical model-free RI. algorithms. We hope this work can serve an initial encouraging step that can inspire more developments in this direction. For future work, the quality of the plans can potentially be further improved by learning more refined cost with the help of additional data (Ziebart et al.) 2009; Wulffeier et al., 2016 Choi et al., 2019; Rosbach et al. 2019). While we have mainly focused continuous control tasks in this work, possible to generalize GPM to handle discrete actions (Tennenholtz & Mannor 2019). The network architecture also provides another axis for potential improvements. We leave the exploration of these interesting directions future work.

---

Published as a conference paper at ICLR 2022
Acknowledgements. We would like to thank Jingchu Liu, Lisen Mu, Jenry Bai, Le Zhao, Break Yang, Lingfeng Sun and anonymous reviewers for discussions and feedback on the project and paper. We would also like to thank the Horizon AI platform team for infrastructure support.
REFERENCES
Arpit Agarwal, Katharina Muelling, and Katerina Fragkiadaki. Model learning for look-ahead exploration in continuous control. CoRR, abs/181 811.0806, 2018.
Susan Amin, Maziar Gomrokchi, Hossein Aboutalebi, Harsh Satija, and Doina Precup. Locally persistent exploration in continuous control tasks with sparse rewards. In Marina Meila and Tong Zhang (eds.), International Conference on Machine Learning, 2021.
Pierre-Lue Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence. 2017.
André Biedenkapp. Raghu Rajan, Frank Hutter, and Marius Lindauer. TempoRL: Learning when to act. In International Conference on Machine Learning, 2021.
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D. Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, Xin Zhang, Jake Zhao, and Karol Zieba. End to end learning for self-driving cars. CoRR, 2016.
Dooseop Choi, Kyoungwook Min, and Jeongdan Choi. Regularizing neural networks for future trajectory prediction via inverse reinforcement learning. CoRR, 2019.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems, 2018.
John D. Co-Reyes, Yuxuan Liu, Abhishek Gupta, Benjamin Eysenbach, Pieter Abbeel, and Sergey Levine. Self-consistent trajectory autoencoder; Hierarchical reinforcement learning with trajectory embeddings. Intermational Conference on Machine Learning, 2018.
Will Dabney, Georg Ostrovski, and Andre Barreto. Temporally-extended s-greedy exploration. In International Conference Learning Representations, 2021.
Peter Dayan and Geoffrey Hinton. Feudal reinforcement learning. In Advances in Neural Informa- tion Processing Systems, 1993.
Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. CARLA: An open urban driving simulator. In Conference on Robot Learning, 2017.
Ben Eysenbach, Russ R Salakhutdinov, and Sergey Levine. Search on the replay buffer: Bridging planning and reinforcement learning. In Advances in Neural Information Processing Systems, 2019.
Carlos Florensa, Yan Duan, and Pieter Abbeel. Stochastic neural networks for hierarchical reinforce- ment learning. In International Conference on Machine Learning, 2017.
Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Matteo Hessel, Ian Osband, Alex Graves, Volodymyr Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, Charles Blundell, and Shane Legg. Noisy networks for exploration. In International Conference on Learning Representations, 2018
C. Daniel Freeman, David Ha, and Luke Metz. Learning to predict without looking ahead: World models without forward prediction. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché- Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems, 2019.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International Conference Machine Learning, 2018.
10

---

Published as a conference paper at ICLR 2022
Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine. Deep reinforcement learning for robotic manipulation with asynchronous off- policy updates. In IEEE Interaational Conference on Robotics and Automation, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International Conference on Machine Learning, 2018
Danijar Hafner, Timothy P. Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning, 2019.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference Learning Representations, 2020.
Mikael Henaff. Explicit explore-exploit algorithms in contimuous state spaces. In NeurlPS, 2019.
Sabrina Hoppe, Zhongyu Lou, Daniel Hennes, and Marc Toussaint. Planning approximate exploration trajectories for model-free reinforcement learning in contact-rich manipulation. IEEE Robotics Autom, Lett., 4(44:4044 2019.
Diederik P. Kingma and Max Welling. Auto-encoding variational Bayes. International Conference on Learning Representations, 2014.
George Konidaris, Scott Kuindersma, Andrew Barto, and Roderic Grupen. Constructing skill trees for reinforcement learning agents from demonstration trajectories. In International Conference on Neural Information Processing Systems, 2010.
Dmytro Korenkevych, A. Rupam Mahmood, Gautham Vasan, and James Bergstra. Autoregressive policies for continuous control deep reinforcement learning. In International Joint Conference on Artificial Intelligence, 2019.
K.J. Kyriakopoulos and G.N. Saridis. Minimum jerk path generation. In IEEE International Conference on Robotics and Automation, 1988.
Sergey Levine and Vladlen Koltun. Guided policy search. In International Conference on Machine Learning, 2013a.
Sergey Levine and Vladlen Koltun. Variational policy search via trajectory optimization. In Advances in Neural Information Processing Systems, 2013b.
Timothy P. Lillicrap. Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In International Conference Learning Representations, 2016.
Kendall Lowrey, Aravind Rajeswaran, Sham Kakade, Emanuel Todorov, and Igor Mordatch. Plan online, learn offline: Efficient learning and exploration via model-based control. In International Conference on Learning Representations, 2019.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529-55 533, February 2015.
Michael Neunert, Abbas Abdolmaleki, Markus Wulfmeier, Thomas Lampe, Jost Tobias Springenberg, Roland Hafner, Francesco Romano, Jonas Buchli. Nicolas Heess, and Martin A. Riedmiller. Continuous-discres reinforcement learning for hybrid control in robotics. Conference on Robot Learning, 2019.
Junhyuk Oh, Satinder Singh, and Honglak Lee, Value prediction network. In Advances in Neural Information Processing Systems, 2017.
11

---

Published as a conference paper at ICLR 2022
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven driven exploration by self-supervised prediction. In International Conference on Machine Learning, 2017.
Karl Pertsch, Youngwoon Lee, and Joseph J. Lim. Accelerating reinforcement learning with learned skill priors. In Conference on Robot Learning, 2020.
Tung Phan-Minh, Elena Corina Grigore, Freddy A. Boulton, Oscar Beijhom, and Eric M. Wolff. Covernet: Multimodal behavior prediction using trajectory sets. In IEEE Conference on Computer Vision and Pattern Recognition, 2020.
Silviu Pitis, Harris Chan, Stephen Zhao, Bradly C. Stadie, and Jimmy Ba. Maximum entropy gain exploration for long horizon multi-goal reinforcement learning, In International Conference on Machine Learning. 2020.
Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Joshua Tobin, Maciek Chociej, Peter Welinder, Vikash Kumar, and Wojciech Zaremba. Multi-goal reinforcement learning: Challenging robotics environments and request for research. ArXiv, abs/1802.09464, 2018a.
Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Y. Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, and Marcin Andrychowicz. Parameter space noise for exploration. In International Conference on Learning Representations, 2018b.
Vitchyr Pong, Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine. Skew-fit: State-covering self-supervised reinforcement learning. In International Conference on Machine Learning, 2020.
Sébastien
Racanière, Theophane Weber, David P. Reichert, Lars Buesing, Arthur Guez, Danilo Jimenez Rezende, Adrià Puigdomènech Badia, Oriol Vinyals, Nicolas Heess, Yujia Li, Raz- van Pascanu, Peter W. Battaglia, Demis Hassabis, David Silver, and Daan Wierstra. Imagination- augmented agents for deep reinforcement learning. In Advances in Neural Information Processing Systems, 2017.
Jette Randlov. Processing Systems, 1999.
Learning macro-actions in reinforcement learning. Advances in Neural Information
Zhizhou Ren, Kefan Dong, Yuan Zhou, Qiang Liu, and Jian Peng.
Exploration via hindsight goal generation. In Advances in Neural Information Processing Systems, 2019.
Nicholas Rhinehart, Rowan McAllister, and Sergey Levine. Deep imitative models for flexible inference, planning, and control. In International Conference on Learning Representations, 2020.
Sascha Rosbach, Vinit James, Simon Großjohann, Silviu Homoceanu, and Stefan Roth. Driving with style: Inverse reinforcement learning in general-purpose planning for automated driving. CoRR, abs/1905.00229, 2019.
Julian Schritwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy P. Lillicrap, and David Silver. Mastering atari, go, chess and shogi by planning with learned model. CoRR. abs/191 0bbs/191.08220 1.08265, 2019.
John Schulman, Nicolas Heess, Theophane Weber, and Pieter Abbeel. Gradient estimation using stochastic computation graphs. In Advances Neural Information Processing Systems, 2015.
Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via self-supervised world models. In International Conference Machine Learning, 2020.
Sahil Sharma, Aravind S. Lakshminarayanan, and Balaraman Ravindran. Learning to repeat: Fine grained action repetition for deep reinforcement learning. In Internationaa Conference Learning Representations, 2017.
Pranav Shyam, Wojeiech Jaškowski, and Faustino Gomez. Model-based active exploration. In
International Conference Machine Learning, 2019.
12

---

Published as a conference paper at ICLR 2022
David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Mare Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, 1lya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of Go with deep neural networks and tree search. Nature, pp- 484 489, January 2016.
David Silver, Hado van Hasselt, Matteo Hessel, Tom Schaul, Arthur Guez, Tim Harley, Gabriel Dulac-Arnold, David Reichert, Neil C. Rabinowitz, André Barreto, and Thomas Degris. The predictron: End-to-end learning and planning. In International Conference on Machine Learning, 2017.
Aravind Srinivas, Allan Jabri, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Universal plan- ning networks: Learning generalizable representations for visuomotor control. In International Conference on Machine Learning, 2018.
Richard S. Sutton. Dyna, an integrated architecture for learning, planning, and reacting. SIGART Bull, 2(4):160-1133 1991.
R.S. Sutton, Precup, and S. Singh. Between MDPs and semi-MDPs: framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 11:1181-211, 1999.
Guy Tennenholtz and Shie Mannor. The natural language of actions. In International Conference on Machine Learning, 2019.
Dhruva Tirumala, Alexandre Galashov, Hyeonwoo Noh, Leonard Hasenclever, Razvan Pascanu, Jonathan Schwarz, Guillaume Desjardins, Wojciech Marian Czarnecki, Arun Ahuja, Yee Whye Teh, and Nicolas Heess. Behavior priors for efficient reinforcement learning. CoRR, abs/2010. 142774 2020.
Marin Toromanoff, É. Wirbel, and F. Moutarde. End-to-end model-free reinforcement learning for urban driving using implicit affordances. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
Alexander Vezhnevets, Volodymyr Mnih, Simon Osindero, Alex Graves, Oriol Vinyals, John Agapiou, and koray kavukcuoglu. Strategic attentive writer for learning macro-actions. In Advances in Neural Information Processing Systems, 2016.
Tingwu Wang and Jimmy Ba. Exploring model based planning with policy networks. In International Conference on Learning Representations, 2020.
Grady Williams, Nolan Wagener, Brian Goldfain, Paul Drews, James M. Rehg, Byron Boots, and Evangelos A. Theodorou. Information theoretic MPC for model-based reinforcement learning. In IEEE International Conference on Robotics and Automation, 2017.
Markus Wulfmeier, Dominic Zeng Wang, and Ingmar Posner. Watch this: Scalable cost-function learning for path planning in urban environments. In International Conference on Intelligent Robots and Systems, 2016.
Sirui Xie, Junning Huang, Lanxin Lei, Chunxiao Liu, Zheng Ma. Wei Zhang, and Liang Lin. NADPEx: An on-policy temporally consistent exploration method for deep reinforcement learning. In International Conference on Learning Representations, 2019.
Wei Xu, Haonan Yu, Haichao Zhang, Yingxiang Hong, Break Yang, Le Zhao, Jerry Bai, and ALF contributors. ALF: Agent Learning Framework. URL https://github.
HorizonRabotic//ar
Haonan Yu, Wei Xu, and Haichao Zhang. TAAC: Temporally abstract actor-critic for continuous control. In Advances in Neural Information Processing Systems, 2021.
Yijie Zhang and Herke Van Hoof. Deep coherent exploration for continuous control. In International Conference on Machine Learning, 2021.
Brian D. Ziebart, Nathan Ratliff, Garratt Gallagher, Christoph Mertz, Kevin Peterson, J. Andrew Bagnell, Martial Hebert, Anind K. Dey, and Siddhartha Srinivasa. Planning- prediction for pedestrians. In International Conference Intelligent Robots and Systems, 2009.
13

---

Published as a conference paper at ICLR 2022
A APPENDIX
A.1
GPM ALGORITHM
The overall procedure of the proposed method is presented in Algorithm T
Algorithm Generative Planning Method (GPM)
Initialize: 0, o' 4 Φ. - 0
for each iteration do
for each environment step do
Told 0 if Taaaa is 0 else Told f p(Tr-1)
time-forward the old plan
Tnow ~ [1(+;)
generate a new plan
1 if liid TTaa is 0 else m~ Categorical Qrai (8, Tnew )) Qt (8, Tola )))
7 = (1 - 11m) Told m Tnew
select actual plan from old and new plans
de - w(T))
extract the primitive action for the current step
st-1 ~ p($141/5,,a, at)
D t DU [(s, a,, ((t,, a.), 8(41)}
save transition to replay buffer
end for
for each train step do
é + Ø Aa (0)
plan value update
0 с 0 - AVaJT
plan generator update
a - a - (a)
> temperature update
- (()
replan threshold update
0' + no + 1)
> target parameter soft-copy
end for end for
J (a) = (B -a log To(a [t,y)s,,00 a) - aH). where To denotes the policy corresponding to
the first step of the plan, which is similar to the one used in Haarnoja et al. (2018). H is the target entropy set the same way as SAC (Haarnoja et al. 2018) for controlling the degree stochasticity
of the policy by adjusting the temperature
A.2 DETAILS ON AND OPERATORS
Here we summarize Table the meaning and implementations of some entities as well as operators that are used in this work to further help understanding.
Meuning/Implemmentttoon
Entity/Operator
Gym Task
CARLA Task
[ag. 04 ....
a, denotes a primitive action
a, denotes a setpoint in ego-cuordinate
p(T)
p(T) T r[1:]
p(T)
TworldBago o o Thitt Togr2world
w(T)
w(T) 7
Table 1: Details on and w.
In Table T Tggobworld an operator for transforming the plan from ego to world coordinate and world2eg the reverse operator.
A.3 NETWORK STRUCTURE AND DETAILS
Network details for different tasks are summarized in Table where we use [K] x denote an MLP with I. FC layers and each with the output size of K For baseline methods, they use the network structure as shown Table for both actor and critic networks. For GPM, we implement the plan generator and plan value network in Figure 2 The plan generator first use MLP that has the same structure as baseline methods as shown in Table S for encoding the observation s to a feature vector 2 is used as the input of the Normal Layer, which uses FCs for generating mean and std vectors of diagonal Gaussian distribution (Haarooja et al. 2018). is passed through a FC layer to the initial state of GRU network (plan RNN), with its hidden size the same the single
14

---

Published as a conference paper at ICLR 2022
layer size in Table Action is detached before inputting the GRU to enforce the training of GRU and improve training stability. A shared FC layer is used for generating the actions after the first step, taking the output from plan RNN as input. The plan value network is implemented with LSTM (value RNN) with hidden size the same as the single layer size in Table 2 An MLP decoder with the same structure those used in other baseline methods (Table estimation, taking the oulput from value RNN as decoder input. The first step uses one such decoder with parameters not shared with others. The rest of the steps uses shared one. The reason for this is that the conditioning state for the first step is different from the rest. ReLU activation is used for hidden layers. Following standard practice, the minimum over the outputs of two value networks is used to mitigate value overestimation (Fujimoto et al.) 2018 Haarnoja et al. 2018).
used for plan value
Task
Network (actor/critic)
learning
batch size
plan length
# parallel actors
Pendulum
100 2
5e
64
InvertedPandulum
[256]
le
256
InvertedDoobblaPendbl...
[256]
1e
CartFoleSwingttp
[256] x
le"
256
sunarlanderContinuous
256 x
le
256
tountaincartontinmuus
256
le-
256
10
BipedalWalker
256
5e
4096
Fetch
[256]
le-3
CARLA
obecrvation, -nccoderreddereee [256]
le
128
20
Table 2: Details network structure and hyper-parameters. encoder for CARLA is provided in Appendix A.5.2 and Figure
More details the observation
A.4 GYM TASK DETAILS
The Gym tasks used in this paper and their corresponding Gym environment names are summarized in Table Fetch tasks are from Plappert et al. (2018a).
Task
Gym Enviroment Name
Pendulum
Pendulum-vo
InvertedPendulum
InvertedPendulum-22
InvertedDouhleeendd.. um
Inverted@Doulleeencui..c
MountaiinCarContinmuus
Mountaiincarconttinuuuswww
CartPoleSwingUp
CartPoleSwing(ppvo)
AumarianderConttimuuu
JumartaanderContthhuuuuuw
BipedalWalker
BipedalWalkeerwv2
FetchReach
FetchReach-vi
FetchPush
FetchPush-vi
Fetchslide
FetchsLide-vi
PetchPickAndPiece
PetchPickandPPace-/
Table 3: Tasks and gym environment names.
A.5 CARLA TASK DETAILS
The CARLA driving task is implemented based on the open source Car Learning to Act (CARLA) simulator (Dosovitskiy et al. (017). More details on sensar/bserattionncttiooowww network structure etc. are presented in the sequel.
A.5.1 SENSORS AND OBSERVATIONS
The following types of sensors are used in CARLA driving task (c.f. Figure 7
Camera: frontal camera capturing RGB image of size 64x64x3;
IMU sensor: 7D vector containing acceleration (3D), gyroscope (3D), compass (ID),
providing some measurements on the status of the current vehicle;
15

---

Published as a conference paper at ICLR 2022
Velocity: 3D vector containing velocity relative ego-coordinate in m/s;
Navigation: the collection of the positions of 8 waypoints d meters away from the ego vehicle along the route, with d {1. 3, 10, 30, 100, 300, 1000, 3000} meters. The positions of the waypoints are represented in 3D ego-coordinate. The sensory reading is a 24D vector;
Goal: 3D position in ego-coordinate representing the target position the ego car should navigate to and stop at;
Previous action: 2D vector representing the previous action. More details on the action space are provided in Appendix|AA5.3
The delta time between two time steps is At 0.05s, i.e., the perception and control rate is 20 Hz.
RGB image
Visual Encoder
Actor Network
policy
IMU
FC
FC
velocity
navigation
21
Network
value
goal
prev action
FC
Observation Encoder
plan/action
Figure Network structures for CARLA task.
A.5.2 CARLA NETWORK STRUCTURE
The network structure used CARLA task is shown Figure 7 shared by all the methods including GPM Given the observation containing multi-modalh signals, an observation encoder is first used to fuse them into a now-dimensional feature vector. This is achieved by processing each modality of signals with dedicated encoder with the same dimensionality for the outputs. More specifically, the visual encoder is a ResNet- 18 based CNN. Each of other modalities encoded by fully connected layer. The outputs from each of the encoder are fused together with a sum operator followed by FC layer to produce the feature vector as input to downstream networks. The observation encoder is shared by both the actor and critic network. During training, the the gradient from the actor is stopped and the observation encoder is only trained by value learning.
This is the common structured
A.5.3 CARLA CONTROLLER AND ACTION SPACE
The CARLA environment with its controller interface is illustrated in Figure takes as input a setpoint in ego-coordinate, specifying immediate position go to for the current step. For the driving case, the setpoint can be represented in the 2D spatial ego-coordinate. Therefore there are two dimensions (r:: lateral, y: longitudinal) in the action space, with (Ir, y) E [-200, 20] x [-20, 20). The actor controls the movement of the car by generating 2D action as the setpoint. Target direction and speed will be computed based on the setpoint and ego-position (origin), which will be passed to PID-controller to produce control signals including throttle, steer, brake and reverse. These signals will be used by the simulator generate the next state of the car. As part of the environment, shrinkage is applied to the setpoint before computing the control signals:
max [|||| 6,0)
(8)
|||||
This is used to balance the portion of the braking behavior w.r.t others in the control space. 5 = 3 is used in this work.
A.5.4 REWARD
The reward in CARLA environment is consisted of the following components:
16

---

Published as a conference paper at ICLR 2022
throttic steer brake reverse
Actor
action
setpoint ego pasition
target direction target speed
PID Controller
Simulator
CARLA Environment
Figure CARLA environment and action space.
Route distance reward: this reward is computed as the distance traveled along route from the previous to the current time step.
this is computed as min mmin(collin coollision peaallte_e______ea 0.5 * max (0, episode reward)), where collision, collision_penaltyy penalty=20,, episode_rewee reward denotes the accumulated rewards up to now within the current episode. Issued only once for a contiguous collision event.
Collision reward:
Red-light reward: this computed min(rod Hight_penailn 0.3 max(0., episode rewordd)_, where _nd__ighht_eenaly-2000 Issued only once for a contiguous red light violation event.
Success reward: success reward of 100 will be received by the agent (ego-car) if its position is within 5 meters to the goal position and its speed is less than 10-3 m/s.
A.5.5
EPISODE TERMINATION CONDITIONS
The episode terminates if either of the following conditions are satisfied:
success: the car's position is within 5 meters to the goal position and its speed is less than 10-3 m/s.
the maximum number of allowed time steps K is reached: this number is computed as K route length
x where the min ciin__elliity==i = 5 m/s and At = 0.05s. The state is marked as non-terminal state, allowing bootstrapping for value learning at this state;
stuck at collision: the car is involved the same collision event and does not move more than meter after 100 time steps. The state is marked as non-terminal state, allowing bootstrapping for value learning at this state.
A.5.6 MAP AND EXAMPLE ROUTES
For CARLA driving task, Town01, which is town with representative layout consisting number of T junctions and intersections. The layout of Town01 is shown in Figure of the town is about 410m 344m. We also spawn 20 other vehicles controlled by autopilot and 20 pedestrians to simulate typical driving conditions involving not only static objects but also dynamic ones.
( (a). The size
AUE LUN
(b)
Figure 9: Map and Example Routes. (a) Map of Town01 (Dosovitskiy et al. 2017). (b) Map with some example routes overlaid. Different routes are rendered with different colors. The beginning of each route is denoted with a A symbol and the end of the route is denoted with a O symbol.
17

---

Published as a conference paper at ICLR 2022
The goal location is randomly selected from the set of waypoints. The agent is spawned random valid waypoint location at the beginning of each episode. A waypoint is valid if it is on the road and is at least 10 meters away from any other actor in the world (vehicles and pedestrians). Some examples routes are shown in Figure bb). b). As can be observed from the figure, the routes constructed this way have a reasonable amount of diversity and cover representative driving scenarios. For example, different routes could include different number of turns, directions for turning, static objects (as the routes are different) and also different dynamic objects (e.g. other vehicles) different space-time locations. The task of the agent is to navigate to the goal position following the route and stay still after arrived at the goal location.
A.6 MORE IMPLEMENTATION DETAILS
A.6.1 PLAN SAVING, SAMPLING AND GENERATION
In practice, for plan value learning, for each training iteration, we sample a batch of (state, plan, rewards, next-plan-state) tuple (st: Tē rf, st s from the replay buffer. More concretely, for each state 8i randomly sampled from the replay buffer, we sample an associated plan T/ as the actions starting from the current step of st to variable number of future steps uniformly sampled in the range of I and maximum plan length L, i.e. Ti- = [aj ... d/-111 where / ~ uniform(1, (1, L). The reward vector is consisted of the rewards along the plan r [ri 1'+11"" 11 74+]]- The plan terminal state s is the state at the time step after the last step of r/, L.e. s = 4/42- Therefore, in effect, we use sampled plan for plan value learning. By constructing plans in this way, we have maximal flexibility of leveraging the action sequences from replay buffer with minimal assumption on the plan distribution. During training, (commit foommit_targot target is set to 0.5L.
Note that in the case of driving (and other similar scenarios), since the plan is a spatial trajectory in the ego-coordinate, and the plan update operator in this case is p(T) = Toorld2ego Tshiit o (r') (c.f. Table To enable the construction of plans from action sequences sampled from the replay buffer (not necessarily aligned with the plan boundary during rollout), we need save the a' w(T') into replay buffer, i.e. D + D U {(s,, d', r(st, az), $t+1)) a' = w(T) actually denotes the setpoint in world coordinate. In this case, since the actions saved in replay buffer are world coordinate, we can construct plans world coordinate by by sampling sequences from the replay buffer, in the same way as described above, i.e. T/ = [ae, 144+1-1
Tego2world (r) = o
where ~~ uniform (1, L). Then the plans in ego-coordinate can be obtained as 7 worrdd2eo (T1). For the standard Gym tasks, since both T worldZego and 7 worldzego are identity operators, we have 7 = meaning we can simply save the current primitive action a = -T( into replay buffer.
A scaling factor can be used for the replanning distribution Categorical( ()). We simply use k= For testing the plan generator after training, the mode of the policy is used, by taking the mean of the Gaussian policy used in the plan generator and taking the mode of Categorical(s) ) for replanning.
A.6.2 PLAN GENERATOR IN AUTONOMOUS DRIVING
We implement ff used in plan generator as follows
a+4s = a145-11 + Doci + g(uet 445-- Aeri
i [1,n- - 1]
where Aaja-1 = (at+111-00 -4+ This actually incorporates a linear prior into f. which is useful for tasks with position-based controls. A smoothing cost is also applied to the plan to minimize the maximum absolute value of its third order derivative max 77 (Kyriakopoulos & Saridis 1988).
In the most basic form, the recurrent plan generator can run at the scale of time-step in order to generate the full plan. In practice, it can be adjusted according to the actual situation, and running at can let the
coarser scale, which has the advantage of oower-computational cost. For example, policy only output a few number (k) of anchor points and interpolate between them to get the full plan using particular form of interpolation methods. Linear interpolation is used in this work.
18

---

Published as a conference paper at ICLR 2022
A.7 MORE RESULTS
A.7.1 ABLATION RESULTS ON PLAN LENGTH AND TARGET COMMITMENT LENGTH
Impact of Plan Length. The results with different plan lengths are shown in Figure T0 a). is observed that the overall the performance is relatively insensitive to plan length, although the performance could be further optimized by tuning this papammerr...tr..
Target Commitment Length. The results with different target commitment lengths are shown in Figure b)). The results indicate that the performance of the proposed method is relatively stable across different target commitment lengths, potentially due to its ability in generating plans with adaptive shapes.
aunartaaderronninuuus
SartpolSSwingup
LunarLanderCrriraauu
CartpoleSwingup
203
209
plan-len 3
320
targetilen
plan-lón
target-len
300
300
target-len
-00
pian-ion
-399
target-lon
plan-len
target-len
0.2
05
24
-200
0.5
26
6.6
15
(8)
(0)
Figure 10: Ablations on (a) plan length and (b) target commitment length. For (a) the target commitment length is set as 1.5. For (b) the plan length is set as 5.
A.7.2 OBSERVATIONAL DROPOUT: THE IMPORTANCE OF ADAPTIVE PLANS
Here we compare the proposed method and action repeat under the setting where a fixed replanning duration is used. For action repeat, this corresponds to the FAR algorithm. For GPM, we also execute the full plan before switching (denoted as GPM-commit). This actually an instance of observational dropout (Freeman et al. 2019), which requires the algorithm to extrapolate controls generated from available observations to those time steps where observations are unavailable. The results are shown in Figure 11 This results highlight the importance of adaptive form of plans performance. Fixed form of plan as used in FAR might be applicable to certain types of tasks (e.g. CartFoleSwinggtp) However, for other tasks, it might be less appropriate and leads to significant performance decreases with increasing plan length (e.g., lunariander raant i nuous). In contrast, the proposed approach shows stable performance across a wide range of plan lengths. By contrasting the performance of FAR with Gom-commit, we highlight the importance on the adaptiveness of the plans.
SeirarlanderContinooo
CartpoleSwingup
CartpoleSwingup
200
200
plan-len 3
1.00
plan-len
plan lon 5
plan-le 5
1re
plan-len
plan-len
plan len 3
-000
plan ln
plan-len
plan-ien
2/1
100
3.0
20
330
20
14
Ci and Stepa
[3]
Cratsnmert Stea
(b)
Contmmex
Figure 11: Observational Dropout for (a) FAR (b) plans of fixed length and execute them to the end before generating the next plan.
Here both approaches generate
Bipeda/Wakker
A.7.3 MPC v.s. GPM
MPC has been typically used together with planning for obtaining the control policy (Lowrey et al., 2019). This strategy is appropriate when a model is given (Lowrey et al.1)019) However, for exploration RL, MPC is less desirable as it essentially switches the plan at every step, not leveraging the temporal property of the plans, as shown on the right, by comparing with variant of using MPC together with GPM (GPM-MPC).
100
19

---

Published as a conference paper at ICLR 2022
A.7.4
EXPLORATION TRAJECTORY VISUALIZATION AND COMPARISON
0-2k stepe
24-4k 24-4kst8p steps
4k-8k steps
apioode and
SAC
5
angllaar velocity (radis)
E2
U 5.
DAR
GPM
HERROOW
beginning
angle (rad)
angle (rad)
(rac)
Figure 12:
Visualization of State Visitation and Evolution during training on Pendul um. Each figure shows collectively all the rollout trajectories within the range of steps depicted on the top.
Since the most rewarding state (0, 0), an effective policy should quickly move close to this state.
episodes
episodes
episodes
SAD
-200
-202
-000
-200
200
200
200
200
200
200
200
250
200
200
DAR
-200
-202
-200
-20
200
200
-200
200
-300
200
200
200
200
200
GPM
0-
Figure 13: Exploration Trajectory Visualization. Trajectory are plotted relative its start position.
20


Reviews:
[{"review_id": "lLalxj5QCmk", "reviewer": "Reviewer_ZK8i", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strengths\nThis paper is well-written and tackles an important problem. The experiments clearly demonstrate that GPM improves exploration over naive baselines (e.g. greedy exploration), and the fact that the model does not need to be trained to explicitly model the dynamics is a nice advantage.\n\n# Weaknesses\nThe main weaknesses of this paper are that it does not demonstrate strong empirical results, that the related works section is missing some important discussion, and that the method is missing some important details. I describe each concern in detail below.\n\n## Method\nAlthough the paper read well, the method section has some important details missing. For example, do the authors use n-step return in Equation 6? If so, how is n-determined? Also, where does P(s_t, a_t) and T(s, \\tau) come from? Does the method assume access to the ground-truth dynamics model T? Also, how was l_commit_target chosen? How sensitive is the method to this hyperparameter, and is there evidence that it’s easier to choose l_commit_target than epsilon directly?\n\n\n## Related work\n\nThe paper could greatly benefit from discussing the relationship to works such as [10-14], since these papers also implicitly learn models through model-free RL.\n\nAnother concern with the writing is that the claim that, “we provide a new aspect and opportunity for improving temporally consistent exploration from the perspective of planning” is a strong overstatement. There have been multiple sub-fields that have explored this idea of using planning to explore, including goal-conditioned reinforcement learning (RL) [1,2,3], hierarchical RL [4,5,6], and model-based RL [7,8,9], as well as many of the papers that the paper cites. I strongly suggest the authors reconsider the phrase of this claim and either remove that claim altogether, or be much more specific with the novelty that is being provided.\n\n## Experiments\n\nIn terms of experiments, the paper does not compare to competitive exploration methods and the tasks, other than CARLA, are relatively easy. In particular, the paper would benefit from comparing to reward-bonus, option-based, goal-directed, and/or other planning exploration, or the paper would need to provide a strong justification for why comparing to these methods is unnecessary. Without these comparisons, I do not see evidence that, “we verify the effectiveness of the proposed method by comparing it with current state-of-the-art methods on a number of benchmark tasks”\n\nFigures 4, 5, and 6 are a bit redundant: they all show that PGM have reasonable exploration and perform better than naive epsilon-greedy, but I think the space could be used more efficiently. For example, the paper would also be strengthened by spending more time describing the CARLA environment in the main paper.\n\nThe paper could also benefit from evaluating more challenging and diverse tasks. Pendulum, InvertedPendulum, InvertedDoublePendulum, and CartpoleSwingUp, are all similar and simple. I suggested the authors add comparisons on (e.g.) Atari domains or more challenging navigation tasks (BipedalWalker’s challenge is in learning _how_ to walk--not _where_ to walk).\n\n## minor comments\n * Equation 1 does not describe the SAC update, which uses a soft update\n\n## references\n\n - [1] Ren, Zhizhou, et al. \"Exploration via Hindsight Goal Generation.\" Advances in Neural Information Processing Systems 32 (2019): 13485-13496.\n - [2] Pong, Vitchyr, et al. \"Skew-Fit: State-Covering Self-Supervised Reinforcement Learning.\" International Conference on Machine Learning. PMLR, 2020.\n - [3] Pitis, Silviu, et al. \"Maximum entropy gain exploration for long horizon multi-goal reinforcement learning.\" International Conference on Machine Learning. PMLR, 2020.\n - [4] Dayan, P. and Hinton, G. E. Feudal reinforcement learning. In Advances in Neural Information Processing Systems, 1993.\n - [5] Florensa, Carlos; Duan, Yan; Abbeel, Pieter.  Stochastic Neural Networks for Hierarchical Reinforcement Learning. In International Conference on Learning Representations (ICLR) 2017.\n - [6] Co-Reyes, John, et al. \"Self-consistent trajectory autoencoder: Hierarchical reinforcement learning with trajectory embeddings.\" International Conference on Machine Learning. PMLR, 2018.\n - [7] Henaff, Mikael. \"Explicit explore-exploit algorithms in continuous state spaces.\" NeurIPS. 2019.\n - [8] Shyam, Pranav, Wojciech Jaśkowski, and Faustino Gomez. \"Model-based active exploration.\" International conference on machine learning. PMLR, 2019.\n - [9] Sekar, Ramanan, et al. \"Planning to explore via self-supervised world models.\" International Conference on Machine Learning. PMLR, 2020.\n - [10] Oh, Junhyuk, Satinder Singh, and Honglak Lee. \"Value Prediction Network.\" NIPS. 2017.\n - [11] Silver, David, et al. \"The predictron: End-to-end learning and planning.\" International Conference on Machine Learning. PMLR, 2017.\n - [12] Schrittwieser, Julian, et al. \"Mastering atari, go, chess and shogi by planning with a learned model.\" Nature 588.7839 (2020): 604-609.\n - [13] Weber, Théophane, et al. \"Imagination-augmented agents for deep reinforcement learning.\" arXiv preprint arXiv:1707.06203 (2017).\n - [14] Srinivas, Aravind, et al. \"Universal planning networks: Learning generalizable representations for visuomotor control.\" International Conference on Machine Learning. PMLR, 2018.", "summary_of_the_paper": "This paper presents a method called generative planning method (GPM) to improve exploration in RL. GPM performs exploration by learning a planner (a map from state to a sequence of action, aka a “plan”)) and performing MPC with a special rule for whether or not to use the latest plan or keep the old plan. The planner is an auto-regressive model (specifically, a stochastic RNN) and is trained to maximize an auto-regressive Q-function. Each time step, the plan from the previous time step is shifted forward by one time step and compared to the newly generated plan. The plans are compared using their predicted Q-values, and the policy switches plans with some probability that increases monotonically with Q(new plan) - Q(old plan). The exact likelihood is determined by a hyperparameter, l_commit_target, that, intuitively, sets a soft target for how long a plan is typically kept. The authors compare GPM to a variety of action-repeat-based exploration methods, from epsilon greedy policies to policies with learned action repeat counts (DAR & TAAC) and find that it is competitive with or outperforms these methods on various low-dimensional robot domains, as well as the image-based CARLA environment. The authors also visualize the trajectories and qualitatively show that GPM improves exploration.", "summary_of_the_review": "My main issue with the paper is that it claims to present a new exploration method, but it only compares to relatively simple exploration methods that only induce exploration via action repeat. The absence of more competitive baselines makes the significance of the paper relatively small. Moreover, some of the claims are overstated, some important methodological details are unclear, and the discussion of certain related works are missing.\n\nAfter the author's response: The additional experiments and updated related work section have addressed my primary concerns. I strongly suggest adding the details about the method from the rebuttal (which I found useful) to the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "U-lzVulnBgr", "reviewer": "Reviewer_nNjd", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n1. As far as I know, the idea is novel and very interesting. I really like the idea of connecting single-step action noise exploration and multi-step intentional exploration via planning under the perspective of planning for exploration.\n2. The topic is definitely very related to the conference and of significant interest.\n3. The paper is generally well-written and easy to follow. The pictorial demonstration is also very clear (fig.1 and fig.2).\n4. The qualitative results (fig.4 & fig.5 & fig.6) are helpful to understand the exploration behaviors and are intuitive and interesting.\n\nWeakness/questions\n1. Testing environment. The tasks tested in the paper are pretty simple in my opinion (e.g. pendulum could be easily solved by a naive a2c agent). Considering the popular testing environment for continuous control are MuJoCo tasks, they should be tested. I am also wondering is there any particular reason why those environments are not considered in the first place? Furthermore, since this paper concerns exploration, it will also be useful to demonstrate the effectiveness in sparse-reward environments.\n2. Presenting. I think there are some issues that could be improved to make the paper more clear. Some suggestions are as follows. 1) some important concepts are not explained well. For example, what does action-repeat mean, what is the dithering phenomenon... Those terms keep appearing in the paper and they should be explained more carefully. 2) terms being used. Throughout the paper, some (I think) similar terms are used differently and maybe interchangeably and I find it hard to understand. For example, temporally extended/coordinated/consistent/persistent...\n3. for the plan generator, why are mu and sigma only generated for the current action? what's the reasoning behind this design choice?\n4. for plan value function, why should be objective optimized in expectation to the length l? isn't that enough you optimize for L? As long as the plan has a high value, why do we care about high-value sub-plans?\n\n\n\n\n", "summary_of_the_paper": "This paper proposes a method for exploration called Generative Planning method (GPM), which generates a multi-step action sequence such that the exploration is more temporally consistent and \"intentional\" compared to regular single-step action noise exploration. The multi-step action sequence is output by a generator with an RNN structure, and the generator is optimized by maximizing the plan value function. The authors show that their method GPM performs better than some other methods in several continuous control tasks and present some interesting qualitative results (e.g. state trajectory) showing that the exploration is more effective.", "summary_of_the_review": "I believe the idea and the approach are quite novel and interesting, however, I do believe some extra experiments should be performed and some improvements over the paper presentation are needed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Od2dWFMw0l", "reviewer": "Reviewer_3VLM", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- the method is (to my knowledge) a novel approach that I see as sitting somewhere between model-based and model-free.\n  It is model-based in the sense that it is predicting actions and future state-action values, a numer of time steps into the future.\n  It is model-free in that there is no explicit learning of the state transitions and rewards.\n  It offers a new integration of planning and control.\n- the method performs well on a set of 8 benchmark tasks, spanning 6 simple systems, a more challenging system (bipedwalker), \n  and a more complex and realistic system building on visual input (CARLA autonomous driving).\n\nWeaknesses:\n- a better understanding of how this compares to model-based methods could be provided.  Like a model-based method, future plans can be generated and tested, although the model is implicit rather than explicit.  Experimental comparisons and discussions would help address this.  Regardless of the actual performance numbers, the results will be of interest to readers.\n\nThe form of the action generator is obviously an important choice.\nFrom my understanding from Table 2, the recurrent models are not shared between actor & critic.\nIs this a critical design choice?  Could both of those networks do well at reconstructing \nthe expected state sequence?  Would a take-home lesson be that it is more important to\npredict next actions rather than next states?  And that working with entirely implicit next-state dynamics\n(not even using a shared latent state for actor & critic) is simpler and more effective than leveraging\na learned latent dynamics model?  If the actor and critic could be conditioned on the real state\n(imagine a setting with thea ability to do state resets, and not needing to count these \"look-ahead\" env steps)\nwould the learning improve?\n\nIs the recurrent critic a key part of the solution, or could a more standard critic architecture also \nbe made to work?\n\nWhat is the benefit of making the replan decision stochastic, as opposed to deterministic?\n\nThere is a large body of literature on the set of topics being discussed,\nand so in many contexts, it could be useful to begin the list with \"e.g.,\"\n\nPresumably there are many variations of the algorithm that could exist, with regard to \nhow many new plans are generated for each environment step; the choice of one new plan per env step is ultimately\njust one arbitrary choice.\n\nHow important is the plan length hyperparameter?\n\nCan you comment on the ability (or not) of the policy to generate multimodal futures?\nPresumably not, given the action parameterization (unimodal multivariate Gaussians), and so does this\never become a limitation?\n\nFigure 2 could perhaps be labeled with \"generative actor\" and \"generative critic\". \nIt took me a while to understand that there was no other policy network, aside from the plan generator.", "summary_of_the_paper": "A generative planning method is proposed, which sits somewhere between model-free and model-based methods.\nNo explicit model is learned. However, an action plan is generated using a recurrent action-plan generator,\npaired with a similarly recurrent critic.  At each environment step, a new action plan is generated, and \nthe current action plan can be abandoned in favor of the new action plan if the estimated benefit is large enough.\nThe method builds on SAC.  Overall the benefits of the temporally extended action plans are:\n(a) temporally-coordinated exploration; (b) more effective than action-repeat; (c) some degree of interpretability\ngiven that an action-sequence plan represents then intent of a policy in a given state.", "summary_of_the_review": "A novel method is presented, which, in the model-free setting, improves upon current alternatives for temporally extended actions,\nin the mod, and model-free SAC. I believe that the results will be of strong interest to the community. The core idea is a creative one that performs well, and will promote future work and discussion.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "1B6wFkEgYue", "reviewer": "Reviewer_356g", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is well-written and easy to follow. The motivation is clear and important: how to improve exploration for model-based RL. The proposed method of planning in a short term and encouraging using the old plan in order to encourage temporally extended exploration is neat and interesting. The evaluations are solid, comparing with 5 baselines on 8 benchmarks. Overall, this is a good paper.\n\nStrength:\n- Well-written and easy to follow\n- The motivation is clear: how to improve exploration for model-based RL\n- Proposed technique is interesting: learning a recurrent model to generate short-term plans at each time step, and only switching to the new plan if it is a lot better than the old plan.\n- Solid evaluations. GPM is compared with 5 baselines on 8 benchmarks, and experiments show that GPM converges faster than the other baselines. Further analyses show that GPM indeed explores more effectively, and it also generates interpretable short-term plans \n\nQuestions:\n- What is the computational overhead of GPM compared with the baselines? Since it needs to generated short-term plans at each time step, does it take longer on average to run each training step? \n- It makes sense to encourage reusing the old plan during training since it encourages exploration. When the learned policy is deployed in test time, is it better to remove this bias towards the old plan? I would like to see a discussion on this from the authors.\n", "summary_of_the_paper": "This paper proposes a method called Generative Planning (GPM), which aims to improve exploration for model-free RL. GPM learns a recurrent model to generate short term plans at each time step, and only decides to switch to the new plan if it is a lot better than the old plan. This encourages temporally extended explorations, and also does it in an adaptive manner. Experiments on a set of continuous control benchmarks show that GPM is able to converge faster than prior approaches, explore more effectively, and generate interpretable short-term plans.\n", "summary_of_the_review": "This paper is well-written and well-motivated. The proposed technique is interesting and the experimental section is strong. Overall, this is a good paper and I recommend acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an alternative approach to epsilon-greedy exploration by instead generating multi-step plans from an RNN, and then stochastically determining whether to continue with the plan or re-plan. The reviewers agreed that this idea is novel and interesting, that the paper is well-written, and that the evaluations are convincing, showing large improvements over epsilon-greedy exploration and more consistently strong performance than other baselines. While the original reviews contained some questions around discussion of related work and the simplicity of the evaluation environments, the reviewers felt these concerns were adequately addressed by the rebuttal. I agree the paper explores a very interesting idea and convincingly demonstrates its potential, and should be of wide interest to the deep RL community especially as it touches on many different subfields of RL: MBRL/planning, exploration, HRL, navigation, etc. I recommend acceptance as a spotlight presentation.
Decision: Accept (Spotlight)