Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a method for exploration called Generative Planning method (GPM), which generates a multi-step action sequence such that the exploration is more temporally consistent and "intentional" compared to regular single-step action noise exploration. The multi-step action sequence is output by a generator with an RNN structure, and the generator is optimized by maximizing the plan value function. The authors show that their method GPM performs better than some other methods in several continuous control tasks and present some interesting qualitative results (e.g. state trajectory) showing that the exploration is more effective.
main_review: Strengths
1. As far as I know, the idea is novel and very interesting. I really like the idea of connecting single-step action noise exploration and multi-step intentional exploration via planning under the perspective of planning for exploration.
2. The topic is definitely very related to the conference and of significant interest.
3. The paper is generally well-written and easy to follow. The pictorial demonstration is also very clear (fig.1 and fig.2).
4. The qualitative results (fig.4 & fig.5 & fig.6) are helpful to understand the exploration behaviors and are intuitive and interesting.

Weakness/questions
1. Testing environment. The tasks tested in the paper are pretty simple in my opinion (e.g. pendulum could be easily solved by a naive a2c agent). Considering the popular testing environment for continuous control are MuJoCo tasks, they should be tested. I am also wondering is there any particular reason why those environments are not considered in the first place? Furthermore, since this paper concerns exploration, it will also be useful to demonstrate the effectiveness in sparse-reward environments.
2. Presenting. I think there are some issues that could be improved to make the paper more clear. Some suggestions are as follows. 1) some important concepts are not explained well. For example, what does action-repeat mean, what is the dithering phenomenon... Those terms keep appearing in the paper and they should be explained more carefully. 2) terms being used. Throughout the paper, some (I think) similar terms are used differently and maybe interchangeably and I find it hard to understand. For example, temporally extended/coordinated/consistent/persistent...
3. for the plan generator, why are mu and sigma only generated for the current action? what's the reasoning behind this design choice?
4. for plan value function, why should be objective optimized in expectation to the length l? isn't that enough you optimize for L? As long as the plan has a high value, why do we care about high-value sub-plans?





summary_of_the_review: I believe the idea and the approach are quite novel and interesting, however, I do believe some extra experiments should be performed and some improvements over the paper presentation are needed.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a method called generative planning method (GPM) to improve exploration in RL. GPM performs exploration by learning a planner (a map from state to a sequence of action, aka a “plan”)) and performing MPC with a special rule for whether or not to use the latest plan or keep the old plan. The planner is an auto-regressive model (specifically, a stochastic RNN) and is trained to maximize an auto-regressive Q-function. Each time step, the plan from the previous time step is shifted forward by one time step and compared to the newly generated plan. The plans are compared using their predicted Q-values, and the policy switches plans with some probability that increases monotonically with Q(new plan) - Q(old plan). The exact likelihood is determined by a hyperparameter, l_commit_target, that, intuitively, sets a soft target for how long a plan is typically kept. The authors compare GPM to a variety of action-repeat-based exploration methods, from epsilon greedy policies to policies with learned action repeat counts (DAR & TAAC) and find that it is competitive with or outperforms these methods on various low-dimensional robot domains, as well as the image-based CARLA environment. The authors also visualize the trajectories and qualitatively show that GPM improves exploration.
main_review: # Strengths
This paper is well-written and tackles an important problem. The experiments clearly demonstrate that GPM improves exploration over naive baselines (e.g. greedy exploration), and the fact that the model does not need to be trained to explicitly model the dynamics is a nice advantage.

# Weaknesses
The main weaknesses of this paper are that it does not demonstrate strong empirical results, that the related works section is missing some important discussion, and that the method is missing some important details. I describe each concern in detail below.

## Method
Although the paper read well, the method section has some important details missing. For example, do the authors use n-step return in Equation 6? If so, how is n-determined? Also, where does P(s_t, a_t) and T(s, \tau) come from? Does the method assume access to the ground-truth dynamics model T? Also, how was l_commit_target chosen? How sensitive is the method to this hyperparameter, and is there evidence that it’s easier to choose l_commit_target than epsilon directly?


## Related work

The paper could greatly benefit from discussing the relationship to works such as [10-14], since these papers also implicitly learn models through model-free RL.

Another concern with the writing is that the claim that, “we provide a new aspect and opportunity for improving temporally consistent exploration from the perspective of planning” is a strong overstatement. There have been multiple sub-fields that have explored this idea of using planning to explore, including goal-conditioned reinforcement learning (RL) [1,2,3], hierarchical RL [4,5,6], and model-based RL [7,8,9], as well as many of the papers that the paper cites. I strongly suggest the authors reconsider the phrase of this claim and either remove that claim altogether, or be much more specific with the novelty that is being provided.

## Experiments

In terms of experiments, the paper does not compare to competitive exploration methods and the tasks, other than CARLA, are relatively easy. In particular, the paper would benefit from comparing to reward-bonus, option-based, goal-directed, and/or other planning exploration, or the paper would need to provide a strong justification for why comparing to these methods is unnecessary. Without these comparisons, I do not see evidence that, “we verify the effectiveness of the proposed method by comparing it with current state-of-the-art methods on a number of benchmark tasks”

Figures 4, 5, and 6 are a bit redundant: they all show that PGM have reasonable exploration and perform better than naive epsilon-greedy, but I think the space could be used more efficiently. For example, the paper would also be strengthened by spending more time describing the CARLA environment in the main paper.

The paper could also benefit from evaluating more challenging and diverse tasks. Pendulum, InvertedPendulum, InvertedDoublePendulum, and CartpoleSwingUp, are all similar and simple. I suggested the authors add comparisons on (e.g.) Atari domains or more challenging navigation tasks (BipedalWalker’s challenge is in learning _how_ to walk--not _where_ to walk).

## minor comments
 * Equation 1 does not describe the SAC update, which uses a soft update

## references

 - [1] Ren, Zhizhou, et al. "Exploration via Hindsight Goal Generation." Advances in Neural Information Processing Systems 32 (2019): 13485-13496.
 - [2] Pong, Vitchyr, et al. "Skew-Fit: State-Covering Self-Supervised Reinforcement Learning." International Conference on Machine Learning. PMLR, 2020.
 - [3] Pitis, Silviu, et al. "Maximum entropy gain exploration for long horizon multi-goal reinforcement learning." International Conference on Machine Learning. PMLR, 2020.
 - [4] Dayan, P. and Hinton, G. E. Feudal reinforcement learning. In Advances in Neural Information Processing Systems, 1993.
 - [5] Florensa, Carlos; Duan, Yan; Abbeel, Pieter.  Stochastic Neural Networks for Hierarchical Reinforcement Learning. In International Conference on Learning Representations (ICLR) 2017.
 - [6] Co-Reyes, John, et al. "Self-consistent trajectory autoencoder: Hierarchical reinforcement learning with trajectory embeddings." International Conference on Machine Learning. PMLR, 2018.
 - [7] Henaff, Mikael. "Explicit explore-exploit algorithms in continuous state spaces." NeurIPS. 2019.
 - [8] Shyam, Pranav, Wojciech Jaśkowski, and Faustino Gomez. "Model-based active exploration." International conference on machine learning. PMLR, 2019.
 - [9] Sekar, Ramanan, et al. "Planning to explore via self-supervised world models." International Conference on Machine Learning. PMLR, 2020.
 - [10] Oh, Junhyuk, Satinder Singh, and Honglak Lee. "Value Prediction Network." NIPS. 2017.
 - [11] Silver, David, et al. "The predictron: End-to-end learning and planning." International Conference on Machine Learning. PMLR, 2017.
 - [12] Schrittwieser, Julian, et al. "Mastering atari, go, chess and shogi by planning with a learned model." Nature 588.7839 (2020): 604-609.
 - [13] Weber, Théophane, et al. "Imagination-augmented agents for deep reinforcement learning." arXiv preprint arXiv:1707.06203 (2017).
 - [14] Srinivas, Aravind, et al. "Universal planning networks: Learning generalizable representations for visuomotor control." International Conference on Machine Learning. PMLR, 2018.
summary_of_the_review: My main issue with the paper is that it claims to present a new exploration method, but it only compares to relatively simple exploration methods that only induce exploration via action repeat. The absence of more competitive baselines makes the significance of the paper relatively small. Moreover, some of the claims are overstated, some important methodological details are unclear, and the discussion of certain related works are missing.

After the author's response: The additional experiments and updated related work section have addressed my primary concerns. I strongly suggest adding the details about the method from the rebuttal (which I found useful) to the paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: A generative planning method is proposed, which sits somewhere between model-free and model-based methods.
No explicit model is learned. However, an action plan is generated using a recurrent action-plan generator,
paired with a similarly recurrent critic.  At each environment step, a new action plan is generated, and 
the current action plan can be abandoned in favor of the new action plan if the estimated benefit is large enough.
The method builds on SAC.  Overall the benefits of the temporally extended action plans are:
(a) temporally-coordinated exploration; (b) more effective than action-repeat; (c) some degree of interpretability
given that an action-sequence plan represents then intent of a policy in a given state.
main_review: Strengths:
- the method is (to my knowledge) a novel approach that I see as sitting somewhere between model-based and model-free.
  It is model-based in the sense that it is predicting actions and future state-action values, a numer of time steps into the future.
  It is model-free in that there is no explicit learning of the state transitions and rewards.
  It offers a new integration of planning and control.
- the method performs well on a set of 8 benchmark tasks, spanning 6 simple systems, a more challenging system (bipedwalker), 
  and a more complex and realistic system building on visual input (CARLA autonomous driving).

Weaknesses:
- a better understanding of how this compares to model-based methods could be provided.  Like a model-based method, future plans can be generated and tested, although the model is implicit rather than explicit.  Experimental comparisons and discussions would help address this.  Regardless of the actual performance numbers, the results will be of interest to readers.

The form of the action generator is obviously an important choice.
From my understanding from Table 2, the recurrent models are not shared between actor & critic.
Is this a critical design choice?  Could both of those networks do well at reconstructing 
the expected state sequence?  Would a take-home lesson be that it is more important to
predict next actions rather than next states?  And that working with entirely implicit next-state dynamics
(not even using a shared latent state for actor & critic) is simpler and more effective than leveraging
a learned latent dynamics model?  If the actor and critic could be conditioned on the real state
(imagine a setting with thea ability to do state resets, and not needing to count these "look-ahead" env steps)
would the learning improve?

Is the recurrent critic a key part of the solution, or could a more standard critic architecture also 
be made to work?

What is the benefit of making the replan decision stochastic, as opposed to deterministic?

There is a large body of literature on the set of topics being discussed,
and so in many contexts, it could be useful to begin the list with "e.g.,"

Presumably there are many variations of the algorithm that could exist, with regard to 
how many new plans are generated for each environment step; the choice of one new plan per env step is ultimately
just one arbitrary choice.

How important is the plan length hyperparameter?

Can you comment on the ability (or not) of the policy to generate multimodal futures?
Presumably not, given the action parameterization (unimodal multivariate Gaussians), and so does this
ever become a limitation?

Figure 2 could perhaps be labeled with "generative actor" and "generative critic". 
It took me a while to understand that there was no other policy network, aside from the plan generator.
summary_of_the_review: A novel method is presented, which, in the model-free setting, improves upon current alternatives for temporally extended actions,
in the mod, and model-free SAC. I believe that the results will be of strong interest to the community. The core idea is a creative one that performs well, and will promote future work and discussion.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a method called Generative Planning (GPM), which aims to improve exploration for model-free RL. GPM learns a recurrent model to generate short term plans at each time step, and only decides to switch to the new plan if it is a lot better than the old plan. This encourages temporally extended explorations, and also does it in an adaptive manner. Experiments on a set of continuous control benchmarks show that GPM is able to converge faster than prior approaches, explore more effectively, and generate interpretable short-term plans.

main_review: This paper is well-written and easy to follow. The motivation is clear and important: how to improve exploration for model-based RL. The proposed method of planning in a short term and encouraging using the old plan in order to encourage temporally extended exploration is neat and interesting. The evaluations are solid, comparing with 5 baselines on 8 benchmarks. Overall, this is a good paper.

Strength:
- Well-written and easy to follow
- The motivation is clear: how to improve exploration for model-based RL
- Proposed technique is interesting: learning a recurrent model to generate short-term plans at each time step, and only switching to the new plan if it is a lot better than the old plan.
- Solid evaluations. GPM is compared with 5 baselines on 8 benchmarks, and experiments show that GPM converges faster than the other baselines. Further analyses show that GPM indeed explores more effectively, and it also generates interpretable short-term plans 

Questions:
- What is the computational overhead of GPM compared with the baselines? Since it needs to generated short-term plans at each time step, does it take longer on average to run each training step? 
- It makes sense to encourage reusing the old plan during training since it encourages exploration. When the learned policy is deployed in test time, is it better to remove this bias towards the old plan? I would like to see a discussion on this from the authors.

summary_of_the_review: This paper is well-written and well-motivated. The proposed technique is interesting and the experimental section is strong. Overall, this is a good paper and I recommend acceptance.
review_title: null

Meta-review: This paper proposes an alternative approach to epsilon-greedy exploration by instead generating multi-step plans from an RNN, and then stochastically determining whether to continue with the plan or re-plan. The reviewers agreed that this idea is novel and interesting, that the paper is well-written, and that the evaluations are convincing, showing large improvements over epsilon-greedy exploration and more consistently strong performance than other baselines. While the original reviews contained some questions around discussion of related work and the simplicity of the evaluation environments, the reviewers felt these concerns were adequately addressed by the rebuttal. I agree the paper explores a very interesting idea and convincingly demonstrates its potential, and should be of wide interest to the deep RL community especially as it touches on many different subfields of RL: MBRL/planning, exploration, HRL, navigation, etc. I recommend acceptance as a spotlight presentation.
Decision: Accept (Spotlight)