Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a method to equip model-based deep reinforcement learning with posterior sampling for exploration where posterior sampling is approximated using a variational distribution approach.
main_review: One key challenge in RL is exploration. Posterior sampling or Thompson sampling has shown good theoretical properties for exploration in RL, and has good performance in certain RL exploration problems. This paper applies the posterior sampling idea to the model-based deep RL setting.

In order to apply posterior sampling to model-based deep RL, one need to maintain an approximate posterior distribution of the learned deep neural net model. The paper proposed a variational approach, called Event based Variational Distributions for Exploration (EVaDE), which consists of three types of Gaussian dropout convolutional layers. The three types of layers attempt to capture three aspects in a game: interaction between objects, random event emphasis, and random object translation. These dropout layers induced variational distributions are used to approximate the posterior distribution which is then used in posterior sampling for exploration.

The proposed dropout layers for variational distributions are appealing, and the ideas to use three types of layers to capture three kinds of effects is very interesting. But little is discussed in the paper on why to consider these three effects. There are some interesting visualizations for the layers in the appendix, but there is no quantitative analysis on whether each layer type indeed performs what it is designed to do. The ablation study is good to suggest the benefits of adding each layer type, but little is shown on what is the effect of each type of the layers. Does a certain type of layer help exploring a specific kind of hard-explored situation among the games?

In the experiments, the paper equips the EVaDE layers to an existing model-based RL method SimPLe and compare EVaDE-SimPLe with four baselines in 12 Atari games. EVaDE-SimPLe achieves good performance compared with the baselines, but It would be better if the paper provides more discussions on why the proposed method performs better in some games but not in some others. Does some games more difficult for exploration? Does the noisy layers hinder the agent's performance in some situations? In particular, in JamesBond, EVaDE-SimPLe performs worse than SimPLe and the ablation study seems to suggest no benefits from any layer in this game. Can EVaDE-SimPLe achieve the same performance if trained longer, or the noisy layers prevent the agent to achieve the best performance when exploration is less important?
summary_of_the_review: Pros
* Good idea to apply posterior sampling to model-based deep RL.
* Novel Gaussian dropout layers capturing objects for important events.
* Good experimental performance in 12 Atari games.

Cons
* Not enough explanations and discussions for the three proposed layers.
* No discussion on why EVaDE-SimPLe performs worse than SimPLe in some games.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper improves a model-based reinforcement learning method by Kaiser et al. (2020) by introducing three different types of neural network layers to the transition and reward model. These modifications are motivated by intuitions about what would be beneficial to model for object-based environments (in Atari games). There are experiments on 12 games in which the proposed method performs the best most of the time. Ablations show that using any one of the three proposed layers is also beneficial.

main_review: This paper proposes a straightforward and easy-to-implement modification to the SimPLe algorithm (Kaiser et al., 2020) - modify three layers in the transition and reward model they are using. This simplicity is good, but it requires much more rigorous experimentation since users of such simple modifications would want to know why and when these modifications work or don’t work. Otherwise, there are infinite tweaks one can do to an architecture - but which one actually helps? My main concern is that there are only 3 independent runs from which it is difficult to judge whether EVaDE is really better than SimPLe. In the ablations, it even seems that sometimes, using just one of the layers outperforms using all three.

summary_of_the_review: A simple modification to an existing architecture is nice, but it’s unclear from the experiments that this particular modification is really helping, due to the small amount of seeds.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studied exploration problems in model-based reinforcement learning. Some new neural network architecture are provided to incorporate object-based domains. 
main_review: This paper proposed some new ideas on neural network architecture construction built on the algorithm of PSRL. But overall I feel this paper does not tackle the key problem of reinforcement learning. Both the exploration strategy and planning algorithm are the same as previous works. The novel design of NN architecture is a bit too specific and lacks of principles. As a research paper, I feel it should provide more principle algorithm design ideas rather than adaptation on one or two specific domains. This may lose the generality. 

The approach described in Section 3 is a bit heuristics and not very related to RL. Why posterior sampling is done by multiplying
each parameter of these EVaDE layers by a perturbation drawn from a Gaussian distribution? What is the posterior here? In section 3.2, dropout is introduced. It's a bit hard to understand how you approximate the posterior of the model in the end. I hope the authors could organize the section in a more mathematic way.  

I respectively disagree on "an interesting aspect of designing for exploration is that the variational distributions can be helpful
even if they are not designed to approximate the posterior well, as long as they assist in perturbing the policy out of local optimums." This is still a very heuristic argument and lacks of theoretical support. PSRL has a strong theoretical guarantee and people then can spend effort on better approximation for posterior distribution. I feel this is a more principle way for exploration.
 
"it is easier to incorporate inductive biases derived from the domain knowledge of the task for learning the model, as the biases can be directly built into the transition and reward functions." I do not quite understand how you can do this and how it is supported. Especially "easier" with respect to what? Has this point been discussed anywhere else in the paper?

"Model-free agents explore the space of policies" What do you mean by that? How about value-based methods? I hope the authors could use accurate and well-supported arguments. 

"object-based" and "event" should be clearly defined in the beginning. 

I feel it should be necessary to have a full algorithmic box for Section 3.5 for self-completeness. How do you interact with the environment and how do you do the planning?
summary_of_the_review: This paper provides a novel neural network architecture design for model-based RL exploration. Overall, I feel it lacks principle explanation and is quite specific to a small set of domains.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an approach for Model-based reinforcement learning relying on Posterior Sampling for Reinforcement learning. Posterior sampling for reinforcement learning uses Thompson sampling to balance exploitation and exploration. However this requires maintaining posterior of dynamic transitions which is often intractable. The authors use dropout, which has been shown to approximate posteriors, to maintain approximate posteriors of the transitions and the reward function. The system model presented in the paper specifically applies to object-based tasks. Only three event convolutional layers use dropout: one for object interaction, one for event weighting and one for event translation. These layers can be inserted into any existing neural network model to provide an approximate posterior. The authors combine this model with a PPO agent trained on the model to show its empirical performance on Atari games with a limited number of interactions with the real environment.
main_review: Strengths:
- Using dropout to approximate posteriors and apply Thompson sampling to MBRL. (I am just curious to know whether it has already been applied in this context, see question 1. below)
- I also like that only 3 additional layers are needed and can be added to any model. 
- the paper is well written and easy to read.
- The experimental setup is well detailled (see specific comments below)
- the different ablation studies and in particular the ablation study of table 2 showing that using the 3 layers is good in general. Even though sometimes only one of the layers leads to better performance, using the 3 layers improves the performance of Simple(30) on most of the games.

Weaknesses:
- the technique seems to be only applicable to object-based tasks. However I like that it incorporates domain knowledge.

Minor comments:
- Please put the object-based task example in the introduction. I had to wait until section 3 to understand what the author were meaning by this.
- Table 1 title: mean scores of achieved -> mean scores achieved

Questions:
1. Are the authors aware of any other work applying dropout to perform Thompson sampling in MBRL?
2. In algorithm 1 it seems that the same posterior sample is used for all the episodes when training the policy with the environment. Isn't there a risk of the policy overfitting to this specific model? For instance the idea of Model ensemble TRPO (https://arxiv.org/abs/1802.10592) was to pick one member of the ensemble at random at each step. I understand that we would not want to do this at each step but we could imagine having a different posterior sample for each different episode. On the other hand, this could seem to be contrary to the idea of Thompson sampling where we would like the policy to use one posterior sample to drive its exploration.
3. Is there a reason explaining that Evade-Simple underperforms Simple(30) in 2 games?
4. Why approximating a posterior for the reward function part only? Wouldn't it be also possible to use it for the dynamics?

Section 4.2
- the agents collect 6400 real environment interactions: please add that this is with the randomly initialized policy
- paragraph 2: I am not sure to understand why the horizon length tradeoff influences the number of iterations.
- "we train the environment model for 45K steps in the first iteration and 15K steps in all subsequent iterations": does the authors mean that they use the past 45K steps to train the environment model? I am not sure, only 6400 interactions are collected before the first training iteration.
- why is "z" changing?
summary_of_the_review: I liked the paper. My grade is explained by the fact that I am not very familiar with the domain of vision and games, hence I might be unaware of important related works. I'll upgrade my score from the discussion with the other reviewers and the authors.
review_title: null

Meta-review: This paper proposes to implement posterior sampling for reinforcement learning for MBRL using three types of noisy convolutional layers inspired by object- and event-based domain knowledge. These layers are used to augment the SimPLe agent (Kaiser et al, 2020), resulting in the EVaDE-SimPLe agent, and experiments demonstrate that the EVaDE-SimPLe outperforms SimPLe on average across twelve Atari games.

The reviewers' opinions on the paper were mixed. The reviews highlighted several strengths of the paper: that using posterior sampling for exploration in MBRL is well-motivated (Reviewers 9oaA, XiQT) and that the simplicity of the proposed layers is appealing (Reviewers trzPm, XiQT). However, the reviewers also generally felt that the proposed method was overly specific to a particular domain (Reviewer gXzj, XiQT) and that there was not enough analysis demonstrating *why* the proposed layers work, in which cases they would not work, or why these modifications might be better than other similar modifications (Reviewers 9oaA, trzP, gXzj). Initially there were also some concerns raised by Reviewer trzP about the validity of the evaluation due to the number of seeds, though these concerns were addressed by the authors during the rebuttal.

I agree with the reviewers that the approach is interesting and that getting posterior sampling to work well in MBRL is an important problem. But I also find myself agreeing that the present approach is not analyzed in sufficient depth (the results are overly focused on just overall performance, rather than analyzing behaviors exhibited by the agents) and that it is unclear how well it would work in other domains (e.g. 3D settings). I therefore feel this work is not quite ready to be presented at ICLR, and recommend rejection.
Decision: Reject