EVaDE : Event-Based Variational Thompson Sampling for Model-Based Reinforcement Learning
Authors: Siddharth Aravindan
Dixant Mittal
Wee Sun Lee
Venue: ICLR 2022 Submitted (2022)
Keywords: Model-based Reinforcement Learning
Thompson sampling
Exploration

Abstract: Posterior Sampling for Reinforcement Learning (PSRL) is a well-known algorithm that augments model-based reinforcement learning (MBRL) algorithms with Thompson sampling. PSRL maintains posterior distributions of the environment transition dynamics and the reward function to procure posterior samples that are used to generate data for training the controller. Maintaining posterior distributions over all possible transition and reward functions for tasks with high dimensional state and action spaces is intractable. Recent works show that dropout used in conjunction with neural networks induce variational distributions that can approximate these posteriors.  In this paper, we propose Event-based Variational Distributions for Exploration (EVaDE), variational distributions that are useful for MBRL, especially when the underlying domain is object-based. We leverage the general domain knowledge of object-based domains to design three types of event-based convolutional layers to direct exploration, namely the noisy event interaction layer, the noisy event weighting layer and the noisy event translation layer respectively. These layers rely on Gaussian dropouts and are inserted in between the layers of the deep neural network model to help facilitate variational Thompson sampling. We empirically show the effectiveness of EVaDE equipped Simulated Policy Learning (SimPLe) on a randomly selected suite of Atari games, where the number of agent environment interactions is limited to 100K. 

Under review as a conference paper at ICLR 2022
EVADE: : EVENT-BASED VARIATIONAL THOMPSON SAMPLING FOR MODEL-BASED REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Posterior Sampling for Reinforcement Learning (PSRL) is well-known algo- rithm that augments model-based reinforcement learning (MBRL) algorithms with Thompson sampling. PSRL maintains posterior distributions of the environment transition dynamics and the reward function to procure posterior samples that are used to generate data for training the controller. Maintaining posterior distributions over all possible transition and reward functions for tasks with high dimensional state and action spaces is intractable. Recent works show that dropout used con- junction with neural networks induce variational distributions that can approximate these posteriors. this paper, we propose Event-based Variational Distributions for Exploration (EVaDE), variational distributions that useful for MBRL, pecially when the underlying domain is object-based based. We leverage the general domain knowledge of object-based domains design three types of event-based convolutional layers to direct exploration, namely the noisy event interaction layer, the noisy event weighting layer and the noisy event translation layer respectively. These layers rely on Gaussian dropouts and are inserted in between the layers of the deep neural network model help facilitate variational Thompson sampling. We empirically show the effectiveness of EVaDE equipped Simulated Policy Learning (SimPLe) on a randomly selected suite of Atari games, where the number of agent environment interactions is limited to 100K.
INTRODUCTION
Model-Based Reinforcement Learning (MBRL) has recently gained popularity for tasks that allow for very limited number of interactions with the environment (Kaiser et al. 2020). These algorithms model of the environment, that is learnt in addition to the policy, improve sample efficiency
in several ways; these include generating artificial training examples (Kaiser et al.] 2020: Sutton
1991), assisting with planning (Nagabandi et al.) 2017 Coulom 2006 Williams al. [0155] Curi et al. 2020) and guiding policy search (Levine & Koltun [2013] Chebotar et al.) 2017). Additionally, is easier to incorporate inductive biases derived from the domain knowledge of the task for learning the model, as the biases can be directly built into the transition and reward functions.
In this paper, we show how domain knowledge can also be used for designing exploration strategies in MBRL. Model free agents explore the space of policies and value functions; MBRL agents, on the other hand, explore the space transition dynamics and reward functions. One method for exploring the space of transition dynamics and reward functions is Posterior Sampling for Reinforcement Learning (PSRL) (Strens) 2000) Osband & Van Roy (017), which uses the Thompson sampling (Thompson, 1933) method of sampling the posterior of the model to explore other plausible models.
Maintaining the posterior is generally intractable and in practice, variational distributions are often used approximation the posterior (Aravindan & Lee 2021 Wang & Zhou 2020 Zhang et al.
2019).
Traditionally, variational distributions designed with two considerations in mind: inference and/or sampling should be efficient with the variational distribution, and the variational distribution should approximate the true posterior accurately possible. For MBRL., we propose to also design the

---

Under review as a conference paper at ICLR 2022
variational distribution to generate trajectories through parts of the state space that may potentially give high returns, for the purpose of exploration.
In MBRL, trajectories the learned model. One way to generate useful exploratory trajectories, is to perturb the reward function in the model so that a different part of the state space appears to contain high rewards, causing the policy to direct the trajectories towards those states. Another method is to perturb the reward function that parts of the state space traversed by the current policy appear sub-optimal, causing the policy to seek new trajectories.
generated in the state space by running policies that are optimized against
We focus on problems where the underlying domain is object-based, i.e., domains where the reward functions depend heavily on the locations of individual objects and interactions between objects, which we call events. An example of such an object based task, is the popular Atari game Breakout. In the game, the agent is rewarded when the ball hits a brick and will not lose a life as long as the paddle successfully hits the ball, both of which can be described as an interaction between two objects. The rewards are determined by the interactions between the ball and the bricks, the wall or the paddle.
For such domains, we present Event based Variational Distributions for Exploration (EVaDE), a set of variational distributions that can help generate useful exploratory trajectories for deep convolutional neural network models. EVaDE comprises three Gaussian dropout (Srivastava al. 2014) based convolutional layers, namely the noisy event interaction layer, the noisy event weighting layer and the noisy event translation layer respectively. The noisy event interaction layer is designed to provide perturbations to the reward function in states where multiple objects appear at the same location, randomly perturbing the value of interactions between objects. The noisy event weighting layer is designed to perturb the output of a convolutional layer at a single location; assuming that the output of the convolutional filters capture events, this would upweight and downweight the reward associated with these events randomly. The noisy event translation layer is designed to perturb trajectories that go through "narrow passages"; small translations can randomly perturb the returns from such trajectories, causing the policy to explore different trajectories.
These EVaDE layers can be used as normal convolutional layers and can be inserted in between the layers of the environment network models. When included in deep convolutional networks, noisy event interaction layers, noisy event weighting layers and noisy event translation layers generate perturbations on possible object interactions, on the importance of different events and on the positional importance of objects/events respectively, through the dropout mechanism which induces variational distributions over the model parameters (Srivastava et al. 2014 Gal & Ghahramani] 2016).
An interesting aspect of designing for exploration is that the variational distributions can be helpful even if they are not designed to approximate the posterior well, as long as they assist in perturbing the policy out of local optimums. After perturbing the policy, incorrect parts of the model will either be corrected by data or left incorrect if they are irrelevant to optimal behaviour.
Finally, we approximate PSRL by equipping the environment models of Simulated Policy Learning (SimPLe) (Kaiser et al. 2020) with EVaDE layers. We conduct experiments to compare EVaDE equipped SimPle (EVaDE-SimPL.e with various popular baselines Atari games. In the experiments conducted, all agents work in the low data regime, where the number of interactions with the real environment restricted to 100K. EVaDE-SimPLe agents achieve human normalized score of 0.78 on average in these games, which is 44% more than the mean score of 0.54 achieved by a recent low data regime method, CURL (Laskin et al., 2020), and 52% more than the mean score of 0.51 achieved by Aanila-SimPPl agents.
suite of 12 randomly selected
BACKGROUND AND RELATED WORK
Posterior sampling approaches like Thompson Sampling (Thompson) 1933) have been one of the more popular methods used to balance the exploration exploitation trade-off. Exact implementations of these algorithms have been shown to achieve near optimal regret bounds (Agrawal & Jia 2017 Jaksch et al. 2010). These approaches, however, work by maintaining posterior distribution over all possible environment models and/or action-valus functions. This is generally intractable in practice. Approaches that work by maintaining an approximated posterior distribution (Osband et al. 2016bb Azizzadenesheli et al. (0118), or approaches that use bootstrap ra-sampling to procure samples, (Osband et al. 2016a Osband & Van Roy 2015) have achieved success in recent times.
2

---

Under review as a conference paper at ICLR 2022
Variational inference procures samples from distributions that can be represented efficiently while also being easy to sample. These variational distributions are updated with observed data to approximate the true posterior accurately possible. Computationally cost effective methods such as dropouts have been known to induce variational distributions over the model parameters (Srivastava et al. Gal & Ghahramani 2016). Consequently, variational inference approaches that approximate
2014
the posterior distributions required by Thompson sampling have gained popularity (Aravindan & Lee
202
Wang & Zhou| 2020 Urteaga & Wiggins 2018 Xie et al. 2019)
Model-based reinforcement learning improves sample complexity at the computational cost of maintaining and performing posterior updates to the learnt environment models. Neural networks have been successful in modelling relatively complex and diverse tasks such as Atari games (Oh jt al.
2013 Ha & Schmidhuber 2018). Over the past few years, variational inference has been used
to re represent environment models, with the intention to capture environment stochasticity (Hafner et al.
2019 Babaeizadeh et al. 2018 Gregor et al. 2019).
SimPLe (Kisser et al., 2020) is one of the first algorithms to use MBRL to train agents to play video games from images. is also perhaps the closest to EVaDE, as not only employs an iterative algorithm to train its agent, but also uses additional convolutional network assisted by an autoregressive LSTM based RNN to approximate the posterior of the hidden variables in the stochastic model. Thus, similar to existing methods (Hainer et al., 2019 Babaeizadeh al. 2018 Gregor et al., 2019), these variational distributions are used for the purpose of handling environment stochasticity rather than improving exploration. To the contrary, EVaDE-SimPL.e is approximation to PSRL, that uses a Gaussian dropout induced variational distribution over deterministic reward functions solely for the purpose of exploration. Unlike SimPLe, which uses the stochastic model generate trajectories to train its agent, EVaDE-SimPP from the variational distribution. Moreover, with EVaDE, these variational distributions are carefully designed so as to explore different object interactions, importance of events and positional importance of objects/events, that we hypothesize are beneficial for learning good policies object-based tasks.
agents optimize for a deterministic environment model sampled
EVENT BASED VARIATIONAL DISTRIBUTIONS FOR EXPLORATION
Event-based Variational Distributions for Exploration (EVaDE) consist of a set of variational dis- tribution designs, each induced by a noisy convolutional layer. These convolutional layers can be inserted after any intermediate hidden layer in deep convolutional neural networks to help us construct approximate posteriors over the model parameters to produce samples from relevant parts of the model space. EVaDE convolutional layers use Gaussian multiplicative dropout to draw samples from the variational approximation of the posterior. Posterior sampling is done by multiplying each parameter, Of of these EVaDE layers by a perturbation drawn from a Gaussian distribution,
(11, (oing) ))) These perturbations are sampled by leveraging the reparameterization trick (Kingma let al. 2015 Salimans et al. 2017] Plappert et al. 2018 Fortunato et al. 2018) using noise sample from the standard Normal distribution, N (0, 1), as shown in Equation I The variance corresponding
to each parameter, (o is trained jointly with the model parameters Aenv-
Pene t O' (1 of cen e'~N(0,)
(1)
When the number of agent-environment interactions is limited, the exploration strategy employed by the agent is critical. In object-based domains, rewards and penalties are often sparse and occur when objects interact. Hence, the agent needs to experience most of the events in order to learn a good environment model. Generating trajectories that contain events is hence a reasonable exploration strategy. Additionally, very common issue with training using a very few number of interactions is that the agent may often get stuck in a local optima, prioritising an event, which relatively important, but may not lead to optimal solution. Generating potentially high return alternate trajectories that do not include that event is another reasonable exploration strategy.
With these exploration strategies in mind, we introduce three EVaDE layers, namely the noisy event interaction layer, the noisy event weighting layer and the noisy event translation layer. All the three layers are constructed with the hypothesis that the channels of the outputs of intermediate layers of deep convolutional neural networks either capture object positions, or events (interaction of multiple objects detected by multi-layer composition of the networks).
3

---

Under review as a conference paper at ICLR 2022
(c)
Figure I: (a) This image shows one noisy event interaction filter acting an input with channels. Here f an m x m noisy convolutional filter, which acts upon input patches the same location across different channels, noisily altering the value of events captured ar those locations. This image shows how the filters of the noisy event weighting layer weight the input channels. The filters f2, fa and fe randomly upweight and downweight the events captured by the channels Cı c2 and C. respectively. The white entries of the filter are entries that are set to zero, while the rest are trainable noisy model parameters. (c) The noisy event translation filter. The filters f1, f2, f's and fe noisily translate events/objects captured by the channels Ci C22 C3 and G. respectively. The white entries of the filter entries that are zero, while the rest trainable noisy model parameters. Gaussian multiplicative dropout is applied to all the non-zero parameters of all filters.
3.1 NOISY EVENT INTERACTION LAYER
The noisy event interaction layer is designed with the motivation of increasing the variety of events experienced by the agent. This layer consists of noisy convolutional filters, each having dimension of m x m X c, where e is the number of input channels to the layer. Every filter parameter multiplied shown in Equation I The filter dimension, m, is hyperparameter that
by Gaussian perturbation can be set so as to capture objects within a small m X m patch of an input channel. Assuming that the input channels capture the positions of different objects, a filter that combines the e input channels locally captures the local object interaction within the m x m patch. By perturbing the filter, different combinations of interactions can be captured; if the filter is used part of the reward function, it will correspondingly reward different interactions depending on the perturbation. The policy optimized for different perturbed reward functions is likely to generate trajectories that contain different events. Note that convolutional filters are equivariant, so the same filter will detect the event any unyee in the image and can result in trajectories that include the event different positions in the image.
We describe the filter in more detail. Every output pixel of the filter, vf,y- repreeentinng (i,j) pixel of the 4hhh output channel, can be computed shown in Equation 2 Here æ is the input to the layer with e input channels, Pat is the m x m patch (represented as a matrix) centred around 24.5- the (i,j) th pixel of the th input channel, 0% is the 1t channel of the th noisy convolutional filter, O the Hadamard product operator, and 1m is an m dimensional column vector whose every entry
Σ am
1m
(2)
Figure Talsooss how this filter is applied over the channels of the input.
3.2 NOISY EVENT WEIGHTING LAYER
Overemphasis on certain events is possibly one of the main causes due to which agents converge to sub-optimal policies in object based tasks. Hence, it would be useful to easily be able increase as well as decrease the importance of an event. For this filter, we assume that each input channel is already detecting an event and design variational distribution over model parameters that directly up-weights or down- weights the events captured by different input channels.
This layer can be implemented with the help of 1 x 1 noisy convolutional filters (each having a dimension of 1 x 1 x c as shown in Figure [b]. where c is the number of input channels. We denote the tth element of the 4thh filter in the layer as 0f.- To implement per channel noisy weighting, set every 0 as trainable model parameter, which has Gaussian dropout variance parameter
4

---

Under review as a conference paper at ICLR 2022
All other weights, i.e., 0% when 1 * k are set to 0. Thus each noisy event weighting layer has c tranable model parameters and trainable Gaussian dropout parameters. A pictorial representation of how this layer acts on its input is presented in Figure lb
associated with it facilitate noisy weighting shown in Equation |
Every output y corresponding to the (i. pixel of the kthh output channel, can be computed using Equation where 0% is the noisy scaling factor for the h input channel.
yt. =
We expect that inducing such a variational distribution that up-weights or downweights events randomly helps the agents learn from different events that are randomly emphasised model samples drawn from the distribution. This may eventually help them in escaping local optima caused by overemphasis of certain events.
by different
3.3 NOISY EVENT TRANSLATION LAYER
In object based domains, agent often has to perform specific sequence of actions to successfully gain some rewards and may be penalized heavily for deviation from the sequence. We refer to the specific sequence of actions as a "narrow passage". A small translation of the positions of the environment or other objects will often cause the agent to be unsuccessful. When random translations of obstacles, events or boundaries are performed within the reward function, the optimized policy may select different trajectory, possibly allowing it to escape from a locally optimal trajectory. We thus design the noisy event translation layer to induce a variational distribution over such model posteriors that can sample variety of translations of relevant objects.
The noisy noft-translation on an input with channels, is performed with the help of convolutional filters, each having dimension of X X These filters compute noisy weighted sum of the corresponding input pixel and the pixels near it to effect a noisy translation of the channel. Similar to the noisy event weighting layer, each filter of the noisy event translation layer acts one input channel. To achieve this, every parameter except the parameters of the 1*t channel of the h filter, & (which has a dimension of x m), and their corresponding dropout variances, is set to O, for all k. Moreover in the channel 0 only the middle column and row contain trainable parameters. Figure 1c shows a detailed pictorial representation of this structure of the filters. A random translation of up to pixels of the imput can be achieved by using (2n 1) x (2n + 1) noisy event translation layer.
Equation shows how yf,5- the (i, output pixel of the th channel, is computed. Here, P.t. is a m X m patch centred at (i.j)th pixel of the hthh input channel, 0% is the k*th channel of the hhth noisy convolutional filter, @ the Hadamard product operator, and 1m is an m dimensional column vector where all the entries are 1.
(0% v - OOOPP :PPt.. 1m
3.4 REPRESENTATIONAL CAPARILITIES OF EVADE EQUIPPED NETWORKS
Ideally, adding EVaDE layers would not cause the network to be unable to represent the true model, even if the layers are added for exploration purposes and do not accurately approximate the posterior. Theorem beloo states that this is indeed the case.
Theorem 1. Let in be any neural network. For any convolutional layer I, let m²(1) x na(l) x call) and m)(() x (1) x c!ll) denote the dimensions of its input und oulput respectively. Then, any function that can be represented by can also be represented by any network à E N. where Ni is the set of all neural networks that can be constructed by adding any combination of EVaDE layers to n, provided that, for every ENaDE layer I added, 1 uses a stride of ma(I) S ma,(), nz(1) S n)) and cx(1) < co)
Proof. The proof follows from the fact that every EVaDE layer i, that is added is capable of repre- senting the identity transformation. A detailed proof is presented in Appendix B
5

---

Under review as a conference paper at ICLR 2022
Next FratFramm
Iepar
Commenire
Ploel tnsbatting
Iepus Addin
Reward
Figure 2: The network architecture of the environment model used to train EVaDE-SimPLe agents.
If the added EVaDE layers results in variational distributions that poorly approximate the posterior, performance can indeed poorer. But with enough data, the correct model should still learnable since it is representable, long the optimization does not get trapped in poor local optimum.
3.5 APPROXIMATE PSRL WITH EVADE EQUIPPED SIMULATED POLICY LEARNING
Simulated Policy Learning (SimPLe) (aiser et al. 2020 is an iterative model based reinforcement learning algorithm, wherein the environment model learnt is used to generate artificial episodes to train the agent policy. In every iteration, the SimPLe agent first interacts with the real environment using its current policy. After being trained on the set of all collected interactions, the models of the transition and reward functions are then used as substitute to the real environment to train the policy of the agent to followed by it in its next interactions with the real environment. PSRL (Strens) 2000 Osband & Van Roy 2017). which augments MBRL with Thompson sampling, has a very similar iterative structure as that SimPLe. However, instead maintaining a single environment model, PSRL maintains a posterior distribution over all possible environment models given the interactions experienced by the agent with the real environment. The agent then optimizes a policy for environment model sampled from this posterior distribution. This policy is used in its real environment interactions of the next iteration. EVaDE equipped SimPLE approximates PSRL, by maintaining an approximate posterior distribution of the reward function with the help of the variational distributions induced by the three EVaDE layers.
Being an approximation PSRL, EVaDE-SimPLe agent has the same iterative training structure where it interacts with the real environment using its latest policy to collect interactions, learns transition model and an approximate posterior over the reward model parameters by jointly optimizing the environment model parameters fenv and the Gaussian dropout parameters of the reward model, rear with the help supervised learning and then optimizes its policy with respect to an environment characterized by the learnt transition function and reward model sample that is procured from the posterior with the help of Gaussian dropout as shown in Equation T This policy then used by the agent to procure more training interactions in the next iteration.
4 EXPERIMENTS
We randomly selected suite of 12 Atari games carry out our experiments. The test suite contains games with easy exploration such as Kangaroo, RoadRunner and Seaquest as well as BankHeist, Frostbite and Qbert, which are hard exploration games (Bellemare et al. 2016).
4.1 NETWORK ARCHITECTURE
the network architecture of the deterministic world model introduced in Kaiser et al. (2020) to train the environment models of the SimPLe agents, but do not augment with the convolutional inference network and the autoregressive LSTM unit. We provide a more detailed description of the network architecture in Appendix [ Readers are also referred to Kaiser
In our experiments
al. (2020) for more details.

---

Under review as a conference paper at ICLR 2022
The architecture of the environment model used by EVaDE agents is shown in Figure 2 This model is very similar to the one used by SimPLe agents, except that has a combination of a 3 x 3 noisy event translation layer, noisy event weighting layer and x noisy event interaction layer inserted before the fifth and sixth de-convolutional layers. The final de-convolutional layer acts as a noisy event interaction filter when computing the reward, while it acts as a normal de-convolutional layer while predicting the next frame. Sharing weights between layers allows us to achieve this. We insert EVaDE layers way that perturbs only the reward function and not the transition dynamies.
We reuse the network architecture proposed in Kaiser et al. (2200) to train the policies in both the SimPLe and WaDDE-SmPLL agents using Proximal Policy Optimization (PPO) (Schulman et al. (017). We disable the Bernoulli dropout while training EVaDE MiDD-SiiPLe agents. All the other hyperparameters used for training the policy network and environment are the same as the ones used in Kaiser et al. 2202))
4.2
EXPERIMENTAL DETAILS
The training regimen that we use to train all the agents is the same and is structured similarly to the the setup used by Kaiser et al. (0020). As in Kaiser et al. (2020), the agents, initialized with a random policy, collect 6400 real environment interactions before starting the first training iteration. In every subsequent iteration, every agent trains its environment model with its collection of real world interactions, refines its policy by interacting with the environment model, if it is vanilla-SiMPLe agent, or a transition model and a reward model sampled from the approximate posterior, if it is an EVaDE-SimPLe agent, and then collects more interactions with this refined policy.
PSRL regret bounds scale linearly with the length of an episode experienced by the agent in every iteration (Osband et al. 2013). Shorter horizons, however, run the risk of the agent not experiencing anything relevant before episode termination. To balance these factors, we set the total number of iterations to 30, instead of 15. We allocate an equal number of environment interactions to each iteration, resulting in 3200 agent-enviromment interactions per iteration. The total number of interactions that each SimPLe and EVaDF-SimPLLe agent procures (about 102K) similar to SimPLe agents trained in Kaiser et al. (2020), which allocates double the number of interactions per iteration, but trains for only 15 iterations. To disambiguate between the different SimPLe agents, we refer to the SimPLe agents trained in our paper and Kaiser et al. (2020) as SimPLe(30) and SimPLe respectively from here on.
We try to keep the training schedule of EVaDE-SimPL.e and SimPL.e(30) similar to the training schedule of the deterministic model in Kaiser et al. (2020) so as to keep the comparisons fair. We train the environment model for 45K steps in the first iteration and 15K steps in all subsequent iterations. Policy training is done with the help of 16 parallel PPO agents, which collect a batch of 50 environment interactions from the learnt model. In every iteration these parallel agents collect z 1000 batches of interactions, where = = 1 in all iterations except iterations 8, 12, 23 and 27 where 2 and in iteration 30, where the real environment. However, the effect of these interactions on the policy (numbering 102K) is minuscule when compared to the 28.8M transitions experienced by the agent when interacting with the learnt environment model.
3. The policy is also trained when the agent interacts with
4.3 RESULTS
We train three independent runs of SimPLe(30) and EVaDE-SimPP agents for every game in the 12 game test suite. We report the average scores achieved by these agents Table (the standard errors of these runs are given in Table 2 along with the scores achieved by the baselines SimPLe (Kaiser jet al. 2020), CURL (Laskin et al. 2020), OverTrained Rainbow (Kielak) 2020) and Dati--fffccient Rainbow (van Hasselt et al.) 2019). For each baseline, report the number of games in which they score more (or less) than E EVaDE-SimPLe, which are counted as wins (or losses) for the bascline. We also report the mean Human Normalized Scores (HNS) achieved by all agents. Additionally, we include some visualizations that help us understand the functionality of the EVaDE layers in Appendix E.3
EVaDE-SimPLo agents achieve the highest score in 5 of the 12 games in the test suite, outperforming every baseline in at least 8 games. Moreover, the effectiveness of the three noisy convolutional filters
7

---

Under review as a conference paper at ICLR 2022
Table Comparison of the mean scores achieved by EVaDE-SimPLe and SimPLe((3) agents with different baselines when the number of agent-environment interactions are restricted to 100K.
Game
SimPLe SimPLe(30)
CURL
OTRainbow Eff. Rainbow EVaDE-SimPLe
BankHeist
34.2
51.9
131.6
182.1
51
236.3
BattleZZne
4031.2
4823
14870
4060.6
10124.6
10427
Breakout
16.4
19.8
4.9
9.84
1.9
25.6
CrazyClimber
62583.6
39591
12146.5
21327.8
16185.3
61949
DemonAttack
208.1
98.6
817.6
711.8
508
145.5
Frostbite
236.9
257.4
1181.3
231.6
866.8
262.1
JamesBond
100.5
255.2
471
112.3
301.6
187.5
Kangaroo
51.2
385
872.5
605.4
779.3
1181
Krull
2204.8
4460
4229.6
3277.9
2851.5
5406
Qbert 1288.8
3753
1042.4
509.3
1152.9
3069
RoadRunner
5640.6
2886
5661
2696.7
9600
8915
Seaquest 683.3
402.3
384.5
286.92
354.1
737.3
Vs EVaDE (W/L) 10L,2W
10L,2W
8L,4W
11L,IW
8L,4W
HNS
0.36
0.51
0.54
0.31
0.38
0.78
Table 2: Scores (mean + standard error) of SimPLE agents when equipped with all three EVaEE filters individually and when equipped with all filters. All scores are averaged over three independent training runs.
Game
SimPle (30)
Inter. Layer
Weight. Layer
Trans. Layer
EVaDE-SimPLe
BankHeist
51.9-=00.6
94.8+-50.6
194.1422.4
168.5 + 26.4
236.3 = 57.6
BattleZone
4823=1098
6375 2580
6146+11565
5761 + 705
10427 530
Breakout
19.8=008
19.5 5.5
19.3 9.4
19.3
25.6 = 4.9
CrazyClimber
39391=13350
19440=1441
71624+3115
63612=2980
61949=4594
DemonAttack
98.6==22.1
146.3 +41.6
143.9 + 6.2
99.4 19.7
145.5 22.3
Frostbite
257.4+:2.8
246.8=4.9
256+:3.1
263 2.7
262.1 12.116..
JamesBond
255.2-18.2
126+881
161.5 86.2
170.8 + 104.1
187.5 443
Kangaroo
385::301
1640-1940
277.1 +: 54.8
1075+:
1181 142
Krull
4460
3790-11200
4300 1:543
4752 + 1727
5406 255
Qbert
37531376
1975 983
936 +263
2959 906
3069 = 997
RoadRunner
2886=1214
7909+860
5064 1268
8365+2150
8915 314
Seaquest
402.3+8.6
590 + 154.5
452.9 + 199
702.7 +76.9
737.3 + 101.5
HNS
0.51
0.58
0.6
0.68
0.78
to improve exploration can be empirically verified as EVaDE-SimPL manages to attain higher mean scores than SimPLe(30) routine. EVaDE-Simbl
of the 12 games, even though both methods follow the same training also scores mean HNS of 0.78, which is 44% higher than the score of 0.54 achieved by the best performing baseline, CURL, and 52% more than the mean HNS of 0.51 achieved by SimPLe(30). Additionally,
agents also surpass the human performances (Brittain et al. 2019) in the games of CrazyClimber, RoadRunner and Krull.
4.4 ABLATION STUDIES
We also conduct ablation studies by equipping SimPLe(30) with just of the three EVaDE layers to ascertain whether all them aid in exploration. We this by just removing the other two layers from the EVaDE environment network model (see Figure Note that when the noisy event interaction filter removed, Gaussian multiplicative dropout is not applied to the sixth de-convolutional layer.
The mean scores achieved when SimPLe(30) is equipped with only the noisy event interaction layer, the noisy event weighting layer or the noisy event translation layer individually along with the scores of SimPLee(30) and WaDPP-imPPLe are presented in Table
8

---

Under review as a conference paper at ICLR 2022
Figure 3: Learning curves of EVaDE- agents, SimPLe(30) agents and agents which only add one of the EVaDE layers.
can be seen that individually, each filter achieves a higher HNS than SimPLee33O), thus indicating that, average, all the filters help in aiding exploration. Moreover, we see that with the exception of the noisy event translation layer, the increase in HNS when the other two EVaDE layers are added
individually is not large. Looking at the learning curves presented in Figure 3 it can possibly be said that an increase scores of SimPLe(30) equipped with one of the EVaDE layers at a particular iteration would mean an increase in scores of IVaDE-SimPLe albeit in later iterations. This pattern can clearly be seen in the games of BankHeist, Frostbite, Kangaroo, Krull and Qbert. This delay in learning could possibly be attributed to the agent wasting its interaction budget exploring areas suggested by one of the layers that is ineffective for that particular game. However, we hypothesise that since all the layers provide different types of exploration, their combination is more often helpful than wasteful. This is validated by the fact that WaDE--imPPL achieves a higher mean HNS than any other agent in this study.
CONCLUSION
In this paper, present Event-based Variational Distributions for Exploration (EVaDE), set of variational distributions over reward functions. EVaDE is composed of three noisy convolutional layers, namely the noisy event interaction layer, the noisy event weighting layer and the noisy event translation layer which are designed to generate trajectories through parts of the state space that may potentially give high returns, especially in object-based domains. These layers can be inserted in between the layers of the environment network models to induce variational distributions over the model parameters that generate perturbations on object interactions, importance of events and positional importance of objects/events through the dropout mechanism. Samples drawn from these variational distributions are used to generate simulations to train the policy of SimPLe agent.
We conduct experiments on a randomly selected test suite of 12 Atari games, where the agents are only allowed 100K interactions with the real environment. EVaDE-SimPLe agents achieve a mean human normalized score (HNS) of 0.78, which is 44 % and 52% more than the mean scores achieved by CURL and vanilla-SmmPLe agents respectively. EVaDE- P-e agents also manage to surpass human performances in three games. We also find that each layer, when added individually to SimPLe results in a higher mean HNS. Also, the three noisy convolutional layers complement each other well, as EVaDF-SimPLe agents achieve higher mean HNS than agents which add only one noisy layer.

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
We include the anonymized codebase that was used to conduct the experiments in the supplementary. We also mention the scores achieved by the individual runs in Table in the appendix. We expect the mean scores of each VaDE agent to fall within the range mentioned in Table 2
REFERENCES
Shipra Agrawal and Randy Jia. Optimistic Posterior Sampling for Reinforcement Learning: Worst-Case Regret Bounds. In I. Guyon, U. V. Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL http:: httptttpppp/pppooeeciiiggg.gos neur ips cc/ pdf
Siddharth Aravindan and Wee Sun Lee. State-Aware Variational Thompson Sampling for Deep Q-Networks. In 20th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2021), pp. 124-132, 2021.
Kamyar Azizzadenesheli, Emma Brunskill, and Animashree Anandkumar. Efficient Exploration through Bayesian Deep Q-Networks. 2018 Information Theory and Applications Workshop (ITA), pp. 1-9. IEEE, 2018.
Mohammad Babacizadeh, Chelsea Finn, Dumitru Erhan, Roy H. Campbell, and Sergey Levine. Stochastic Variational Video Prediction. In International Conference on Learning Representations, 2018. htt.htttt://ppeeee
ew n forum? d=rk4 9Mg-9MM
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying Count-Based Exploration and Intrinsic Motivation. In Advances in Neural Information Processing Systems, pp. 1471-1479, 2016.
Mare Brittain, Joshua Bertram, Xuxi Yang. and Peng Wei. Prioritized Sequence Experience Replay. CoRR, abs/1905. 2019.
12726
Yevgen Chebotar, Mrinal Kalakrishnan, Ali Yahya, Adrian Li, Stefan Schaal, and Sergey Levine. Path Integral Guided Policy Search. In 2017 IEEE international conference on robotics and automation (ICRA), pp. 3381-3388. IEEE, 2017.
Rémi Coulom. Efficient Selectivity and Backup Operators in Monte-Carle Tree Search. In Interna- tional conference on computers and games, pp. 72 83. Springer, 2006.
Sebastian Curi, Felix Berkenkamp. and Andreas Krause. Efficient Model- -ased Reinforcement Learn- ing through Optimistic Policy Search and Planning. Advances in Neural Information Processing Systems, 33, 2020.
Meire Fortunato, Mohammad Azar, Bilal Piot, Jacob Menick, Matteo Hessel, Ian Osband, Alex Graves, Volodymyr Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, Charles Blundell, and Shane Legg. Noisy Networks for Exploration. In International Conference on Learning Representations, 2018.
net./ http//ormmmmddyy/ccePere
Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. In international conference on machine learning, pp. 1059- PMLR, 2016.
Karol Gregor, George Papamakarios, Frederie Besse, Lars Buesing, and Theophane Weber. Temporal Difference Variational Luto--Encoder In International Conference on Learning Representations, 2019. JRLlhttps: /ttp:tts///opeeeevieee.. wtt.et//ooouumm..d.aaiaaiiaieiie
David Ha and Jürgen Schmidhuber. Recurrent World Models Facilitate Policy Evolution. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 2455 2467, 2018.
10

---

Under review as a conference paper at ICLR 2022
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning Latent Dynamies for Planning from Pixels. In International Conference on PMLR, 2019.
Machine Learning, pp.
Thomas Jaksch, Ronald Ortner, and Peter Auer. Near- Year-Optimal Regret Bounds for Reinforcement Learning. Journal of Machine Learning Research, 1(AAp)) 2010.
Eukasz Kaiser, Mohammad Babaeizadeh, Piotr Mitos, Blažej Osiński, Roy H. Campbell, Konrad
Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, Afroz Mohiuddin, Ryan Sepassi, George Tucker, and Henryk Michalewski. Model Based Reinforcement Learning for Atari. In International Conference on Learning Representations, 2020. URL https:// openreview Creenreveewnee/forrreeeeeeeeeeee
www. met//formm iid=SIxdiiii SISSSxxxCCP JH
Kacper Kielak. Importance of Using Appropriate Baselines for Evaluation of Data Dafi-eeffc in Deep Reinforcement Learning for Atari. CoRR, abs/2003.10181, 2020. URL htt.t:h/taaaattt
org/aba/2003.111 10181
Diederik P Kingma, Tim Salimans, and Max Welling. Variational Dropout and the Local Reparam- cterization Trick. In Proceedings of the 28th International Conference on Neural Information Processing Pestems-Volumte 2. pp. 1375-2588 2015.
Michael Laskin, Aravind Srinivas. and Pieter Abbeel. CURL: Contrastive Unsupervised Repre- sentations for Reinforcement Learning. In International Conference on Machine Learning, pp. 5639-5650. PMLR, 2020.
Sergey Levine and Vladlen Koltun. Guided Policy Search. In International conference on machine learning, pp. 1-9. PMLR, 2013.
Anusha Nagabandi, Gregory Kahn, Ronald S. Fearing, and Sergey Levine. Neural Network Dy- namies for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning. CoRR, Abs//708.00556, 2017. URL http:::/a.xi......//// ott/ abaa 1008. 02596
Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard Lewis, and Satinder Singh. Action-Condition Video Prediction Using Deep Networks Atari Games. In Proceedings of the 28th International Conference Neural Information Processing Systems-Volume 2, pp. 2863-28711 2015.
Ian Osband and Benjamin Van Roy. Bootstrapped Thompson Sampling and Deep Exploration. arXiv preprint 1rXiv:1507.000
2015.
Ian Osband and Benjamin Van Roy. Why is Posterior Sampling Better than Optimism for Reinforce- ment Learning? In International Conference Machine Learning, pp. 2701-2710, 2017.
Ian Osband, Daniel Russo, and Benjamin Van Roy. (More) Efficient Reinforcement Learning via
Posterior Sampling. In Advances in Neural Information Processing Systems, pp. 3003-3011, 2013.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep Exploration via Bootstrapped DQN. In Advances in Neural Information Processing Systems, pp. 4026-4034, 201
Ian Osband, Benjamin Van Roy, and Zheng Wen. Generalization and Exploration via Randomized Value Functions. In Proceedings of the 33rd International Conference on Machine Learning- Volume 48, pp. 2377-2388 JMLR. org, 2016b.
Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Y. Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, and Marcin Andrychowicz. Parameter Space Noise for Exploration. In International Conference Learning Representations, 2018. URL /ttp:///////eeeeeiiie/ net htt///oommmiidyyyBBa
Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution Strategics as a Scalable Alternative to Reinforcement Learning. arXiv preprint arXiv:1703.03864 2017. 2017.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms. CoRR, abs/11077 001177770044444 06347,, 2017. URL htttp://axxiv.ogg// orgbbs//
1707. 06347
11

---

Under review as a conference paper at ICLR 2022
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A Simple Way to Prevent Neural Networks from Overfitting. Journal of Machine Learning Research, 15(56):1929-199) 1958, 2014. URL http:// wtw..org./.................o.eeo http://tt...pppee///////////
sri srivastava 4a ht.ml
Malcolm Strens. A Bayesian Framework for Reinforcement Learning. In International Conference on Machine Learning. volume 2000, pp. 943-950, 2000.
Richard Sutton. Dyna, an Integrated Architecture for Learning, Planning, and Reacting. ACM Sigart Bulletin, 2(4): 160-163, 1991.
William R Thompson. On the Likelihood that One Unknown Probability Exceeds Another in View of the Evidence of Two Samples. Biometrika, 25(3/4):285-29) 1933.
lñigo Urteaga and Chris Wiggins. Variational Inference for the Multi-Aree' Contextual Bandit. In International Conference on Artificial Intelligence and Statistics, pp. 698-706. PMLR, 2018.
Hado van Hasselt, Matteo Parametric Models in
Hessel, and John Aslanides.
When to In NeurIPS,
Use
Reinforcement Learning?
http:://ppprr....... wttpppprrr.ips.cccppppee..e 9579 when-to- parameerricc.mm momerric-models in rei.norreoeottt----e---ee--r--ee learning
pp.
14322-14333, 2019.
URL.
Ashish Vaswani, Samy Bengio, Eugene Brevdo, Francois Chollet, Aidan N. Gomez, Stephan Gouws, Llion Jones, łukasz Kaiser, Nal Kalchbrenner, Niki Parmar, Ryan Sepassi, Noam Shazeer, and Jakob Uszkoreit. Tensor2Tensor for Neural Machine Translation. CoRR, abs/180 1as./7403.74166 2018.
URL http:: org/ abs 1803. 07416
Zhendong Wang and Mingyuan Zhou. Thompson Sampling via Local Uncertainty. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 10115-10125. PMLR, 13-18 Jul 2020. URL
mlr. www..press/vvl1. 22zess/vl13/wang22aa html
Grady Williams, Andrew Aldrich, and Evangelos Theodorou. Model Predictive Path Integral control Using Covariance Variable Importance Sampling. arXiv preprint arXiv: /509.01 149, 2015.
Sirui Xie, Junning Huang, Lanxin Lei, Chunxiao Liu, Zheng Ma, Wei Zhang, and Liang Lin. NADPEx: An On-Policy Temporally Consistent Exploration Method for Deep Reinforcement Learning. International Conference on Learning Representations, 2019. URL https://
openreview net/
ic9mm
Ruiyi Zhang, Zheng Wen, Changyou Chen, Chen Fang, Tong Yu, and Lawrence Carin. Scalable Thompson Sampling via Optimal Transport. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 87-96. PMLR, 2019.
12

---

Under review as a conference paper at ICLR 2022
A EVADE-SIMPLE AS AN APPROXIMATION OF PSRL
We present the pseudocode EVaDE-SimPLa Algorithm | As mentioned in Section an EVaDE-SimPLe agent has the same iterative training structure as SimPLe and PSRL. Lines 6- 10 of the algorithm show the first step of each iteration where the agent interacts with the real environment using its latest policy to collect interactions. The agent then updates its posterior distribution over the environment model parameters by jointly optimizing the environment model parameters fenv which include the parameters of the transition and reward function and the Gaussian dropout parameters of the reward network Oene with the help of supervised learning (line 11). sample from this approximate posterior distribution is then acquired with the help of Gaussian dropout shown in lines 12-17 of the algorithm. As shown in the subsequent lines 19-26, the agent updates its policy by optimizing the parameters of the policy network, 6x by interacting with this environment sample. This optimized policy is used by the agent to procure more training interactions by interacting with the real environment in the next iteration.
Algorithm Approximate PSRL with EVaDE equipped Simulated Policy Learning
1: Initialize agent policy, environment model and dropout parameters 0nr fenu: Jen respectively
2: Initialize empty real environment interaction dataset Dreat < {
3: for iteration ... 7 do
4: < 0
5: while kieea rorld interactions not collected do
D Interact with real environment
6: if terminal then
7:
Start new episode, initialize start state Policy(s. (s, 0x))
8:
9:
5', r t Interact Real. World a) Dreat Dreat U {(s, a, r,);,,)));;;+
10:
Oenos Tenv <- Supervised Learn(0 Deen Dreat) for layer in the environment model do if i is an EVaDE layer then
D Learn a variational posterior Draw sample from the posterior
12:
13:
14:
Sample e¹ ~ N))
Øi env (1 (1 + Tenue")
16:
else
18:
< 0. Dsim + 0, steps + 0
b Train policy with environment sample
19:
while keam interactions not completed do if 8 is terminal or then
20:
21:
Start new episode, initialize start state Policy(s, 0x))
22:
23:
s', r < Interact Env Sample(Oenv: 8, a) Daim Dsim J [(s, a, s')), s', steps steps
24:
25:
steps mod update_ fillutefffrueccceaaa = then
26:
0x +- cainforcement__Lermff Daim)
27:
return 0g
B PROOF OF THEOREM
We provide the proof for Theorem which is restated below, in this section.
Theorem. Let n be any neural network. For any convolutional layer I, let m²(1) (I) x ci(l) and ml(l) (1) x co (I) denote the dimensions of its input and output respectively. Then, any function that can be represented by n can also be represented by any network
N,, where N is the set of all neural networks that can be constructed by adding any combination of EVaDE layers to provided that, for every EVaDE layer added, ī uses stride of 1, mz(l) I m.()), (1)) S n.()) and (i()) &oll).
13

---

Under review as a conference paper at ICLR 2022
B.1 NOTATIONS
NEURAL NETWORKS
Any function f represented by k-layer neural network in is an ordered composition of the functions f2, ... .fx computed by its layers N1, .2, .N respectively, i.e., f - fk o fk-1 o.... fI.
CONVOLUTIONALL LAYERS
Any m X n convolutional layer has a total of m X n X (1) x (1) learnable parameters, where (1) and (I) are the number of channels in the input and output of the layer respectively. The parameters of any convolutional layer can be partitioned into co filters, where each filter has m x n x (() parameters, and is responsible for computing one output channel.
We denote the set of parameters of any convolutional layer by 0. We denote the set of parameters of the 4tth filter by Oz, and the parameters of the 1t channel of this filter by 0% We denote the (i, ji parameter of the nth channel of the ttth filter by 0!.5., For noisy convolutional layers, we have a learnable Gaussian dropout parameter attached every parameter of the convolutional layer (see Equation 1.. We use oke of and d. denote the dropout parameters of the htth filter, the 1th channel of the th filter and the (i. parameter of the 4th channel of the hthh filter respectively.
STRIDES
stride is hyperparameter of a convolutional layer, that determines the number of pixels of the input that each convolutional filter moves, compute the next output pixel.
B.2 IMPLICATIONS THE CONSTRAINTS IN THEOREM 1
Theorem states that every EVADE layer added uses stride of 1 and satisfies the constraints m²(1) < (.) ny (i) I ()) and c; (i) I (T). This means that for any inserted EVaDE layer, every output dimension is at least as large as corresponding input dimension. This eventually implies that for every EVaDE layer, input and output dimensions match, i.e., m,,)) (i),), ni; = (i) and (1) = (i).
To see why. let us assume that the EVaDE layers [,, I,.--.la T. are inserted, in order, in between the consecutive layers of m,
layers N, and N141 of neural network As N; and Ne+1 must have m;(N++1) = (N,(N) n.) = (N;) and (N++1) = (N,.... This implies that the dimensions of the input to layer In match the dimensions of the output of the layer 4 i.e., m²(() (L)), nT,))) (T.) and (t) (T,) However, under the constraints imposed in Theorem D every output dimension is greater than or equal to its corresponding input dimension for every EVaDE layer. Thus, matching the output dimensions of l with the input dimensions of i is only possible if the input and output dimensions match for every EVaDE layer 1y.-. ... le that is added.
With the above implications, the constraint of using a stride of forces SAME padding for every EVaDE layer, and also ensures that patches centred around every (i.j) tht the input are used to compute the outputs. This an important implication that will help us prove the claims that all EVaDE layers can represent the identity transformation.
pixel of every channel in
B.3 CLAIMS
We prove the three following claims by construction, i.e., showing that there is combination of parameters using which these layers can perform the identity transformation.
Claim 1. The noisy event interaction layer can represent the identity transformation.
Proof. Let us assume an m X m noisy event interaction layer. With the help of the observations made in the previous section, and the constraints also ensure that the number of filters in this layer is equal to the number of input channels.
ensured of using patches centred around every input a!,, Vi, j.
14

---

Under review as a conference paper at ICLR 2022
The identity transformation can be achieved with the following parameter assignments.
The dropout parameter of set to zero.
corresponding to every convolutional layer parameter 0% is
The layer parameter corresponding the central entry of the kth layer of the k* convolu- tional filter, i.e., o%
1 1. is set to vk.
All other convolutional layer parameters set to O.
As stated in Equation the event interaction layer computes the outputs vi.s Vi,j, k using the following equation.
Σ 1=0
1 0h
PPP...) 1.
Vi.j
Applying the above parameter assignments, get v . ttth filter, that is 1, aligns with =².g- This is the required identity transformation.
the only non-zero parameter in the
Claim 2. The noisy event weighting layer represent the identity transformation.
Proof. The noisy event weighting layer uses input channels. Consequently, 05. is just a single trainable parameter instead of grid of trainable parameters as in the other two EVaDE layers.
convolutional filters, where is the number of
The identity transformation can be achieved with the following parameter assignments.
The dropout parameter of corresponding to every convolutional layer parameter 0 is set to zero.
The layer parameter corresponding to the 4 layer of the th convolutional filter, i.c., 0 is set to vk.
AAll other convolutional layer parameters
to O.
This is valid assignment, as the only parameters set are trainable, while the other parameters are forced to be set to 0 by construction (see Section 3.2).
As stated in Equation 3 the event interaction layer computes every output vi using the following equation.
wf.s 02
Setting 0 1 and of 0 Vk. yields vtts = 2t, Vi,jkk, which is the identity transformation required.
Claim 3. The noisy event translation layer can represent the identity transformation.
Proof. In this case, can use the parameter assignments stated in the proof of Claim to produce an identity transformation. This is possible, since we construct the noisy event translation layer with the same structure of an m convolutional layer with the number of filters equalling the number of input channels. Moreover, the only non-zero parameter (which is set to 1) in the 4thh filter, gt. T is in the middle row and middle column of its hthh channel, making it valid assignment for the noisy event translation layer (see Section 3.3
As stated in Equation 4 the event interaction layer computes every output vi, using the following equation.
Vi. = - (0% Pat. 1m
15

---

Under review as a conference paper at ICLR 2022
As in the case with the noisy event interaction layer, substituting these assignments, get vi, oy Vi k, which is the identity transformation required.
B.3.1 PROOF OF THEOREM
We have to prove that all elements from N, the set of neural networks that can be constructed by adding any combination of EVaDE layers the neural network in, can represent the functions represented by k-layered neural network in.
Let ń be general element from N, that adds the EVaDE layers L, (2, .. bm in order, after the layers Ni. Naz:".. Ni.. of the neural network n, where ij-1 ij ij+1 V2 < j i1 2 0, fm S k. Adding an EVaDE layer after No refers to it being added after the input layer and before the first layer of n. Note that more than one EVaDE layer can be added after any layer N, of
1 and
Also, let f1, f2. - fo be the functions computed by the layers N1, N2, NN of n respectively. Thus the function represented by is f fı. o fx-1 o.... f1.
Let f1. f.. fm be the functions computed by the EVaDE layers 1 1......i Im respectively. Thus the ..... f1. As all fi. fz, ... fm
function computed by ñ is f = f& o fk-1 . fm o fin. fi fú o... can learn to represent the identity transformation, f can learn to represent This implies that n can represent any function represented by i.
C VARIATIONAL DISTRIBUTIONS USING DROPOUTS
Variational methods are used to approximate inference and/or sampling when using intractable posterior distributions. These methods work by using variational distributions that facilitate easy sampling and/or inference, while approximating the true posterior as closely as possible.
These methods require the user to define two distributions, the prior p(0), and the variational distribution (0). Given a set of training samples D = (X, Y), where is the set of input samples and Y the of corresponding labels, variational methods work to minimize the KL-divergence between the learnt variational distribution (0) and the true posterior D). This is equivalent to maximizing the Evidence Lower Bound (ELBO) shown below.
KL(q(0),P((DD) = / g(0) log (0) d0
D)
Now,
(( =p(0]X, p(0)p(XXX)))) p(0)p(Y X, (
p(X,Y)
p(X,
p(O)p(Y|X, 0)p(X).
P(X,Y)
Substituting the value for D),
KL(g(O),p(0)) ()) = q()) log q(0)p(X,Y)
d8
p(O)p(Y ())(
p())p(Y XX, 0) do + ((0) log P(X.Y)
(()
(()) log
P(X)
/ ((0) log
p(0)p(Y dl + log P(X,Y)
(0)
P(X)
/ ((0) log
(()) d0 / ((() lo ( 0)d8 log P(X,Y)
(())
P(X)
(TT(g(0),p(0)) (()) log ( 0)d0 log P(X,Y)
P(X)
16

---

Under review as a conference paper at ICLR 2022
Since P(X.Y) and P(X) are constants with respect to 8, the set of parameters that minimize P() are the same as the ones that maximize the ELBO, i.e.,
arg min KL(g(0), = arg max / (10))log -KT((()))(0))
C.I DROPOUTS AS VARIATIONAL DISTRIBUTIONS
Gal Ghahramani (2016) introduces the usage of dropout mechanism induce variational distributions, samples from which are used to approximate the ELBO. The first term of the BLBO can be re-written as,
fa(e)
((00)
where every (z,, y4) is training example from D.
This integral can be approximated by averaging out the prg-babbabiilties using several samples drawn from the variational distribution q(0) (Equation 5
I (()) Σ
log plu x4, 0)d0 R
log (, where 0, ~ (())
(5)
Neural networks that different types of dropouts help maintain variational distributions g()) that approximate posteriors over deep Gaussian processes (Gal & Ghahramani [016 Kingma et al.) 2015). Procuring a sample from this posterior using (0) is easy, as random dropped out network corresponds to a sample from the posterior over these deep Gaussian processes.
D NETWORK ARCHITECTURES
In this section, we detail the network architectures used for training the environment models of SimPLe (Kaiser et al. 2020) and EVaDe-SimPLe, and the policy network architectures used by both the methods.
D.1 ENVIRONMENT NETWORK ARCHITECTURE
D.1.1 SIMPLE
In our experiments we use the network architecture of the deterministic world model introduced in Kaiser et al. (2020) to train the environment models of the SimPLe agents, but do not augment it with the convolutional inference network and the autoregressive LSTM unit.
Given four consecutive game frames and action input, the network jointly models the transition and reward functions, as it predicts the next game frame and the reward using the same network. The network consists of dense layer, which outputs pixel embedding of the stacked input frames. This layer followed by a stack of six x convolutional layers, each with a stride 2. These layers are followed by six 4 x 4 de-convolutional layers. For 1 < i the 1th de-convolutional layers, take in as input, the output of the previous layer, as well as the output of the 6 - 7th convolutional layer. The last de-convolutional layer takes in input the output of its previous layer and the dense pixel embedding layer. An embedding of the action input is multiplied and added to the input channels of every de-convolutional layer. The outputs from the last de-convolutional layer passed through softmax function to predict the next frame. The outputs from the last de-convolutional layer is also combined with the output of the last convolutional layer and then passed through fully connected layer with 128 units followed by the output layer to predict the reward.
D.1.2 EVADE-SIMPLE
The architecture the environment model used by EVaDE agents is shown in Figure 2 This model resembles the model of SimPLe agents until the fourth de-convolutional layer. All the stand-alone
17

---

Under review as a conference paper at ICLR 2022
EVDEE layers that use, use stride of and SAME padding as keep the size of the inputs and outputs of the layer same. As the EVaDE layers are added only to the reward function, we split the network into two parts, predicts the reward (the reward network) respectively. We denote the last two de-convolutional layers in each part d dg and d' dg respectively.
that predicts the next frame (the transition network) and one that
As shown in Figure 2 in the transition network, the outputs of the fourth de-convolutional layer and the first convolutional layer are passed d3. d' takes as inputs the outputs of dž and the pixel embedding layer.
The reward network adds combination of 3 x 3 noisy event translation layer, noisy event weighting layer and a 1 x noisy event interaction layer which are inserted before both dğ and de- dz shares weights with dř. and takes in the outputs the previous event interaction layer and the first convolutional layer as inputs. Likewise, dg shares weights with dg. and takes in the outputs of the previous event interaction layer and the pixel embedding layer as imputs. Moreover, we also apply Gaussian multiplicative dropout the weights of dg. make it act event interaction layer. As with SimPLe agents, an embedding of the action input is multiplied and added to the input channels of every de-convolutional layer (also shown in Figure 2
The outputs of d' are passed through a softmax to predict the next frame, while the outputs of d'g are combined with the output of the last convolutional layer and passed through fully connected layer with 128 units followed by the output layer to predict the reward.
D.2 POLICY NETWORK
The policy network for both SimPLe and EvADE-SimPP agents consists of two convolutional layers followed by a hidden layer and an output layer. The inputs to the policy network are four consecutive game frames, which are stacked and passed through two 5 x convolutional layers, both of which use stride of 2. These convolutional layers are followed by fully connected layer with 128 hidden units, which is followed by the output layer, that predicts the stochastic policy, i.e., the probabilities corresponding to each valid action, and the value of the current state of the agent.
E EXPERIMENTAL DETAILS
E. HUMAN NORMALIZED SCORE
We use the human normalized scores from Brittain et al. (2019) as defined in Equation 6 to compare agents.
Scoreagent Scorerandom
HNS
(6)
Scorepuman Scorerandom
where Scoreagent ScorChuman and Scorerandom denote the scores achieved by agent being evaluated, human and an agent which acts with a random policy respectively.
We also list the baseline scores achieved by humans and random agents, as listed in Brittain et al. (2019) in Table 3 for easy access.
E.2
MORE EXPERIMENTAL DETAILS
We present the scores achieved by all three independent runs of all agents trained in Table Figure shows the learning curves as shown in Figure 3 with error bars equal to a width of 1 standard error on each side.
18

---

Under review as a conference paper at ICLR 2022
Table 3: Baseline human and random values used to calculate Human Normalized Scores
Game
Human Score
Random Score
BankHeist
753.1
14.2
BattleZone
37187.5
2360
Breakout
30.5
1.7
CrazyClimber
35829.4
10780.5
DemonAttack
1971
152.1
Frostbite
4334.7
65.2
JamesBond
302.8
29
Kangaroo
3035
52
Krull
2665.5
1598
Qbert
13455
163.9
RoadRunner
7845
11.5
Seaquest
42054.7
68.4
19

---

Under review as a conference paper at ICLR 2022
Table 4: Scores achieved by every independent run of every SimPLE agent and when equipped with different EVaDE layers
Game
SimPLe(300
Inter. Layer
Weight. Layer
Trans. Layer
EVaDE-SimPL.e
133.1
85
232.2
218.4
155.3
BankHeist
9.375
12.5
195.3
128.8
205.9
13.13
186.9
154.7
158.4
347.8
4156
1313
9250
4438
10844
BattleZone
6969
8031
4250
6000
9375
3344
9781
4938
6844
11063
20.09
8.563
29.78
14.45
35.38
Breakout
18.25
25.03
27.56
23.64
20.91
20.94
24.81
0.625
19.69
20.5
54569
57534
75300
69494
55194
CrazyClimber
51244
58522
74141
59838
59934
12959
62266
65431
61503
70719
55.31
134.1
215.3
129.7
169.1
DemonAttack
112.7
155.5
71.41
105.9
100.9
127.7
142.2
152.2
62.5
166.4
261.3
256.6
250
263.4
268.4
Frostbite
251.9
241.6
259.4
258.1
249.4
259.1
242.2
259.1
267.5
268.4
268.8
12.5
59.38
371.9
232,8
JamesBond
240.6
282.8
332.8
117.2
101.6
256.3
82.81
92.19
23.44
228.1
987.5
3294
293.8
25
1144
Kangaroo
56.25
1588
362.5
1481
956.3
112.5
37.5
175
1719
1444
5639
6124
5150
5548
5569
Krull
4873
3103
3290
7266
4906
2868
2142
4460
1443
5744
3002
3935
516.4
1193
1082
Qbert
4151
1133
1430
4190
3916
4106
857
873.4
3494
4208
2793
8794
3034
5709
8666
RoadRunner
831.3
8744
4763
6763
8541
5034
6188
7397
12622
9538
392.5
791.3
221.3
649,4
536.3
Seaquest
419.4
286.3
288.1
604.4
813.8
395
692.5
849.4
854.4
861.9
20

---

Under review as a conference paper at ICLR 2022
201
16:.0
Direccion Leve
13:00
90.0
8:00
Donnshitak
Plccar)
eq
oor Lanee
611
Layer
541
2:000
30
254
en Lors
Stapert
(301
18:30
Layt
3000
Desulaee
Figure 4: Learning curves of EVaDE-SimPLee agents, SimPLee(30) agents and agents which only add one of the EVaDE layers with error bars of standard error.
E.3 VISUALIZATIONS OF THE EVADE LAYERS
We present some visualizations of the input and output feature maps of the noisy event interaction layer, the noisy weighting layer and the noisy translation layer. All these visualizations were obtained from the final trained model, with the learned weights and variances.
In Figures Saarres5nandde and show illustrations of output feature map detecting interactions between the right facing green-coloured enemy ships and the right facing blue-coloured divers given different input images from the game of Seaquest. We also show two input feature maps, which seem to capture the positions of these objects at the same locations. We observe that the pixels in the output feature map in Figure33aan are brighter the locations where the two objects are close each other, whereas in the same feature map these pixels are dimmer when the two objects are separated by some distance (in Figure 6).
In Figures 7 and we show two feature maps before and after passing them through a noisy event weighting layer. The inputs for these visualizations were taken from the game of Breakout. The input feature maps to the noisy event weighting layer seem to capture the bricks from the input image. The output feature map in Figure is an upweighted version its input, as the pixels seem be brighter. On the other hand, the output feature map in Figure & seems to weight its input feature map, as the pixels seem a lot dimmer. The weighting factors for the input-output pairs shown in Figures and g are 1.93 and 0.57 respectively.
21

---

Under review as a conference paper at ICLR 2022
(a) Input Frame
(b) Input Feature Map
(c) Input Feature Map
(d) Oulput Feature Map
Figure 5: This figure shows output feature map(channel) that captures interactions between two input feature maps when passed through the noisy event weighting layer. Here, the interaction between the blue diver and the green enemy is captured in (d).
(a) Input Frame
(b) Input Feature Map
(c) Input Feature Map (d) Output Feature Map
Figure 6: This figure shows the same output feature map of the noisy event interaction layer as shown in Figure [5] but when there is no interaction between the blue and green objects. The blue and green objects in (a) are separated by some distance, and thus the interaction is not shown in (d).
(a) Input Frame
(b) Input Feature Map
(c) Output Feature Map
Figure 7: This figure shows an output feature map (channel) that up-weights the corresponding input feature map when passed through the noisy event weighting layer.
22

---

Under review as a conference paper at ICLR 2022
(a) Input Frame
(h) Input Feature Map
(c) Output Feature Map
Figure 8: This figure shows output feature map (channel) that down-weights the corresponding input feature map when passed through the noisy event weighting layer.
(a) Input Frame
(b) Input Feature Map (c) Output Feature Map
Figure 9:
This figure shows the function of the noisy translation layer. The output feature map translates the input pixels to its top, bottom, left and right different degrees
show the input and output feature maps of game state from Krull, before and after passing it through a noisy event translation layer. The input feature map seems to capture different objects from the input image. The translation effect in output feature map can be seen clearly as every light pixel in the input seems to have lightened up the pixels to its top, bottom, left and right to different degrees.
In Figure 9
E.4 CODEBASE USED AND HYPERPARAMETERS
We build our SimPLe and EVaDE-SimPLe agents by utilizing the implementation of SimPLe agents from Vaswani et al. (2018). To keep the comparison fair, we use the same hyperparameters as used by Vaswani et al. (2018) to train all our agents. The codebase in Vaswani et al. (2018) uses an Apache 2.0 license, thus allowing for public use and extension of their codebase.
E.5 COMPUTATIONAL HARDWARE USED
We train our agents on cluster NVIDIA RTX 2080 Ti GPUs with an Intel Xeon Gold 6240 CPU. The total time taken to train 3 independent runs of all 5 algorithms on the test suite of 12 games was around 79 days and 20 hours.
23


Reviews:
[{"review_id": "tgcCZmSM1y", "reviewer": "Reviewer_9oaA", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "One key challenge in RL is exploration. Posterior sampling or Thompson sampling has shown good theoretical properties for exploration in RL, and has good performance in certain RL exploration problems. This paper applies the posterior sampling idea to the model-based deep RL setting.\n\nIn order to apply posterior sampling to model-based deep RL, one need to maintain an approximate posterior distribution of the learned deep neural net model. The paper proposed a variational approach, called Event based Variational Distributions for Exploration (EVaDE), which consists of three types of Gaussian dropout convolutional layers. The three types of layers attempt to capture three aspects in a game: interaction between objects, random event emphasis, and random object translation. These dropout layers induced variational distributions are used to approximate the posterior distribution which is then used in posterior sampling for exploration.\n\nThe proposed dropout layers for variational distributions are appealing, and the ideas to use three types of layers to capture three kinds of effects is very interesting. But little is discussed in the paper on why to consider these three effects. There are some interesting visualizations for the layers in the appendix, but there is no quantitative analysis on whether each layer type indeed performs what it is designed to do. The ablation study is good to suggest the benefits of adding each layer type, but little is shown on what is the effect of each type of the layers. Does a certain type of layer help exploring a specific kind of hard-explored situation among the games?\n\nIn the experiments, the paper equips the EVaDE layers to an existing model-based RL method SimPLe and compare EVaDE-SimPLe with four baselines in 12 Atari games. EVaDE-SimPLe achieves good performance compared with the baselines, but It would be better if the paper provides more discussions on why the proposed method performs better in some games but not in some others. Does some games more difficult for exploration? Does the noisy layers hinder the agent's performance in some situations? In particular, in JamesBond, EVaDE-SimPLe performs worse than SimPLe and the ablation study seems to suggest no benefits from any layer in this game. Can EVaDE-SimPLe achieve the same performance if trained longer, or the noisy layers prevent the agent to achieve the best performance when exploration is less important?", "summary_of_the_paper": "The paper proposes a method to equip model-based deep reinforcement learning with posterior sampling for exploration where posterior sampling is approximated using a variational distribution approach.", "summary_of_the_review": "Pros\n* Good idea to apply posterior sampling to model-based deep RL.\n* Novel Gaussian dropout layers capturing objects for important events.\n* Good experimental performance in 12 Atari games.\n\nCons\n* Not enough explanations and discussions for the three proposed layers.\n* No discussion on why EVaDE-SimPLe performs worse than SimPLe in some games.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "SG9ToenJPFx", "reviewer": "Reviewer_gXzj", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposed some new ideas on neural network architecture construction built on the algorithm of PSRL. But overall I feel this paper does not tackle the key problem of reinforcement learning. Both the exploration strategy and planning algorithm are the same as previous works. The novel design of NN architecture is a bit too specific and lacks of principles. As a research paper, I feel it should provide more principle algorithm design ideas rather than adaptation on one or two specific domains. This may lose the generality. \n\nThe approach described in Section 3 is a bit heuristics and not very related to RL. Why posterior sampling is done by multiplying\neach parameter of these EVaDE layers by a perturbation drawn from a Gaussian distribution? What is the posterior here? In section 3.2, dropout is introduced. It's a bit hard to understand how you approximate the posterior of the model in the end. I hope the authors could organize the section in a more mathematic way.  \n\nI respectively disagree on \"an interesting aspect of designing for exploration is that the variational distributions can be helpful\neven if they are not designed to approximate the posterior well, as long as they assist in perturbing the policy out of local optimums.\" This is still a very heuristic argument and lacks of theoretical support. PSRL has a strong theoretical guarantee and people then can spend effort on better approximation for posterior distribution. I feel this is a more principle way for exploration.\n \n\"it is easier to incorporate inductive biases derived from the domain knowledge of the task for learning the model, as the biases can be directly built into the transition and reward functions.\" I do not quite understand how you can do this and how it is supported. Especially \"easier\" with respect to what? Has this point been discussed anywhere else in the paper?\n\n\"Model-free agents explore the space of policies\" What do you mean by that? How about value-based methods? I hope the authors could use accurate and well-supported arguments. \n\n\"object-based\" and \"event\" should be clearly defined in the beginning. \n\nI feel it should be necessary to have a full algorithmic box for Section 3.5 for self-completeness. How do you interact with the environment and how do you do the planning?", "summary_of_the_paper": "This paper studied exploration problems in model-based reinforcement learning. Some new neural network architecture are provided to incorporate object-based domains. ", "summary_of_the_review": "This paper provides a novel neural network architecture design for model-based RL exploration. Overall, I feel it lacks principle explanation and is quite specific to a small set of domains. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "KzkRYgcbxYN", "reviewer": "Reviewer_trzP", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposes a straightforward and easy-to-implement modification to the SimPLe algorithm (Kaiser et al., 2020) - modify three layers in the transition and reward model they are using. This simplicity is good, but it requires much more rigorous experimentation since users of such simple modifications would want to know why and when these modifications work or don’t work. Otherwise, there are infinite tweaks one can do to an architecture - but which one actually helps? My main concern is that there are only 3 independent runs from which it is difficult to judge whether EVaDE is really better than SimPLe. In the ablations, it even seems that sometimes, using just one of the layers outperforms using all three.\n", "summary_of_the_paper": "This paper improves a model-based reinforcement learning method by Kaiser et al. (2020) by introducing three different types of neural network layers to the transition and reward model. These modifications are motivated by intuitions about what would be beneficial to model for object-based environments (in Atari games). There are experiments on 12 games in which the proposed method performs the best most of the time. Ablations show that using any one of the three proposed layers is also beneficial.\n", "summary_of_the_review": "A simple modification to an existing architecture is nice, but it’s unclear from the experiments that this particular modification is really helping, due to the small amount of seeds.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "IFNq_DvgSNc", "reviewer": "Reviewer_XiQT", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Using dropout to approximate posteriors and apply Thompson sampling to MBRL. (I am just curious to know whether it has already been applied in this context, see question 1. below)\n- I also like that only 3 additional layers are needed and can be added to any model. \n- the paper is well written and easy to read.\n- The experimental setup is well detailled (see specific comments below)\n- the different ablation studies and in particular the ablation study of table 2 showing that using the 3 layers is good in general. Even though sometimes only one of the layers leads to better performance, using the 3 layers improves the performance of Simple(30) on most of the games.\n\nWeaknesses:\n- the technique seems to be only applicable to object-based tasks. However I like that it incorporates domain knowledge.\n\nMinor comments:\n- Please put the object-based task example in the introduction. I had to wait until section 3 to understand what the author were meaning by this.\n- Table 1 title: mean scores of achieved -> mean scores achieved\n\nQuestions:\n1. Are the authors aware of any other work applying dropout to perform Thompson sampling in MBRL?\n2. In algorithm 1 it seems that the same posterior sample is used for all the episodes when training the policy with the environment. Isn't there a risk of the policy overfitting to this specific model? For instance the idea of Model ensemble TRPO (https://arxiv.org/abs/1802.10592) was to pick one member of the ensemble at random at each step. I understand that we would not want to do this at each step but we could imagine having a different posterior sample for each different episode. On the other hand, this could seem to be contrary to the idea of Thompson sampling where we would like the policy to use one posterior sample to drive its exploration.\n3. Is there a reason explaining that Evade-Simple underperforms Simple(30) in 2 games?\n4. Why approximating a posterior for the reward function part only? Wouldn't it be also possible to use it for the dynamics?\n\nSection 4.2\n- the agents collect 6400 real environment interactions: please add that this is with the randomly initialized policy\n- paragraph 2: I am not sure to understand why the horizon length tradeoff influences the number of iterations.\n- \"we train the environment model for 45K steps in the first iteration and 15K steps in all subsequent iterations\": does the authors mean that they use the past 45K steps to train the environment model? I am not sure, only 6400 interactions are collected before the first training iteration.\n- why is \"z\" changing?", "summary_of_the_paper": "This paper proposes an approach for Model-based reinforcement learning relying on Posterior Sampling for Reinforcement learning. Posterior sampling for reinforcement learning uses Thompson sampling to balance exploitation and exploration. However this requires maintaining posterior of dynamic transitions which is often intractable. The authors use dropout, which has been shown to approximate posteriors, to maintain approximate posteriors of the transitions and the reward function. The system model presented in the paper specifically applies to object-based tasks. Only three event convolutional layers use dropout: one for object interaction, one for event weighting and one for event translation. These layers can be inserted into any existing neural network model to provide an approximate posterior. The authors combine this model with a PPO agent trained on the model to show its empirical performance on Atari games with a limited number of interactions with the real environment.", "summary_of_the_review": "I liked the paper. My grade is explained by the fact that I am not very familiar with the domain of vision and games, hence I might be unaware of important related works. I'll upgrade my score from the discussion with the other reviewers and the authors.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to implement posterior sampling for reinforcement learning for MBRL using three types of noisy convolutional layers inspired by object- and event-based domain knowledge. These layers are used to augment the SimPLe agent (Kaiser et al, 2020), resulting in the EVaDE-SimPLe agent, and experiments demonstrate that the EVaDE-SimPLe outperforms SimPLe on average across twelve Atari games.

The reviewers' opinions on the paper were mixed. The reviews highlighted several strengths of the paper: that using posterior sampling for exploration in MBRL is well-motivated (Reviewers 9oaA, XiQT) and that the simplicity of the proposed layers is appealing (Reviewers trzPm, XiQT). However, the reviewers also generally felt that the proposed method was overly specific to a particular domain (Reviewer gXzj, XiQT) and that there was not enough analysis demonstrating *why* the proposed layers work, in which cases they would not work, or why these modifications might be better than other similar modifications (Reviewers 9oaA, trzP, gXzj). Initially there were also some concerns raised by Reviewer trzP about the validity of the evaluation due to the number of seeds, though these concerns were addressed by the authors during the rebuttal.

I agree with the reviewers that the approach is interesting and that getting posterior sampling to work well in MBRL is an important problem. But I also find myself agreeing that the present approach is not analyzed in sufficient depth (the results are overly focused on just overall performance, rather than analyzing behaviors exhibited by the agents) and that it is unclear how well it would work in other domains (e.g. 3D settings). I therefore feel this work is not quite ready to be presented at ICLR, and recommend rejection.
Decision: Reject