Escaping Stochastic Traps with Aleatoric Mapping Agents
Authors: Augustine N. Mavor-Parker
Kimberly A Young
Caswell Barry
Lewis Griffin
Venue: ICLR 2022 Submitted (2022)
Keywords: Curiosity
Neuroscience
Acetylcholine
Uncertainty
Reinforcement learning
Intrinsic Rewards

Abstract: When extrinsic rewards are sparse, artificial agents struggle to explore an environment. Curiosity, implemented as an intrinsic reward for prediction errors, can improve exploration but fails when faced with action-dependent noise sources. We present aleatoric mapping agents (AMAs), a neuroscience inspired solution modeled on the cholinergic system of the mammalian brain. AMAs aim to explicitly ascertain which dynamics of the environment are unpredictable, regardless of whether those dynamics are induced by the actions of the agent. This is achieved by generating separate forward predictions for the mean and aleatoric uncertainty of future states with reducing intrinsic rewards for those states that are unpredictable. We show AMAs are able to effectively circumvent action-dependent stochastic traps that immobilise conventional curiosity driven agents.

Under review as a conference paper at ICLR 2022
ESCAPING STOCHASTIC TRAPS WITH ALEATORIC MAP- PING AGENTS
Anonymous authors Paper under double-bling review
ABSTRACT
When extrinsic rewards sparse, artificial agents struggle explore envi- ronment. Curiosity, implemented as an intrinsic reward for prediction errors, can improve exploration but fails when faced with action-dependent noise sources. We present aleatoric mapping agents (AMAs), neuroscience inspired novel form of curiosity modeled on the cholinergic system of the mammalian brain. AMAs aim to explicitly ascertain when dynamics of the environment are unpredictable, even if those dynamics are induced by the actions of the agent. This is achieved by generating separate forward predictions for the mean and aleatoric uncertainty of future states, reducing intrinsic rewards for those transitions that are unpredictable. We demonstrate that in range of environments AMAs are able to circumvent action deeionndependeen stochastic traps that immobilise conventional curiosity driven agents.
INTRODUCTION
Efficient exploration is a central problem in reinforcement learning. Exploration is particular chal- lenging in environments with sparse rewards- requiiine agents to navigate with limited guidance (e.g. Sutton & Barto (2018); Pathak al. (2017); Burda et al. (2018b), see Weng (2020) for review). A notable exploration method that effectively deals with sparse rewards is curiosity driven learning-wher agents are equipped with self-supervised forward prediction model that employs prediction errors as intrinsic rewards Schmidhuber (1991b); Pathak et al. (2017) 7): Schmidhuber (1991a). Curiosity is built upon the intuition that in unexplored regions of the environment, the forward prediction error of the agent's 's internal model will be large Schmidhuber (1991b); Pathak et al. (2017). As a result, agents are rewarded for visiting regions of the state space that they have not previously occupied. If, however, particular state transition is impossible to predict, it will trap curious agent Burda et al. (2018b); Schmidhuber (1991a). a). This is referred to as the noisy TV problem (e.g. Burda et al. (201 8b); Schmidhuber (1)),)), the etymology being that naively curious agent could dwell on the unpredictability of noisy TV screen.
Several existing curiosity-like methods Burda et al. (2018b); Pathak al. (2017: 2019) aim avoid noisy TVs or "stochastic traps" Shyam et al. (2019). Nevertheless, employing dynamics based prediction errors as intrinsic rewards difficult as current methods either fail when stochastic traps are
or require an ensemble of dynamics models Pathak et al. (2017; 2019): Shyam al. (2019); Burda al. (2018a). This work presents aleatoric mapping agents (AMAs), which uses aleatoric uncertainty estimation Kendall & Gal (2017) to escape stochastic traps.
AMAs are both inspired by and build upon proposals developed within neuroscience Yu & Dayan (2005), that suggest expected uncertainties in top down predictions are signalled by the modulation of cortical acetylcholine, We build on this theory by algorithmically demonstrating a functional purpose for expected aleatoric uncertainties that is consistent with, but also further specifies, Yu & Dayan (2005)'s predictions of cholinergic activity in the Posner task. Our work adds further credence to Yu & Dayan (2005)'s is call for experimental tests of whether acetylcholine signals epistemic or aleatoric uncertainties. Accordingly, a blueprint for a VR mouse task, along with theoretical predictions of cholinergic uncertainty signalling in such task, is presented in Section 5.

---

Under review as a conference paper at ICLR 2022
2 BACKGROUND
2.1 EPISTEMIC AND ALEATORIC UNCERTAINTIES
The uncertainty of statistical model can be described as the sum of two theoretically distinct types of uncertainty: epistemic uncertainty and aleatoric uncertainty (e.g. Hora (1996), see Hüllermeier Waegeman (2021) for a review). Epistemic uncertainty measures the errors of model's prediction that can be minimised with additional experience and learning Hüllermeier & Waegeman (2021). As result, an agent using epistemic uncertainties intrinsic rewards tends to value dynamics it has not previously encountered, and hence cannot predict accurately, but could learn to predict in the future (e.g. Osband et al. (2016)). More concretely, epistemic uncertainty can be considered to be the expected information gain of observing the next predicted state Mukhoti et al. (2021). On the other hand, prediction errors that are due to aleatoric uncertainties are, by definition, result of unpredictable processes Hüllermeier & Waegeman (2021). Therefore, any agents that receives intrinsic rewards for aleatoric dynamics risks being trapped, as exemplified by the noisy TV problem Schmidhuber (1991a); Burda et al. (2).
Direct estimation of epistemic uncertainty surrounding future states would be an ideal basis for a curious agent but tractable epistemic uncertainty estimation with high dimensional data is an unsolved problem Gal (2016). We implicitly incentivise agents to seek epistemic uncertainties by removing the aleatoric component from the total prediction error. This is similar to methods that separate epistemic and aleatoric uncertainties in return predictions Clements et al. (2019), or within latent variable model Depeweg et al. (201 alllowwgg for the construction of policies that are rewarded for exploring their environments and punished for experiencing alcatoric uncertainty. However, as far are aware, framework to reduce intrinsic rewards for those state transitions with aleatoric uncertainty.
the first to compute alcatoric uncertainties within scalable curiosity
2.2 CURIOSITY AND INTRINSIC MOTIVATION IN REINFORCEMENT LEARNING
Curiosity-driven Pathak et al. (2017) agents assign value to states of the environment that they deem to be "interesting" Still & Precup (2012); Schmidhuber (1997). How a curiosity based method computes whether state is "interesting" Still & Precup (2012); Schmidhuber (1997) is usually its defining characteristic. The original formulation of curiosity used prediction errors directly as intrinsic rewards Schmidhuber (1991b). The noisy TV problem quickly emerged when using this naïve approach in stochastic environments Schmidhuber (1991a). In order to evade the allure of stochastic traps, the first proposed solution to the noisy TV problem implements "interesting" Still Precup (2012); Schmidhuber (1997) as prediction errors that reduce over time Schmidhuber (1991a); Kaplan & Oudeyer (2007). Others consider "interesting" Still & Precup (2012); Schmidhuber (1997) to mean a high dependency between present and future states and actions (i.e. "interesting" things are predictable Still & Precup (2012) or controllable Mohamed & Jimenez Rezende (2015)).
Inverse dynamics feature (IDF) curiosity Pathak et al. (2017) rejuvenated interest in using one step prediction errors as intrinsic rewards. IDF curiosity avoids stochastic traps by computing prediction errors with features that aim only contain information concerning stimuli the agent can affect Pathak et al. (2017). Further experiments Burda et al. (2) showed that simple one-step prediction errors also work effeetively within a random representation space generated by feeding state observations through a randomly initialised network. Burda et al. (201 8a) also showed the (IDF) approach is vulnerable action-dependent noisy TVs-demonstratte by giving the agent 'remote control' to noisy TV in the environment that could induce unpredictable environment transitions. This motivated random network distillation (RND) Burda al. (2018b), which removes dynamics from network to learn to predict the output of another fixed randomly initialised network at each state, using the resulting error as intrinsic rewards.
the prediction problem affogether-instrueetin
Other exploration methods explicitly leverage uncertainty quantification for exploration. The canon- ical approach is "optimism under uncertainty", which in its most basic form means weighting the value of state-actions pairs inversely to the number of times they have been experienced (Sutton & Barto, 2018, 8, p. 36). Known as count based methods Strehl & Littman (2008); Bellemare et al. (2016), this approach Rocktäschel (2020). However, is not feasible to count state visitations in many environments where there is large number of unique states Bellemare et al. (2016). "Pseudo-count" methods exchange
shown to reliably evade noisy TVs in minigrid environments Raileanu &
2

---

Under review as a conference paper at ICLR 2022
tabular look up tables for density models to estimate analogous intrinsic reward to counts in large state spaces Bellemare et al. (2016). Nevertheless, prediction error based methods for exploration remain a key component in state of the art algorithms deployed in high dimensional state spaces (e.g. Badia et al. (2019)).
Attempts have been made to reward Epistemic uncertainty directly. This typically requires a posterior distribution over model parameters, which is intractable without approximations such as ensembles or variational inference (e.g. Houthooft et al. (2016)). Osband et al. (2016) instantiated an ensemble Lakshminarayanan et al. (2017) approach into the final layer of a deep Q-network-reward agents for epistemic value uncertainty. Pathak al. (2019) use the variance of ensemble predictions being used as intrinsic rewards, while Shyam et al. (2019) reward experience of epistemic uncertainty within an ensemble of environment models. Lastly, some uncertainty estimation methods have recently been developed that enforce a smoothness constraint in the representation space Mukhoti et al. (2021); van Amersfoort et al. made from learned representations- but these approaches have not yet been adopted in reinforcement learning. All in all, uncertainty often plays a role in different formulations of curiosity, which is why we looked at models acetylcholine- neuromodulator associated with uncertainty- for inspiration in building curiosity models capable of avoiding distractions.
its
for sensible estimations of uncertainty to be
2.3 ACETYI.CHOLINE
In the mammalian brain acetylcholine is implicated in range of processes including learning and memory, fear, novelty detection, and attention Ranganath & Rainer (2003); Pepeu & Giovannini (2004); Acquas et al. (1996); Barry et al. (2012); Yu & Dayan (2005); Hasselmo (2006); Giovannini et al. (2001); Parikh et al. (2007). Traditional views-Supported by the rapid increase in cholinergic tone in response to environmental novelty and demonstrable effects on neural plasticity- emphasised its role as a learning signal, generating physiological changes that favour encoding of new information over retrieval Hasselmo (2006).
Notably, Yu & Dayan (2003) proposed alternative perspective, suggesting that acetylcholine signals the expected uncertainty of top down predictions, while modulation of norepinephrine is result of unexpected uncertainties. More concretelty, Yu & Dayan (2003)'s model can be seen as favouring bottom up sensory input over top down predictions if predictions are believed to be maacuratee-consistenn with evidence that shows acetylcholine inhibits feedback connections and strengthens sensory inputs Hasselmo (2006). However this approach does not explicitly separate epistemic and alcatoric uncertainties Yu & Dayan (2005). In contrast, the utility of quantifying epistemic uncertainties for exploration has been widely recognised in the RI. literature (e.g. Oshand et al. (2016); Pathak et al. (2019)). Here we demonstrate a potential use of aleatoric uncertainties in exploring agents both biological and artificial. Namely, aleatoric uncertainties can be used to divert attention away from unpredictable dynamics when using prediction errors as intrinsic rewards. This is similar to model proposed by Parr & Friston (2017), suggesting acetylcholine may indicate expected uncertainties in top down predictions within an MDP.
In this context we propose extension to Yu & Dayan (2005)'s dichotomy. Specifically, suggest that in the mammalian brain, cortical acetylcholine signals expected alcatoric uncertainties, while norepinephrine is modulated by epistemic uncertainties both expected and unexpected. This formulation is attractive in an MI. framework, providing a means to avoid stochastic traps. while also being consistent with empirical biological data Hasselmo (2006); Yu & Dayan (2003). Testing this hypothesis calls for experiments to clarify the nature of the uncertainty signalled by acetylcholine, which we present in Section 5.
3 METHOD
AMAs operate in the arena of Markov decision processes that consist of states s E S, actions a E A, and rewards r E R C R Sutton & Barto (2018). At each timestep t the agent selects an action via a stochastic policy a ~ T(-[st) Szepesvári (2010) and then receives a reward r1+1 and state $t+1 generated via the transition function p( rsti/s,, ae) of the environment Sutton & Barto (2018). The objective of the agent is to learn a stochastic policy æ, parametrised by ç. that generates an action as function of the current state 8t, which aims maximise the expectation of the sum of discounted
3

---

Under review as a conference paper at ICLR 2022
future rewards (e.g. Mnih al. (2016)).
max EE Sig
Σ T O==
(I)
Where T' is the episode length and 7 is the discount factor. Following other curiosity based methods, the total reward is the sum of the intrinsic reward provided by the intrinsic reward module of the agent and extrinsic rewards provided by the environment (e.g. Pathak et al. (2017); Badia et al. (2019); Raileanu & Rocktäschel (2020); Burda et al. (2018b)).
rz = Br +rē
Where the superscripts and e indicate intrinsic and extrinsic rewards, and 3 is hyperparameter that regulates the influence of intrinsic rewards on the policy. In previous works Burda et al. (2018a), the intrinsic reward ri is equal to the mean squared forward prediction error of a curiosity module. To avoid stochastic traps subtract the aleatoric uncertainty- which is constrained have diagonal covariance Kendall & Gal (2017)ffrom the prediction error, so that agents are not surprised by transitions that were previously learnt to be unpredictable.
= || 1 TI (E(+1)
(3)
Where ur+1 is the predicted mean of the next state, Σ1+1 is the predicted aleatoric uncertainty of the next state and is a hyperparameter that regulates by how much the predicted uncertainty of the next state effects intrinsic rewards. The states being predicted are usually image observations but AMAs also work with learned representations of state observations
show in Section 4.4. To learn predict the mean of the next state j*+1 and its aleatoric uncertainty Êς+1. follow Kendall & Gal (2017)-fittin)
diagonal covariance Gaussian distribution to the elements of the next state. The predictions are made by a heauee neural network-winn a mean prediction head parameterised by f and variance prediction head g parametrised by ø. As employed in previous works, the separate heads of the houble-headed deep network share feature extracting parameters Kendall & Gal (2017). The prediction network performs heteroscedastic aleatoric uncertainty estimation Kendall Gal (2017), which a reinforcement learning context means the prediction heads conditioned on the current state and action.
N
p(s
II 1: ar). go ):)) 1=1
(4)
Where N is the total number of states observed during training. While Kendall & Gal (2017) use maximum a posteriori inference with zero-mean Gaussian prior on the network parameters {0, ø}. we found empirically that the resulting regularisation terms in the cost function did not improve results. Accordingly, we simply perform maximum likelihood estimation with the likelihood presented in Equation (4), resulting in the following cost function Kendall & Gal (2017).
(0, の) = (St+1 - - " (8.41 11++1 + A Alog(de( (É.+1)
(5)
The first term is the familiar mean squared error divided by the uncertainry. The second term blocks the explosion of predicted aleatoric uncertainties Kendall & Gal (2017). We follow Kendall Gal (2017)'s prescription of estimating log Σ instead of Σ to ensure stable optimisation. Furthermore, the hyperparameter was added tempering the model's '' aleatoric uncertainty budget (e.g. Depeweg et al. (2;18); Clements et al. (2019); Eriksson & Dimitrakakis (2019)). We use the predicted mean and alcatoric uncertianty of the next state-waie are being Iearned online with Equation (5)-to compute intrinsic rewards according to Equation (3). Lastly, we would like highlight that the policy network is separate to the state prediction network as in other curiosity based methods Pathak et al. (2017).
4 EXPERIMENTS
This section a range experiments with noisy TVs. Extra details such as the hyperparameters and architectures used are in the appendix. Shaded regions are standard error of the mean.
4

---

Under review as a conference paper at ICLR 2022
4.1 NOISY MNIST
First we completed a supervised learning task, similar to the noisy MNIST environment introduced by Pathak et al. (2019). The environment does not elicit any actions from an agent. Instead, the prediction network simply needs to learn one step mappings between pairs of MNIST handwritten digits. The first images in the pairs are randomly selected Os Is. When the first image is O then the second image is the exact same image (these are the deterministic transitions). When the first image is 1, then the second image is random digit from 2-9 (these the stochastic transitions). A prediction model capable of avoiding noisy TVs should eventually learn to compute equal intrinsic rewards for both types of transitions Pathak et al. (2019). We trained two different neural networks
Input
Output
Pred.
Sigma
Error Baflrr Contection
AMA Error
Noisy MNIST
9
0.1
0.00
Q
0.5
0.06
MSF Stochaslir
AMA Stochasiic
Detenministi
0
0
6
0.02
.0022
Updates
200
400
600
800
(a)
(b)
Figure 1: AMAs can learn to ignore stochastic transitions. (a) Example transitions from the Noisy MNIST environment along with associated predictions. The two rows show stochastic transitions where AMA's predicted variance high in the majority of the image allowing intrinsic reward be small despite the stochastic transition. (b) Two reward curves for MSE and AMA are plotted where stochastic is the + (2...., 9} transitions and deterministic is the 0 - transitions.
on this task (dapted from Liao (2020)). one with a mean squared error (MSE) loss function-as a baseline- and the other with the AMA loss function Kendall & Gal (2017). The networks are equivalent except that the AMA network has two prediction heads. Both networks contain skip connection from the input layer to the output layer and were optimised with Adam at a learning rate of 0.001 and a batch size of 32 Kingma & Ba (2015). The uncertainty budget hyperparameter À and the uncertainty weighting hyperparameter 1) were set to for the AMA network.
The MSE prediction network is unable to reduce prediction errors for the stochastic transitions, causing it to produce much larger intrinsic rewards than the deterministic transitions, consistent with Pathak et al. (2019). On the other hand, the AMA prediction network is able eut its losses by attributing high variance to the stochastic transitions, making them just rewarding as the deterministic transitions.
4.2 DEEP LEARNING MINIGRID
Next we test AMA on the Gym MiniGrid environment Shevvaler--oiiveed et al. (2018), which allows for lesoteee-limited deep reinforcement learning. The agent receives tensor observations describing its receptive field at each timestep. The channels of the observations represent semantic features (e.g. blue door, grey wall, empty, etc.) of each grid tile. The action space is discrete (containing actions: turn left, turn right, move forward, pick up, drop, toggle objects and done) allowing the agent to move around the environment as wells well as open close doors. We used singleton environments but with different seeds for each run rus-resulting in different room configurations- standard error regions for Figure 2. We measure exploration by counting the number of unique states visited throughout training. An action-dependent noisy TV was added, inspired by other minigrid experiments with noisy TVs Raileanu & Rocktäschel (2020), by setting approximately half of the state observation to uniformly sampled integers within the range of possible minigrid values. When the agent selects the 'done' action the noisy TV activated in the next observation. This is the only effect of the 'done" action.
generate the
5

---

Under review as a conference paper at ICLR 2022
With Noisy TV
Without Noisy TV
100
Statos Wisted
100
Visted
8
60
rooms
40
40
20
20
Stape
Training Steps
100K 2006
B00K-40000000000000 500K
(a)
(b)
30
Viiteed
25
25
20
rooms15
15
10
Training Step
Training Stops
100k 250k
50k: 100k 150K 2004 250k
(e)
(c)
(d)
A2C without Intrinsic Rewards
AMA
MSE
Figure 2: AMA agents effectively explore sparse reward minigrid environments that contain action dependent stochastic traps. (c) and (d) panel show performance on the casiest environment, containing four rooms, while the (a) and (b) show performance
more challenging environment with six rooms. AMA and MSE have similar exploration performance when the noisy TV is absent, outperforming a no-intrinsic-reward baseline but when noisy TV is present only the AMA curiosity approach is able to significantly explore the environment. (e) shows an example six room environment.
We perform policy optimisation with a synchronous advantage actor critic (A2C) implementation
recommended by the gym minigrid README Mnih al. (2016); Willems (2020). For the minigrid experiments we train on intrinsic and extrinsic rewards with their relative weighting being equal. The actor critic weights 0,001, while the intrinsic reward module was optimised with the Adam optimizer at learning rate of 0.001 for the AMA agent and 0.0001 for the MSE agent Kingma & Ba (2015). Hyperparameters were optimised for overall performance with and without the noisy TV, see appendix for details. The forward prediction module works in the observation space as opposed to a learned feature space as is implemented in other curiosity driven methods Pathak et al. (2017); Burda et al. (2018b); Raileanu & Rocktäschel (2020). Pixel based curiosity was chosen due to simplicity. The forward prediction model is double headed CNN, which builds upon previous intrinsic motivation implementation on minigrid Railcanu & Rocktäschel (2020). The uncertainty budget À of the AMA network was set to 0.1, the uncertainty weighting 1) was set to and intrinsic rewards were clipped below zero.
optimised with RMSProp Tieleman & Hinton (2014) a learning rate of
We perform experiments in four and six room configurations of the environment. Without a noisy TV both AMA and MSE reward functions generate visit more states compared to the no intrinsic reward baseline. On the other hand, the presence of noisy TV profoundly affects the performance of the MSE curiosity agent, greatly reducing the number of states visited. In contrast, AMA agents are almost unaffected by the presence of an action dependent noisy TV.
4.3 MARIO AND SPACE INVADERS
Next we examine whether our findings from the MNIST and minigrid experiments scale to games from Atari Bellemare et al. (2013) and Gym Retro Nichol et al. (2018); Kauten (2018). We adapt the proximal policy optimisation (PPO) Schulman al. (2017) curiosity implementation from Burda et al. (2018a), using pixel based in our AMA system. We extend Burda et al. (2018a)'s 's U-Net Ronneberger et al. (2015) to use two output heads to predict the mean and variance of future states. For our baselines we use official implementations from Burda et al. (2018b) and Burda et al. (2018a). We leave all PPO hyperparameters equal to their values from Burda et al. (2018a). We set the uncertainty budget hyperparameter À to and the uncertainty weighting hyperparameter 17 to 2. For the these experiments we do not clip intrinsic rewards. To simulate the noisy TV thought experiment, we extend the environment's action space with an action that induces grayscale tiled CIFAR-10 Krizhevsky et al.

---

Under review as a conference paper at ICLR 2022
Space Invaders with Nolay TV
Space Invaders without Noisy TV
Mario with Noisy TV
without Noisy TV
3062
3003
602
100
200
Frarten
Pump
2M
GH
BM
10M
4M4
SM
2010
2004
230c 000 00CL 3.2
(a)
(b)
(c)
(d)
Pixel AMA
Pixel MSE
IDF
RND
Figure 3: Pixel AMA performs significantly better than all baselines with noisy TV (a) and (c) and without the distracting noisy TV AMA nearly matches its most directly comparable method Pixel MSE (b) and (d). No extrinsic rewards were used for policy optimisation. In Mario distance covered and extrinsic reward are equivalent. The y-axis plots extrinsic rewards per episode.
(2014) images (examples in appendix) in place of game frames for the next observation and sends the ultimate action of the action space the game emulator. The choice of the ultimate action is arbitrary. In the appendix we also demonstrate robustness (with the same hyperparameters) to uniform noise.
We compare AMA three alternatiee intrinsic reward methods: random network distillation (RND) Burda et al. (201 (b), inverse dynamics feature (IDF) curiosity Pathak et al. (2017) and MSE pixel based curiosity Burda al. (2018a). Policy optimisation is done with intrinsic rewards only but we measure performance with extrinsic rewards following Burda et al. (201 1a). Compared to the relatively weak baseline of pixel based curiosity, the Space Invaders and Mario experiments show similar results to the minigrid experiments MSE and AMA pixel based curiosity have comparable performance when no noisy TV is present (Figure 4(b) and (d)), while with the noisy TV AMA greatly outperforms MSE pixel based curiosity (Figure 4(a) and Figure 4(c)). Unsurprisingly, random network distillation and inverse dynamics curiosity maintain their superiority over pixel based methods without noisy TV (Figure 4(b) and 4(d)). However, unlike AMA curiosity, both these baselines are vulnerable to action dependent noisy T'ss (Figure 4(a) and Figure 4(c)). While RND has previously been shown to be able to evade stochastic traps Burda al. (201 8b), it seems that if the number of novel states in the trap is large then RND remains vulnerable to trapping.
4.4 BANK HEIST: AN ATARI GAME WITH A NATURAL STOCHASTIC TRAP
Although Atari games are famously deterministic Machado et al. (2018), we identified a naturally ocurring stochastic trap in the Bank Heist gameplay videos of the original IDF curiosity paper. The objective of Bank Heist is to simultaneously avoid police cars and navigate to banks distributed across four 2D mazes. The mazes can be entered and exited through the sides of the screen. With each enter/exit the bank locations reset stochastically. Additionally, the getaway car has the option to drop dynamite in order destroy police cars.
When trained on purely intrinsic rewards, IDF curiosity will perpetually enter and exit the maze while also dropping dynamite. This creates high prediction error as it is impossible predict when the dynamite will explode and where the banks will regenerate. Coincidentally, an agent stuck in this stochastic trap achieves high extrinsic reward rrentttnn the bank positions causes banks to occasionally regenerate on top of the getaway car increasing game score but is it actually exploring? We measured the average number of maze pixels crossed by the agent's car in an episode. In doing so, we see that as extrinsic rewards rise-i.e the point at which the agent begins to regenerate the banks-the average coverage of the maze sharply decreases.
Next we integrated the AMA prediction paradigm into the IDF approach, predicting the mean and aleatoric variance of future state representations and computing rewards with Equation (3) with AMA hyperparameters À and 17 set to 1. With the AMA approach the visited more positions in the maze as it not susceptible to the stochastic trap but performed poorly in terms of extrinsic rewards as more exploration does not necessarily correlate with better extrinsic reward.
7

---

Under review as a conference paper at ICLR 2022
Bank Heist Maze Coverage
Rewards
3000
Pirel Caverage
250
2500
2000
150
1500
50
Training Steps
1k
2k
3k
AMA IDF
MSE IDF
Figure 4: Pixel AMA performs significantly better than all baselines with noisy TV (a) and (c) and without the distracting noisy TV AMA nearly matches its most directly comparable method Pixel MSE (b) and (d). No extrinsic rewards were used for policy optimisation. In Mario distance covered and extrinsic reward are equivalent. The y-axis plots extrinsic rewards per episode.
This section demonstrates that noisy TVs can be inherent issue even in environments with simple dynamics. Further, the results highlight the pitfalls of measuring exploration performance with extrinsic reward (which the agent trained aptimise)-althoughh we believe extrinsic reward correlates well with coverage for Space Invaders and Mario.
A PROPOSED TEST FOR THE ALEATORIC MODEL OF ACETYLCHOLINE
Inspired by Yu & Dayan (2005), we propose that in the mammalian brain acetylcholine signals aleatoric uncertainty surrounding future states. However, neuroscience data that elucidates the specific nature of the uncertainty signalled by acetylcholine. As result, this section proposes ID rodent VR task designed test the specific nature of cholinergic uncertainty signalling in the mammalian brain which we hope will be picked up by experimental neuroscientists. To supplement our experimental proposal, we compute theoretical predictions of cholinergic activity within either an aleatoric or epistemic acetylcholine model two competing interpretations of Yu & Dayan (2005)'s work. The aleatoric model uses aleatoric uncertainties as theoretical acetylcholine signal Kendall & Gal (2017), while the epistemic model uses ensemble variance as an acetylcholine signal Pathak al. (2019).
not aware of any experimental
The proposed task places animal in a VR corridor containing series of spatial landmarks and two reward zones which it must respond in order to have a chance of receiving a reward. Responding in reward zone A causes the animal to teleport to random position along the track. Responding in reward zone B causes the animal to teleport to a fixed position on the track.
To compute how both models predict the cholinergic signal should respond in the proposed rodent VR experiment, we simulate the task with a simple multi-armed bandit environment. In our bandit model of the task agent predicts 1D function by sampling minibatches from different regions of the input. In one region of sample space the function takes a simple sinuisoidal form, analogous to zone A of the VR track, in second region the function consists of points randomly sampled from a standard normal distribution at each timestep, analogous to zone B (Figure 5(c)). As described previously, we applied
models to this task, the first acetycholine was identified with aleatoric uncertainty, while in the second-as as a control control-acetylchan acetylcholine tracks epistemic uncertainty.
We trained an action value based multi-armed bandit to maximise intrinsic rewards for two kinds of forward prediction models: a double headed network trained to optimise the AMA objective and an ensemble of networks where each member minimising their own MSE (e-g. Pathak al. (2019)). The aleatoric model uses the AMA reward function whereas the epistemic model is (intrinsically) rewarded for variance in ensemble predictions. We plot both models' uncertainties each reward zone over time recovering a clear prediction of cholinergic activity in both cases. The aleatoric uncertainty AMA remains high in reward zone but decreases in reward zone A. On the other hand, the epistemic model shows decrease in acetylcholine in both reward zones time. We hope these clear and distinct predictions on the nature of cholinergic uncertainty signalling will be tested by the experimental neuroscience community in task similar to the we propose.
8

---

Under review as a conference paper at ICLR 2022
Epistemic Acetylcholine
Aleatoric Acetylcholine
0.01
Acctyccolie
Stinulus Value
0.008
0.008
0.004
0.002
Training Step
Traning Stap
20k
60k
50k
100k
150K
200k
2506
300
Deterministic Zone
Stochastic Zone
Stochastic Zone
Deterministic Zane
Corrider Position
(a)
(b)
(c)
Figure 5: Predictions on theoretical experiment to illuminate the epistemic or alcatoric nature of cholinergic signalling in the brain. (a) and (b) show that the epistemic model predicts acetycholine will eventually decrease in both zones while the aleatoric model predicts acetylcholine decreasing to zero in the stable zone but remaining high the noisy zone. Panel (c) shows our 1D model of the proposed animal experiment where the bandit samples different 'corridor positions' and receives scalar stimuli, which trying to predict-using the resulting prediction errors as intrinsic rewards.
LIMITATIONS
The AMA reward function implicitly rewards epistemic uncertainty by assuming the total uncertainty can be decomposed into epistemic and aleatoric uncertainties. While theoretically true Kendall & Gal (2017), there is no guarantee that AMAs are able surgically subtract those errors due to aleatoric dynamics from the total prediction error. Additionally, aleatoric uncertainty estimates are not guaranteed to be reliable for of distribution data, meaning intrinsic rewards could become less reliable the further the agent travels into novel territory Mukhoti et al. (2021). ). In practice we find that stochastic policy-and clipping in the case of minigrid-ffisee offsets potentially deceptive intrinsic rewards. Furthermore, trust region methods in the retro games (i.c. using PPO Schulman al. (2017) instead of A2C Mnih et (2016) may also compensate for ocassionally deceptive rewards suggested by the fact that intrinsic reward clipping was not necessary for the retro game experiments. Finally, we note that like all curiosity approaches (e.g. Burda et al. (2018a)), our method generates non-stationary rewards, which is known to make learning difficult for RL agents.
7 CONCLUSION
We have shown AMAs are able to avoid action-dependent stochastic traps that destroy the exploration capabilities of conventional curiosity driven agents Burda et al. (2018a). AMAs tractably avoid stochastic traps by decreasing intrinsic rewards in regions with high estimated aleatory. The success of AMAs further cements proposal from theoretical neuroscience Yu & Dayan (2005) that experi- mentalists should complete experiments to clarify the nature of uncertainty signalled by acetylcholine. Accordingly, we present an experimental design along with theoretical predictions of cholinergic activities from competing models. Completing the animal experiment proposed has the potential to benefit neuroscience and reinforcement learning alike Hassabis et al. (2017). Future work could also analyse an alternative hypothesis that novelty based methods such as random network distillation could be accurate model of acetylcholine Barry al. (2012); Burda et al. (2018b). Future RL research should aim to integrate the AMA approach into curiosity methods that operate in feature spaces besides pixels et al. (2018b)), with the aim of achieving state of the exploration even when noisy TVs are present.
within those methods that circumvent dynamics altogether (e.g. Burda
REFERENCES
Elio Acquas, Catriona Wilson, and Hans C Fibiger. Conditioned and unconditioned stimuli increase frontal cortical and hippocampal acetylcholine release: effects of novelty, habituation, and fear. Journal of Neuroscience, 16(9):3089-33333 1996.

---

Under review as a conference paper at ICLR 2022
Adria Puigdomènech Badia, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Martin Arjovsky, Alexander Pritzel, Andrew Bolt, et al. Never give up: Learning directed exploration strategies. In International Conference Learning Representations, 2019.
Caswell Barry, James G Heys, and Michael E Hasselmo. Possible role of acetylcholine in regulating spatial novelty effects on theta rhythm and grid cells. Frontiers in neural circuits, 6:5, 2012.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems, 29:1471- 14479, 2016.
Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling, The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A Efros. Large- scale study of curiosity-drivee learning. International Conference on Learning Representations, 2018a.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations, 2018b.
Maxime Shevalier-Boisveer Lucas Willems, and Suman Pal. Minimalistic gridworld environment for openai gym.
wwwpeeecch/gyymmmneee mampriimeemgmmmmmiiiiiiiii. 2018.
William Clements, Benoit-Marie Robaglia, Bastien Van Delft, Reda Bahi Slaoui, and Sébastien Toth. Estimating risk and uncertainty in deep reinforcement learning, arXiv preprint 19Xiv:1905.090 096388,
2019.
Stefan Depeweg, Jose-Miguel Hernandez-Lobato, Finale Doshi-Velez and Steffen Udluft. Decom- position of uncertainty bayesian deep learning for efficient and risk-sensitive learning. In International Conference on Machine Learning, pp. 1184-1193, PMLR, 2018.
Hannes Eriksson and Christos Dimitrakakis. Epistemic risk-sensitive reinforcement learning. arXiv preprint arXiv:1906.00 06273, 2019.
Yarin Gal. Uncertainty in deep learning. University of Cambridge, 1(3), 2016.
MG Giovannini. A Rakovska, RS Benton, M Pazzagli, L Bianchi, and G Pepeu. Effects of novelty and habituation on acetylcholine, gaba, and glutamate release from the frontal cortex and hippocampus of freely moving rats. Neuroscience, 106(1):43-555 2001.
Demis Hassabis, Dharshan Kumaran, Christopher Summerficld, and Matthew Botvinick. Neurosciencce-ispired artificial intelligence. Neuron, 95(2):245-258, 258, 2017.
Michael Hasselmo. The role of acetylcholine in learning and memory. Current opinion in neurobiology, 16(6):710-715, 2006.
Stephen C Hora. Aleatory and epistemic uncertainty in probability elicitation with an example from hazardous waste management. Reliability Engineering & System Safery, 54(2-3):217-25) 1996.
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration. Advances in Neural Information Processing Systems, 29:1 1111119-1111111111 2016.
Eyke Hüllermeier and Willem Waegeman. Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine Learning, 110(3)4457-56, 2021.
Frederic Kaplan and Pierre-Y Oudeyer. In search of the neural circuits of intrinsic motivation. Frontiers in neuroscience, 1:17, 2007.
Christian Kauten. Super Mario Bros for OpenAl Gym. GitHub, 2018. URL https://github.com/ paatenjj/gym--uupe--mmrr-oo-oo
10

---

Under review as a conference paper at ICLR 2022
Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? In Advances in neural information processing systems, pp. 5574-5584, 2017.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster), 2015.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. The cifar-10 dataset. online: http:///ww.. cs. toronto. edu/kriz/cifar homl, 55(5), 2014.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in Neural Information Processing Systems, 30, 2017.
Xingyu Liao. pytorch simple autoencoder. http:///gthbbccomo//////ignnnggg hwwp://wwwppooooro---ggenbbbbbb beginner/blob/ Aat.:///-AAuoEEnoor//sipppppp autoencoder.py. 2020.
Marlos C Machado, Marc G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523-566 2018.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. International conference on machine learning, pp. 1928-19323 2016.
Shakir Mohamed and Danilo Jimenez Rezende. Variational information maximisation for intrinsically motivated reinforcement learning. Advances in Neural Information Processing Systems, 28:2125- 2133, 2015.
Jishnu Mukhoti, Andreas Kirsch, Joost van Amersfoort, Philip HS Torr, and Yarin Gal. Deterministic neural networks with appropriate inductive biases capture epistemic and aleatoric uncertainty. arXiv preprint arXiv:2102.1 1/582, 2021.
Alex Nichol, Vicki Pfau, Christopher Hesse, Oleg Klimov, and John Schulman. Gotta learn fast: A new benchmark for generalization in rl. arXiv preprint (XX.v:8804.03722 2018.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. In Advances in neural information processing systems, pp. 4026 4034, 2016.
Vinay Parikh, Rouba Kozak, Vicente Martinez, and Martin Sarter. Prefrontal acetylcholine release controls cue detection on multiple timescales. Neuron, 56(1):141-15) 2007.
Thomas Parr and Karl J Friston. Uncertainty, epistemics and active inference. Journal of The Royal Society Interface, (1133))221107766 2017.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International Conference on Machine Learning, pp. 2778-77822 27877. PMLR, 2017.
Deepak Pathak, Dhiraj Gandhi, and Abhinay Gupta. Self-supervised exploration via disagreement. In International conference on machine learning, pp. 1062-5077. PMLR, 2019.
Giancarlo Pepeu and Maria Grazia Giovannini. Changes in acetylcholine extracellular levels during cognitive processes. Learning & memory, 11(1):21-27, 2004.
Roberta Raileanu and Tim Rocktäschel. Ride: Rewarding impact-driven exploration for procedurally- generated environments. In International Conference on Learning Representations, 2020. URL
Charan Ranganath and Gregor Rainer. Neural mechanisms for detecting and remembering novel events. Nature Reviews Neuroscience, 4(3): 190:199222222 2003.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234-241, Springer, 2015.
11

---

Under review as a conference paper at ICLR 2022
Jürgen Schmidhuber. Adaptive confidence and adaptive curiosity. In Institut fur Informatik, Technische Universitat Munchen, Arcisstr. 21, 800 Munchen 2. Citeseer, 1991a.
Jürgen Schmidhuber. A possibility for implementing curiosity and boredom in model-building neural controllers. In Proc. of the international conference on simulation of adaptive behavior: From animals to animats, pp. 222-227, 1991b.
Jürgen Schmidhuber. What's interesting?, 1997.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Pranav Shyam, Wojciech Jaškowski, and Faustino Gomez. Model-based active exploration. In International Conference Machine Learning, pp. 5779-5788, 2019.
Susanne Still and Doina Precup. An cnfformttion-theorete approach to periosity-drivess reinforcement learning. Theory in Biosciences, 131(3):139-144 2012.
Alexander L Strehl and Michael L Littman. An analysis of model-based interval estimation for markov decision processes. Journal of Computer and System Sciences, 74(8):1309-133,, 2008.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Csaba Szepesvári. Algorithms for reinforcement learning. Synthesis lectures on artificial intelligence and machine learning, 4(1):1-103, 2010.
Tijmen Tieleman and Geoffery Hinton. Rmsprop gradient optimization. URL http://www. cs. toronto. _lideesseiddddddooeoo.e pdf, 2014.
Joost van Amersfoort, Lewis Smith, Andrew Jesson, Oscar Key, and Yarin Gal. On feature collapse and deep kernel learning for single forward pass uncertainty. arXiv preprint arXi::2102.11409 2021.
Lilian Weng.
Exploration lilianweng-giin githigggttttbbbhh. githubbio//illlogg 2020. URL
strategies in deep
reinforcement learning.
interratioo-sratggies-- in receppreinfrremmet--eaammm
Lucas Willems. RI. Starter Files. http://gittubb.ccmmmsssilllllee starter- files, 2020.
Angela Yu and Peter Dayan. Expected and unexpected uncertainty: Ach and ne in the neocortex.
Advances in neural information processing systems, pp. 173-180, 2003.
Angela Yu and Peter Dayan. Uncertainty, neuromodulation, and attention. Neuron, 46(4):681 2005.
12

---

Under review as a conference paper at ICLR 2022
8 APPENDIX
8.1 NOISY MNIST
We use three random seeds for the repeats of the MNIST experiments. The results in the graph show test set performance. The hyperparameters used are listed below. The learning rate was manually tuned so that the identity transformation was learned for the deterministic transitions (hence very low loss for the MSE and AMA network) and the AMA network produced sensible uncertainty estimates for the stochastic transitions.
Hyperparameter
Value
MSE Learning Rate
0.001
AMA Learning Rate
0.0001
Batch Size
32
AMA uncertainty budget
I
AMA uncertainty coefficient 7)
1
Table I: Noisy MNIST hyperparameters
Input
westht (754-124)
Siee (194)
weight (784-784)
Mas
Limeer
Linee
weight (70401940)
weigt: ('84-704)
Max
(764)
RetU
ReLU
weight
Figure 6: MNIST image to image architecture. For the MSE network the second log variance branch was discarded. skip connection was provided from the input the final layer for both AMA and MSE.
8.2 MINIGRID
We used seeds for the minigrid experiments. We tuned different hyperparameter via grid search: curiosity learning rate [0.01, 0.001, 0.0001] and intrinsic reward scaling (before normalisation) [1, 10, 100) for the AMA and MSE modules on the six room environment (by summing up the novel states visited with and without a noisy TV). We used different seeds for the grid search and the final results. We also adapt an implementation of the Welford algorithm from stack overflow for normalising rewards The architecture for forward prediction is adapted from the implementation from Raileanu & Rocktäschel (2020) but in preliminary experimentation we ended up changing their prediction architecture dramatically.
13

---

Under review as a conference paper at ICLR 2022
Hyperparameter
Value
AMA learning rate
0.001
MSE learning rate
0.0001
AMA reward scaling
MSE reward scaling
1
AMA normalise rewards
True
MSE normalise rewards
False
RMS Prop a
0.99
RMS Prop
1. 1.000e-8
number of actors
16
unroll length
5
discount factor
0.99
policy learning rate
0.001
GAE À
0.95
entropy coefficient value loss coefficient
0.01
0.5
max grad norm
0.5
AMA uncertainty budget X
0.1
AMA uncertainty coefficient 77
1
Table 2: Minigrid hyperparameters
Comudd
weight (16-060-2)
Ska (10)
ReLLI
MouPoolFd
Convöd
ReLUi
Linee
weight (eivall
(540
Taxhs
Linner
weight (v66))
bls (7)
Figure 7: Actor critic architecture for the policy network in the minigrid experiments.
14

---

Under review as a conference paper at ICLR 2022
weight (32x343x3)
Nas (32)
Tanh
weight (147x621)
weight (147H521)
Saxx (1423
(147)
Tanh
Linear
Linear
weight (14tx10r)
weight (147x1977)
Figure 8: Curiosity forward prediction architecture for the minigrid experiments. For the MSE
baseline the variance predictions not used and loss is computed via standard MSE.
Hyperparameter
Value
global learning rate
0.0001
normalise rewards
True
number of PPO epochs number of actors unroll length discount factor 7 GAE A
128
128
0.99
0.95
entropy coefficient
0.001
value loss coefficient
0.5
policy gradient clip range
[-1.1, 1.1]
Pixel AMA uncertainty budget
Pixel AMA uncertainty coefficient 7)
IDF AMA uncertainty budget X
IDF AMA uncertainty coefficient !
Table 3: Retro game policy hyperparameters
8.3
ATART
For each run in the Space Invaders and Mario experiments we used five seeds method. For the Bank Heist experiments we used three seeds per method.
We use the official implementations Burda et al. (2018b;a) for the baselines we compare to (with their default hyperparameters). For AMA and Pixel MSE we adapt from Burda al. (2018a). The hyperparameters used for our AMA experiments can be found in Table 3. We did not change the PPO/vanilla curiosity hyperparameters from the original implementation we adapted and only changed the AMA hyperparameters. The hyperparameters were chosen by first exploring different configurations on smaller minigrid environments and evaluating promising configurations on the Atari environments.
The UNet architecture used for the forward predictions is described below. We duplicate the decoder head to create two headed output but only describe the encoder and decoder here. For the Pixel MSE baselines we use identical architectures but don't use the uncertainty predictions and train on MSE only. There are UNet style residual connections between the corresponding encoder and
15

---

Under review as a conference paper at ICLR 2022
Layer Type
Filters
Kernel Size
Stride
Conv2d
32
8
Conv2d
64
8
Conv2d
4
Dense (512 Units)
N/A
N/A
N/A
Conv2d Transpose
64
4
2
Conv2d Transpose
32
8
Conv2d Transpose
4
8
Table 4: Retro game forward prediction hyperparameters
Hyperparameter
Value
epistemic learning epistemic batch size aleatoric learning rate aleatoric batch size aleatoric uncertainty budget aleatoric uncertainty coefficient y)
0.0001
32
0.001
1000
I
1
greedy c
0.1
Table 5: Bandit hyperparameters.
decoder layers. Leaky ReLU activations used in the encoder layers and Tanh activations used in the decoder layers. Batch normalisation is used throughout the hidden layers. Action information concatenated at each layer. See supplementary code for further details.
To integrate AMA into the IDF approach, we did not share any representations between mean and variance prediction heads, instead we used two prediction MLPs for the mean and variance. Leaky ReLU is used throughout hidden layers and action information is concatenated at each layer. We used five layers with 512 units each and UNet style residual connections.
8.4 BANDIT
We performed 3 repeats to produce the standard eгror regions show in the graph. Learning rate was tuned by hand, observing how well the network performed in making predictions as bandit sampled different regions of the environment. The intrinsic reward method for the epistemic bandit is based on Pathak et al. (2019); Lakshminarayanan et al. (2017). We use an action value based bandit algorithm with e-greedy exploration (Sutton & Barto, 2018, p. 31).
8.5 PLOT FORMATTING
For the plots we use Weight and Biases² including their built in smoothing function. For all results we use the exponential moving average (with smoothing parameter of 0.59) except the minigrid plots where we used the moving average smoother (with smoothing parameter of 0.1).
(https:/wandd.ai
16

---

Under review as a conference paper at ICLR 2022
(188-113
wrrgle (1800-190)
vedgge (180-
welee (180--000)
Liwer
LEWA
May
Figure 9: AMA prediction network for bandit tasks.
Linear
weight (100<1>
bas (100)
Tanh
Linear
weight (1200x 1003 (1200)
Linear
weight (1300k 1200)
blos (1300)
Tanh
Linear
wolght (100+1300)
blos (1co)
Linear
100)
blas
Figure 10: Epistemic prediction network for the bandit task.
17

---

Under review as a conference paper at ICLR 2022
Figure 11: Four examples of frames that agent might see when interacting with the CIFAR noisy TV. This is a complex noise distribution picked to test the limits of the heteroscedastic alcatoric uncertainty estimation.
Space Inveders with Noisy TV
Mario with
Dizance Covered
800
200
402
100
200
222
4M
BM
30M
200M000 -SCDEEOE 1 2.200.44
Pixel
IDF
Figure 12: AMA is robust to Noisy TVs of very random unifrom noise distribution (random pixels from 0-255) while other basclines are also trapped by this additional noisy TV that significantly different the noise distribution used for the results used in the main text.
8.6 DIFFERENT NOISE DISTRIBUTION FOR RETRO GAMES
For the CIFAR noisy TV tiled random CIFAR image (from the training set) for each frame observed on the noisy TV. This required around around 2.5 tiles to fill the 84 x 84 pixels of the retro game frames. An example frame can be seen in Figure 11.
8.7 HYPERPARAMETER ANALYSIS
The AMA curiosity method contains two hyperparameters À and 1). We analysed Mario, Space Invaders and Minigrid how sensitive performance is to the values of these hyperaparameters. We find that decent results be achieved with tuning on the retro game environment (Figure 14 and Figure 15), while in the six room minigrid environment it is important to tune À (Figure 16).
8.8 EVENTUAL DECREASE IN EXPLORATION DURING MARIO TRAINING
In initial experiments we noticed the AMA agent lost motivation to explore its environment after reaching peak extrinsic reward. To ensure that this not inherent problem with AMA,
18

---

Under review as a conference paper at ICLR 2022
Without Noisy TV,
With Noisy TV
1000
Distance Covered
Ditarcca Covered
1000
eta=0.5
800
Varlable Lambda
200
200
Training Steps
Training Steps 100
40
60
100
120
1000
Dictanca Covered
1000
800
eta=1
Varlable Lambda
600
400
200
200
Training Steps
0
Training Stops
100
120
20
40
60
100 120
-oorttir peraat
Distance Covered
1000
Disance Coverod
eta=2 Variable Lambda
800
600
600
400
200
Training Steps
Training Steps
20
40
60
80
100
120
60
Figure 13: Overall, the mario results are not very sensitive to the setting of n..
19

---

Under review as a conference paper at ICLR 2022
Without Noisy TV,
With Noisy TV
Disanca Covared
1000
1000
800
Lambda=10
Variable Eta
500
600
200
200
Treining
0
Tra ning Stops
20
60
100
120
20
40
50
80
100
120
Distance Covered
1000
Disance Coverse
Lambda=1
800
800
Variable Eta
400
200
Training Steps
Training Steps
20
60
80
100
120
Distance Covered
Distance Covered
Lambda=0.1
800
800
Variable Eta
600
400
200
200
Training Steps A6
Training Steps
100 120
Figure 14: As the uncertainy penalty (A) increases performance with the noisy TV becomes poor (expected) because higher uncertainty penalty moves AMA more towards MSE curiosity. Otherwise for Mario results are not very sensitive to hyperparameter values.
Without Molay TV. Eta-1, Verisbte Lambda
With Noday TV, Ete-1, Veriable Lambd.
60
20
Traiinn Steas
50k 150K 200.20000220000000 300k3300
50k 1006
2004 250K 100% 2004
Without TV, Lambds:001. Variable
Statin Weltee
153
60
Training Steps
5tk 1008 2004 www.kktkkkoooo. www.k/5kk.k0kk.o0
50k 10N.10022000250033000355
Figure 15: Performance on minigrid not very sensitive to 17 but requires some tuning.
20

---

Under review as a conference paper at ICLR 2022
Marlo IDF Curlosity without Nolsy TV
Marlo Pixel Curlosity without Nolsy TV
Seed 1
Seed
Seed 3
Seed
Seed 2
Seed
1200
Extrinsic Reward
Extrinsic Reward
3000
1000
2500
www
800
2000
600
1500
400
1000
200
500
Frames
1M
2M
3M
4M
5M
1M
2M
3M
4M
5M
Figure 16: Repeats ran for more frames Mario for MSE Pixel curiosity and MSE IDF curiosity, 3/3 of seeds see decrease in performance for IDF curiosity and 2/3 in the pixel curiosity case.
ran the other curiosity methods for further frames and found similar eventual decreases in extrinsic reward (Figure 16). Presumably the cause of this is that once a sigmificant portion of the environment has been explored the agent no longer motivated to return those regions (as prediction error decreases) this issue has been noted by previous authors Pathak et al. (2017).
8.9 POTENTIAL NEGATIVE SOCIAL IMPACTS
The work presented very far from any real world deployment. it were be deployed in any real world context then extensive testing would need to be done to understand how the curiosity agents would behave in novel environments as erratic behaviours could be dangerous in, for example, robotic control context. The AMA objective is overarching (like other curiosity methods) and so care should be taken if deploying something like a recommender system as the agent could find certain behaviours intrinsically rewarding that you might not have intended it to (like the noisy TV problem). Lastly, although the AMA system contains notions of uncertainty quantification, that does not mean it is able completely understand the limits of its predictions and should not be overconfident in its abilities to do so.
8.10 HARDWARE
The experiments were performed three different machines depending their availability: A 32 core CPU with one GeForce GTX TITAN X, 12 core CPU with two GeForce GTX TITAN Xs and one core CPU with two GeForee 2080Ti GPUs.
We list times here on the 2080Ti machine, the other machines were as much as 2x slower. The minigrid experiments took around 40 minutes per run, the Space Invader experiments took around 12 hours per run, the Mario experiments took around hour and 20 minutes per run.
8.11 LICENSING
The repository from Willems (2020) has an MIT license. The code used from Chevalier-Boisver et al. (2018) has an Apache License 2.0. Raileanu & Rocktäschel (2020) has a creative commons license. Besides those listed we are aware of any further code licensing. We adapted copyright free rat silhouette image for the bandit figure
21

---

Under review as a conference paper at ICLR 2022
8.12 FURTHER CODE ACKNOWLEDGEMENTS
Although we did not directly use their code we would like to acknowledge the following open source contributions that provided useful reference when implementing Kendall and Gal's Kendall & Gal (2017) aleatoric uncertainty estimation algorithms:
https: btgp:///oooo.ooooooooooo//aaaaaa Do-We-Need-ing -a--aaaasaa-----e-eeeeeeeeee
What-Unnertainties for- Computer-Vision
https//gihhbbbcommmmoooiddddddee htpp://gitabbccomnnniBvWwwttt
22


Reviews:
[{"review_id": "wjeBuj5Tmt1", "reviewer": "Reviewer_DE7e", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Strengths*\n\nThe paper provides a very nice, comprehensive review of the existing curiosity techniques, which was a pleasure to read.\n\nI like the neuroscientific inspiration of the proposed algorithm. The field is full of claims about neuro-AI synergy but short of good examples. This work, further developed, may become a good example of such synergy.\n\nThe idea of explicitly accounting for aleatoric uncertainty in deciding on future actions seems highly meaningful and, potentially, fruitful.\n\nIn this formulation, the uncertainty is estimated with a single agent and not with ensembles of agents, as it was done before. This makes the model more useful.\n\nThe authors have performed an impressive set of experiments, involving a significant number of baselines.\n\nI like the idea of proposing a test, which experimental neuroscientists may then use to improve our understanding of the brain.\n\n*Weaknesses*\n\nProvided the close relation of the proposed method to prior work (as mentioned by the authors), it would be reasonable to expect a strong Results part of the paper (see below).\n\nMost of the experiments in this paper were performed in environments specially altered to introduce stochastic traps. The framework’s performance in unaltered environments did not exceed that of the baselines.\n\nThe exploration has been previously developed as a tool with a final goal of maximizing rewards in scarce environments. In this work, the authors have reported improvements in exploration but there are no experiments indicating improvement of total reward in standard unaltered environments (the authors do explicitly discuss this issue). It is unclear whether additional exploration, as suggested here, leads to higher rewards in the standard or real-world environment, which is the goal of exploration in the first place.\n\nIn the proposed experiment for establishing the role of acetylcholine in the brain, the prediction seemingly boils down to the fact that epistemic uncertainty decays over time, whereas aleatoric uncertainty remains constant. These predictions seem to directly follow from the definitions of such uncertainties, so it is unclear whether the proposed design of the experiment is necessary.\n\n*Specific comments*\n\nIn Equation (5), is the last sign (+) correct?\n\nAlthough the notation is clear, consider introducing the variables before using them.\n\nWhereas the introduction is generally great, it may be useful to talk a bit more about Angela Yu and Peter Dayan’s model, as it forms the main neuroscientific motivation for the proposed framework, so that the readers won’t have to read their paper.\n\nI would personally consider adding a short description of how Equation (5) was derived. It’s pretty clear for me, but may or may not be clear for everyone. The derivation is super simple and won’t take up a lot of space.\n\n*Suggestions to the authors*\n\nI found this work highly promising and would love to see future developments based on your framework. Perhaps, something along these lines:\n\nIt would be great to see the improvement of the received reward in standard or real-world environments. The claim here, supported by the literature, is that stochastic traps are the major problem in curiosity-driven exploration. This implies that there are enough standard environments where stochastic traps affect the efficiency of baseline curiosity-based algorithms. Such environments can be used to showcase the benefits of the proposed algorithm when it comes to rewards which, consequently, would skyrocket the impact of this work.\n\nIt would also be nice if the author follow up on their proposed acetylcholine experiment and collaborate with experimental neuroscientists to test this hypothesis. Determining that acetylcholine encodes one type of uncertainty or another would be an important result and another great way to ascertain the high impact of this work.\n", "summary_of_the_paper": "This paper extends the works on curiosity in artificial agents by incorporating an explicit prediction of (non-reducible) aleatoric uncertainty into the agent’s action choice computation, such that the agents have a preference against the environmental states where uncertainty cannot be reduced by learning. The proposed framework has enabled efficient exploration in a set of tasks where non-reducible uncertainty has been introduced; the authors have also used it to propose a test to clarify the role of acetylcholine – the neurotransmitter related to uncertainty – in the brain.", "summary_of_the_review": "An impressive amount of work has been done; the writing is mostly easy to follow, and the results seem promising; however, provided that the proposed framework is closely related to the existing ones, further research towards practical results is needed. The better exploration, proposed here, is important, but it may or may not lead to better-performing agents. I, therefore, think that the paper is better suited for a conference workshop rather than for the main track. Thus, overall, I tend to recommend rejection.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "a9sM5ziTWCg", "reviewer": "Reviewer_j6wR", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main weakness of the current work is that it misses relevant work which does the same thing while it seems to misinterpret certain parts of the literature. As an example, the authors state that \"tractable epistemic uncertainty estimation with high dimensional data is an unsolved problem\", citing Gal 2016 which demonstrates a method for estimating epistemic uncertainty in high dimensional data (that's MC Dropout). Also, there is a similar approach using Deep Ensembles to assess epistemic uncertainty. Although far from optimal, such approaches are already being used in epistemic uncertainty driven exploration. For example, in Planning to Explore ([https://arxiv.org/abs/2005.05960](https://arxiv.org/abs/2005.05960)) the authors use epistemic uncertainty estimation as an intrinsic reward. In fact, the authors in Related Work last paragraph, they cover a few efforts using epistemic uncertainty estimation as intrinsic rewards, while they don't mention what's the shortcomings of such work which they aim to solve with their proposal.\n\nIn terms of experimentation, the selected baselines are not sufficient to demonstrate how much better this method is. I'd expect to see a comparison with MC Dropout or Deep Ensembles based methods of assessing epistemic uncertainty (in the presence of heteroscedastic noise), or DUE ([https://arxiv.org/pdf/2102.11409.pdf](https://arxiv.org/pdf/2102.11409.pdf)).\n\n**Detailed comments:**\n\nC1: \"this work presents aleatoric mapping agents which use single network deterministic uncertainty estimation\" - is the reference correct there? Kendall & Gal work uses a Bayesian Neural Network, whereas deterministic uncertainty estimation was proposed here [https://openreview.net/forum?id=Fu7D6kQPzs4](https://openreview.net/forum?id=Fu7D6kQPzs4) (not cited in the text)\n\nC2: For the epistemic and aleatoric uncertainty you cite Hullermeier & Waegeman. Can you also cite the earlier references? e.g. \"Hora S (1996) Aleatory and epistemic uncertainty in probability elicitation with an example from hazardous waste management. Reliability Engineering and System Safety 54(2–3):217–223\"\n\nC3: \"However, as far as we are aware, we are the first to compute aleatoric uncertainties with a scalable curiosity framework to reduce intrinsic rewards for those state transitions with aleatoric uncertainty\" - I don't understand what is the novelty here. There is a lot of work disentangling epistemic from aleatoric uncertainty which works at scale. Can you please state as precise as possible what's the contribution in terms of the computation of aleatoric uncertainty?\n\nC4: \"We implicitly incentivise agents to seek epistemic uncertainties by removing the aleatoric component from the total prediction error\" - If I'm not mistaken that's the common way to disentangle the epistemic from the aleatoric uncertainty which is usually derived by the law of total variance. What is the novelty here?\n\nC5: \"Possible because regularisation terms could absorb some epistemic uncertainty\" - What does it mean to absorb some epistemic uncertainty? Epistemic uncertainty can be reduced or increased. Do you mean that through the regularization, epistemic uncertainty is reduced? If so, can you explain why this can happen?\n\nC6: In the Minigrid experiments, can you try to implement the intrinsic reward based on the epistemic uncertainty as evaluated from ensemble Kendall & Gal 2017 models? That should be the Variance over expectations of the deep ensemble components $Var_{\\theta \\sim p(\\theta \\mid D)}[ \\mu^\\theta_{s_{t+1}} ]$.\n\nC7: It's very difficult to follow the experimental protocol in section 5. How do you estimate the Acetylcholine in the simulations?\n\nFinally, I'd like to suggest to the authors to try to use \\citep instead of \\cite whenever possible. Also try to use a more accessible colour for the citations as currently, it makes it challenging to read.", "summary_of_the_paper": "This work proposes a method to disentangle epistemic from aleatoric uncertainty for avoiding the noisy TV problem which occurs when intrinsically motivated agents get rewarded for visiting states that have high irreducible uncertainty.", "summary_of_the_review": "My recommendation is influenced by the following issues:\n- The novelty of the contribution is not clear to me. I'd like to understand C3-C6 because overall it seems to me that epistemic uncertainty driven intrinsic rewards have been used before and here it's not compared thoroughly against such baselines (which are mentioned in related work)\n- C7 - I don't understand how this section contributes towards understanding what type of uncertainty acetylcholine signals.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "PI-hcdBNkb-", "reviewer": "Reviewer_QuEW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I am surprised that this idea was not explored in an earlier paper. Taking the prediction error minus the variance looks like the very first solution to try in order to prevent agents to stay watching a noisy TV. While my knowledge to this domain is limited, to my mind this is the first paper suggesting this idea. Probably other reviewers will prove me wrong. \n\n\nI expect this approach to work well, and I trust the good experimental results exposed in the paper. I believe this could have a nice impact on the curiosity/exploration efforts in the RL community.\n\n\nHowever, there are many weaknesses that makes this paper not ready for a publication:\n\nFirst, all the mathematical justifications are unclear and slovenly written with too much errors in equations:\n\nEq 3) is wrongly reported. It should be $S_{t+1}$ in the left of the noise term, and $\\hat{S_{t+1}}$ in the right of the variance term.\n\nIn Eq 5), $\\mu_{S_{t+1}}$ should actually be $S_{t+1}$ (except it's an average over a batch of states $S_{t+1}$ with the same previous $S_t$, but I guess it's not)\n\nIn Eq 6) $\\hat{S}$ should be $\\hat{\\mu}$ for consistency.\n\nAlso I feel that there is a lot of paragraphes just to describe a very simple gaussian model used to fit $S_{t+1}$ given $S_{t}$ by maximum log-likelihood (in Eq 5). Sigma is the estimated variance, and mu is the estimated mean. Eq 5 is just the log of Eq 4 (with lambda = -1/pi).\nThen in Eq 7, it's just R = error - estimated variance. No need for $\\eta$, the point is that these terms should have the same expectation at convergence of the fitted model with $\\eta=1$.\n\n\nThe bio-inspired aspect needs more content to be convincing: \n\nfor example, reporting models or experimental results from the biological papers (for ex, the actual original curves that fig 5 is trying to reproduce).\n\nWhat I suggest:\n\nThe experimental section looks good to me. But all the method section needs to be re-written from scratch. After introducing the MDP notations, I would directly start with the final equation 7: we take the error minus the variance.\n\nThen, show that the mean of this bonus is zero when the model is learned (so whatever is the unpredictability, the bonus will go to zero).\nOptionally, look at the sub-optimality behaviour of the bonus: is the error higher or lower than the variance? How does it affect the agent's curiosity?\n\nFinally explain how the mean and variance are learned, here with a two-headed network trained by maximum LLH.\nAlso, discuss what happen if the action are taken into account in the learned model (so far the bonus is on-policy and this should appear somewhere and be discussed). Here everything works as long as the agent is not changing his policy, but depending on the learning dynamics of the policy, the model may never converge, this would not happen with action taken into account.\n\n\n\n", "summary_of_the_paper": "This paper suggests an intrinsic bonus for exploration that avoids noisy TV by adding a penalty for the estimated variance of the reached state $S_{t+1}$ given previous state $S_{t}$.\n\nFor this, they fit an independent normal model of the new state $S_{t+1}$ with mean ($\\mu_{t+1}$) and var ($\\sigma_{t+1}^2$) predicted from the previous state $S_{t}$.\nThen, the bonus is given by the error of prediction $(S_{t+1} - \\hat{\\mu_{t+1}})^2$ minus the variance penalty $\\sigma_{t+1}^2$. \n\nThat way, a noisy TV will always be fitted with an higher variance that should have a similar order than the error of prediction, so both predictable and unpredictable transitions bonus will end up to zero (in average) after learning the model of dynamics of the whole environment.\n", "summary_of_the_review": "(+) Nice and novel idea\n(-) Too poor mathematical justifications and unclear description of the method.\n\nI sincerely hope the paper will be improved for a further submission because I believe the idea has an high potential, that would be missed if it appears in its present slovenly form.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Authors present a method to disentangle epistemic from aleatoric uncertainty for avoiding the noisy TV problem during self-driven exploration. This is an important area where we need more ideas and experiments. The authors present a biologically inspired approach and through experiments. Although it doesn't present the state-of-the-art exploration in well-known RL environments, I acknowledge that new solutions to problems that were previously intractable often would face such an issue. The prediction to discriminate neuroscientific modulations that directly encode epistemic and aleatoric uncertainty is bold but not very specific. Unfortunately, as the reviewers noted, the manuscript in the current form doesn't quite meet the bar yet. I suggest comparing methods for directly estimating uncertainty. I also suggest adding discussion on the estimation bias for the epistemic uncertainty for the proposed method. I strongly encourage the authosr to continue this interesting line of work.
Decision: Reject