Escaping Stochastic Traps with Aleatoric Mapping Agents

When extrinsic rewards are sparse , artificial agents struggle to explore an environment . Curiosity , implemented as an intrinsic reward for prediction errors , can improve exploration but fails when faced with action-dependent noise sources . We present aleatoric mapping agents ( AMAs ) , a neuroscience inspired novel form of curiosity modeled on the cholinergic system of the mammalian brain . AMAs aim to explicitly ascertain when dynamics of the environment are unpredictable , even if those dynamics are induced by the actions of the agent . This is achieved by generating separate forward predictions for the mean and aleatoric uncertainty of future states , reducing intrinsic rewards for those transitions that are unpredictable . We demonstrate that in a range of environments AMAs are able to circumvent action-dependent stochastic traps that immobilise conventional curiosity driven agents . 1 INTRODUCTION . Efficient exploration is a central problem in reinforcement learning . Exploration is particular challenging in environments with sparse rewards—requiring agents to navigate with limited guidance ( e.g . Sutton & Barto ( 2018 ) ; Pathak et al . ( 2017 ) ; Burda et al . ( 2018b ) , see Weng ( 2020 ) for a review ) . A notable exploration method that effectively deals with sparse rewards is curiosity driven learning—where agents are equipped with a self-supervised forward prediction model that employs prediction errors as intrinsic rewards Schmidhuber ( 1991b ) ; Pathak et al . ( 2017 ) ; Schmidhuber ( 1991a ) . Curiosity is built upon the intuition that in unexplored regions of the environment , the forward prediction error of the agent ’ s internal model will be large Schmidhuber ( 1991b ) ; Pathak et al . ( 2017 ) . As a result , agents are rewarded for visiting regions of the state space that they have not previously occupied . If , however , a particular state transition is impossible to predict , it will trap a curious agent Burda et al . ( 2018b ) ; Schmidhuber ( 1991a ) . This is referred to as the noisy TV problem ( e.g . Burda et al . ( 2018b ) ; Schmidhuber ( 1991a ) ) , the etymology being that a naively curious agent could dwell on the unpredictability of a noisy TV screen . Several existing curiosity-like methods Burda et al . ( 2018b ) ; Pathak et al . ( 2017 ; 2019 ) aim to avoid noisy TVs or “ stochastic traps ” Shyam et al . ( 2019 ) . Nevertheless , employing dynamics based prediction errors as intrinsic rewards is difficult as current methods either fail when stochastic traps are action-dependent , or require an ensemble of dynamics models Pathak et al . ( 2017 ; 2019 ) ; Shyam et al . ( 2019 ) ; Burda et al . ( 2018a ) . This work presents aleatoric mapping agents ( AMAs ) , which uses aleatoric uncertainty estimation Kendall & Gal ( 2017 ) to escape stochastic traps . AMAs are both inspired by and build upon proposals developed within neuroscience Yu & Dayan ( 2005 ) , that suggest expected uncertainties in top down predictions are signalled by the modulation of cortical acetylcholine . We build on this theory by algorithmically demonstrating a functional purpose for expected aleatoric uncertainties that is consistent with , but also further specifies , Yu & Dayan ( 2005 ) ’ s predictions of cholinergic activity in the Posner task . Our work adds further credence to Yu & Dayan ( 2005 ) ’ s call for experimental tests of whether acetylcholine signals epistemic or aleatoric uncertainties . Accordingly , a blueprint for a VR mouse task , along with theoretical predictions of cholinergic uncertainty signalling in such a task , is presented in Section 5 . 2 BACKGROUND . 2.1 EPISTEMIC AND ALEATORIC UNCERTAINTIES . The uncertainty of a statistical model can be described as the sum of two theoretically distinct types of uncertainty : epistemic uncertainty and aleatoric uncertainty ( e.g . Hora ( 1996 ) , see Hüllermeier & Waegeman ( 2021 ) for a review ) . Epistemic uncertainty measures the errors of a model ’ s prediction that can be minimised with additional experience and learning Hüllermeier & Waegeman ( 2021 ) . As a result , an agent using epistemic uncertainties as intrinsic rewards tends to value dynamics it has not previously encountered , and hence can not predict accurately , but could learn to predict in the future ( e.g . Osband et al . ( 2016 ) ) . More concretely , epistemic uncertainty can be considered to be the expected information gain of observing the next predicted state Mukhoti et al . ( 2021 ) . On the other hand , prediction errors that are due to aleatoric uncertainties are , by definition , a result of unpredictable processes Hüllermeier & Waegeman ( 2021 ) . Therefore , any agents that receives intrinsic rewards for aleatoric dynamics risks being trapped , as exemplified by the noisy TV problem Schmidhuber ( 1991a ) ; Burda et al . ( 2018a ) . Direct estimation of epistemic uncertainty surrounding future states would be an ideal basis for a curious agent but tractable epistemic uncertainty estimation with high dimensional data is an unsolved problem Gal ( 2016 ) . We implicitly incentivise agents to seek epistemic uncertainties by removing the aleatoric component from the total prediction error . This is similar to methods that separate epistemic and aleatoric uncertainties in return predictions Clements et al . ( 2019 ) , or within a latent variable model Depeweg et al . ( 2018 ) —allowing for the construction of policies that are rewarded for exploring their environments and punished for experiencing aleatoric uncertainty . However , as far as we are aware , we are the first to compute aleatoric uncertainties within a scalable curiosity framework to reduce intrinsic rewards for those state transitions with aleatoric uncertainty . 2.2 CURIOSITY AND INTRINSIC MOTIVATION IN REINFORCEMENT LEARNING . Curiosity-driven Pathak et al . ( 2017 ) agents assign value to states of the environment that they deem to be “ interesting ” Still & Precup ( 2012 ) ; Schmidhuber ( 1997 ) . How a curiosity based method computes whether a state is “ interesting ” Still & Precup ( 2012 ) ; Schmidhuber ( 1997 ) is usually its defining characteristic . The original formulation of curiosity used prediction errors directly as intrinsic rewards Schmidhuber ( 1991b ) . The noisy TV problem quickly emerged when using this naïve approach in stochastic environments Schmidhuber ( 1991a ) . In order to evade the allure of stochastic traps , the first proposed solution to the noisy TV problem implements “ interesting ” Still & Precup ( 2012 ) ; Schmidhuber ( 1997 ) as prediction errors that reduce over time Schmidhuber ( 1991a ) ; Kaplan & Oudeyer ( 2007 ) . Others consider “ interesting ” Still & Precup ( 2012 ) ; Schmidhuber ( 1997 ) to mean a high dependency between present and future states and actions ( i.e . “ interesting ” things are predictable Still & Precup ( 2012 ) or controllable Mohamed & Jimenez Rezende ( 2015 ) ) . Inverse dynamics feature ( IDF ) curiosity Pathak et al . ( 2017 ) rejuvenated interest in using one step prediction errors as intrinsic rewards . IDF curiosity avoids stochastic traps by computing prediction errors with features that aim to only contain information concerning stimuli the agent can affect Pathak et al . ( 2017 ) . Further experiments Burda et al . ( 2018a ) showed that simple one-step prediction errors also work effectively within a random representation space generated by feeding state observations through a randomly initialised network . Burda et al . ( 2018a ) also showed the ( IDF ) approach is vulnerable to action-dependent noisy TVs—demonstrated by giving the agent a ‘ remote control ’ to a noisy TV in the environment that could induce unpredictable environment transitions . This motivated random network distillation ( RND ) Burda et al . ( 2018b ) , which removes dynamics from the prediction problem altogether—instructing a network to learn to predict the output of another fixed randomly initialised network at each state , using the resulting error as intrinsic rewards . Other exploration methods explicitly leverage uncertainty quantification for exploration . The canonical approach is “ optimism under uncertainty ” , which in its most basic form means weighting the value of state-actions pairs inversely to the number of times they have been experienced ( Sutton & Barto , 2018 , p. 36 ) . Known as count based methods Strehl & Littman ( 2008 ) ; Bellemare et al . ( 2016 ) , this approach was shown to reliably evade noisy TVs in minigrid environments Raileanu & Rocktäschel ( 2020 ) . However , it is not feasible to count state visitations in many environments where there is a large number of unique states Bellemare et al . ( 2016 ) . “ Pseudo-count ” methods exchange tabular look up tables for density models to estimate an analogous intrinsic reward to counts in large state spaces Bellemare et al . ( 2016 ) . Nevertheless , prediction error based methods for exploration remain a key component in state of the art algorithms deployed in high dimensional state spaces ( e.g . Badia et al . ( 2019 ) ) . Attempts have been made to reward Epistemic uncertainty directly . This typically requires a posterior distribution over model parameters , which is intractable without approximations such as ensembles or variational inference ( e.g . Houthooft et al . ( 2016 ) ) . Osband et al . ( 2016 ) instantiated an ensemble Lakshminarayanan et al . ( 2017 ) approach into the final layer of a deep Q-network—rewarding its agents for epistemic value uncertainty . Pathak et al . ( 2019 ) use the variance of ensemble predictions being used as intrinsic rewards , while Shyam et al . ( 2019 ) reward experience of epistemic uncertainty within an ensemble of environment models . Lastly , some uncertainty estimation methods have recently been developed that enforce a smoothness constraint in the representation space Mukhoti et al . ( 2021 ) ; van Amersfoort et al . ( 2021 ) —allowing for sensible estimations of uncertainty to be made from learned representations—but these approaches have not yet been adopted in reinforcement learning . All in all , uncertainty often plays a role in different formulations of curiosity , which is why we looked at models acetylcholine—a neuromodulator associated with uncertainty—for inspiration in building curiosity models capable of avoiding distractions . 2.3 ACETYLCHOLINE . In the mammalian brain acetylcholine is implicated in a range of processes including learning and memory , fear , novelty detection , and attention Ranganath & Rainer ( 2003 ) ; Pepeu & Giovannini ( 2004 ) ; Acquas et al . ( 1996 ) ; Barry et al . ( 2012 ) ; Yu & Dayan ( 2005 ) ; Hasselmo ( 2006 ) ; Giovannini et al . ( 2001 ) ; Parikh et al . ( 2007 ) . Traditional views—supported by the rapid increase in cholinergic tone in response to environmental novelty and demonstrable effects on neural plasticity— emphasised its role as a learning signal , generating physiological changes that favour encoding of new information over retrieval Hasselmo ( 2006 ) . Notably , Yu & Dayan ( 2003 ) proposed an alternative perspective , suggesting that acetylcholine signals the expected uncertainty of top down predictions , while modulation of norepinephrine is a result of unexpected uncertainties . More concretelty , Yu & Dayan ( 2003 ) ’ s model can be seen as favouring bottom up sensory input over top down predictions if predictions are believed to be inaccurate—consistent with evidence that shows acetylcholine inhibits feedback connections and strengthens sensory inputs Hasselmo ( 2006 ) . However this approach does not explicitly separate epistemic and aleatoric uncertainties Yu & Dayan ( 2005 ) . In contrast , the utility of quantifying epistemic uncertainties for exploration has been widely recognised in the RL literature ( e.g . Osband et al . ( 2016 ) ; Pathak et al . ( 2019 ) ) . Here we demonstrate a potential use of aleatoric uncertainties in exploring agents both biological and artificial . Namely , aleatoric uncertainties can be used to divert attention away from unpredictable dynamics when using prediction errors as intrinsic rewards . This is similar to a model proposed by Parr & Friston ( 2017 ) , suggesting acetylcholine may indicate expected uncertainties in top down predictions within an MDP . In this context we propose an extension to Yu & Dayan ( 2005 ) ’ s dichotomy . Specifically , we suggest that in the mammalian brain , cortical acetylcholine signals expected aleatoric uncertainties , while norepinephrine is modulated by epistemic uncertainties both expected and unexpected . This formulation is attractive in an ML framework , providing a means to avoid stochastic traps , while also being consistent with empirical biological data Hasselmo ( 2006 ) ; Yu & Dayan ( 2003 ) . Testing this hypothesis calls for experiments to clarify the nature of the uncertainty signalled by acetylcholine , which we present in Section 5 .

Summary: This paper suggests an intrinsic bonus for exploration that avoids noisy TV by adding a penalty for the estimated variance of the reached state $S_{t+1}$ given previous state $S_{t}$.  For this, they fit an independent normal model of the new state $S_{t+1}$ with mean ($\mu_{t+1}$) and var ($\sigma_{t+1}^2$) predicted from the previous state $S_{t}$. Then, the bonus is given by the error of prediction $(S_{t+1} - \hat{\mu_{t+1}})^2$ minus the variance penalty $\sigma_{t+1}^2$.   That way, a noisy TV will always be fitted with an higher variance that should have a similar order than the error of prediction, so both predictable and unpredictable transitions bonus will end up to zero (in average) after learning the model of dynamics of the whole environment.