Brain Bandit: A Biologically Grounded Neural Network for Efficient Control of Exploration
Authors: Chen Jiang
Jiahui An
Yating Liu
Ni Ji
Venue: ICLR 2025 Oral (2025)
Keywords: explore-exploit
stochastic Hopfield network
Thompson sampling
decision under uncertainty
brain-inspired algorithm
reinforcement learning
Primary area: applications to neuroscience & cognitive science
TL;DR: We demonstrate that a brain-inspired stochastic Hopfield network can achieve efficient, human-like, uncertainty-aware exploration in bandit and MDP tasks.

Abstract: How to balance between exploration and exploitation in an uncertain environment is a central challenge in reinforcement learning. In contrast, humans and animals have demonstrated superior exploration efficiency in novel environments. To understand how the brain’s neural network controls exploration under uncertainty, we analyzed the dynamical systems model of a biological neural network that controls explore-exploit decisions during foraging. Mathematically, this model (named the Brain Bandit Net, or BBN) is a special type of stochastic continuous Hopfield network. We show through theory and simulation that BBN can perform posterior sampling of action values with a tunable bias towards or against uncertain options. We then demonstrate that, in multi-armed bandit (MAB) tasks, BBN can generate probabilistic choice behavior with a flexible uncertainty bias resembling human and animal choice patterns. In addition to its high efficiency in MAB tasks, BBN can also be embedded with reinforcement learning algorithms to accelerate learning in MDP tasks. Altogether, our findings reveal the theoretical foundation for efficient exploration in biological neural networks and propose a general, brain-inspired algorithm for enhancing exploration in RL.

Published as a conference paper at ICLR 2025
BRAIN BANDIT: A BIOLOGICALLY GROUNDED NEU- RAL NETWORK FOR EFFICIENT CONTROL OF EXPLO- RATION
Chen : An Yating Liu N ji
Chinese Institute for Brain Research, Beijing
Chinese Academy of Medical Sciences & Peking Union Medical College
"China Agricultural University
chen. j jiang3
@mail 3 www.mmclll.... ca, fanjiahui, liuyating, niji)ecibr. ac.
ABSTRACT
How to balance between exploration and exploitation in uncertain environment is a central challenge in reinforcement learning. In contrast, humans and animals have demonstrated superior exploration efficiency in novel environments. To un- derstand how the brain's 'sin's neural network controls exploration under uncertainty, we analyzed the dynamical systems model of a biological neural network that controls explore-exploi decisions during foraging. Mathematically, this model (named the Brain Bandit Net, or BBN) special type of stochastic continuous Hopfield network. We show through theory and simulation that BBN can per- form posterior sampling of action values with a tunable bias towards or against uncertain options. We then demonstrate that, in multi-armed bandit (MAB) tasks, BBN can generate probabilistic choice behavior with a flexible uncertainty bias resembling human and animal choice patterns. In addition to its high efficiency in MAB tasks, BBN can also be embedded with reinforcement learning algorithms accelerate learning in MDP tasks. Altogether, our findings reveal the theoreti- cal foundation for efficient exploration in biological neural networks and propose general, brain-inspired algorithm for enhancing exploration in RL. The code is available at /ttpss//giiaab.com/hhennnniiiiie
INTRODUCTION
The explorc-exploit (E-E) dilemma, originally described in the context of animal foraging (Stephens & Krehs| 1986 Charnoy] 1976). has become an important problem across many fields including psy- chology, neuroscience and reinforcement learning (RI))Addico)
al.) 2017). Despite the devel- opment of numerous algorithms, sample-efficient exploration in RL remains difficult for complex, sparse-reward tasks (Sutton & Barto 2018). Meanwhile, studies in humans and animals have re- vealed a diverse array of exploration strategies (Wilson et al.) 2021 Schulz & Gershman 2019). In addition, excitingly, recent research has begun to reveal the biological neural networks that give rise to the rich and flexible exploration behaviors (Costa et al. 2019 Tomov et al. 2020 Hogeveen al. 2022 Costa & Averbeck 2020). Based on recent findings the biological neural network that controls exploration, we built the Brain Bandit Network (BBN), stochastic Hopfield network for controlling exploratory action selection under input uncertainty. We show theoretically that the BBN model can perform Bayesian posterior sampling while implementing a tunable bias that ranges from optimistic, neutral, and conservative in the face uncertainty.
Our main contributions are four-fold:
1. We propose a biologically grounded, scalable network model for solving the E-E dilemma.
2. We analytically show that BBN implements a hybrid between Bayesian posterior sampling and uncertainty-diredt exploration.
"Currett address: MeGill University. 1 'Coresponding author.

---

Published as a conference paper at ICLR 2025
3. We show that BBN can closely approximate human and animal behavior in bandit tasks under a variety of conditions.
4. We show that BBN can drive highly efficient exploration in bandit and MDP tasks, promis- ing further application to more complex RL problems.
2 BACKGROUND AND RELATED WORK
2.1 THE EXPLORATION PROBLEM IN REINFORCEMENT LEARNING
The domain of efficient exploration in reinforcement learning focuses on balancing immediate wards (exploitation) and information gathering for future rewards (exploration). A classic example is the Multi-Armed Bandit (MAB) problem. introduced by (Robbins 1952) in 1952 and widely used to model this tradeoff (Lai Robbins 1985 Berry & Fristedt 1985; Agrawal 1995 Auer et al.]
1995 Sutton & Barto [999). Conventional methods inject noise into action selection (Sutton &
Barto 1999), but these dithering algorithms can be inefficient. Alternative methods like Upper Con-
fidence Bound (UCB) employ optimism in the face of uncertainty (OFU) by biasing for uncertain
choices (Lai & Robbins 1985 Agrawal 1995 Auer et al. 1995). Thompson sampling (Thomp-
son 1933) makes decisions based on posterior samples rather than optimistic estimates. Optimistic Thompson Sampling (O-TS), combining UCB and Thompson sampling. reshapes the posterior dis- tribution optimistically and exhibits strong empirical and theoretical performance (Chapelle Li,
2011 May et al. 2012). More recent methods leverage deep networks to learn the exploration bonus
(Zhou et 2020 Ban et al. 2022 or the posterior variance (Zhang et al. 202T).
2.2
BIOLOGICAL SOLUTIONS TO THE EXPLORE-EXPLON DILEMMA
Early work on the explore-exploit tradeoff, rooted in Optimal Foraging Theory and the Marginal Value Theorem (Stephens & Krebs 1986 Charnov 1976), suggests that animals achieve near-
optimal balance between exploiting known resources and exploring uncertain options. Cognitive scientists have used bandit tasks to study this tradcoff in humans and animals (Addicott et al. 2017
Cohen et al.] 2007 Wang al. 2023 Beron et al. (022). Two main strategies emerge: random exploration, involving stochastic action choices, and directed exploration, leveraging uncertainty to guide actions (Wilson al. 2021
Schulz & Gershman 2019). Humans and animals often combine these strategies flexibly, ad adjusting based on task horizon, option novelty, developmental stage, and mental state (Gershman 2018 Bartumeus al. 2016 Wilson et al.
2014 Cockburn et al. 2022
Addicott et al. 2017 Fan et al. 2023 Waltz et al.) [2020). Ad- Mizell et al. 3024| et al. |
ditionally, they exhibit persistent exploration, repeating previous choices regardless of value (Beron et al. 2022 Laurie et al. 2024). 'Tese strategies resemble algorithms like Thompson sampling and Optimism the Face of Uncertainty (OFU), but with key differences (Wilson al. (02T).
To understand the brain's solution to the E-E dilemma, neuroscientists have identified neurobio- logical mechanisms that control explore-exploit decisions (Daw et al. [0066] Costa et al. 2019 Hogeveen al. 2022). Recent studies in C. elegans (Flaveil et al. 2213: Ji et al.) 2021) have
vealed a compact recurrent network governing the transitions between behavioral states analogous to exploration and exploitation (Fig. []] This minimal network provides a unique opportunity to explore the algorithmic principles the brain uses to solve the E-E problem.
MODEL
3.1 THE BRAIN-INSPIRED BANDIT NETWORK (BBN) STOCHASTIC CONTINUOUS HOPFIELD NETWORK
To model the biological neural network that controls E-E decisions during foraging (Fig. g Ji al., 2021)) we define a set of N neurons whose temporal dynamics are described by the following stochastic differential equations (or Langevin equations):
da: dt
Σ
Wiy Wizs(x;) + b, + (:dW(t)
2

---

Published as a conference paper at ICLR 2025
(a) Model architecture
(b) Hopfield energy
(c) Neural activities
20.0
12.5
10
-10.0
10.0
5-
-0.55
5,0
2.5
X1
%
5
10
15
20
0.0
1000
2000
3000
4000
5000
P(Action 1)
P(Action 2)
X1
time
Figure 1: The Brain-inspired Bandit Network (BBN) (a) Architecture of the 2-D BBN model. (b) Hopfield energy (or Lyapunov function) plotted over the state space of BBN. Heatmap represents the Hopfield energy; red and green curves are the nullclines, and white dots represent simulated network states. (c) Evolution of network states over time
Where f(x) = Ite-mie-e,, WA, < O, and dW(t) is the Wiener process. Here, Way (xz) represents
the inhibitory interaction between neurons; b, is the baseline activity of neuron i; 1 and a.dW (1) are the deterministic and the stochastic components of the external input, respectively. a, is the standard deviation of the Wiener noise. We term this type of stochastic continuous Hopfield network with all negative weights the Brain-inspire Bandit Network (BBN), for reasons that will become clear later.
Assuming approximately symmetric weights i.e., Wij Wi the deterministic part of the model is essentially continuous Hopfield network (Hopfield 1982 1984) with exclusively inhibitory connections. hence associated with a Hopfield energy or Lyapunov function of the form:
I f(x)dx Σ (ma)
E
Σ (.
Wigs (xt) f (x;) + Σ
xif (Ir))
2
N
Σ If (zx)
= Eónt Eest
Here, have decomposed the Hopfield energy into Eint, dependent only on internal network parameters, and gext which embodies influence from external inputs I, With suitable parameters (see Appendix B. the model can have up to N local energy minima attractor states exhibiting winner-take-al dynamics (Fig. and Fig. 12 Stochastic noise induces transitions between these attractor states, consistent with experimental findings in foraging networks (Ji al. 202I).
3.2 THE BBN IMPLEMENTS BAYESIAN POSTERIOR SAMPLING
Hinton and Sejnowski (Hinton & Sejnowski 1983) have demonstrated that a discrete Hopfield net- work with stochastically activating units (i.e. Ising network) can implement Bayesian inference by sampling from the posterior distribution. Here we extend this conclusion to continuous Hop- field networks. Briefly, using Kramers' escape theory (Kramers) 1940 Langer 1968 Hänggi al.) 1990), we can approximately compute the mean first passage time (MFPT), defined here as the expected time leave an attractor state A and crossing the nearby saddle point S, as:
2 w exp
AEA DA
(TA)
Wb II, w!
'Wiie the original Hopfield network study (Hopfeel 1984 required weight symmetry to prove absolute stability of the energy (or Lyapunov) function. Later work Maasuuoka [992] that the global convergence of the Hopfield energy function still holds for networks with asymmetric weights.
Chen & Amari (2001) have shown
3

---

Published as a conference paper at ICLR 2025
Where 7 is the friction coefficient (equivalent to in Eq. I and w, are the angular frequencies (i.e. cigenvalues of the Hessian matrix) at the center (i.e. energy minimum) of the attractor. wp and wp are the angular frequencies of the saddle point, with wj associated specifically with the unstable mode. A E is the energy difference between the saddle point and the center of the attractor and AAA-S Fs F.A. D is the diffusion constant, which in thermodynamics scales with the magnitude of the stochastic noise.
The equilibrium probability of the network being in a given attractor state A can be approximated by its stability, measured via the MFPT, relative to the other attractors. This translates to:
PA (TAI) Σ1 (TAi)
where (IE(I.....N) = II, w/t
1
Σ2
aj a1
exp
AFAi Das
AEAl
wb II, wir
(4)
Assuming identical biophysical parameters and inputs for all neurons, the angular frequencies w of the N attractors permutations of each other and there is single saddle point defined by x NN (f)) + + I. This leads to ai = -aj Vi. Further, by substituting AEA = (Es -Ex + Exst into Eq. we have:
PA ~
1+ Σ exp ([
Es EE--ET) ET
Es Ein
E
(5)
DAs
DA1
DAi
DAL
Now if we define the probability of an attractor state in the absence of external inputs as its prior probability as: Peite exp /Da): and the probability of the state given input data (e.g. sensory evidence) as: P A.) exp (E (Di)), we have:
1(A
1+ E
(potor /PRrier * A1)(1AA)/P(( |
(6)
Eq. 6 reveals close connection between the Hopfield-energy-base formulation of attractor state probability and Bayesian inference. Specifically, we consider PAi as the probability of hypothe- sis being true decision i being optimal, then Eq. [eesenttally computes the Bayesian posterior of given external evidence.
3.3 THE BBN CAN EXIIBIT OPTIMISTIC, NEUTRAL, OR CONSERVATIVE BIASES ON INPUT UNCERTAINTY
In Kramers' theory, the diffusion constant D from thermal fluctuations typically isotropic (E I² D = o²)) However, in our model, input to each neuron can have different levels of uncertainty, making the overall noise anisotropic. Recent studies (Zhu al., 2018 Yang et al.) (2023) show that anisotropic noise affects escape efficiency (the rate at which model leaves one of its attractor states (i.e. 1/MFPT)) by interacting with local attractor curvature. Starting from local energy minimum at Zp. the model evolves as:
((x)) f
((EVVVE)) + (H+
Here, Ho is the Hessian matrix evaluated the attractor bottom, and Σ is the noise covariance matrix. Since both matrices are diagonal in our model, the escape efficiency is highest when the dimensions of largest input noise align with those of highest curvature. To capture this effect, define isotropic noise Σ σ²I I that yields the same efficiency E:
TT(H, Tr(H;)
HH = 2 H.) = THH
where ग.
DreTT
(8)
4

---

Published as a conference paper at ICLR 2025
Here, tfff represents the effective diffusion constant and H, PHj HA, Vi where i
permutation matrix. Substituting Eq. into Eq. 4 we have:
PAI
+ exp
(TH(H))AFA (T(HI Tr(H
(a) Attractor probability
(b) Slope
(c) Intercept
Attractor state probability
1.0
5.0
optimistic neutral
4,5
conservative
0.6
8 3.0
intercept
0. 0,33,0;-255
2.5
L
0,330
1,25.0;
2.0
3
m,
1.5
-0.5 0.0
0.5
1.0
1.5
2.5
3,0
3.5
0.0
0.2
0.4
0.6
0.8
1.0
Mean Input difference
Total uncertainty
Relative uncertainty
Figure 2: BBN implements Bayesian posterior sampling with a tunable bias towards, neutral to, against input uncertainty. (a) Sigmoidal dependence of attractor state probability on the difference in mean input values. (b) Slope of the state probability curve in (a) as a function of total input uncertainty (defined as Vo + o for the three types of networks. (c) Intercept of the state probability curve function of relative input uncertainty (defined as σ 02).
While all N attractors have equal energy and share common set of angular frequencies, their Hessian matrices are non-identical and can interact differently with non-isotropic noise (i.e., * e.). If PAi corresponds to the attractor state with the highest input noise, the following scenarios can occur (assuming j * 1):
1. Tr < Tr(H,E) and PAI > Paj (Optimistic). 2. Tr(H, = Tr(H, and PAI Pay (Neutral). E) > Tr(H,E) and PAI < PAj (Conservative).
These regimes are termed Optimistic, Neutral, and Conservative, respectively. Fig. iilustrates the input dependence of attractor state probabilities under these three regimes.
Parameter sensitivity analyses (Fig. 3(a-b) and Fig. II in Appendix A reveal that the three pa- rameter regimes span a wide range parameter combinations, obviating the need for fine-tuning. By adjusting the baseline activity b, synaptic threshold k; or inhibitory synaptic weight w either individually or in pairs - one can ficxibly modulate the uncertainty bias from highly optimistic (PA1 1) to neutral (PA1 N highly conservative (PA1 0).
3.4 (OPTIMISTIC) UNCERTAINTY BIAS PRESERVED IN HIGHER DIMENSIONS
The theoretical analysis above predicts that the uncertainty bias of BBN should scale well high dimensions. To verify this empirically, we progressively increased network dimension by adding more neurons, while keeping all network parameters in Eq. [nncaaneed. Strikingly, for BBN that is optimistic at N = 2. scaling up to N = 10 did not alter its optimistic bias (Fig. c()). In contrast, a BBN that neutral in 2D became mildly optimistic becomes mildly optimistic at N > 5. Thus, with increasing network dimension, the model exhibits a tendency to bias towards attractor states with higher input uncertainty.
increased, while a conservative BBN
To understand this empirical phenomenon, we examined sate-transition dynamics near the saddle point for a perfectly neutral 3D BBN (i.e., H, = cf, Vi) (Fig. [3}. With isotropic noise, the network exhibited equal probability of entering any attractor state. However, with highly anisotropic noise, preferentially entered the attractor state along the dimension of highest noise, creating a bias towards high-uncertainty states. This makes conservative bias harder maintain and optimistic bias more prominent in high-ding dige--imessional models (Fig. [4 To incorporate this effect into our theoretical framework, we need to combine escape rates analysis (Kramers 1940 Zhu et al. 2018) with theory
5

---

Published as a conference paper at ICLR 2025
Theory
(b) Simulation
(c) Multi-dimensional model
80
1.0
1.0
0.8
Hiseal
F0
0.8
0.7
Median(A)
Probability
1/
0.6
0.5
0.4
0.4
-0.4
03
50
5.0
0.2
0.
0.0
60
60
23
80
70
8.0
85
Dimensionality (N)
Figure 3: Parameter dependence and multi-dimensional model. (a) Theoretically derived and (b) numerically simulated attractor state probability function of network parameters and k. The color scale corresponds to the probability that the network samples the attractor state driven by the highest input uncertainty, which is an indicator of the network's uncertainty bias. (c) Equilib- rium attractor state probabilities in high dimensional BBN models. Three colored lines correspond attractor states driven by the highest (orange), median (green), or lowest (blue) levels of input uncertainty. The network parameters remain unchanged as the dimensionality N increases.
of dynamics around saddle points (Daneshmand al., 2018 challenge aim address in future work.
4 EXPERIMENTAL EVALUATION
4.1 UNCERTAINTY-AWARE EXPLORATION IN MULTI-ARMED BANDIT TASK
Given BBN's 's ability to infer and sample from posterior distribution with a tunable uncertainty bias, a natural application of BBN control action choice given external, uncertain evidence. We thus adapted the BBN model to play multi-armed bandit (MAB) games and compared its performance with classic bandit algorithms.
4.1.1 RUNNING BBN BANDIT GAMES
To make the BBN model play bandit games, we (1) define BBN model with N neurons, each corresponding to one of the N bandit arms; (2) pick network parameters that yield "optimistic" exploration for a 2-D BBN, and simply apply the same parameters to all neurons in the N-D model; (3) prior to each bandit trial, assign network input I by sampling from the reward memory buffer and numerically simulate the network for T steps using the Runge-Kutte method; (4) at the end of simulation, select the arm a whose corresponding neuron has the highest activation value; (5) collect the reward ra and add it to the memory buffer for arm a; (6) repeat (3)-(5) for the trial till the game ends. The pseudocode along with detailed task parameters are presented in Appendix B.
4.1.2 BBN IMPLEMENTS ENCERTAINTY-AWAR POSTERIOR SAMPLING
To reveal BBN's exploration strategies, we examined the dependence of choice probability on tal and relative reward uncertainty for BBN agents with optimistic,, neutral, or conservative biases, as well as classic algorithms Thompson Sampling (TS) and Upper Confidence Bound (UCB). As shown in Fig. 4 (a-b). TS exhibited a constant intercept regardless of relative uncertainty (RU) and a decreasing slope with increasing total uncertainty (TU), indicating sensitivity only to total uncer- tainty; UCB exhibited constant slope with varying TU and an increasing intercept with increasing RU, indicating sensitivity only to relative uncertainty. In contrast, BBN with optimistic parameters showed variation in both slope and intercept with changes in TU and RU. These results indicate that BBN implements a hybrid algorithm combining posterior sampling (like TS) with tunable bias towards high uncertainty (akin to UCB).

---

Published as a conference paper at ICLR 2025
(a) Slope
(b) Intercept
1.00
BBNoopiimistic) BBN(neutral)
Intercept
Slope
intercept
0.25
BBN(conservative)
0.00
Thompson sampling
4.0
-0.25
UCB
-0.50
35-
-1.00
0.00
0.25
0.50
6.75
1.00
1.25 1.50
Total uncertainty
Relative uncertainty
Figure Exploratory behavior BBN, Thompson sampling and UCB in 2-armed bandit games (a) Slope of the choice probability curve as function of total uncertainty. (b) Intercept of the choice probability curve as function of relative uncertainty.
4.1.3 EFFICIENT EXPLORATION IN BANDIT TASKS
We next compared the empirical performance of BBN-driven exploration comparison against UCB. Thompson sampling, and Optimistic Thompson Sampling (OTS, (Hu et al.) (023 in both 2-armed bandit and 3-armed bandit games. Each agent played 10,000 game blocks of 20 trials each
in 2-armed bandit games and 30 trials each in 3-armed bandit games. Fig. S (a-b) presents the probability of choosing the optimal arm as trial number increases. BBN (with optimistic param- eters) consistently outperformed other algorithms in 2-armed bandits and topped the performance in 3-armed bandit games. The other 'hybrid' algorithm, OTS, performed close to BBN 3-armed bandits, but did poorly in 2-armed bandits.
(a) 2-armed bandit
(b) 3-armed bandit
0.5
475
UCRL2
LSE 75
Nopptmall
#25
800
0.50
0:55
e-groody UCB
Curnulative
LBE ICB
H4M0OUn)
675
350
T5
0.40
BBN(Ours)
0,45
10
15
10
35
25
80 740
Figure 5: BBN achieves efficient exploration in both bandit tasks. (a) The probability of choosing optimal action over trials in 2-armed bandit games. (b) The probability of choosing optimal action over trials in 3-armed bandit games. (c) Cumulative regret the Aix (MDP task, see Fig. [6)
4.2 BBN CLOSELY APPROXIMATES BANDIT CHOICE BEHAVIOR IN HUMANS AND ANIMALS
The results above indicate that BBN exhibits similar hybrid strategies as previously reported in humans (Wilson al. 2014 Gershman 2018). We thus asked whether BBN can accurately model human and animal choice patterns in bandit tasks. We first compiled several publicly available datasets of humans playing bandit games (detailed list Appendix CC. We performed optimization on two network parameters b and & to minimize the difference between the choice probability curves output by BBN and the human datasets. shown Fig. 6 (a-b), BBN can closely to both the intercept and the slope of human choice probability curves. In contrast, Thompson sampling failed to to the diverse intercepts across human groups, and UCB consistently yielded slopes that are much higher than those observed in human data.
We next extended the above analyses to a dataset in which mice played switching blocks of 2- armed bandit games (Beron al. (022). In this dataset, the reward for each arm is sampled from a Bernoulli distribution. In addition, the mean reward for each arm is not static, with a small prob- ability (0.02) of being reversed before each trial starts. Based on results from (Beron et al. 2022

---

Published as a conference paper at ICLR 2025
(a) Slope
(b) Intercept
3.0
0.3
Fan23_high anxiety(8BN)
2.5
0.2
Gershman19
Fan23, anxiety(BBN)
Human
2.0
0.1
Zajkowski 17(BBN)
0.0
Mizell 24(8BN)
1.5
(en23+GGrshman11((CC))
44
1.0
-0.
0.5
-03
-0.3
-0.2
-6.1
0.0
0.2
0.3
Model
Model
(c) Choice optimality
(d) Action switching
P(optimal)
Piswitch)
0.5
0.1
mouse model
0.0
0.0
-10
10
20
-10
0
10
20
Block Position
Block Position
Figure 6: The choice pattern of BBN closely approximates humans and animals in MAB tasks. (a-b) BBN-fitted versus actual slope and intercept values extracted from human data. (c-d) The probability of choosing the optimal arm and switching to another arm upon block transition in mice playing the 2-armed bandit game.
used the last five rewards inputs to the BBN model to drive choice behavior. As shown in Fig. 6) (c-d), paremeter-tuned BBN generated choice and switching behavior that closely approximated those exhibited in the mice study.
4.3 EFFICIENT EXPLORATION IN MDP PROBLEMS
Building on the strong performance of BBN in MAB tasks, we next applied BBN to MDP problems. Unlike bandit problems with immediate rewards and no state transitions, MDP tasks require sequen- tial decision-makinn under delayed rewards and unknown transition probabilities (Bellman) 1966.
Bertsekas 2012). Among existing methods, UCRL2 (Auer et al. 2008) extends OFU to MDPs, while PSRL (Strens 2000; Osband al. 2013)) generalizes posterior sampling to RL. Hybrid algo- rithms like Optimistic Thompson Sampling (OTS) (Agrawal & Jia 2017 Tiapkin et al. 2022; Hu
al. 2023) aim to improve exploration efficiency but face challenges such computational cost and uncertainty estimation.
We consider finite-houe horizon MDP with state space S, action space A, horizon H. rewards rl and transition probabilities P.a conditioned on states 8, actions a, and step I. The expected total return step / under policy be estimated iteratively using the Bellman equation:
Q1+11 = M4 Σ Ts'na Pass D'v''
where y E(r) is the reward. Estimating uncertainty in Q-values remains open issue in RI. Donoghue et al. (O'Donoghue et al.] 2018) proposed the Uncertainty Bellman Equation (UBE) provide upper bound on the variance of Q-value posteriors. For tabular state space, this method effectively propagates local variance estimates to global value uncertainty.
4.3.1 RUNNING BBN MDP TASKS
To apply BBN to drive cetion-selection in MDP tasks, we (1) define a BBN model with N neu- rons, each corresponding to of the N discrete actions, select network parameters that belong to the "optimistic" regime for a 2D network; (2) initialize state-action values to i.i.d. Gaussian distri- butions; (3) sample input values for each neuron from the distributions of state-action values and
8

---

Published as a conference paper at ICLR 2025
perform numerical simulation of the BBN network for T steps using the Runge- Kutta method; (4) at the end the simulation, select action a whose corresponding neuron has the highest activation value; (5) collect the reward Ta and move to the next state (6) Repeat (3)-(5) till the episode ends; (7) Update the distribution of state-action values using the Uncertainty Bellman Equation (UBE) et al. 2018). (8) repeat (3)-(7) for the next episode till the game ends. We present the pseudo-code for the Algorithm in Appendix B.4
We first compared the exploration efficiency of the BBN-based algorithm (UBE_BB] on the SixArms (Strehl & Lituman 2008) task, with additional implementation details presented in Ap- pendix B.5 We compare our model to PSRL(Osband et al.) 2013). UCRL2 (Auer al. 2008 and OTS-MDP (Hu et al., 2023). We also specifically tested the role of BBN by replacing it with UCB (UBE UCB) or Thompson sampling (UBE TS). In PSRL,, we maintain Gaussian distribution for the rewards and Dirichlet distribution for the transition probabilities. In the OTS-MDP and BBN models, we follow()) (Hu et al. 2023) and limit our uncertainty estimation the reward for simplic- ity. As shown in Fig. (c), the cumulative regret is lowest in UBE-BBN, which demonstrates the potential of BBN promoting highly efficient exploration.
4.3.2
GRID WORLD
(a) FourRooms:
(c) Visitation Counts: grid size 23x23
PSRL
>20
UCLR2
15
(b) State Coverage: grid size 23x23
10
1.0
UBE-UCB
0.5
UBE-BBN (Ours)
159
Episode #
150
450
episodes
(d) Effect of Action Persistence: grid size 103x103
(e) # of Episodes before Reaching Reward State
10
# episodes to 14 jereward
1000
Fraction failure
reach reward
600
0.4
0.6
200
No persistence
With persistence
size
Figure 7: BBN-enhanced RL agent exhibits efficient exploration the FourRooms task. (a) The FourRooms environment. The agent starts at the red point and can receive reward only at the blue point. (b) The percent of grids covered (i.e. the coverage rate) by agents driven by various (c) Display of visitation counts over the course
exploration algorithms over the period of training. of training. (d) Visitation counts for the UBE-BBN agent with or without action persistence, (e) Number of episodes taken till first reaching the reward state for different agents. Pink and purple the UBE-BBR BBN agents with and without action persistence respectively. Blue is PSRL and green is UBE UCB

---

Published as a conference paper at ICLR 2025
We next evaluated the exploration efficiency of BBN sparse-reward MDP tasks, specifically the FourRooms task. In this task, an b-by-N grid world divided into four compartments connected by narrow passages (Fig. (a)). The agent starts from the upper left corner (red dot) and explores the environment to learn state-action values. First, we conducted reward-free exploration by assuming rewards at any state. Exploration efficiency was measured the coverage rate (ratio of visited states to total states) over episodes. Fig. (b) shows that UBE-BBN achieved the fastest coverage rate among all methods. Fig. (c) provides examples of cumulative visitation counts for each method during training. We then varied the environment size and repeated the experiments. UBE- BBN scaled well with grid size, while other algorithms faltered (Fig. 19 Appendix Additional comparisons with more methods in different conditions are in Fig. 23 in Appendix E Trajectories
(visitation counts in a single episode) Fig. 24 reveal that UBE-BBN excelled extended deep exploration, covering hard-to-reach states effectively. Finally, we enhanced action persistence in UBE-BBN by allowing the BBN model to inherit activity states from the previous step (Fig. [25]. This modification leveraged the Hopfield network's 's persistence property, instilling action correlation within episodes. As shown in Fig. 25 adding persistence further boosted UBE-BBN's exploration efficiency in the FourRooms task at large grid sizes.
Parameter sensitivity in MDP tasks: We additionally performed parameter sensitivity analysis for the SixArms and FourRooms task (as shown in Fig. [8] in Appendix [..T] and demonstrated that a broad range of "optimistic" network parameters yielded high performance on these tasks. Hence, optimistic BBN generally delivers good performance in these MDP tasks without requiring parameter fine-tuning,
5 DISCUSSION
We have demonstrated both theoretically and empirically that the BBN architecture can drive flex- ible and efficient exploration in ways similar humans and animals. However, several limitations and open questions remain regarding its practical application. First, simulating the stochastic dif- ferential equations incurs high computational costs. This issue may be circumvented by analytically computing the attractor probabilities using Eq. 4 or by employing neuromorphic hardware. Sec- ond, given the development of many hybrid TS and OFU methods in the RI. community (Hu al.] 2023 Tiapkin et al.) 2022 Agrawal & Jial 2017). it's intriguing to consider what gives rise to BBN's superior performance. One possibility is that BBN, as a system of coupled Langevin equations, ef- fectively implements Langevin sampling of the posterior distribution. Langevin sampling has been shown to enjoy faster mixing and convergence rates than other sampling methods and is particu- larly well-suited for approximate Bayesian inference (Welling & Teh (00IT) Third, the current BBN algorithm lacks the ability to estimate uncertainty associated with state-action values, relying instead on separate algorithm (in this case, the UBB) to generate value distributions. How biolog- ical neural networks compute and encode uncertainty remains an outstanding question, especially in sequential decision settings. Recent studies have suggested that distributed population code (Dehaene et al. 2021) or a spatiotemporal activity pattern could encode uncertainty levels (Savin & Denève 2014). We hope future experimental and theoretical studies will provide more insights into how the brain estimates and utilizes uncertainty. Lastly, given that humans and animals can flexibly modulate their uncertainty bias a context-dependent manner, a valuable extension for the BBN algorithm would be to integrate contextual information into the network input. Expanding the BBN model to include upstream neurons found in the biological foraging network might help implement context-dependent E-E decisions (Fig. S).
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Merideth A Addicott, John M Pearson, Maggie M Sweitzer, David I. Barack, and Michael L. Platt. A primer on foraging and the explore/exploit trade-off for psychiatry research. Neuropsychophar- macology, 42(10):1931-193) 2017.
Rajeev Agrawal. Sample mean based index policies by o (log n) regret for the multi-armed bandit problem. Advances in applied probability, 10544:0054-1078, 1995.
Shipra Agrawal and Randy Jia. Optimistic posterior sampling for reinforcement learning: worst- case regret bounds. Advances in Neural Information Processing Systems, 30, 2017.
Peter Auer, Nicolo Cesa-Bianchi Yoav Freund, and Robert E. Schapire. Gambling in a rigged casino:
The adversarial multi-armed bandit problem. In Proceedings of IEEE 36th annual foundations of computer science, pp. 322-331. IEEE, 1995.
Peter Auer, Thomas Jaksch, and Ronald Ortner. Near-optimal regret bounds for reinforcement learning. Advances in neural information processing systems, 21, 2008.
Yikun Ban, Yuchen Yan, Arindam Banerjee, and Jingrui He. EE-net: Exploiaation-explortioo neural networks in contextual bandits. In International Conference Learning Representations, 2022.
URL htttps:///pppreeeeeee
net/ wtt.//orrmmmo////// cctidd=x_hhvvssRrrr
Frederic Bartumeus, Daniel Campos, William S Ryu, Roger Lloret-Cabot, Vicenç Méndez, and Jordi Catalan. Foraging success under uncertainty: search tradeoffs and optimal space use. Ecology letters, 19(11):1299111) 2016.
Richard Bellman. Dynamic programming, science, 153(3731):34 37, 1966.
Celia C Beron, Shay Q Neufeld, Scott W Linderman, and Bernardo L Sabatini. Mice exhibit stochas- tic and efficient action switching during probabilistic decision making. Proceedings of the Na- tional Academy of Sciences, 119(15) (0t)))22119111111
13961 119, 2022.
Donald A Berry and Bert Fristedt. Bandit problems: sequential allocation of experiments (mono- graphs on statistics and applied probability). London: Chapman and Hall, 5(71-87):7-7, 1985.
Dimitri Bertsekas. Dynamic programming and optimal control: Volume I, volume 4. Athena scien- tific, 2012.
Olivier Chapelle and Lihong Li. An empirical evaluation of thompson sampling. Advances in neural information processing systems, 24, 2011.
Eric L Charnov. Optimal foraging, the marginal value theorem. Theoretical population biology. (2):129-136) 1976.
Tianping Chen and Shun Ichi Amari. Stability of asymmetric hopfield networks. IEEE Transactions on Newral Networks, 12(1):19--163, 2001.
Jeffrey Cockburn, Vincent Man, William A Cunningham, and John P O'DDerrty.. Novelty and un- certainty regulate the balance between exploration and exploitation through distinct mechanisms in the human brain. Neuron, 110(16):2611-2702 2022.
Jonathan D Cohen, Samuel M McClure, and Angela J Yu. Should i stay or should go? how the human brain manages the trade-off between exploitation and exploration. Philosophical Trans- actions of the Royal Society B: Biological Sciences, 362(148) (62(1181))933-44 1):33-942, 2007.
Vincent D Costa and Bruno B Averbeck. Primate orbitofrontal cortex codes information relevant for managing explore-exploit tradeoffs. Journal of Neuroscience, 40(12):2553-256, 2020.
Vincent Costa, Andrew Mitz, and Bruno Averbeck. Subcortical substrates of explore-cxploit decisions in primates. Neuron, 103(3)5533-553 2019.
Hadi Daneshmand, Jonas Kohler, Aurelien Lucchi, and Thomas Hofmann. Escaping saddles with
stochastic gradients. International Conference on Machine Learning, pp. 1155-1164 PMLR, 2018.

---

Published as a conference paper at ICLR 2025
Nathaniel D Daw, John P O' O'doherty, Peter Dayan, Ben Seymour, and Raymond J Dolan. Cortical substrates for exploratory decisions in humans. Nature, 441(7095):866 8778999 2006.
Guillaume P Dehaene, Ruben Coen- Coen-Cagli, and Alexandre Pouget. Investigating the representation
of uncertainty in neuronal circuits. PLOS Computational Biology, 17(2):e100813) 2021.
Haoxue Fan, Samuel J Gershman, and Elizabeth A Phelps. Trait somatic anxiety is associated with reduced directed exploration and underestimation of uncertainty. Nature Human Behaviour, 7()): 102-113, 2023.
Steven W Flavell, Navin Pokala, Evan Z Macosko, Dirk R Albrecht, Johannes Larsch, and Cornelia I Bargmann. Serotonin and the neuropeptide pdf initiate and extend opposing behavioral states in elegans. Cell,
2013.
Samuel Gershman. Deconstructing the human algorithms for exploration. Cognition, 173:34-42 2018.
Samuel Gershman. Uncertainty and exploration. Decision, 6(3)277, 2019.
Peter Hänggi. Peter Talkner, and Michal Borkovec. Reaction-rate theory: fifty years after kramers. Reviews of modern physics, 62(2):251, 1990.
Geoffrey E Hinton and Terrence J Sejnowski. Optimal perceptual inference. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, volume 448, pp. 448-453. Citeseer, 1983.
Jeremy Hogeveen, Teagan S Mullins, John D Romero, Elizabeth Eversole, Kimberly Rogge- Obando, Andrew Mayer, and Vincent Costa. The neurocomputational bases of explore- exploit decision-making. Neuron, 110(11):
2022.
John Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences, 2958))255422533 1982.
John J Hopfield. Neurons with graded response have collective computational properties like those of two-state neurons. Proceedings of the national academy of sciences, 81800)3008-3302, 1984.
Bingshan Hu, Tianyue Zhang, Nidhi Hegde, and Mark Schmidt. Optimistic thompson sampling- based algorithms for episodic reinforcement learning. In Uncertainty in Artificial Intelligence, pp. 990-899, PMLR, 2023.
Ni Ji, Gurrein K Madan, Guadalupe I Fabre, Alyssa Dayan, Casey M Baker, Talya S Kramer, Ijeoma Nwabudike, and Steven W Flavell. A neural circuit for flexible control of persistent behavioral states. Elife, 10:e62889, 2021.
Hendrik Anthony Kramers. Brownian motion in field of force and the diffusion model of chemical reactions. Physica, 7(4):284-304, 1940.
Tze Leung Lai and Herbert Robbins. Asymptotically efficient adaptive allocation rules. Advances in applied mathematics, 6(1):4-22 1985.
IS Langer. Theory of nucleation rates. Physical Review Letters, 21(14))973, 1968.
Veldon-Janm Laurie, Akram Shourkeshti, Cathy S Chen, Alexander B Herman, Nicola Grissom, and R Becket Ebitz. Persistent decision-making in mice, monkeys, and humans. bioRxiv, pp- 2024-05, 2024.
Kiyotoshi Matsuoka. Stability conditions for nonlinear continuous neural networks with asymmetric connection weights. Neural nenworks, 5(3):495-500, 1992.
Benedict C May, Nathan Korda, Anthony Lee, David S Leslie, and Nicolo Cesa- Bianchi. Optimistic bayesian sampling contextual bandit problems. Journal of Machine Learning Research, 13(6), 2012.
12

---

Published as a conference paper at ICLR 2025
Jack-Morgan Mizell, Siyu Wang. Alec Frisvold, Lily Alvarado, Alex Surrell-kupnny Waitsang Ke- ung. Caroline E Phelps, Mark H Sundman, Mary-Kathry Franchetti, Ying- hui Chou, et al. Dif- ferential impacts of healthy cognitive aging directed and random exploration. Psychology and Aging, 3981):88, 2024.
lan Osband, Daniel Russo, and Benjamin Van Roy. (more) efficient reinforcement learning via posterior sampling. Advances in Neural Information Processing Systems, 26, 2013.
Brendan O' DDnooguu, Ian Osband, Remi Munos, and Volodymyr Mnih. equation and exploration. International conference on machine learning, pp. 3836-3845, 2018.
The uncertainty bellman
Herbert Robbins. Some aspects of the sequential design of experiments. 1952.
Cristina Savin and Sophie Denève. Spatio-temporal representations of uncertainty in spiking neural networks. Advances in neural information processing systems, 27, 2014.
Eric Schulz and Samuel J Gershman. The algorithmic architecture of exploration in the human brain. Current opinion neurobiology, 55:7-14, 2019.
Eric Schulz, Charley M Wu, Azzurra Ruggeri, and Björn Meder. Searching for rewards like child means less generalization and more directed exploration. Psychological science, 30(11):1561
2019.
David Stephens and John Krebs. Foraging theory, volume 6. Princeton university press, 1986.
Alexander L Strehl and Michael Littman. An analysis of model-based interval estimation for markov decision processes. Journal of Computer and System Sciences, 74(8):1309-1333 2008.
Malcolm Strens. A bayesian framework for reinforcement learning. ICMI, volume 2000, pp-
943-950, 2000.
Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. Robotica, 17(2): 229-235 1999.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
William R Thompson. On the likelihood that one unknown probability exceeds another view of the evidence of two samples. Biometrika, 25(3-4):288) 4):285-294, 1933.
Daniil Tiapkin, Denis Belomestny, Daniele Calandriello, Eric Moulines, Remi Munos, Alexey Nau- mov, Mark Rowland, Michal Valko, and Pierre Ménard. Optimistic posterior sampling for inforcement learning with few samples and tight guarantees. Advances Neural Information Processing Systems, 35:10737-10751, 2022.
Momchil S Tomov, Van Q Truong, Rohan A Hundia, and Samuel Gershman. Dissociable neural correlates of uncertainty underlie different exploration strategies. Nature communications, 11(1): 2371, 2020.
James A Waltz, Robert C Wilson, Matthew A Albrecht, Michael Frank, and James M Gold. Differ- ential effects of psychotic illness on directed and random exploration. Computational Psychiatry (Cambridge, Mass.), 4:18, 2020.
Siyu Wang, Blake Gerken, Julia R Wieland, Robert C Wilson, and Jean-Marc Fellous. The effects of time horizon and guided choices on celloreexpplott decisions in rodents. Behavioral neuro- science, 137(2):127, 2023.
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference machine learning (ICML-I1), pp. 681-688. Citeseer, 2011.
Robert C Wilson, Andra Geana, John M White, Elliot A Ludvig, and Jonathan D Cohen. Humans directed and random exploration to solve the explore-exploit dilemma. Journal of experimental psychology: General, 143(4),0074, 2014.
13

---

Published as a conference paper at ICLR 2025
Robert C Wilson, Elizabeth Bonawitz, Vincent D Costa, and R Becket Ebitz. Balancing exploration and exploitation with information and randomization. Current opinion behavioral sciences, 38: 49-56, 2021.
Ning Yang, Chao Tang, and Yuhai Tu. Stochastic gradient descent introduces effective landscape- dependent regularization favoring flat solutions. Physical Review Letters, 130(23):237001, 2023.
Wojciech K Zajkowski, Malgorzata Kossut, and Robert C Wilson. A causal role for right frontopolar cortex in directed, but not random, exploration. Elife, 6re27430, 2017.
Weitong Zhang, Dongruo Zhou, Lihong Li, and Quanquan Gu. Neural thompson sampling, 2021. URL Utt.hhtpp.//axx....o////////00.0 00827
Dongruo Zhou, Lihong Li, and Quanquan Gu. Neural contextual bandits with ucb-based exploration, 2020. URL Utt.tttt///aax...c.//a////.1....
Zhanxing Zhu, Jingfeng Wu, Bing Yu, Lei Wu, and Jinwen Ma. The anisotropic noise in stochastic gradient descent: Its behavior of escaping from sharp minima and regularization effects. arXiv preprint arXiv:1803.00/99 2018.
14

---

Published as a conference paper at ICLR 2025
APPENDIX
A SUPPLEMENTAL FIGURES
Biological neural model
b. 3-dimensional BBN model
Action
SN
SN
SN
Sensory Neurons
IN
Interneurons
Motor command neurons
5-HT
PDF
X₁
X
Dwell (Exploit)
Roam (Explore)
Action (myopically optimal)
Action
Figure 8: (a) biological neural network in C. elegans that controls the exploration state (roaming) and exploitation state (dwelling). (b) Architecture of the 3-D BBN model.
Phase plane
Hopfield energy
States distribution
20
0.4
X1
9.5
Optimistic
15
15
-10.0
0.3
* 10
110
-00.5
0.2
-11.0
-11.5
10
20
10
20
network state
X1
20
10
20
15
X2
Neutral
Density
0.2
10
0.1
-10.5
0
10
20
10
20
-11.0
10
20
X1
X1
network state
20
10
20
X1
Conservative
15
8
15
-85
X2
9.00
0.3
10
10
-9.5
0.2
-10.0
0
10
15
20
0
5
10
15
20
10.5
20
network state
X1
X1
Figure 9: Stability analysis on the three types of BBN models that generate optimistic, neutral or conservative bias to uncertainty.
15

---

Published as a conference paper at ICLR 2025
Slope shift
Intercept shift
0-35.0,25
1.0
choice probability
0.8
0-25.6-15
choice probability
0.8
Optimistic
0.6
0.6
0.4
0.4
4-20,--00
0.2
0.0
0.0
--12
-1.5
-1.0
-0.5
0.0
0.5
1.0
1.5
-1.5
11.0
-0.5
0.0
0.5
1.0
1.5
value difference
difference
1.0
6,-35,,-22
1.0
choice probability
0.8
choice probability
8-13---1a
Neutral
6f1228of=12
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
115
11.0
0.5
0.0
0.5
1.0
1.5
-1.5
110 -1.0-0.5 -0,5
0.0
0.5
1.0
1.5
value difference
value difference
1.0
0=25,0=25
1.0
@-22.6j=83
Conservative
choice probability
0.8
2-25/0)-115
0.-3 2:=2
choice probability
0.8
R-15.6-=1
8-115500-00
6/-128
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
-1.5
-1.0
-0.5
0.0
0.5
1.0
1.5
-1.5
-1.0
005
0.0
0.5
1.0
1.5
value difference
value difference
5.0
neutral
4.0
conservative
2
3.5
intercept
3.0
2.5
2.0
2
1.5
-3
1.0
2.5
3.5 total uncertainty
3.0
4.0
0.0
0.2
0.4
0.6
0.8 1.0
relative uncertainty
Figure 10: Slope and intercept shift. (Left column) The slope decreases the total uncer- tainty increases while relative uncertainty is kept unchanged. (Right column) The intercept in- creases (optimistie), stays unchanged (neutral), or decreases (conservative) relative uncertainty increases and total uncertainty kept unchanged.
16

---

Published as a conference paper at ICLR 2025
35
08
08
30
06
0.
Theory
7.
2.
04
15
0.3
42
10
02
90
200
31.0
0.0
7.0
5.0
11.0
130
15.0
2.0
35
955
11.0
0.0
07
50
0.0
30
Simulation
25
05
0.6
15
06
04
0.4
0.4
60
0.4
03
1.0
02
05
01
70
80
90
70
90
13.0
110
150
20
35
Figure 11:
Attractor state probability as a function of network parameters Warm colors indi-
cate a higher chance of finding the network in the state that receives greater input uncertainty.
17

---

Published as a conference paper at ICLR 2025
Conservative (b 5)
Neutral (b = 5.5)
Optimistic
(b 6. 65)
State dynamics
10
5 X3
X
X3
5
10
10
X
10
10
X2
10
X
X1
15
X
X1
State probability (theory)
0.8
06-
04-
0.4
0.4
02
1.0
State probability (simulation)
Hit00)
Mitt00
Modunikl
Maxk01
Medbownt
Maris!
0.8
Hedianont
66
04-
0.2
0.0
Network dimension (nt)
Figure 12: Uncertainty bias multi-dimmensionll BBN models. (Top row) State dynamics of 3-D BBN model with conservative, neutral and optimistic uncertainty biases. The concentration of state dynamics reveal the three attractor states, which are visited with different relative proportion in the three types of BBNs. Input noise is strongest along the Z-direction is the largest and lowest along the X-direction. (Middle row) Probability of attractor states with the highest (orange), median (green), and lowest (blue) input uncertainty the network scales from 2D to 10-D under the servative, neutral, or optimistic parameter regimes, computed from numerical simulations. For the same type of network, internal model parameters kept the same as the dimensionality increases. The dotted curve indicates perfectly equal partition of probability among all N states. (Bottom row) Theoretically predicted state probability for the same network models presented above in the middle row, presented in the same format.
18

---

Published as a conference paper at ICLR 2025
2 - oz =0.7
o =2, oz =0.2.
o0,= =0
10
X3 5
X3 5
x
0
0
2
5
10 X
X2
10
Figure 13: State entry dynamics near the saddle point for 3D BBN. Red points show simulated state dynamics when the network is initialized from the saddle point. Green circle denotes the saddle point, green erosses denote the attractor centers, and the projected 2D histogram reveal the relative occupancy of the three attractors (pink indicates high state probability).
2D model
3D
4D
PAI (theoretical)
Density
PAi (numerical)
Figure 14: Theoretical vs. simulated attractor state probability in multi-dimennsinal BBNs.
B METHOD DETAILS
B. PARAMETER SELECTION
Parameter
Definition
Suggested range
w
inhibitory weights activity baseline threshold of sigmoid slope of sigmoid leak current or decay rate time constant
[2, 4J, increase to make states more stable
[5.5, 71, increase makes network optimistic
[6, 8], increase makes network conservative
[1, 2), increase amplifies uncertainty bias
0.5
Table 1: Internal Model Parameters
Parameter
Definition
Suggested range
mean input variance of input
scale raw input values to [-2, 2] scale input variance to [0. 2]
2
Table 2: External Parameters
19

---

Published as a conference paper at ICLR 2025
Parameter
Definition
Suggested range
N
number of neurons
typically equal number of actions choices
T
total simulation steps step length
[400, 1000]
dt
0.1 0.2 if using suggested parameter ranges
Table 3: tab: Hyperparameters
In this section, we list the primary parameters used in BBN (Tables 2 and provide a principled way determine optimal parameters for environments.
Based on our experience and past literature (May et al. 2012 Hu et al. 2023 Agrawal & Jia 2017) optimistic bias generally promotes efficient exploration. In addition, sensitivity analysis MDP tasks (Fig. 18) showed that a broad range of "optimistic" parameters yielded high performance, obviating the need for extensive fine-tuning. Further, we have shown that network parameters that yield optimistic bias for 2D BBN preserve such bias in higher dimensions (Fig. 3(c) and Fig. 12 Thus, the steps to set up a --dimensional BBN model are:
(1) Define a BBN model with N interconnected neurons; (2) Select internal network parameters from the "optimistic" regime based sensitivity analysis results presented in (Fig. a(abb) and Fig. I.I or use the parameter ranges suggested below as starting point;
(3) Verify that the 2D network has two attractors and exhibits optimistic bias by numerically sim- ulating the model under anisotropic 2D Gaussian noise with μr [0, 0). 0.1)); tune the parameters if necessary using the tips provided below;
(4) Apply these parameters all neurons in the ND network;
(5) Scale the input to the network (typically past rewards or Q-values) to a range that permits the existence of multiple attractors (use suggested range or verify empirically).
We found that simulation step number of T=400 is sufficient for bandit and MDP tasks t. Below are sample network dynamics in the first episode of 2-armed bandit game. Multiple transitions occurred between the attractor states, reflecting equal state probability as expected for equal uncer- tainty for the two arms.
Neural activities
20
X1
X2
M My of poatt
par m the with
15
network state
10
5
50
100
150
200 time
250
300
350
400
Figure 15: State dynamics BBN in two-armed bandit game
20

---

Published as a conference paper at ICLR 2025
B.2 RUNNING BBN IN BANDIT GAMES
To make the BBN model play bandit games, we
(1) define BBN model with N neurons, each corresponding to one of the N bandit arms;
(2) pick network parameters that yield "optimistic" exploration for 2-D BBN, and simply apply the parameters to all neurons in the N-D model;
(3) each trial, sample input from the reward memory buffer and numerical simulation of the network for 7 steps using the Runge-Kutts Kutta method;
(4) at the end of the simulation, select the arm whose corresponding neuron has the highest acti- vation value;
(5) collect the reward la and add memory buffer for arm a;
(6) repeat (3)-(5) for the next trial till game ends.
B.3 RUNNING BBN MDP TASKS
Here we consider the tabular case MDP, so the states and Q-values are parameterized as entries in a lookup table, where each state-action pair maps Q-value. To implement BBN in action selection, the agent needs to estimate the uncertainty of Q-values for BBN's imput. However, how to estimate uncertainty of the cumulative rewards in MDP tasks remains open issue in the RL community because the choice of an action affects both the currett immediate reward and subsequent state transfer. O D'Donoghue et al. (2018) gave an upper bound on the variance of posterior distribution of the Q-values by proposing the Uncertainty Bellman Equation (UBE), which connects the uncertainty any time-step the expected uncertainties subsequent time-steps. We leverage the upper bound on the variance by UBE to obtain uncertainty estimation for Q-values.
Here we present detailed steps apply BBN to drive action selection in MDP tasks: (1) define BBN model with N neurons, each corresponding to one of the N discrete actions, select network parameters that belong to the "optimistic" regime for a 2D network;
(2) initialize state-action values to i.i.d. Gaussian distributions;
(3) sample input values for each neuron from the distributions of state-action values and perform numerical simulation of the BBN network for T steps using the Runge-Kutta method;
(4) at the end of the simulation, select action a whose corresponding neuron has the highest activa- tion value;
(5) collect the reward ra and move to the next state
(6) Repeat (3)-(5) till the episode ends;
(7) Update the distribution of state-action values using the uncertainty bellman equation (UBE) CooorihmmO"DDnoghhn et al. 2018)).
(8) repeat (3)-(7) for next episode till game ends. We present the pseudo-code for the Algorithm
in Appendix B.4 The pseudocode along with detailed task parameters are presented below.
B.4 PSEUDOCODES
Algorithm [ presents the pseudocode BBN in Multi-armed Bandit Games and Algorithm 2 presents the pseudocode of UBE-BBN in playing MDP tasks.
21

---

Published as a conference paper at ICLR 2025
Algorithm 1: BBN for multi-armed bandit games
Input
The horizon of the multi-armed bandit game H; The number of arms A
The total simulation steps for BBN model T;
Output:
The selected arm at each trial h;
Initialize the model parameter for BBN model; Initialize the value for each neuron zy; for h 1, .... H do for 1 = 1, 2, ..., do sample I, from reward history for each arm
da MJ; + Σ ((r)) + b, - 4;; 37
34 Ti + dx;; end
select an arm a + argmax(I;); receive a reward ra ~ N M(aa oa); add ra to reward history of arm a: end
22

---

end end Published as a conference paper at ICLR 2025
Algorithm 2: UBE-BBN for MDP tasks
Input
The horizon of the MDP task H:
The maximum episode т
The number of total states S
The number of actions A The total simulation steps for BBN model T;
Output:
The selected action at each timestep t;
Initialize the model parameter for BBN model; Initialize the value for each neuron 24; for iter 1, 2, ..., T do for h = 2,
do current state for f 1, 2, T do
sample I, from Qsi ~ NQQ varQ
dx: Mis Σ wiy fx;z + b, I;; 1F1
Ti dt
Tt t vi + dx;;
end
select an action receive a reward reo N (Hsa, sa); move to next state s update Psast;
argmaz(z;);
end
update Q values using dynamic programming: for h = H, H-l, do for S do
for a E A do
Aag Ersai
Q sa t Pssa + Eea at's P Poas Qutt employ the Uncertainty Bellman Equation (UBE):
var variisa Eve ast Peas var
end end
B.5 BANDIT AND MDP TASK PARAMETERS
Bandit parameters for performance comparison We chose use Gaussian bandits where re- ward values are sampled from A ((4, (14) For 2-armed bandit games, the reward mean μ for both arms are sampled from Gaussian distribution /(, 12) at the beginning of each block. The reward variance is and 4 respectively. For 3-armed bandit games, the reward mean for all arms are sampled from Gaussian distribution (00, 12) at the beginning each block. The reward variance are 1,225.2 respectively. Note that while we chose to use Gaussian bandits here, the model can be extended to non-Gaussian input distributions and performs well empirically non-Gaussian (e.g. Bernoulli) bandit tasks.
Bandit parameters for fitting to mice data We follow the bandit parameters in (Beron et al. 2022. The mean rewards the Bernoulli bandits are 0.8 and 0.2 respectively.
SixArms SixArms & Littman 2008) consists of seven states and six actions. The agent starts in state 0. We consider episodic case, the state is reset every 20 steps. A transition of
23

---

Published as a conference paper at ICLR 2025
the form (a. r)r,, where is action, is the transition probability, and is the reward for taking the transition.
<3, 1, 800;
4
(0-2, 0)
0-3, 50)
(4-5, 1, 0
0, 1,
10
<3. 0.05, 0
(4, 0)
5, 1, 03
0-3 1. 0
(1. 0.15 0
(4, 1, 1660
5
0
2
(1, 1, 133)
4, 0.03, 0}
0. 1 0
22-5, 1,
(0-4, 1, 0)
(, 0.01, 0)
(2, 0.1, 0)
(5, 1. 6000
6
33 3-5, 1.0
0-1 1 1,
3
(2 1, 300)
Figure 16: SixArms.
For more detailed parameters for each algorithm used in our experiments, please refer to our code: http:///sshabbbooommmooooooououu
C DATASETS FOR MODEL FITTING
Gershman19 is from (Gershman 2019). In their experiment, participants were given a choice be- tween two arms, labeled either "safe" (S) or "risky" (R). The safe arms always return deterministic rewards, while the risky arms sample rewards from a Gaussian distribution. There are four types of bandit settings: RS, SR, RR, and SS, which are denoted by compound labels (c.g.. "SR" denotes trials in which the left arm is safe and the right arm is risky). The reward mean for both risky arms and safe arms are sampled from a Gaussian distribution A(0 102) at the beginning of each block. The reward variance for risky arms is 16, and for safe arm is O. By comparing the slope and intercept of the choice probability curve for each type, we can quantify the degree of randomness and preference for uncertainty.
Fan23 Fan et al. 2023) further explored the relationship between trait somatic anxiety and differ- ent exploration strategies in decision-making They used the same experimental design Gersh- man ( Gershman 2019) and evaluated the anxicty for each individual. In Fig (a-b), the slope and intercept of human data in (Gershman, 2019) are drawn directly from the paper. And for humans with high or low anxiety, we split the 40% of the population with the highest "somatic anxiety" score and the 40% with the lowest "somatic anxiety" score in the collected data from (Fan al. 2023), and then performed probit regression respectively.
Mizell24 from (Mizell al. 2024) involved younger adults (ages 18-25) and older adults (ages 65-74) making decisions between two virtual slot machines to measure exploration behaviors called Horizon Task. The rewards are sampled from Gaussian distribution. Participants first completed instructed trials, sampling the slot machines under two conditions: unequal information (one drawn from one machine and three from the other) and equal information (two drawn from each machine). They then made free choices in either short horizon (one choice) or a long horizon (six choices) condition. The task assessed directed exploration (choosing the more informative option) and ran- dom exploration (choosing the lower reward option). We use unequal information condition of the collected data to fit our model.
Zajkowsk17 1 is from (Zajkowski et al. 2017). Participants also performed Horizon Task, where they made explore-exploit decisions between two virtual slot machines under two conditions: un- equal information and equal information. The task involved 160 games, each consisting of 5 or 10 choices, with the key manipulation being the horizon length: short (5 choices) or long (10 choices). Continuous theta-burst transcranial magnetic stimulation (TMS) was used to selectively inhibit the right frontopolar cortex (RFPC) when participants performed the Horizon Task. We use unequal information condition of the collected data fit our model.
24

---

Published as a conference paper at ICLR 2025
D FURTHER RESULTS ON BANDIT TASKS
D.1 LIMITED MEMORY BUFFER SIZE
BBN doesn't need all the past experience in the memory buffer. For example, in the experiment of fittigg to mice behavior, we only used the last reward histories since the reward for each bandit will change over time. We also performed additional experiments to if the limited memory buffer would hurt the performance in bandit tasks. We limited the buffer size to 8 for each arm. Fig 17 shows BBN with limited memory buffer size still consistently outperforms other methods.
(a) 2-armed bandit
(b) 3-armed bandit
0.60
Ploptimal)
0.625
Ploptimal)
0.35
0.600
0.50
0.575
E-greedy
C-greedy
0.45
UCB
0.550
OTS
0.40
OTS
BBN(Ours)
BBN(Ours)
0.500
BBN immited size)
0.35
BBN(Limited size)
0.475
10
20
10
15
20
25
30
trials
trials
Figure 17: BBN with limited memory buffer size achieve similar efficient exploration in bandit tasks
E
FURTHER RESULTS ON MDP TASKS
E.
PARAMETER SENSITIVITY ANALYSIS
(a) SixArms
(b) FourRooms
7.0
3.4
0.83
7.0
0.82
6.5
3.2
6.5
0.81
6.0
3.0
6.0
0.80
Goverage
5.
Regre
5.5
0079
2.8
5.0
5.0
0.78
-0.77
4.5
4.5
0.77
4.0
4.0
0.75
6.0
6.5
7.0 k
7.5
8.0
6.0
8.5
7.0
7.5
8.0
k
Figure 18: Parameter sensitivity analysis of UBE-BBN with different parameter combinations evaluated in two MDP tasks. Performance in the SixArms task was evaluated by the cumulated regret the agent, while performance the FourRooms grid world task was evaluated by the coverage rate. broad range of "optimistic network parameters" generally yielded high performance on these tasks.
25

---

Published as a conference paper at ICLR 2025
E.2
PERFORMANCE VARIATIONS OF GRID WORLD TASKS
As shown in Fig 19 UBE-BBN yields fastest coverage rate among all the methods on different environments. Fig 20 gives examples of cumulative visitation counts for more algorithms during training. Only UBE-BBN covers all states with less than 450 episodes.
(a) Grid World size 15*15
(b) Grid World size 19*19
(c) Grid World size 23*
1.0
0.8
0.8
Coverreee Rale
Cocongg: A40
0.
0.
3.0
Eploades
150
252
Eptoodes
200
230
300
150
Eptidda
150
250
500
Figure 19: Learning curves on different sizes of FourRooms environments.
Fig 24 shows the trajectories of agents, which the visitation counts in single episode. As shown, e-greedy, UCRL2, UBE-TS only perform exploration around the starting state, failing to do "deep" exploration. PSRL, OTS-MDP and UBE-UCB
perform "deep" exploration, but they all act deterministically, so they will be stuck at a certain state. UBE-BBN is also driven by uncertainty like UBE-UCB to perform "deep" exploration, but with stochastic sampling of action choices, it will not be stuck at certain state.
Action persistence further boosts BBN performance. BBN with persistence refers taking neu- ron values at the end of last step the starting point for the next step, while BBN without per- sistence refers to initializing neuron values each step. We compare the different behavior of the BBN model with and without persistence across four different grid sizes: 15x15, 19x19, 23x23, and 103x103. The results presented here show the trajectories during the first episode of exploration, and the exploration length corresponds to the number of states in the grid world. As shown in Fig. 25 for the same exploration length, the BBN model with persistence explores larger portion of the grid world.
26

---

Published as a conference paper at ICLR 2025
a)
Four rooms (reward-free, 23x23)
b]
Four rooms (random reward, 23x23)
Hellsodes to reward
1000
Faadio failure to
E-greedy
800
02
reach reward
0.4
400
0.8
UCLR2
1.0
room2
Reward location
UCRL2
OTS-MDP
USC_UCD
>00
URF FARRRROOO
USE stenper
PSRL
15
Visitation count
10
UBE-TS
UBE-UCB
UBE-BBN
(Ours)
50
150
450
episodes
Comparison of exploration efficiency across different exploration algorithms in Four Rooms) (a) Visitation counts in reward free setting; (b) Number episodes until first en- counter of the reward state.
Figure 20:
27

---

Published as a conference paper at ICLR 2025
Room Size: 51x51
UBE_T TS
Visitation Conttioofeequunnn frequency
UCRL2
UBE-BBN (Ours)
245-255
355-365 Episode #
385-395
Figure 21: Trajectories (visitation counts in single episode) of UBE-TS, UCRL2, and UBE- BBN in expanded Four Rooms task with reward
28

---

Published as a conference paper at ICLR 2025
Four rooms with reward penalty (23x23)
E-greedy
start
penalty
UCLR2
eward
OTS-MDP
>20
15
Visitation count
UBE-TS
10
UBE-UCB
UBE-BEN (Ours)
150 episodes
450
Figure 22:
Comparison of visitation counts across algorithms in Four Rooms game with
reward and penalty).
29

---

Published as a conference paper at ICLR 2025
Nine rooms (reward-free, 23x23)
e-greedy
UCLR2
OTS-MDP
>20
PSRL
15
Visitation count
10
UBE-TS
5
UBE-UCB
UBE-BBN (Ours)
50
150
450
episodes
Figure 23: Comparison of visitation counts across algorithms in a Nine Rooms game.
30

---

Published as a conference paper at ICLR 2025
E-greedy
>=10
UCRL2
OTS-MDP
6
PSRL
3
UBE-TS
UBE-UCB
UBE-BBN(Ours)
100
200
300
400
episodes
Figure 24: Agent trajectories (visualized through visitation counts) in single episodes over the course of training.
31

---

Published as a conference paper at ICLR 2025
No Persistence
With Persistence
15*15
19*19
23*23
103*103
Grid world size
Figure 25: Trajectories (visitation counts in a single episode) of BBN with/without persistence.
32


Reviews:
[{"review_id": "ZGFfSTj6vG", "reviewer": "Reviewer_siYM", "summary": "This paper presents a simple recurrent neural network model of exploratory action selection.", "questions": "The pseudo code for the MDP tasks includes P_{sas'}. Does this mean that it is assumed that the agent has access to the true dynamics of the environment?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper extends the concept of Boltzmann machine to continuous valued recurrent neural networks. I could not follow all the details of the mathematics, but the analysis seems to be new and helpful.\nThe simulation results are compared with human behavioral data.", "weaknesses": "The implementation of bandit assumes the availability of the memory of the all past experience in the memory buffer, which may be impractical as a cognitive model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7TcqEQaWQ8", "reviewer": "Reviewer_aJhq", "summary": "# Summary\nThe authors present a biologically-inspired neural network model called Brain Bandit Network (BBN) that addresses the explore-exploit problem in sequential decision making tasks. \nThe model itself is a stochastic continuous Hopfield network, which they show can implement a form of Bayesian posterior sampling and a tunable input uncertainty bias. \nThey demonstrate the BBN's effectiveness in MAB tasks, showing that it approximates human and animal choice patterns, and on some MDP exploration benchmarks.", "questions": "# Suggestions for improvement:\n* L169 -- L174: What is MFPT? The lead up to these equations needs better development, esp. for someone not already familiar with this literature\n* L202/208: Define escape efficiency\n* L322/323: Define relative and total uncertainty\n* L438 and elsewhere: Explain the rationale behind the choice of T steps for BBN simulation.\n* Citation format seems wrong", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "# Strengths\n* Novelty: As far as I can tell, this is a novel combination of Stochastic Hopfield Networks applied to decision making / explore-exploit tasks. The connections with neuroscience are interesting.\n* Well written/Thorough: Paper is quite clear and well written, and as such has high pedagogical value. The authors present extensive experiments comparing BBN against baseline methods on multiple tasks, providing empirical support for their claims. (Minor concerns addressed below)", "weaknesses": "# Weaknesses\n* Niche/limited applicability: The method itself is quite niche, needs an SDE solver, and has several hyperparameters that require niche/expert knowledge to tune. \n* Tuning the numerous hyperparameters requires specialized knowledge, making it difficult for non-experts to utilize effectively. Are there principled ways to determine optimal BBN parameters for new environments? \n* The performance of this model in high-dimensions needs more exposition. Section 3.4 provides a start, but more extensive experiments and theoretical analysis are needed to understand the behavior and performance of BBN in high-dimensional settings. (ideally supported by some theoretical footing).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WGYrINM7Kq", "reviewer": "Reviewer_8itA", "summary": "In this work, the Authors propose a framework addressing the exploration-exploitation tradeoff using a stochastic continuous Hopfield network. They provide a derivation to show that the proposed framework implements a form of Bayesian inference while also displaying an uncertainty-related bias. The comparison of the proposed model with human and mouse data on multi-armed bandit task shows the similarity between the two; additional considerations and experiments are provided for two MDP tasks.", "questions": "How were the states and Q-values parameterized in the task? Please provide a Methods-style description of the model. This may be particularly important in the light of the next question.\n\nOne thing that I couldn’t understand is: What makes the model explore more than the baselines? In particular, how would the exploration in this task compare to the random walk? How would that result change based on the parameterization of the stare and the Q-function (e.g., tabular vs. deep learning)?\n\nFinally, I didn’t quite understand why is the higher state coverage considered a positive thing in this task. Specifically, as the task is not rewarded (Figure 8 b-c), the exploration seems unnecessary. When rewards are present, their utility may be different depending on the reward and the task.\n\nOverall, I think this is an interesting paper relevant to the ICLR community. I believe that it offers sufficient novelty and significance through the remarkable match between the model’s predictions and human behavior. Text-related issues can be addressed during the remaining time.\n\n________________\nPost-rebuttal: questions addressed by the Authors; raising the score to 8.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The work deals with an interesting and important problem of the exploration-exploitation tradeoff, particularly focusing on the aspect of human decision-making. As human decision-making, shaped by evolution, is expected to be optimal, at least in certain ways, understanding its mechanics may be useful for the design of future ML algorithms. This work combines the study of exploration-exploitation tradeoff with recurrent neural networks and bandit tasks, which are all relevant and interesting to the ICLR community.\n\nThe quality of the research in this work is high: it starts with the thorough derivations of the analytic predictions for the system’s behavior and then continues with numerical experiments where the model’s predictions are first validated and then comparisons are performed with human and animal data. The text is mostly well-written, following the standard structure of machine-learning papers and introducing concepts in a logical sequence, making it easy to follow.\n\nThe result regarding the similarity between the model’s predictions and human/mouse data on exploration/exploitation in multi-armed bandit tasks is novel and interesting. It is particularly strong as it spans multiple datasets from multiple works.", "weaknesses": "While the derivations for the analytic results are thorough and lengthy, most of them appear to be either existing or intuitive. The attractor structure of the Hopfield network with two neurons and inhibitory connections is known. Hopfield networks have been shown to be able to implement Bayesian inference, although the derivations have been provided for the discrete case. The fact that the probability of the network’s activity to stay in an attractor depends on the relative orientation of its Hessian matrix and the noise direction is clear and intuitive. Of course, none of that diminishes the novelty of the result regarding the similarity between the model’s behavior and human decision-making.\n\nThe MDP part may need a fuller description (ideally a Methods section; e.g. in the Appendix) to clarify what gives rise to the observed phenomena. The uncertainty Bellman equation is also worth mentioning in the text for the clarity. Please see the specific questions regarding the Gridworld task below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)