Modeling Human Exploration Through Resource-Rational Reinforcement Learning
Authors: Marcel Binz
Eric Schulz
Venue: NeurIPS 2022 Accept (2022)
Keywords: Exploration
Meta-Learning
Cognitive Science
Resource-Rationality

Abstract: Equipping artificial agents with useful exploration mechanisms remains a challenge to this day. Humans, on the other hand, seem to manage the trade-off between exploration and exploitation effortlessly. In the present article, we put forward the hypothesis that they accomplish this by making optimal use of limited computational resources. We study this hypothesis by meta-learning reinforcement learning algorithms that sacrifice performance for a shorter description length (defined as the number of bits required to implement the given algorithm). The emerging class of models captures human exploration behavior better than previously considered approaches, such as Boltzmann exploration, upper confidence bound algorithms, and Thompson sampling. We additionally demonstrate that changing the description length in our class of models produces the intended effects: reducing description length captures the behavior of brain-lesioned patients while increasing it mirrors cognitive development during adolescence.

Modeling Human Exploration Through Resource-Rational Reinforcement Learning
Marcel Binz
Eric Schulz
MPI for Biological Cybernetics
MPI for Biological Cybernetics
Tübingen, Germany marcel. binz@tue mpg. de
Tübingen, Germany eric. www.ssctuuluu..e....... www.:/ettueeee.p..emmmmm de
Abstract
Equipping artificial agents with useful exploration mechanisms remains a challenge this day. Humans, the other hand, seem manage the trade-off between exploration and exploitation effortlessly. In the present article, we put forward the hypothesis that they accomplish this by making optimal use of limited computa- tional resources. We study this hypothesis by meta-learning reinforcement learning algorithms that sacrifice performance for shorter description length (defined as the number of bits required to implement the given algorithm). The emerging class of models captures human exploration behavior better than previously considered ap- proaches, such as Boltzmann exploration, upper confidence bound algorithms, and Thompson sampling. We additionally demonstrate that changing the description length in our class of models produces the intended effects: reducing description length captures the behavior of brain-lesioned patients while increasing mirrors cognitive development during adolescence.
Introduction
Knowing how to efficiently balance between exploring unfamiliar parts of an environment and exploiting currently available knowledge is an essential ingredient of any intelligent organism. In theory, it is possible obtain Bayes-optimal solution to this exploration-exploiiee exploitation dilemma by solving an augmented problem known as a Bayes-adapting Markov decision process (BAMDP, Duff, 2003). However, BAMDPs intractable to solve in general and analytical solutions only available for a few special cases [Gittins 1979]. The intractability of this optimal solution prompted researchers develop body of heuristic strategies [Auer et al. 2002 Kaufmann et al. 2012 Russo ett al. 2017 Russo and Van Roy, 2014). Most of these heuristics can be divided into two broad categories: directed and random exploration strategies [Wilson al. 2014 Schulz and Gershman 2019]. Directed exploration strategies provide bonus rewards that encourage the agent to visit parts of the environment that ought be explored, whereas random exploration strategies inject some form of stochasticity into the policy.
Having access vast amount of cxisting exploration strategies leads to the question: which of them should we use when building artificial agents? To answer this question, we may take inspiration from how people approach the exploration-exploitttii extensively in the multi-armed bandit setting [Mehlhorn et al., (215)] Wilson et al.) 2022 Brändle et al. 2021]. Prior work indicates that people substantially deviate from the Bayes-optimal strategy even for the simplest bandit problems [Steyvers et al.) 2009 Zhang and Angela 2013] Binz and Endres 2019). They, however, use uncertainty estimates to intelligently guide their choices through combination of both directed and random exploration [Wilson et al. 2014 Gershman 2018 The question of when and why individuals rely particular exploration strategy has been under-explored far.
dilemma. Human exploration has been studied
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

We take the first steps towards answering these questions by looking at human exploration from resource-rational perspective [Gershman et 2015 Lieder and Griffthhs 2020 Binz al. 2022.. More specifically, we investigate the hypothesis that people solve the exploration-explotaatio dilemma by making optimal use of limited computational resources. To test this hypothesis, we devise family of resource-rational reinforcement learning algorithms by combining ideas from meta-learning [engio et al.) 1991 Schmidhuber et al. [996] and information theory [Hinton and Van Camp [993]. The resulting model which we call resource-rational RL (RR-RL*) implements a free-standing reinforcement learning algorithm that achieves optimal behavior subject to the constraint that it can be implemented with given number of bits.
We demonstrate that RR-RL² captures many aspects of human exploration by reanalyzing data from three previously conducted psychological studies. First, we show that it explains human choices in a two-armed bandit task better than traditional approaches, such as Thompson sampling [Thompson [933]. upper confidence bound (UCB) algorithms (Kaufmann et al. (012), and mixtures thereof [Gesshman 2018). We then verify that the manipulation of computational resources in our class of models matches the manipulation of resources in human subjects in two different contexts. Taken together, these results enrich our understanding of human exploration and provide insights into how to improve the exploratory capabilities of artificial agents.
2 Methods
We start by describing the general problem setting considered this article and its optimal solution, followed by brief summary of the meta-reinforcemmet learning framework. We then show how to augment the standard meta-reiiforcemment learning objective with an information- allowing to construct reinforcement learning algorithms that trade-off performance against the number of bits required to implement them.
constraint,
2.1 Notation and Preliminaries
Each task considered in this article can be interpreted as a multi-armed bandit problem. In a k-armed bandit problem, an agent repeatedly interacts with slot machines that are associated with a reward distribution p(ralau (4,, w) with unknown parameters w. In each time-step, the agent selects an action ar and is provided reward r, based on the associated reward distribution.
The goal of an agent is to select actions such that the sum of rewards over a finite horizon H is maximized. We assume that the agent additionally has access prior distribution p(w) over bandit problems that it may encounter, which can be updated after observing a history observations h4 ; (01, Pl.-.. at-1, 11- by applying Bayes' rule:
t-1
p(w(h)) a p(w) II
plmmlams w)
m=1
The policy that optimally trades-off exploration and exploitation can be obtained by reasoning how the agent's beliefs about reward functions evolve with future observations [Martin] 1967). Formally, this is accomplished by transforming the original bandit problem into corresponding BAMDP defined by the tuple (H. A, H, R). this augmented problem, H. represents the set of all possible histories. while A and H correspond to the action space and the horizon of the original bandit problem. The transition probabilities T and reward function are given by:
T(h lar, he) p(re lat, h2) [he+1 = (he, az, rz)]
(2)
R(ar, h4) Em(relauth)) [re]
(3)
with the marginal reward probabilities:
p(relar,he) ht) = f p/lb:///lllllllll w)p(w)h))ds
(4)
The policy that maximizes the sum rewards in the BAMDP corresponds to the Bayes-optimal policy for the original bandit problem.

---

2.2 Meta-Reinforvement Learning
While the BAMDP formalism provides a precise recipe for deriving a Bayes-optimal policy, finding an analytical expression of this policy is typically not possible. Recent work on meta-reinforremeent learning, however, has shown that it possible learn an approximation to it [Wang al. 2016 Ortega et al. 2019 Zintgraf et al. 2019|. Duan et al. [2016] refer to this approach as RT. because it uses traditional reinforcement learning algorithm to meta-learn another free-standing reinforeement
learning al algorithm.
RL parametrizes the to-be-learned reinforcement learning algorithm with general- purposs function approximator. Typically, this function approximator takes the form of a recurrent neural network that receives the last action and reward as inputs, uses them update its hidden state, and produces policy that is conditioned on the new hidden state. Let W denote the parameters of this recurrent neural network. In an outer-loop meta-learning process, the network is then trained on the prior distribution over bandit problems p(w) to find the cestory-dependere policy x Taa W) that maximizes the sum of obtained rewards. If the meta-learning procedure has successfully converged to optimum, RI. implements a free-standing reinforcement learning algorithm that mimics the Bayes-optimal policy, Importantly, learning at this stage fully implemented through the forward dynamics of the recurrent neural network and no further updating of its parameters is required.
2.3 Risource-Rationa RI.2
We transform RL22 into resource-rational algorithm by augmenting its meta-learning objective with an information-theorets constraint and refer to this resource-rational variant as RR-RI. More precisely, 2 is obtained by solving the following optimization problem:
H Σ
max Be A)p(c) I ())))) be, W) A
r
s.t. KL (g( ( (p
(5)
The first component of Equation 5 corresponds the standard meta-reinforcement learning objective, while the second component ensures that the Kullback-Leibler (KL) divergence between stochastic parameter encoding q( (W)A) and prior p( (W) remains smaller than some constant C. The KL term can be interpreted as the description length of neural network parameters, i.e., the number of bits required to store them RR-RL.2 therefore optimally trades-off performance against the number of bits required to implement the emerging reinforcement learning algorithm. Note that the objective from Equation can also be motivated by a PAC- PAC-Bayes bound on generalization performance to unseen tasks (Ynn et al. 2019 Rothfuss et al.) 2020 Jose and Simeone, 2020). While we focus on the resource-rational interpretation in the present article, we believe that both of these perspectives are complementary.
In practice, we solve a sample-based approximation of the optimization problem in Equation S using standard on-policy actor-critic algorithm [Mnih et al. 2016 [u et al. 2017]. We rely on a dual gradient ascent procedure [Haarnoja et al.) 2018 to ensure that the constraint is satisfied. Appendix A contains a complete description of the network architecture, choices of prior and encoding distribution, and the meta-learning procedure. A public implementation of our model and the following analyses is available under www.-raattttol-eeeeeeeereeeeeete
http://ggttuubboooooo
Which exploration strategies RR-RL implements will partially depend on its available computational resources. Models with limited resources must implement strategies that rely on simple computations, such as computing average rewards or noisy estimates thereof. Models with access to more resources, on the other hand, can spend some of them to compute more complex statistics, such as uncertainty estimates, and incorporate these into their decision dociision-makin making process. Moreover, what strategies are resource-rational not only depends on the computational resources of the decision-make but also on the characteristics of the particular problem under consideration. Problems with short task horizon, for instance, require less exploration and thereby make exploitation more appealing, while problems with longer horizon allow for the application of more sophisticated exploration strategies.
'The desired coding length can, for example, be achieved using back coding (Hinonn and Van Camp
1993 or minimal random coding (Havasi et al. 2018
3

---

(a) Exploration Strategies
(b) Embedding
Boltzmann
UCB
Thompson
Humans
RR-RL2
Dimension 2
$
2
-2
101
102
103
104
-5
5
Description Length
Dimension 1
Figure 1: Illustration of exploration strategies implemented by RR-RL2 (also see Equation c and the corresponding explanation the main text). (a) Probit regression coefficients obtained by fitting the hybrid model to data simulated by RR-RL with varying description lengths (depicted on logarithmic scale). The blue line shows the influence the estimated mean on choices (Boltzmann exploration), the orange line shows the influence of the option's 's uncertainty estimates (UCB-based based exploration), estimated mean (which corresponds to
and the green line shows the influence of the uncertainty Thompson sampling in this particular task). (b) UMAP embeddings of probit regression coefficients for RR-RL and human participants.
Modeling Human Exploration
We now demonstrate that RR-RL² explains human choices on both qualitative and quantitative level. We first show that varying its description length leads to set of diverse exploration strategies, allowing to capture individual differences in human decision-making. When reanalyzing data from a two-armed bandit benchmark [Gershman] 2018), we furthermore find that RR-RI. beats previously considered algorithms in terms fitting human behavior by large margin.
Experimental Design: The behavioral data-set of Gershman (2188) contains records of 44 partici- pants who each played 20 two-armed bandit problems with episode length of H 10. The mean reward for each arm a was drawn from p(w.) = (O, 10) at the beginning of the task and the reward in each time-step from p(ra (a,, w) N (Wa,+ 1).
Analysis: To analyze the set emerging exploration strategies, we adopted method proposed by Gershman (0188. He assumed that agent uses Bayes' rule as described in Equation to update its belief's over unobserved parameters. If prior and reward are both normally distributed, the posterior will also be normally distributed and the corresponding updating rule is given by the Kalman filtering equations. Let p(wal/t) = N(H,t, da.t) ) be the posterior distribution at time-step t. Based on the parameters of this posterior distribution, he then defined the following probit regression model:
V p(A.- = 0//,, w) = $ Wi Wi W2RU, + W W3
(6)
TU,
V. = M0.1 - 11,t
RU, Fo.t T1.t
TU, /8. + of,
with & denoting the cumulative distribution function of a standard normal distribution. Equation 6 also referred to as the hybrid model it contains several known exploration strategies special exploration strategy for w = [W1, O, O]. a variant of the UCB [W1, W2, 0], and Thompson sampling for [0, 0, 1].
cases. We can recover a Boltzmann-
algorithm for
Fitting the coefficients of the hybrid model to behavioral data allows us to inspect how much a given agent relied on a particular exploration strategy. Previously, Gershman [2188] has applied this form of analysis to human data, which revealed that people rely on mixture of directed and random exploration. In this article, extend this approach to artificial data generated by RR-RL?.

---

(a) Model Comparison
(b) Posterior Probabilities
7500
Pytric
Beltemann
7000
LCS
Thorppoe
BIC
RL"
6500
10
20
6000
10-522
5500
Saltrmann
RR-RL2
Hybrid
Boltzmann
UCB
Thompson
RL?
LCB
Thampson
RL
22
Participant
Figure 2: Model comparison results on the two-armed bandit data from Gershman
2018 (a)
Bayesian information criterion (BIC) values for the aggregated data of all participants. Lower values correspond better fit to human behavior. (b) Posterior probabilities for each model and participant. Higher values correspond to a better fit to human behavior.
Results: We trained RR-RL² with a targeted description length of {1,2.... 10000)) nats on the same distribution used in the original experimental study and examined how the description length of these models influences their exploration behavior using the previously described probit regression analysis. Figure [ (a) illustrates the results of this analysis. We find that RR-RL2 implements Boltzmann-lice exploration strategy for description lengths between 1 and 100 nats. Note that behavior at this stage is quite noisy as indicated by the small probit regression weights (average regression coefficient of 0.15). Beginning from 100 nats, we observe a rise of the factor corresponding to Thompson sampling, which continues to rise until the limit of 10000 nats. Between 100 and 1000 nats, we additionally find minor influences of a Boltzmann-like exploration strategy (average regression coefficient of 0.19). For a description length of 1000 nats and larger, Boltzmann- like exploration diminishes and is replaced with minor influences of a CCCB-baee strategy (average regression coefficient of 0.23).
Having established that different styles of exploration emerge in RR-RL² depending its description length, we next set out to test how well it explains human choices. In order to do so, we conducted Bayesian model comparison [Bishop] 2006). A detailed summary of our comparison procedure is provided in Appendix B We used the Bayesian information criterion (BIC, [Schwarz 1978) as an approximation to the model evidence. The resulting BIC values for each candidate model illustrated in Figure (a). We find that the BIC value for RR-RL2 is substantially lower compared to that of the hybrid model (5562.63 against 6158.91) when aggregated across all participants; all other models provide a less adequate fit to human choices. The majority of participants (n = 32) was best described by RR-RL2 and the protected exccedance probability (PXP), which measures the probability that a particular model is the most frequent within set of alternatives [Rigoux et al. 2014], also favored RR-RL" decisively (PXP 1). We provide detailed illustration the posterior probabilities for each model and participant in Figure (b).
Finally, compared the probit regression coefficients of human participants the ones of RR-RL2. Figure (b) shows a two-dimensional UMAP embedding [Mcinnes et al., [2018) of these coefficients. The figure reveals of diverse exploration strategies within the human population and confirms that RR-RL2 captures the overall variability in human exploration appropriately.
Manipulating Computational Resources
RR-RL2 also makes precise predictions about what should happen if computational resources are actively manipulated. Do these predictions align with the actual behavior of people? Providing answers to this question is non-trivial because we cannot simply ask a person to use an algorithm with shorter or longer description length. There are, however, two types of studies that can provide insights. The first type consists of lesion studies that compare the behavior of healthy participants to that of participants suffering from brain damage, whereas the second type consists of developmental
5

---

(a) Humans
(b) RR-RL2
low-risk
high-risk
low-risk
high-risk
1.0
1.0
Choice Probability
Choice Probability
0.8
0.8
0.6
0.6
Td
0.4
0.4
0.2
0.2
0.0
0.0
Healthy
Lesion
KL 10000
KL = 2000
Figure 3: Probability of selecting low- and high-risks arms in the Iowa Gambling Task. (a) Human data taken from Bechara al.) [1994). The probability of selecting an inferior high-risk arm is increased in participants with vmPFC damage. (b) Data simulated from RR-RL* with large and small description length. The probability of selecting an inferior high-risk arm is increased for models with fewer bits, mirroring the results of the original study.
studies that investigate how behavior evolves during cognitive development. We next take a look at an example of each of them and demonstrate that RR-RLa reproduces their key findings.
4.1 Damage to Ventromedial Prefrontal Cortex
There has been a long history of analyzing people with brain lesions cognitive neuroscience [Damasio and Damasio 1989]. We focused particular study conducted by Bechara et al. [1994] for the purpose of this article and predicted that reducing the description length of RR-RL- correspond to the behavior of brain-lesioned patients.
should
Experimental Design: To probe decision-makinn in clinical populations, Bechara et al. 1994 introduced an experimental paradigm called the Iowa Gambling Task (IGT). The IGT involves 100 choices in a single four-armed bandit problem. Two of the arms are high-risk arms, while the other two are low-risk arms. High-risk arms result in constant positive reward of 100 but have chance to yield a noisy penalty with an expected value of 125. Low-risk arms result in a constant positive reward of 50 but have chance to yield noisy penalty with an expected value of 25. A complete list of trials is printed in Table DI Bechara et al. [994] used the IGT to compare the decision decision-making of healthy participants to that of participants with ventromedial prefrontal cortex (vmPFC) damage.
Analysis: The focus our analysis was the proportion of selected low- and high-risk arms across the entire experiment. High-risk arms cause an average loss of 25 points per trial, while low-risk arms provide an average payoff of points per trial. We should therefore expect an agent to select the superior low-risk arms with higher frequency. Healthy participants are indeed able to learn about the structure the task and will after a while start to sample the superior low-risk arms. Participants with vmPFC damage, however, continue to sample to the inferior high-risk illustrated in Figure (a). This pattern striking because performance these subjects remains worse than chance regardless of how often they interact with the task.
Results: RR-RL2 requires a distribution over bandit problems for meta-learning, but participants in the IGT only encountered a single bandit task. Therefore, we cannot directly use the task of the original study for meta-learning as we have done in the previous example. We instead constructed distribution over bandit problems that maintains the key characteristics of the IGT:
The positive reward component was independently sampled for each arm from uniform distribution with a minimum value of 0 and maximum value of 150.
The mean across all trials of the negative reward component was also sampled from uniform distribution with a minimum value of 0 and maximum value of 150.

---

(a) Humans
(b) RR-RL2
directed
random
directed
random
0.3
0.3
0.2
0.2
A plexplore)
A plexplore)
0.1
0.1
0.0
0.0
-0.1
-0.1
15
20
25
0
5000 Description Length
10000
Age
Figure 4: Illustration of strategic directed and random exploration in the horizon task. (a) Human data explorstar
from Somerville al. (0017). During adolescence, people start engage more in strategic directed exploration, whereas strategic random exploration remains constant over time. (b) Data simulated from RR-RL2 with varying description lengths. Like in the human data, we observe an increase in strategic directed exploration, but no change in strategic random exploration.
The negative reward component had an occurrence probability sampled randomly from uniform distribution with a minimum value of 0.05 and maximum value of 0.95.
We furthermore added additive noise sampled from zero-mean normal distribution with standard deviation of 10 to the negative reward component each time-step.
We trained RR-RL2 with targeted description length of (100, 200 ..... 10000} nats the previously described distribution. When tested on the IGT, we find that RL. replicates the pattern reported by Bechara et al. [1994). Models with high description length successfully solve the task by selecting low-risk arms in the majority of time-steps. If description length is however sufficiently reduced, RR-RI.2 predominately samples high-risk arms. We illustrate this behavior for example models in Figure (b). Figure D2 D2 provides more detailed picture of how description length mediates choice behavior.
In summary, our analysis sheds light on why brain-lesioned patients display below-chance perfor- mance in the IGT. Intuitively, any resource-limited agent must primarily devote its computational resources to things that are easy to estimate. In the IGT, the deterministic positive reward compo- nent is easier estimate than the noisy negative component. An agent with significantly restricted resources will thus focus on the positive component while ignoring the negative. In turn, the agent will assign higher estimated payoff's to the inferior high-risk arms and therefore select them more frequently. We found that RR-RI.2 implements this behavior and that reducing its description length captured participants with lesioned vmPFC.
4.2 Developmental Trajectories
People are not born with fully-developed cognitive abilities but instead develop them during their lifetime. In this section, we tested whether increasing the description length of RR-RL matches the behavioral trajectories of people as they grow up. To test this hypothesis, we reanalyzed data collected by (Smerville et al. [2017], who studied changes in exploration behavior between early adolescence and adulthood.
Experimental Design: In their study, Somerville al. [2017] made use of an experimental paradigm known as the horizon task [Wilson et al. 2014]. Each task was based on two-armed bandit problem and involved four forced-choice trials, followed by either one or six free-choice trials. Participants were aware of the number of remaining choices and could use this information to guide their behavior. The mean reward of one of the arms was drawn randomly from (40, 60), while the mean reward for the other determined by sampling the difference the first arm from {4, 8, 12, 20, 30). The
7

---

arrangement of arms as well as the sign of their difference was randomized. In each time-step, the observed reward was sampled from a normal distribution with the corresponding mean value and standard deviation of 8. The addition of forced-choice trials allowed to control the amount of information that was available to participants. They either provided an equal amount of information for both arms (i.e., two observations each) or an unequal amount of information (i.e., single observation from one arm, three from the other). In total. Somervilee et al. [2017) collected data for 147 participants between the ages of 12.08 and 28, completing 160 bandit tasks each.
Following Somerville et al. [017], we used the decision in the first free-choice trial to distinguish between different types of exploration. In the unequal information condition, choice was classified as directed exploration if it corresponded to the option that was observed fewer times during the forced-choie trials. In the equal information condition, a choice was classified as random exploration it corresponded to the option with the lower estimated mean. We refer to an exploration behavior as strategic if it occurs more frequently in the long horizon tasks compared to the short horizon tasks. (Somerville et al. [2077] found as shown in Figure 4 (a) - that strategic directed exploration emerges during adolescence, whereas strategic random exploration is age-invariant.
Analysis:
To quantify these effects, they fitted two independent linear regression models, using the probability of engaging in directed and random exploration as dependent variables. Both models used age, the corresponding horizon, and the interaction between the two as regressors. They found significant effect of horizon in both conditions, indicating that participants engaged more in both directed and random exploration tasks with a longer horizon. Furthermore, they found a significant interaction effect between horizon and age for directed exploration but not for random exploration, confirming that strategic directed exploration increases during cognitive development, while strategic random exploration remains constant over time.
Results: We trained RR-RL² with targeted description length of {100, 200,.... 10000} nats on the same distribution used in the original experimental study. Figure (b) visualizes how strategic directed and random exploration change as the description length of RR-RL increases. Matching the main result of the experimental study, we find that strategic directed exploration increases with description length, while strategic random exploration remains unaffected. We repeated the previously described regression analysis on data simulated by RR-RL² to quantify this conclusion (replacing age as regressor with description length). The outcome of this analysis mirrored the results of the original study. We found a significant effect of horizon on both directed (Fi.194 = 56.50, p < 0.001, n/² = 0.20) and random exploration (F1, = 6.80, = 0.01, 12² = 0.03). This means that RR- made more exploratory decisions of both types if it was beneficial to do so. We also found significant interaction effect between horizon and description length on directed exploration (F = 17.48, p < 0.001, n = 0.06) but not on random exploration (Fi,194 = 0.02. p = 0.89). These results confirm that description length and age have comparable qualitative effects on the development of strategic exploration.
However, when comparing the effect sizes of our analysis to those from the experimental study, we find that the interaction effect between description length and horizon on directed exploration only amounts to around half of the effect between age and horizon. We speculated that part of this difference comes from a mismatch between the distribution used to train our models and what kind of tasks people expect in the experiment. People, for instance, might assume that task rewards are noisier than they are, which would require more exploratory choices, and, in turn, lead to stronger effects. We tested this hypothesis by retraining RR-RI.2 on the same distribution but with the standard deviation of the reward noise increased by 50%. While this modification increased the effect size of the interaction effect on directed exploration, it did not close the gap entirely, suggesting that there are additional - currently undiscovered factors that contribute to the development of strategic directed exploration during adolescence. Furthermore, we found that, while strategic random exploration was unaffected by both age and description length. humans had a higher base rate of engaging in random exploration than our models. We, however, believe that this issue could be resolved by adding an E- e-grredy error model to our models as doing so would naturally increase the base rate of random exploration.
8

---

5
General Discussion
The
dilemma is one of the core challenges in reinforcement learning. How do humans arbitrate between exploration and exploitation, and which kind of exploration strategies do they engage in? We have put forward the hypothesis that people tackle this problem in a resource- rational manner. To test this hypothesis, we proposed method for meta-learning reinforcement learning algorithms with limited description length. The resulting class of models which we coined RR-RL 2 makes precise predictions about how people make decisions. We have put these predictions to a rigorous test by comparing our model to data from three psychological studies. RR-RL* displayed key elements of human decision-maan making in all three of them:
1. It captured human exploration in a two-armed bandit task on both a qualitative and quantita- tive level.
2. Reducing its description length aligned with decision-making in brain-lesioned patients.
3. Increasing its description length reflected changes in exploration behavior attributed cogni- tive development.
In summary, our results demonstrate that is possible meta-learn resource-rational reinforcement learning algorithms and that human exploration is well-characterized by these very algorithms.
5.1 Limitations and Future Work
We have focused on comparing RR-RI.2 human exploration in the simple multi-armed bandit setting. In the real world, however, people face much more sophisticated challenges that call for richer repertoire of exploration strategies [Schulz et al.) 2019 Brändle et al.] 2021]. This criticism is not necessarily shortcoming of the proposed model, which could in principle be applied to more complex tasks, but rather one regarding the experimental research in cognitive psychology, which has predominately focused multi-armed bandit problems. In future work, we intend to develop new experimental paradigms that allow us to compare RR-RL" against human behavior in more complex settings. Potential paradigms may include contextual bandits, grid world problems, and multi-agent scenarios.
RR-RI.2 also places a constraint on particular type of computational resource: the description length of the reinforcement learning algorithm in use. People, on the other hand, are subject to a variety of additional computational constraints. They can, for instance, only run algorithms with finite computation time or only store restricted amount of chunks in their short-term memory (Collins) al. 2014]. Future work should aim unify all of these constraints in common framework.
would also be interesting investigate how RR-RL could be implemented in a biologically plausible way. We believe that replacing the standard recurrent neural networks used in this article with spiking neural networks could provide one promising path towards this goal. Earlier work of Bellec et al. [2018) showed that it is indeed possible to train spiking neural networks in meta-learning setting and would therefore provide an excellent starting point for this endeavor.
Finally, there are many alternative ways of how one could apply meta-learning to our setting. It is, for example, possible to meta-learn hyperparameters of standard reinforcement learning algorithms [Doyal 2002 Luksys et al. 2009]. or to use model-agnostic meta-learning to find optimal weight initializations for neural networks that are trained by gradient descent [Finn al., 2017). Future work should compare these and other models against each other. For this to be effective, we believe that a set of commonly-accepted benchmarks needs to be established.
5.2 Conclusion
Many applications could benefit from the availability of human- like agents. Having access to such agents may be especially valuable in cooperative self-play scenarios, where training with them is crucial for successful coordination with people [Carroll et al., 2019 Strouse et al. [021]. The traditional path towards constructing agents that learn and think like people is to take inspiration from the cognitive processes of the human mind and incorporate them into existing systems [Lake] jtt al. 2017]. In this article, we have pursued different approach. Instead of hard-coding cognitive processes directly into agents, we have identified two computational principles meta-lcarning

---

and resource rationality that give rise to many aspects of human behavior. The presented approach very general, easy adapt to new domains, and can be scaled more complex problem settings without major modifications.
Finally, want emphasize that low description lengths might not only be biological necessity, but also a feature [Gigerenzer et al. 1999 Zador [019]. Implementing an algorithm in just a few bits strong form of regularization and could, in turn, produce exploration strategies that are applicable across domains, Hence, we believe that constructing artificial systems with such constraints could lead towards more generally capable agents.
Acknowledgments and Disclosure of Funding
This work was funded by the Max Planck Society, the Volkswagen Foundation, as well as the Deutsche Fonschunggsemmeisschatt (DFG, German Research Foundation) under Germany's Excel- Ience Stattgy-EXCC00//1333977776
10

---

References
Hirotugu Akaike. A new look at the statistical model identification. IEEE transactions on automatic control, 19(6):716-723, 1974.
Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47(2-3):2352255 2002.
Antoine Bechara, Antonio R Damasio, Hanna Damasio, and Steven W Anderson. Insensitivity to future consequences following damage to human prefrontal cortex. Cognition, 50(1-3)77-15, 1994.
Guillaume Bellec, Darjan Salaj, Anand Subramoney, Robert Legenstein, and Wolfgang Maass. Long short-term memory and learning-to-li to-arrrt--rr- in networks of spiking neurons. Advances in neural information processing systems, 31, 2018.
Y Bengio, S Bengio, and J Cloutier. Learning synaptic learning rule. In CCCNN-91-Seatlee International Joint Conference on Neural Networks, volume 2, pages 969-vol. IEEE, 1991.
Marcel Binz and Dominik Endres. Where do heuristics come from? Proceedings of the 41th
Annual Meeting of the Cognitive Science Society, pages 1402-1408, 2019.
Marcel Binz, Samuel Gershman, Eric Schulz, and Dominik Endres. Heuristics from bounded meta-learned inference. Psychological Review, (Advance online publication), 2022.
Christopher Bishop. Machine learning and pattern recognition. Information science and statistics.
Springer, Heidelberg. 2006.
Franziska Brändle, Marcel Binz, and Eric Schulz. Exploration beyond bandits. 2021.
Micah Carroll, Rohin Shah, Mark K Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. On the utility learning about humans for human-ai coordination. Advances in Neural Information Processing Systems, 32:5174-5111 2019.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint 1:0.vv1406..000 1078, 2014.
Anne GE Collins, Jaime K Brown, James M Gold, James A Waltz, and Michael J Frank. Work- ing memory contributions to reinforcement learning impairments in schizophrenia. Journal of Neuroscience,
2014.
H. Damasio and A.R. Damasio. Lesion Analysis in Neuropsychology. Oxford University Press, 1989.
Kenji Doya. Metalearning and neuromodulation. Neural networks, 15(4-6):95-506, 2002.
Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, llya Sutskever, and Pieter Abbeel. RI 2: Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv:1611.00 1.007997 2016.
Michael O Duff. Optimal learning: Computational procedures for bayes-adaptive markov decision processes. 2003.
Ben Eysenbach, Russ R Salakhutdinov, and Sergey Levine. Robust predictable control. Advances in Neural Information Processing Systems, 34, 2021.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pages 1126-1135. PMLR, 2017.
Andrew Gelman, John B Carlin, Hal S Stern, and Donald B Rubin. Bayesian data analysis. Chapman and Hall/CRC, 1995.
Samuel Gershman. Deconstructing the human algorithms for exploration. Cognition, 173:34 42, 2018.
Samuel J Gershman, Eric J Horvitz, and Joshua B Tenenbaum. Computational rationality: A converging paradigm for intelligence in brains, minds, and machines. Science, 34962245) 2015.
11

---

G. Gigerenzer, P.M. Todd, and ABC Research Group. Simple Heuristics that Make Us Smart. Oxford University Press, 1999.
John Gittins. Bandit processes and dynamic allocation indices. Journal of the Royal Statistical Society. Series B (Methodologicall) pages 148-177, 1979.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv: I8/2:::812.0000 1812.05905, 2018.
Marton Havasi, Robert Peharz, and José Miguel Hermández-Lobato. Minimal random code learning: Getting bits back from compressed model parameters. International Conference on Learning Representations, 2018
Geoffrey E Hinton and Drew Van Camp. Keeping the neural networks simple by minimizing the description length the weights. In Proceedings of the sixth annual conference on Computaaiooall learning theory, pages 5-13, 1993.
Sharu Theresa Jose and Osvaldo Simeone. Transfer meta-learning: Information--teeoreiee bounds and information meta-risk minimization, 2020.
Emilie Kaufmann, Olivier Cappé, and Aurélien Garivier. On bayesian upper confidence bounds for bandit problems. In Artificial intelligence and statistics, pages 592-600, 2012.
Diederik Kingma and Max Welling. arXiv:1312.61 14, 2013.
Auto-encoding variational bayes.
arXiv preprint
Durk P Kingma, Tim Salimans, and Max Welling, Variational dropout and the local reparameterizattoo trick. Advances in neural information processing systems, 28:2555--583, 2015.
Brenden M Lake, Tomer D Ullman, Joshua Tenenbaum, and Samuel Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40, 2017.
Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint arXiv: 18885000009 2018
Falk Lieder and Thomas Griffiths. Resource-rational analysis: Understanding human cognition as the optimal use of limited computational resources. Behavioral and Brain Sciences, 43, 2020.
Gediminas Luksys, Wulfram Gerstner, and Carmen Sandi. Stress, genotype and norepinephrine in the prediction of mouse behavior using reinforcement learning. Nature neuroscience, 12(9): 1180-1186, 2009.
James John Martin. Bayesian decision problems and Markov chains. Wiley, 1967.
Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint 1802.0030 2018
Katja Mehlhorn, Ben R Newell, Peter M Todd, Michael D Lee, Kate Morgan, Victoria A Braithwaite, Daniel Hausmann, Klaus Fiedler, and Cleotilde Gonzalez. Unpacking the exploration-eexloiiatioon tradeoff: A synthesis of human and animal literatures. Decision, 2(3):191, 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pages 9928-1937. PMLR, 2016.
Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov. Variational dropout sparsifies deep neural networks. In International Conference on Machine Learning, pages 2498-244 2508. PMLR, 2017.
Pedro A Ortega, Jane X Wang, Mark Rowland, Tim Genewein, Zeb Kurth-Nelson, Razvan Pascanu, Nicolas Heess, Joel Veness, Alex Pritzel, Pablo Sprechmann, et al. Meta-learning of sequential strategies. arXiv preprint arXiv: /905. 100.000000000000 2019.
Lionel Rigoux, Klaas Enno Stephan, Karl Friston, and Jean Daunizeau. Bayesian model selection for group studies- revisste. Neuroimage, 84:971-985, 2014.
12

---

Jonas Rothfuss, Vincent Fortuin, and Andreas Krause. Pacoh: pac-guarantees. arXiv preprint arXiv:2002.05551, 2020.
meta-learning with
Daniel Russo and Benjamin Van Roy. Learning to optimize via information-directed sampling. In Advances in Neural Information Processing Systems, pages 1583-1591, 2014.
Daniel Russo, Benjamin Van Roy, Abbas Kazerouni, Ian Osband, and Zheng Wen. A tutorial on thompson sampling. arXiv preprint arXiv: ToX.:v/707.000.00 2017.
Juergen Schmidhuber, Jieyu Zhao, and MA Wiering. Simple principles of metalearning. Technical report IDSIA, 69:1-23, 1996.
Eric Schulz and Samuel Gershman. The algorithmic architecture of exploration in the human brain. Current opinion in neurobiology, 55:7-14, 2019.
Eric Schulz, Rahul Bhui, Bradley C Love, Bastien Brier, Michael T Todd, and Samuel J Gershman. Structured, uncertainty-drivet exploration real-world consumer choice. Proceedings of the National Academy of Sciences, 116229)13903-13988 2019.
Gideon Schwarz. Estimating the dimension of model. The annals of statistics, pages 461-4644 464, 1978.
Leah H Somerville, Stephanic F Sasse, Megan C Garrad, Andrew T Drysdale, Nadine Abi Akar, Catherine Insel, and Robert C Wilson. Charting the expansion of strategic exploratory behavior during adolescence. Journal of experimental psychology: general, 146(2):155, 2017.
Mark Steyvers, Michael D) Lee, and Eric-Jan Wagenmakers. A bayesian analysis of human decision- making on bandit problems. Journal of Mathematical Psychology, 53(3):168-17) 2009.
DJ Strouse, Kevin McKee, Matt Botvinick, Edward Hughes, and Richard Everett. Collaborating with humans without human data. Advances in Neural Information Processing Systems, 34, 2021.
William R Thompson. On the likelihood that unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3/4):255-994, 1933.
Jane X Wang, Zeb
Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning reinforcement learn. arXiv preprint 161iv: 1.05763, 2016.
Robert C Wilson, Andra Geana, John M White, Elliot A Ludvig, and Jonathan D Cohen. Humans use directed and random exploration to solve the exploit dilemma. Journal of Experimental Psychology: General, 143(6):2074, 2014.
Robert C Wilson, Elizabeth Bonawitz, Vincent D Costa, and R Becket Ebitz. Balancing exploration and exploitation with information and randomization. Current Opinion in Behavioral Sciences, 38: 49-56, 2021.
Yuhuai Wu, Elman Mansimov, Roger B Grosse, Shun Liao, and Jimmy Ba. Scalable trust-region method for deep reinforcement learning using coonecker-factored approximation. Advances in neural information processing systems, 30:5279-5288 2017.
Mingzhang Yin, George Tucker, Mingyuan Zhou, Sergey Levine, and Chelsea Finn. Meta-learning without memorization. arXiv preprint arXiv:1912.03822 2019.
Anthony M Zador. A critique of pure learning and what artificial neural networks can learn from animal brains. Nature communications, (01)):177,0099
Shunan Zhang and Yu Angela. Forgetful bayes and myopic planning: Human learning and decision-making.ne in bandit setting. In Advances in neural information processing systems, pages 2607-2615, 2013.
Luisa Zintgraf, Kyriacos Shiarlis, Maximilian Igl. Sebastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: A very good method for bayes-adaptive deep el via meta-learning. arXiv preprint 1110::9110008448. 2019.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you descrihe the limitations of your work? [Yes] See Section 5.1
(c) Did you discuss any potential negative societal impacts of your work? [No] There are no immediate negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as URL)? [No] We did not provide code at the time of the initial submission, but will make publicly available once the paper is accepted.
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] Information to reproduce our results is presented in the main text and in Appendix A
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [Yes] We fitted Bayesian linear regression models mapping description length of model to the measure of interest and report the resulting uncer- tainty over the mean (see for example Figures [ and D2). This method assumes that models with a similar description length score similar on the respective measure.
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, cloud provider)? [N/A] No special compute resources were used during this project. Models were trained using standard CPUs on an internal cluster.
4. If you using existing assets (e.g., code, data, models) curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A] We used existing data from experiments conducted in previous work. This information can be found in the original articles.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A] We used existing data from experiments conducted in previous work. This information can be found in the original articles.
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A] We used existing data from experiments conducted in previous work. This information can be found in the original articles.
14


Reviews:
[{"review_id": "yPkclkgM5g", "reviewer": "Reviewer_oRHY", "summary": "The paper explores computational foundations underlying human exploration and focuses on the idea that model description length may be a critical factor determining the resulting type of exploration. The authors first summarise theoretical foundation of the model and then move on to comparing their proposed model (RL^3) to other models in their ability to fit behavioural data in normal participants as well as behavioural data in lesioned patients as well as changes with development. They show that their model outperforms other models regarding fit to behavioural data and that manipulating description length qualitatively reproduces the effect of brain lesions in Iowa Gambling Task as well as the effects of development in the Horizon task.", "questions": "For the first weakness, perhaps it would be better to link with more cognitive computational neuroscience papers (especially those linking working memory capacity to reinforcement learning, e.g. work of Collins and Frank), where biological correlates are a lot more straightforward, and at least provide discussion on plausible biological computation of quantities like KL.\n\nFor the second weakness, at the very least I would expect some discussion mentioning the possibility that simple models with dynamic control of exploration (and possibly other) metaparameters as opposed to full meta-learning with control of description length may be another feasible (and much simpler, even if not normative) solution to the problem of accurately reproducing behavioural data.\n\nIn Fig. 1 interpretation that \"Between 100 and 1000 nats, we additionally find minor influences of a Boltzmann-like exploration strategy.\" I don't really understand what exactly this is based on? How do you define minor influences?\n\nFor Fig. 3 I would rather put data from D2 there as opposed to picking 2 \"example models\". If using choice probability as the only behavioural measure is already quite limiting, then picking 2 matching data points seems even more so, It's also curious that 'lesion group' performance seems to be better (at least in this one measure) than performance with any description length (<100) that best matches Boltzmann sampling. Again I'm very curious how a simple RL algorithm with dynamic metaparameter control or RL + working memory algorithms would perform... ", "limitations": "Some of them yes, but some others not yet - see suggestions above in 'questions'.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper discusses a really interesting hypothesis regarding what is the relevant factor determining the type and scale of exploration in the RL with metalearning set up. To my knowledge it is novel and it could provide some very stimulating ideas for the exploration-exploitation field. The authors also show great effort trying to link their results to behavioural (and lesion + developmental) studies and show a good understanding and review of relevant literature, both in machine learning and neuroscience.\n\nIn my opinion, weaknesses mainly fall into 2 categories:\n- firstly, the level of methodological description is somewhat limited, and although in principle the suggested approach could be biologically plausible, it is hard to imagine (nor is there any discussion), how this could be biologically implemented. \n- secondly, I feel that the benchmarks the authors' model is compared to are extreme / theoretical cases as opposed to using more empirical/heuristic ideas of how RL metaparameters (including those controlling exploration) should be controlled, e.g. Doya \"Metalearning and neuromodulation\" (Neural Networks 2002) or Luksys et al. 2009 Nat Neuroscience paper on modelling of stress, which incidentally also doesn't use bandit tasks (whose excessive use the authors rightly criticised). It may well be that while theoretical prototypes such as Boltzmann or Thompson may not do very well in detailed comparison with behavioural data, more empirical approaches, like those in the mentioned papers may do better, despite their simplicity and no need to compute complex quantities like KL. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "tvSxlX1Dqpc", "reviewer": "Reviewer_xjvi", "summary": "The paper examines human exploration from the perspective of rationality under limited resources. The focus is on exploration in multi-armed bandit problems where the objective is to maximise the sum of rewards over a finite horizon. It is assumed that the agent has a prior distribution over bandit problems it may encounter.\n\nThe authors investigate the hypothesis that people explore optimally within the contraints they have on their computational resources. The authors devise a family of algorithms that achieve optimal exploration subject to a constraint on the description length of the algorithm. The authors then examine three data sets -- these are exploration data collected  from human subjects under different circumstances -- and demonstrate that their model captures certain characteristics of these data sets. \n", "questions": "When analyzing the data from Gershman (2018) -- the authors examined how well each model fitted the behavioral data from a two-armed bandit task -- my understanding is that C was a free parameter and that this would create a very flexible model that can fit many different types of exploration behaviour well. Would the authors agree? And would it be possible to examine the model in a predictive context? E.g., fit the parameters on some part of the data and examine how well the fitted model explains the remaining data?\n\nWhen analyzing the data by Bechara et al. (1994) -- 100 choices in a single four-armed bandit problem -- the authors focused on the proportion of low- and high-risk arms selected across the entire experiment. The model (I believe) can make more precise predictions than that. Is a more rigorous test of the model possible and have the authors attempted that? \n\n", "limitations": "The main limitation is that all experimental data examined is on bandit problems. This is acknowledged by the authors.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper explores a useful point of view and I found the results to be interesting and informative. \n\nThe model that is developed puts together ideas from various existing models and solution methods to create a specific approach to exploration under limited computational resources. The primary novelty in the paper is the analysis of human exploration data --  from very different experiments -- using this model. \n\nI have some concerns about the flexibility of the model (and consequently, the interpretation of the results) and some of the design choices. \n\nWhen analyzing the data from Gershman (2018) -- the authors examined how well each model fitted the behavioral data from a two-armed bandit task -- my understanding is that C was a free parameter and that this would create a very flexible model that can fit many different types of exploration behaviour well. \n\nWhen analyzing the data by Bechara et al. (1994) -- 100 choices in a single four-armed bandit problem -- the authors focused on the proportion of low- and high-risk arms selected across the entire experiment. The model (I believe) can make more precise predictions than that and I wonder whether a more rigorous test of the model would have been possible. \n\nThe paper is generally clear, especially at high-level exposition of the motivations and ideas.\n\nAdditional  comments for the authors:\n\nFigure 2a is misleading. The text in the  body of the  paper and the figure caption both note that the figure is showing Bayesian information criterion (BIC) values for the aggregated data. But the figure actually shows the difference in BIC compared to RL3, with RL3 value in the figure corresponding to a value of 0. In other  words, there is a mismatch in what the figure shows and what the text says about what the figure shows. Figure 2a somewhat misleading. The text in the  body of the  paper and the caption notes that the figure is showing Bayesian information criterion (BIC) values for the aggregated data. But the figure actually shows the difference in BIC compared to RL3  (with RL3 in the figure showing 0). \n\nCorrections to some references:\n\n\"Hanna Damasio. Lesion analysis. Neuropsychology., 1989.\" The correct  reference is: Damasio, H., Damasio, A.R., 1989. Lesion Analysis in Neuropsychology. Oxford Univ. Press, Oxford.\n\n\"Gerd Gigerenzer and Peter M Todd. Simple heuristics that make us smart. Oxford University Press, USA, 1999.\"  The authors should be Gerd Gigerenzer and Peter M Todd, & the ABC Research Group.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "d7rpgNghsSr", "reviewer": "Reviewer_m1ih", "summary": "This paper develops a model called \"RL3\" which learns a free-standing RL algorithm by optimizing a resource-constrained objective function. The key contribution is to show how human-like patterns of exploration arise from the optimized algorithm. Using data from a two-armed Gaussian bandit task, they show that the learned algorithm quantitatively matches human behavioral data, and fits the data better than a number of previously proposed models. Importantly, the learned algorithm produces a mixture of directed, random, and Boltzmann sampling similar to what people produce in the task. Another interesting observation is that the mix of directed/random/Boltzmann changes with the capacity limit. The authors then show that reducing the capacity limit produces behavior similar to what has been observed in ventromedial prefrontal cortex patients. Finally the authors show that they can reproduce the developmental trajectory of directed vs. random exploration.", "questions": "- I understand that space is limited so I don't expect the authors to address this, but I think it would be very interesting to apply this approach to contextual bandits.\n- I'm also curious about whether the implication here is that the brain contains many pretrained RL algorithms for different tasks, or a few that generalize across tasks.", "limitations": "I don't see any potential negative impacts. The authors devote a section of their discussion to limitations, though it's somewhat vague. I would have liked to see more specific predictions for future experimental work.", "rating": 9, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Overall, I really liked this paper. it presents an interesting and novel idea, the writing is clear, and the analysis is rigorous.\n- The paper is quite rich, addressing data from healthy humans, lesion patients, and developmental trajectories.\n- Not only can the model reproduce aspects of human exploration behavior, but the theory sheds light on which phenomena should be observed to emerge at different capacity limits (this should be testable).\n\nWeaknesses:\n- The meta-learning setup for the IGT is somewhat contrived. It might have made more sense to use tasks where there is a clearer distribution for meta-learning.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "OjiIlhk-hsD", "reviewer": "Reviewer_6vWe", "summary": "This paper presents RL^3: a combination of meta reinforcement learning with information bottlenecks. RL^3 explains human exploration behavior as resource rational -- humans rely on limited description lengths (defined here as the amount of information in the weights of a reinforcement learning algorithm parameterized by a neural network) to balance exploration with exploitation in various multi-armed bandit tasks. By reanalyzing psychological data with both healthy adults, as well as clinical populations and adolescents, the paper shows that exploration behavior can be better explained using these resource rational principles as opposed to classic, hard-coded exploration strategies (such as Thompson sampling, for example).", "questions": "* While reasonable, this is a rather limited view of both description length, as well as requiring fairly strong assumptions about the form of the decision-making process underlying explore/exploit. For example, the presented method could not obviously extend to consider object-oriented or model-based exploration, which is likely critical for more ecological/complex tasks than bandit settings. Indeed, much of the work in AI for exploration has focused on these concepts. How do the authors see this method as being applicable to these more complex settings?\n* The fit of RL^3 to both healthy adults and the lesioned population is very good, but the fit to the adolescent data is significantly worse. Both the effect size, but also the base rate of exploration, seem to be dramatically different from the human groups. Could the authors speculate on why this fit is not great, and whether there are changes to the model that could be made to improve the fit?\n* One major missing piece for me was some intuition on how classic exploration strategies come out of the RL^3 approach. In particular, why is Thompson sampling seemingly the optimal strategy for exploration when a sufficiently high description length is allowed? And why does it require a high description length in the first place? Adding some intuition for this to the paper would help – especially in the first experiment for discussing Figure 1.\n", "limitations": "Limitations are adequately addressed by the authors. No ethical limitations to consider, but limitations of the approach are reasonably well described (although see my other comments for other potential important limitations).", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n* (Significance) This paper presents a compelling argument for how human exploration is more complex than simply exploiting or exploring. Cognitive scientists in particular will be inspired by this work to think about how to both move towards more complex exploration tasks, and in considering how to further use meta-learning techniques to model richer kinds of cognitive phenomena\n* (Originality) Resource rationality itself is not new, but this is a very nice, straightforward application of the idea to demonstrate the flexibilty and variability of human exploration behavior. \n* (Quality) The method is able to capture not just one experiment, but three separate sets of experiments separated by decades, and across exceptionally diverse populations (from lesioned groups, to healthy adults, to adolescents) without requiring fine-tuning or separate hyperparameters.\n* (Clarity) The paper is well presented. The motivation is clear (with some minor exceptions outlined below), the method is well described, the results are fairly interpreted, and the conclusions are well laid out given the results.\n\nWeaknesses\n* (Significance) While the paper attempts to provide motivation for why these findings may be important to the ML/AI community, I am not entirely convinced of their arguments. In particular, the presented method does not seem obviously applicable to more complex settings that are commonly explored in ML/AI, often involving objects (like in videogames) and models. I think this contribution is important for computational cognitive science, but I am not sure it will have a big impact on the ML/AI audience.\n* (Clarity) The paper title is both grandiose and also exceptionally vague. I would ask the authors to choose a paper title that is more indicative of the content of the paper. Key words like “resource rational” and “meta-learning” are some examples that I think could be used to make a more informative title.\n* (Clarity) The abstract introduces terms that are not defined until the second page, specifically “description length”. Without context, description length could refer to an exceptionally large number of things, and is therefore quite confusing when mentioned in isolation. I would recommend changing the abstract such that description length is not mentioned, or is mentioned and unpacked in a clause or a single sentence to give the reader an understanding of what this specifically refers to.\n* (Clarity) RL^3 seems like a poor method name given that RL^2 was so named since it was using reinforcement learning to train a reinforcement learning algorithm. Since this approach is not doing an extra meta-level of optimization, and is instead limiting the information in the weights of the inner RL algorithm, RL^3 is a very poor choice of name. What about RR-RL^2 for resource rational RL^2? Or IB-RL^2 for information bottleneck RL^2? \n* (Clarity) Figure 1a is somewhat difficult to interpet as a standalone figure. I think I understand it as behaving more like Thompson sampling as the description length is increased, but is that what’s actually happening? Some extra description in the figure caption would really help with the interpretation. (See also my question in the “questions” section below).\n* (Clarity) Figure 2a - is RL^3 really at 0 BIC? Is that theoretically possible? Or is the delta BIC with respect to the RL^3 model? If the latter, the caption should be updated accordingly.\n* (Quality) Experimental results on development are much less compelling than for the other two experiments. The model is clearly doing something distinctly different from humans in that its base probability of exploration is 0 rather than a constant value. While the authors address some reasoning for why the effect size is lower for the model, there is no discussion of the differences in why the overall likelihood of exploration for people is so much higher. Could the authors comment on this?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept