Introduction
Knowing how to efficiently balance between exploring unfamiliar parts of an environment and exploiting currently available knowledge is an essential ingredient of any intelligent organism. In theory, it is possible to obtain a Bayes-optimal solution to this exploration-exploitation dilemma by solving an augmented problem known as a Bayes-adaptive Markov decision process (BAMDP, Duff, 2003 ). However, BAMDPs are intractable to solve in general and analytical solutions are only available for a few special cases (Gittins, 1979 ) . The intractability of this optimal solution prompted researchers to develop a body of heuristic strategies (Auer et al., 2002 ; Kaufmann et al., 2012 ; Russo et al., 2017 ; Russo and Van Roy, 2014 ) . Most of these heuristics can be divided into two broad categories: directed and random exploration strategies (Wilson et al., 2014 ; Schulz and Gershman, 2019 ) . Directed exploration strategies provide bonus rewards that encourage the agent to visit parts of the environment that ought to be explored, whereas random exploration strategies inject some form of stochasticity into the policy.

Having access to a vast amount of existing exploration strategies leads to the question: which of them should we use when building artificial agents? To answer this question, we may take inspiration from how people approach the exploration-exploitation dilemma. Human exploration has been studied extensively in the multi-armed bandit setting (Mehlhorn et al., 2015 ; Wilson et al., 2021 ; Brändle et al., 2021 ) . Prior work indicates that people substantially deviate from the Bayes-optimal strategy even for the simplest bandit problems (Steyvers et al., 2009 ; Zhang and Angela, 2013 ; Binz and Endres, 2019 ) . They, however, use uncertainty estimates to intelligently guide their choices through a combination of both directed and random exploration (Wilson et al., 2014 ; Gershman, 2018 ) . The question of when and why individuals rely on a particular exploration strategy has been under-explored so far.

We take the first steps towards answering these questions by looking at human exploration from a resource-rational perspective (Gershman et al., 2015 ; Lieder and Griffiths, 2020 ; Binz et al., 2022 ) . More specifically, we investigate the hypothesis that people solve the exploration-exploitation dilemma by making optimal use of limited computational resources. To test this hypothesis, we devise a family of resource-rational reinforcement learning algorithms by combining ideas from meta-learning (Bengio et al., 1991 ; Schmidhuber et al., 1996 ) and information theory (Hinton and Van Camp, 1993 ) . The resulting model – which we call resource-rational RL 2 (RR-RL 2 ) – implements a free-standing reinforcement learning algorithm that achieves optimal behavior subject to the constraint that it can be implemented with a given number of bits.

We demonstrate that RR-RL 2 captures many aspects of human exploration by reanalyzing data from three previously conducted psychological studies. First, we show that it explains human choices in a two-armed bandit task better than traditional approaches, such as Thompson sampling (Thompson, 1933 ) , upper confidence bound (UCB) algorithms (Kaufmann et al., 2012 ) , and mixtures thereof (Gershman, 2018 ) . We then verify that the manipulation of computational resources in our class of models matches the manipulation of resources in human subjects in two different contexts. Taken together, these results enrich our understanding of human exploration and provide insights into how to improve the exploratory capabilities of artificial agents.