Modeling Human Exploration
We now demonstrate that RR-RL 2 explains human choices on both a qualitative and quantitative level. We first show that varying its description length leads to a set of diverse exploration strategies, allowing us to capture individual differences in human decision-making. When reanalyzing data from a two-armed bandit benchmark (Gershman, 2018 ) , we furthermore find that RR-RL 2 beats previously considered algorithms in terms of fitting human behavior by a large margin.

Experimental Design: The behavioral data-set of Gershman ( 2018 ) contains records of 44 participants who each played 20 two-armed bandit problems with an episode length of H=10 . The mean reward for each arm a was drawn from p(\omega_{a})=\mathcal{N}(0,10) at the beginning of the task and the reward in each time-step from p(r_{t}|a_{t},\omega)=\mathcal{N}(\omega_{a_{t}},1) .

Analysis: To analyze the set of emerging exploration strategies, we adopted a method proposed by Gershman ( 2018 ) . He assumed that an agent uses Bayes’ rule as described in Equation 1 to update its beliefs over unobserved parameters. If prior and reward are both normally distributed, the posterior will also be normally distributed and the corresponding updating rule is given by the Kalman filtering equations. Let p(\omega_{a}|h_{t})=\mathcal{N}(\mu_{a,t},\sigma_{a,t}) be the posterior distribution at time-step t . Based on the parameters of this posterior distribution, he then defined the following probit regression model:

 p(A_{t}=0|h_{t},\mathbf{w}) =\mathbf{\Phi}\left(\mathbf{w}_{1}V_{t}+\mathbf{w}_{2}RU_{t}+\mathbf{w}_{3}\dfrac{V_{t}}{TU_{t}}\right) (6) V_{t} =\mu_{0,t}-\mu_{1,t} RU_{t} =\sigma_{0,t}-\sigma_{1,t} TU_{t} =\sqrt{\sigma_{0,t}^{2}+\sigma_{1,t}^{2}}

with \mathbf{\Phi} denoting the cumulative distribution function of a standard normal distribution. Equation 6 is also referred to as the hybrid model as it contains several known exploration strategies as special cases. We can recover a Boltzmann-like exploration strategy for \mathbf{w}=[\mathbf{w}_{1},0,0] , a variant of the UCB algorithm for \mathbf{w}=[\mathbf{w}_{1},\mathbf{w}_{2},0] , and Thompson sampling for \mathbf{w}=[0,0,1] .

Fitting the coefficients of the hybrid model to behavioral data allows us to inspect how much a given agent relied on a particular exploration strategy. Previously, Gershman ( 2018 ) has applied this form of analysis to human data, which revealed that people rely on a mixture of directed and random exploration. In this article, we extend this approach to artificial data generated by RR-RL 2 .

Results: We trained RR-RL 2 with a targeted description length of \{1,2,\ldots,10000\} nats on the same distribution used in the original experimental study and examined how the description length of these models influences their exploration behavior using the previously described probit regression analysis. Figure 1 (a) illustrates the results of this analysis. We find that RR-RL 2 implements a Boltzmann-like exploration strategy for description lengths between 1 and 100 nats. Note that behavior at this stage is quite noisy as indicated by the small probit regression weights (average regression coefficient of 0.15 ). Beginning from 100 nats, we observe a rise of the factor corresponding to Thompson sampling, which continues to rise until the limit of 10000 nats. Between 100 and 1000 nats, we additionally find minor influences of a Boltzmann-like exploration strategy (average regression coefficient of 0.19 ). For a description length of 1000 nats and larger, Boltzmann-like exploration diminishes and is replaced with minor influences of a UCB-based strategy (average regression coefficient of 0.23 ).

Having established that different styles of exploration emerge in RR-RL 2 depending on its description length, we next set out to test how well it explains human choices. In order to do so, we conducted a Bayesian model comparison (Bishop, 2006 ) . A detailed summary of our comparison procedure is provided in Appendix B . We used the Bayesian information criterion (BIC, Schwarz, 1978 ) as an approximation to the model evidence. The resulting BIC values for each candidate model are illustrated in Figure 2 (a). We find that the BIC value for RR-RL 2 is substantially lower compared to that of the hybrid model ( 5562.63 against 6158.91 ) when aggregated across all participants; all other models provide a less adequate fit to human choices. The majority of participants ( n=32 ) was best described by RR-RL 2 and the protected exceedance probability (PXP), which measures the probability that a particular model is the most frequent within a set of alternatives (Rigoux et al., 2014 ) , also favored RR-RL 2 decisively (PXP \approx 1 ). We provide a detailed illustration of the posterior probabilities for each model and participant in Figure 2 (b).

Finally, we compared the probit regression coefficients of human participants to the ones of RR-RL 2 . Figure 1 (b) shows a two-dimensional UMAP embedding (McInnes et al., 2018 ) of these coefficients. The figure reveals a set of diverse exploration strategies within the human population and confirms that RR-RL 2 captures the overall variability in human exploration appropriately.