RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Methods
We start by describing the general problem setting considered in this article and its optimal solution, followed by a brief summary of the meta-reinforcement learning framework. We then show how to augment the standard meta-reinforcement learning objective with an information-theoretic constraint, allowing us to construct reinforcement learning algorithms that trade-off performance against the number of bits required to implement them.
Notation and Preliminaries
Each task considered in this article can be interpreted as a multi-armed bandit problem. In a k -armed bandit problem, an agent repeatedly interacts with k slot machines that are associated with a reward distribution p(r_{t}|a_{t},\omega) with unknown parameters \omega . In each time-step, the agent selects an action a_{t} and is provided a reward r_{t} based on the associated reward distribution.

The goal of an agent is to select actions such that the sum of rewards over a finite horizon H is maximized. We assume that the agent additionally has access to a prior distribution p(\omega) over bandit problems that it may encounter, which can be updated after observing a history of observations h_{t}\coloneqq\left(a_{1},r_{1},\ldots,a_{t-1},r_{t-1}\right) by applying Bayes’ rule:

p(\omega|h_{t})\propto p(\omega)\prod_{m=1}^{t-1}p(r_{m}|a_{m},\omega) (1)

The policy that optimally trades-off exploration and exploitation can be obtained by reasoning how the agent’s beliefs about reward functions evolve with future observations (Martin, 1967 ) . Formally, this is accomplished by transforming the original bandit problem into a corresponding BAMDP defined by the tuple (\mathcal{H},\mathcal{A},H,T,R) . In this augmented problem, \mathcal{H} represents the set of all possible histories, while \mathcal{A} and H correspond to the action space and the horizon of the original bandit problem. The transition probabilities T and reward function R are given by:

 T(h_{t+1}|a_{t},h_{t}) =p(r_{t}|a_{t},h_{t})\delta\left[h_{t+1}=\left(h_{t},a_{t},r_{t}\right)\right] (2) R(a_{t},h_{t}) =\mathbb{E}_{p(r_{t}|a_{t},h_{t})}\left[r_{t}\right] (3)

with the marginal reward probabilities:

p(r_{t}|a_{t},h_{t})=\int p(r_{t}|a_{t},\omega)p(\omega|h_{t})d\omega (4)

The policy that maximizes the sum of rewards in the BAMDP corresponds to the Bayes-optimal policy for the original bandit problem.
Meta-Reinforcement Learning
While the BAMDP formalism provides a precise recipe for deriving a Bayes-optimal policy, finding an analytical expression of this policy is typically not possible. Recent work on meta-reinforcement learning, however, has shown that it is possible to learn an approximation to it (Wang et al., 2016 ; Ortega et al., 2019 ; Zintgraf et al., 2019 ) . Duan et al. ( 2016 ) refer to this approach as RL 2 because it uses a traditional reinforcement learning algorithm to meta-learn another free-standing reinforcement learning algorithm.

RL 2 parametrizes the to-be-learned reinforcement learning algorithm with a general-purpose function approximator. Typically, this function approximator takes the form of a recurrent neural network that receives the last action and reward as inputs, uses them to update its hidden state, and produces a policy that is conditioned on the new hidden state. Let \mathbf{W} denote the parameters of this recurrent neural network. In an outer-loop meta-learning process, the network is then trained on the prior distribution over bandit problems p(\omega) to find the history-dependent policy \pi(a_{t}|h_{t},\mathbf{W}) that maximizes the sum of obtained rewards. If the meta-learning procedure has successfully converged to its optimum, RL 2 implements a free-standing reinforcement learning algorithm that mimics the Bayes-optimal policy. Importantly, learning at this stage is fully implemented through the forward dynamics of the recurrent neural network and no further updating of its parameters is required.
Resource-Rational RL 2
We transform RL 2 into a resource-rational algorithm by augmenting its meta-learning objective with an information-theoretic constraint and refer to this resource-rational variant as RR-RL 2 . More precisely, RR-RL 2 is obtained by solving the following optimization problem:

\max_{\mathbf{\Lambda}}~{} \mathbb{E}_{q(\mathbf{W}|\mathbf{\Lambda})p(\omega)\prod p(r_{t}|a_{t},\omega)\pi(a_{t}|h_{t},\mathbf{W})}\left[\sum_{t=1}^{H}r_{t}\right] s.t. KL\left[q(\mathbf{W}|\mathbf{\Lambda})||p(\mathbf{W})\right]\leq C (5)

The first component of Equation 2.3 corresponds to the standard meta-reinforcement learning objective, while the second component ensures that the Kullback–Leibler (KL) divergence between a stochastic parameter encoding q(\mathbf{W}|\mathbf{\Lambda}) and a prior p(\mathbf{W}) remains smaller than some constant C . The KL term can be interpreted as the description length of neural network parameters, i.e., the number of bits required to store them. 1 1 1 The desired coding length can, for example, be achieved using bits-back coding (Hinton and Van Camp, 1993 ) or minimal random coding (Havasi et al., 2018 ) . RR-RL 2 therefore optimally trades-off performance against the number of bits required to implement the emerging reinforcement learning algorithm. Note that the objective from Equation 2.3 can also be motivated by a PAC-Bayes bound on generalization performance to unseen tasks (Yin et al., 2019 ; Rothfuss et al., 2020 ; Jose and Simeone, 2020 ) . While we focus on the resource-rational interpretation in the present article, we believe that both of these perspectives are complementary.

In practice, we solve a sample-based approximation of the optimization problem in Equation 2.3 using a standard on-policy actor-critic algorithm (Mnih et al., 2016 ; Wu et al., 2017 ) . We rely on a dual gradient ascent procedure (Haarnoja et al., 2018 ) to ensure that the constraint is satisfied. Appendix A contains a complete description of the network architecture, choices of prior and encoding distribution, and the meta-learning procedure. A public implementation of our model and the following analyses is available under https://github.com/marcelbinz/resource-rational-reinforcement-learning .

Which exploration strategies RR-RL 2 implements will partially depend on its available computational resources. Models with limited resources must implement strategies that rely on simple computations, such as computing average rewards or noisy estimates thereof. Models with access to more resources, on the other hand, can spend some of them to compute more complex statistics, such as uncertainty estimates, and incorporate these into their decision-making process. Moreover, what strategies are resource-rational not only depends on the computational resources of the decision-maker but also on the characteristics of the particular problem under consideration. Problems with a short task horizon, for instance, require less exploration and thereby make exploitation more appealing, while problems with a longer horizon allow for the application of more sophisticated exploration strategies.
Manipulating Computational Resources
RR-RL 2 also makes precise predictions about what should happen if computational resources are actively manipulated. Do these predictions align with the actual behavior of people? Providing answers to this question is non-trivial because we cannot simply ask a person to use an algorithm with a shorter or longer description length. There are, however, two types of studies that can provide insights. The first type consists of lesion studies that compare the behavior of healthy participants to that of participants suffering from brain damage, whereas the second type consists of developmental studies that investigate how behavior evolves during cognitive development. We next take a look at an example of each of them and demonstrate that RR-RL 2 reproduces their key findings.
General Discussion
The exploration-exploitation dilemma is one of the core challenges in reinforcement learning. How do humans arbitrate between exploration and exploitation, and which kind of exploration strategies do they engage in? We have put forward the hypothesis that people tackle this problem in a resource-rational manner. To test this hypothesis, we proposed a method for meta-learning reinforcement learning algorithms with limited description length. The resulting class of models – which we coined RR-RL 2 – makes precise predictions about how people make decisions. We have put these predictions to a rigorous test by comparing our model to data from three psychological studies. RR-RL 2 displayed key elements of human decision-making in all three of them:

It captured human exploration in a two-armed bandit task on both a qualitative and quantitative level.

Reducing its description length aligned with decision-making in brain-lesioned patients.

Increasing its description length reflected changes in exploration behavior attributed cognitive development.

In summary, our results demonstrate that it is possible to meta-learn resource-rational reinforcement learning algorithms and that human exploration is well-characterized by these very algorithms.
Limitations and Future Work
We have focused on comparing RR-RL 2 to human exploration in the simple multi-armed bandit setting. In the real world, however, people face much more sophisticated challenges that call for a richer repertoire of exploration strategies (Schulz et al., 2019 ; Brändle et al., 2021 ) . This criticism is not necessarily a shortcoming of the proposed model, which could in principle be applied to more complex tasks, but rather one regarding the experimental research in cognitive psychology, which has predominately focused on multi-armed bandit problems. In future work, we intend to develop new experimental paradigms that allow us to compare RR-RL 2 against human behavior in more complex settings. Potential paradigms may include contextual bandits, grid world problems, and multi-agent scenarios.

RR-RL 2 also places a constraint on a particular type of computational resource: the description length of the reinforcement learning algorithm in use. People, on the other hand, are subject to a variety of additional computational constraints. They can, for instance, only run algorithms with finite computation time or only store a restricted amount of chunks in their short-term memory (Collins et al., 2014 ) . Future work should aim to unify all of these constraints in a common framework.

It would also be interesting to investigate how RR-RL 2 could be implemented in a biologically plausible way. We believe that replacing the standard recurrent neural networks used in this article with spiking neural networks could provide one promising path towards this goal. Earlier work of Bellec et al. ( 2018 ) showed that it is indeed possible to train spiking neural networks in a meta-learning setting and would therefore provide an excellent starting point for this endeavor.

Finally, there are many alternative ways of how one could apply meta-learning to our setting. It is, for example, possible to meta-learn hyperparameters of standard reinforcement learning algorithms (Doya, 2002 ; Luksys et al., 2009 ) , or to use model-agnostic meta-learning to find optimal weight initializations for neural networks that are trained by gradient descent (Finn et al., 2017 ) . Future work should compare these and other models against each other. For this to be effective, we believe that a set of commonly-accepted benchmarks needs to be established.
Conclusion
Many applications could benefit from the availability of human-like agents. Having access to such agents may be especially valuable in cooperative self-play scenarios, where training with them is crucial for successful coordination with people (Carroll et al., 2019 ; Strouse et al., 2021 ) . The traditional path towards constructing agents that learn and think like people is to take inspiration from the cognitive processes of the human mind and incorporate them into existing systems (Lake et al., 2017 ) . In this article, we have pursued a different approach. Instead of hard-coding cognitive processes directly into our agents, we have identified two computational principles – meta-learning and resource rationality – that give rise to many aspects of human behavior. The presented approach is very general, easy to adapt to new domains, and can be scaled to more complex problem settings without major modifications.

Finally, we want to emphasize that low description lengths might not only be a biological necessity, but also a feature (Gigerenzer et al., 1999 ; Zador, 2019 ) . Implementing an algorithm in just a few bits acts as a strong form of regularization and could, in turn, produce exploration strategies that are applicable across domains. Hence, we believe that constructing artificial systems with such constraints could lead us towards more generally capable agents.
Acknowledgments and Disclosure of Funding
This work was funded by the Max Planck Society, the Volkswagen Foundation, as well as the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany’s Excellence Strategy–EXC2064/1–390727645.
Checklist
For all authors…

Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? [Yes]

Did you describe the limitations of your work? [Yes] See Section 5.1 .

Did you discuss any potential negative societal impacts of your work? [No] There are no immediate negative societal impacts.

Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]

If you are including theoretical results…

Did you state the full set of assumptions of all theoretical results? [N/A]

Did you include complete proofs of all theoretical results? [N/A]

If you ran experiments…

Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [No] We did not provide code at the time of the initial submission, but will make it publicly available once the paper is accepted.

Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Information to reproduce our results is presented in the main text and in Appendix A .

Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [Yes] We fitted Bayesian linear regression models mapping description length of a model to the measure of interest and report the resulting uncertainty over the mean (see for example Figures 1 , 4 and D2 ). This method assumes that models with a similar description length score similar on the respective measure.

Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [N/A] No special compute resources were used during this project. Models were trained using standard CPUs on an internal cluster.

If you are using existing assets (e.g., code, data, models) or curating/releasing new assets…

If your work uses existing assets, did you cite the creators? [Yes]

Did you mention the license of the assets? [N/A]

Did you include any new assets either in the supplemental material or as a URL? [N/A]

Did you discuss whether and how consent was obtained from people whose data you’re using/curating? [N/A]

Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]

If you used crowdsourcing or conducted research with human subjects…

Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A] We used existing data from experiments conducted in previous work. This information can be found in the original articles.

Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] We used existing data from experiments conducted in previous work. This information can be found in the original articles.

Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A] We used existing data from experiments conducted in previous work. This information can be found in the original articles.
A.1 Meta-Learning Procedure
We employed a standard on-policy actor-critic procedure [Mnih et al., 2016 , Wu et al., 2017 ] to optimize a sample-based approximation of Equation 2.3 using the Adam optimizer with a learning rate of 3\text{\times}{10}^{-4} . We simultaneously updated a dual parameter to satisfy the constraint on description length as suggested in prior work [Haarnoja et al., 2018 , Eysenbach et al., 2021 ] . Models were trained on one million batches of size 32 . By the end of meta-learning, all models have converged. We additionally scaled rewards to roughly fall within the range of [-1,1] to further stabilize training. Pseudocode for the meta-learning procedure is provided in Algorithm 1 .
A.2 Model Architecture
The model architecture consists of a single gated recurrent unit (GRU, Cho et al., 2014 ) layer with a hidden size of 128 . Inputs to this GRU layer correspond to the action and reward from the previous time-step. Its outputs were then transformed by two linear layers, projecting to the policy and value estimate respectively.
A.3 Prior and Encoding Distributions
The prior over network weights corresponds to a variational dropout prior [Kingma et al., 2015 ] . The encoding distribution is parametrized by a set of independent normal distributions over network weights. We adopted the approximation suggested by Molchanov et al. [ 2017 ] to estimate the KL divergence between the encoding and prior distribution and obtained gradients with respect to \mathbf{\Lambda} using the reparametrization trick [Kingma and Welling, 2013 ] .
C.1 Model Comparison Based on AIC
We complemented our Bayesian model comparison from the main text with a model comparison that relies on the Akaike information criterion (AIC, Akaike, 1974 ):

AIC=2|\theta|-2\max_{\theta}\sum_{n=1}^{N}\sum_{t=1}^{H}\log p(A_{t}^{i,n}=a_{t}^{i,n}|h_{t}^{i,n},\theta,m) (11)

This measure offers a simple approximation to the out-of-sample predictive accuracy of a model [Gelman et al., 1995 ] . Figure C2 visualizes the AIC values for each model on the aggregated data of all participants. The results of this analysis are fully consistent with the results based on BIC reported in the main text.
Figure 1: Illustration of exploration strategies implemented by RR-RL 2 (also see Equation 6 and the corresponding explanation in the main text). (a) Probit regression coefficients obtained by fitting the hybrid model to data simulated by RR-RL 2 with varying description lengths (depicted on a logarithmic scale). The blue line shows the influence of the estimated mean on choices (Boltzmann exploration), the orange line shows the influence of the option’s uncertainty estimates (UCB-based exploration), and the green line shows the influence of the uncertainty-scaled estimated mean (which corresponds to Thompson sampling in this particular task). (b) UMAP embeddings of probit regression coefficients for RR-RL 2 and human participants.; Illustration of exploration strategies implemented by RR-RL 2 (also see Equation 6 and the corresponding explanation in the main text). (a) Probit regression coefficients obtained by fitting the hybrid model to data simulated by RR-RL 2 with varying description lengths (depicted on a logarithmic scale). The blue line shows the influence of the estimated mean on choices (Boltzmann exploration), the orange line shows the influence of the option’s uncertainty estimates (UCB-based exploration), and the green line shows the influence of the uncertainty-scaled estimated mean (which corresponds to Thompson sampling in this particular task). (b) UMAP embeddings of probit regression coefficients for RR-RL 2 and human participants.
Figure C1: Performance and description lengths in the two-armed bandit task. (a) Regrets for models with varying description lengths. Depicted on a logarithmic scale to ensure comparability with Figure 1 . The solid line shows a moving average over a window of fifty. (b) Targeted description lengths C plotted against description lengths of the converged models.; Figure C1: Performance and description lengths in the two-armed bandit task. (a) Regrets for models with varying description lengths. Depicted on a logarithmic scale to ensure comparability with Figure 1 . The solid line shows a moving average over a window of fifty. (b) Targeted description lengths C plotted against description lengths of the converged models.
Figure D1: Performance and description lengths in the meta-learning version of the Iowa Gambling Task. (a) Regrets for models with varying description lengths. The solid line shows the mean prediction of a Bayesian polynomial regression model. Shaded contours represent the standard deviation of the mean. (b) Targeted description lengths C plotted against description lengths of the converged models.; Figure D1: Performance and description lengths in the meta-learning version of the Iowa Gambling Task. (a) Regrets for models with varying description lengths. The solid line shows the mean prediction of a Bayesian polynomial regression model. Shaded contours represent the standard deviation of the mean. (b) Targeted description lengths C plotted against description lengths of the converged models.
Figure E1: Performance and description lengths in the horizon task. (a) Regrets for models with varying description lengths. The solid line shows the mean prediction of a Bayesian polynomial regression model. Shaded contours represent the standard deviation of the mean. (b) Targeted description lengths C plotted against description lengths of the converged models.; Figure E1: Performance and description lengths in the horizon task. (a) Regrets for models with varying description lengths. The solid line shows the mean prediction of a Bayesian polynomial regression model. Shaded contours represent the standard deviation of the mean. (b) Targeted description lengths C plotted against description lengths of the converged models.
Figure 2: Model comparison results on the two-armed bandit data from Gershman ( 2018 ) . (a) Bayesian information criterion (BIC) values for the aggregated data of all participants. Lower values correspond to a better fit to human behavior. (b) Posterior probabilities for each model and participant. Higher values correspond to a better fit to human behavior.; Model comparison results on the two-armed bandit data from Gershman ( 2018 ) . (a) Bayesian information criterion (BIC) values for the aggregated data of all participants. Lower values correspond to a better fit to human behavior. (b) Posterior probabilities for each model and participant. Higher values correspond to a better fit to human behavior.
Figure C2: Akaike information criterion (AIC) values for the two-armed bandit data from Gershman [ 2018 ] . Lower values correspond to a better fit to human behavior.; Figure C2: Akaike information criterion (AIC) values for the two-armed bandit data from Gershman [ 2018 ] . Lower values correspond to a better fit to human behavior.
Figure D2: Probability of selecting low- and high-risks arms in the Iowa Gambling Task for all description lengths. Mean choice probabilities are illustrated as solid lines obtained by fitting a Bayesian polynomial regression model to the underlying data. Shaded contours represent the standard deviation of the mean.; Figure D2: Probability of selecting low- and high-risks arms in the Iowa Gambling Task for all description lengths. Mean choice probabilities are illustrated as solid lines obtained by fitting a Bayesian polynomial regression model to the underlying data. Shaded contours represent the standard deviation of the mean.