Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper examines human exploration from the perspective of rationality under limited resources. The focus is on exploration in multi-armed bandit problems where the objective is to maximise the sum of rewards over a finite horizon. It is assumed that the agent has a prior distribution over bandit problems it may encounter.

The authors investigate the hypothesis that people explore optimally within the contraints they have on their computational resources. The authors devise a family of algorithms that achieve optimal exploration subject to a constraint on the description length of the algorithm. The authors then examine three data sets -- these are exploration data collected  from human subjects under different circumstances -- and demonstrate that their model captures certain characteristics of these data sets. 

strengths_and_weaknesses: The paper explores a useful point of view and I found the results to be interesting and informative. 

The model that is developed puts together ideas from various existing models and solution methods to create a specific approach to exploration under limited computational resources. The primary novelty in the paper is the analysis of human exploration data --  from very different experiments -- using this model. 

I have some concerns about the flexibility of the model (and consequently, the interpretation of the results) and some of the design choices. 

When analyzing the data from Gershman (2018) -- the authors examined how well each model fitted the behavioral data from a two-armed bandit task -- my understanding is that C was a free parameter and that this would create a very flexible model that can fit many different types of exploration behaviour well. 

When analyzing the data by Bechara et al. (1994) -- 100 choices in a single four-armed bandit problem -- the authors focused on the proportion of low- and high-risk arms selected across the entire experiment. The model (I believe) can make more precise predictions than that and I wonder whether a more rigorous test of the model would have been possible. 

The paper is generally clear, especially at high-level exposition of the motivations and ideas.

Additional  comments for the authors:

Figure 2a is misleading. The text in the  body of the  paper and the figure caption both note that the figure is showing Bayesian information criterion (BIC) values for the aggregated data. But the figure actually shows the difference in BIC compared to RL3, with RL3 value in the figure corresponding to a value of 0. In other  words, there is a mismatch in what the figure shows and what the text says about what the figure shows. Figure 2a somewhat misleading. The text in the  body of the  paper and the caption notes that the figure is showing Bayesian information criterion (BIC) values for the aggregated data. But the figure actually shows the difference in BIC compared to RL3  (with RL3 in the figure showing 0). 

Corrections to some references:

"Hanna Damasio. Lesion analysis. Neuropsychology., 1989." The correct  reference is: Damasio, H., Damasio, A.R., 1989. Lesion Analysis in Neuropsychology. Oxford Univ. Press, Oxford.

"Gerd Gigerenzer and Peter M Todd. Simple heuristics that make us smart. Oxford University Press, USA, 1999."  The authors should be Gerd Gigerenzer and Peter M Todd, & the ABC Research Group.


questions: When analyzing the data from Gershman (2018) -- the authors examined how well each model fitted the behavioral data from a two-armed bandit task -- my understanding is that C was a free parameter and that this would create a very flexible model that can fit many different types of exploration behaviour well. Would the authors agree? And would it be possible to examine the model in a predictive context? E.g., fit the parameters on some part of the data and examine how well the fitted model explains the remaining data?

When analyzing the data by Bechara et al. (1994) -- 100 choices in a single four-armed bandit problem -- the authors focused on the proportion of low- and high-risk arms selected across the entire experiment. The model (I believe) can make more precise predictions than that. Is a more rigorous test of the model possible and have the authors attempted that? 


limitations: The main limitation is that all experimental data examined is on bandit problems. This is acknowledged by the authors.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper explores computational foundations underlying human exploration and focuses on the idea that model description length may be a critical factor determining the resulting type of exploration. The authors first summarise theoretical foundation of the model and then move on to comparing their proposed model (RL^3) to other models in their ability to fit behavioural data in normal participants as well as behavioural data in lesioned patients as well as changes with development. They show that their model outperforms other models regarding fit to behavioural data and that manipulating description length qualitatively reproduces the effect of brain lesions in Iowa Gambling Task as well as the effects of development in the Horizon task.
strengths_and_weaknesses: This paper discusses a really interesting hypothesis regarding what is the relevant factor determining the type and scale of exploration in the RL with metalearning set up. To my knowledge it is novel and it could provide some very stimulating ideas for the exploration-exploitation field. The authors also show great effort trying to link their results to behavioural (and lesion + developmental) studies and show a good understanding and review of relevant literature, both in machine learning and neuroscience.

In my opinion, weaknesses mainly fall into 2 categories:
- firstly, the level of methodological description is somewhat limited, and although in principle the suggested approach could be biologically plausible, it is hard to imagine (nor is there any discussion), how this could be biologically implemented. 
- secondly, I feel that the benchmarks the authors' model is compared to are extreme / theoretical cases as opposed to using more empirical/heuristic ideas of how RL metaparameters (including those controlling exploration) should be controlled, e.g. Doya "Metalearning and neuromodulation" (Neural Networks 2002) or Luksys et al. 2009 Nat Neuroscience paper on modelling of stress, which incidentally also doesn't use bandit tasks (whose excessive use the authors rightly criticised). It may well be that while theoretical prototypes such as Boltzmann or Thompson may not do very well in detailed comparison with behavioural data, more empirical approaches, like those in the mentioned papers may do better, despite their simplicity and no need to compute complex quantities like KL. 
questions: For the first weakness, perhaps it would be better to link with more cognitive computational neuroscience papers (especially those linking working memory capacity to reinforcement learning, e.g. work of Collins and Frank), where biological correlates are a lot more straightforward, and at least provide discussion on plausible biological computation of quantities like KL.

For the second weakness, at the very least I would expect some discussion mentioning the possibility that simple models with dynamic control of exploration (and possibly other) metaparameters as opposed to full meta-learning with control of description length may be another feasible (and much simpler, even if not normative) solution to the problem of accurately reproducing behavioural data.

In Fig. 1 interpretation that "Between 100 and 1000 nats, we additionally find minor influences of a Boltzmann-like exploration strategy." I don't really understand what exactly this is based on? How do you define minor influences?

For Fig. 3 I would rather put data from D2 there as opposed to picking 2 "example models". If using choice probability as the only behavioural measure is already quite limiting, then picking 2 matching data points seems even more so, It's also curious that 'lesion group' performance seems to be better (at least in this one measure) than performance with any description length (<100) that best matches Boltzmann sampling. Again I'm very curious how a simple RL algorithm with dynamic metaparameter control or RL + working memory algorithms would perform... 
limitations: Some of them yes, but some others not yet - see suggestions above in 'questions'.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents RL^3: a combination of meta reinforcement learning with information bottlenecks. RL^3 explains human exploration behavior as resource rational -- humans rely on limited description lengths (defined here as the amount of information in the weights of a reinforcement learning algorithm parameterized by a neural network) to balance exploration with exploitation in various multi-armed bandit tasks. By reanalyzing psychological data with both healthy adults, as well as clinical populations and adolescents, the paper shows that exploration behavior can be better explained using these resource rational principles as opposed to classic, hard-coded exploration strategies (such as Thompson sampling, for example).
strengths_and_weaknesses: Strengths
* (Significance) This paper presents a compelling argument for how human exploration is more complex than simply exploiting or exploring. Cognitive scientists in particular will be inspired by this work to think about how to both move towards more complex exploration tasks, and in considering how to further use meta-learning techniques to model richer kinds of cognitive phenomena
* (Originality) Resource rationality itself is not new, but this is a very nice, straightforward application of the idea to demonstrate the flexibilty and variability of human exploration behavior. 
* (Quality) The method is able to capture not just one experiment, but three separate sets of experiments separated by decades, and across exceptionally diverse populations (from lesioned groups, to healthy adults, to adolescents) without requiring fine-tuning or separate hyperparameters.
* (Clarity) The paper is well presented. The motivation is clear (with some minor exceptions outlined below), the method is well described, the results are fairly interpreted, and the conclusions are well laid out given the results.

Weaknesses
* (Significance) While the paper attempts to provide motivation for why these findings may be important to the ML/AI community, I am not entirely convinced of their arguments. In particular, the presented method does not seem obviously applicable to more complex settings that are commonly explored in ML/AI, often involving objects (like in videogames) and models. I think this contribution is important for computational cognitive science, but I am not sure it will have a big impact on the ML/AI audience.
* (Clarity) The paper title is both grandiose and also exceptionally vague. I would ask the authors to choose a paper title that is more indicative of the content of the paper. Key words like “resource rational” and “meta-learning” are some examples that I think could be used to make a more informative title.
* (Clarity) The abstract introduces terms that are not defined until the second page, specifically “description length”. Without context, description length could refer to an exceptionally large number of things, and is therefore quite confusing when mentioned in isolation. I would recommend changing the abstract such that description length is not mentioned, or is mentioned and unpacked in a clause or a single sentence to give the reader an understanding of what this specifically refers to.
* (Clarity) RL^3 seems like a poor method name given that RL^2 was so named since it was using reinforcement learning to train a reinforcement learning algorithm. Since this approach is not doing an extra meta-level of optimization, and is instead limiting the information in the weights of the inner RL algorithm, RL^3 is a very poor choice of name. What about RR-RL^2 for resource rational RL^2? Or IB-RL^2 for information bottleneck RL^2? 
* (Clarity) Figure 1a is somewhat difficult to interpet as a standalone figure. I think I understand it as behaving more like Thompson sampling as the description length is increased, but is that what’s actually happening? Some extra description in the figure caption would really help with the interpretation. (See also my question in the “questions” section below).
* (Clarity) Figure 2a - is RL^3 really at 0 BIC? Is that theoretically possible? Or is the delta BIC with respect to the RL^3 model? If the latter, the caption should be updated accordingly.
* (Quality) Experimental results on development are much less compelling than for the other two experiments. The model is clearly doing something distinctly different from humans in that its base probability of exploration is 0 rather than a constant value. While the authors address some reasoning for why the effect size is lower for the model, there is no discussion of the differences in why the overall likelihood of exploration for people is so much higher. Could the authors comment on this?

questions: * While reasonable, this is a rather limited view of both description length, as well as requiring fairly strong assumptions about the form of the decision-making process underlying explore/exploit. For example, the presented method could not obviously extend to consider object-oriented or model-based exploration, which is likely critical for more ecological/complex tasks than bandit settings. Indeed, much of the work in AI for exploration has focused on these concepts. How do the authors see this method as being applicable to these more complex settings?
* The fit of RL^3 to both healthy adults and the lesioned population is very good, but the fit to the adolescent data is significantly worse. Both the effect size, but also the base rate of exploration, seem to be dramatically different from the human groups. Could the authors speculate on why this fit is not great, and whether there are changes to the model that could be made to improve the fit?
* One major missing piece for me was some intuition on how classic exploration strategies come out of the RL^3 approach. In particular, why is Thompson sampling seemingly the optimal strategy for exploration when a sufficiently high description length is allowed? And why does it require a high description length in the first place? Adding some intuition for this to the paper would help – especially in the first experiment for discussing Figure 1.

limitations: Limitations are adequately addressed by the authors. No ethical limitations to consider, but limitations of the approach are reasonably well described (although see my other comments for other potential important limitations).
review_title: null

final_score: rating: 9: Very Strong Accept: Technically flawless paper with groundbreaking impact on at least one area of AI/ML and excellent impact on multiple areas of AI/ML, with flawless evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper develops a model called "RL3" which learns a free-standing RL algorithm by optimizing a resource-constrained objective function. The key contribution is to show how human-like patterns of exploration arise from the optimized algorithm. Using data from a two-armed Gaussian bandit task, they show that the learned algorithm quantitatively matches human behavioral data, and fits the data better than a number of previously proposed models. Importantly, the learned algorithm produces a mixture of directed, random, and Boltzmann sampling similar to what people produce in the task. Another interesting observation is that the mix of directed/random/Boltzmann changes with the capacity limit. The authors then show that reducing the capacity limit produces behavior similar to what has been observed in ventromedial prefrontal cortex patients. Finally the authors show that they can reproduce the developmental trajectory of directed vs. random exploration.
strengths_and_weaknesses: Strengths:
- Overall, I really liked this paper. it presents an interesting and novel idea, the writing is clear, and the analysis is rigorous.
- The paper is quite rich, addressing data from healthy humans, lesion patients, and developmental trajectories.
- Not only can the model reproduce aspects of human exploration behavior, but the theory sheds light on which phenomena should be observed to emerge at different capacity limits (this should be testable).

Weaknesses:
- The meta-learning setup for the IGT is somewhat contrived. It might have made more sense to use tasks where there is a clearer distribution for meta-learning.
questions: - I understand that space is limited so I don't expect the authors to address this, but I think it would be very interesting to apply this approach to contextual bandits.
- I'm also curious about whether the implication here is that the brain contains many pretrained RL algorithms for different tasks, or a few that generalize across tasks.
limitations: I don't see any potential negative impacts. The authors devote a section of their discussion to limitations, though it's somewhat vague. I would have liked to see more specific predictions for future experimental work.
review_title: null

Meta-review: metareview: This paper presents a resource constrained variant of the RL^2 algorithm, and studies how well it models human exploration behavior in bandit tasks.  The resource constraint is the policy description length (in bits for the learned parameters).  Based on the resource constraint, the algorithm produces a mix of Boltzmann-like and Thompson-sampling exploration.  The space of exploration strategies from the algorithm is compared to human behavior data in three human populations.  The fit to human data is substantially better than many alternatives that have been considered in previous papers.

The reviewers found many strong contributions in this paper.  A particular strength was the ability of the proposed model to fit human behavior data from three substantially different populations in the past literature (xjvi, ). The reviewers also appreciated the clarity of the presentation that mixed ideas from neuroscience and machine learning (xjvi, orHY), and the motivation.  Reviewers raised multiple detailed questions about the approach and results.  The author response addressed each of the concerns in detail, and many reviewer suggestions were incorporated to improve the paper presentation.  No reviewers raised additional concerns after the author discussion, and remaining questions were limited to potential directions for future work.  Reviewers expressed interest in this work and in potential directions for future work that builds on the presented ideas.

Four reviewers indicate to accept this paper for its novel contribution of a better model of human exploration behavior in bandit tasks.  The paper is therefore accepted.
confidence: Certain
Decision: Accept