# Adaptive tuning of human learning and choice variability to unexpected uncertainty

Junseok K. Lee<fnref n="1" />,²*, Marion Rouault<fnref n="1" />,², Valentin Wyart<fnref n="1" />,²,³*

Human value-based decisions are notably variable under uncertainty. This variability is known to arise from two distinct sources: variable choices aimed at exploring available options and imprecise learning of option values due to limited cognitive resources. However, whether these two sources of decision variability are tuned to their specific costs and benefits remains unclear. To address this question, we compared the effects of expected and unexpected uncertainty on decision-making in the same reinforcement learning task. Across two large behavioral datasets, we found that humans choose more variably between options but simultaneously learn less imprecisely their values in response to unexpected uncertainty. Using simulations of learning agents, we demonstrate that these opposite adjustments reflect adaptive tuning of exploration and learning precision to the structure of uncertainty. Together, these findings indicate that humans regulate not only how much they explore uncertain options but also how precisely they learn the values of these options.

Copyright © 2023
The Authors, some
rights reserved;
exclusive licensee
American Association
for the Advancement
of Science. No claim to
original U.S. Government
Works. Distributed
under a Creative
Commons Attribution
License 4.0 (CC BY).

## INTRODUCTION

Human decisions exhibit a pervasive variability under uncertainty (1). In the context of value-based decisions, the source of this variability has classically been assigned to exploration (2–5), i.e., purposeful bias and variance in choice policy aimed at reducing uncertainty about the values of choice options. Beside this well-described source of decision variability, recent work has shown that the computations used to learn option values from obtained rewards suffer from imprecisions because of limited cognitive resources (6–9). Learning imprecisions result in decision variability, which can be mistaken for exploration, but these two sources of decision variability are dissimilar in nature. Exploration drives decision variability through the probabilistic selection of options that do not maximize the expected value, whereas learning imprecisions reflect random noise in the reinforcement learning process that updates option values. In practice, these differences allow decomposing human decision variability into two separate sources through detailed computational modeling of human behavior (6).

Both exploration and imprecise computations entail substantial reward costs. First, by selecting an option that does not maximize the expected value, exploration temporarily foregoes the exploitation of the best available option (10). Second, relying on imprecise computations means that the option with the highest subjective value is less likely to be the objectively best option available. Despite these similar reward costs, the two sources of decision variability have very different cognitive benefits. Exploration reduces uncertainty about option values, whereas imprecise learning reduces demands in terms of cognitive and neural resources (11–13). It is well known that humans arbitrate the "explore-exploit" trade-off under uncertainty in terms of its costs and benefits (3, 4). However, these findings have been obtained using reinforcement learning models, which assign all decision variabilities to

exploration. Whether humans simultaneously regulate learning imprecisions in terms of their specific costs and benefits remains unknown.

The costs and benefits of exploration and learning imprecisions depend on the dominant form of uncertainty in the environment: expected versus unexpected uncertainty (14, 15). Expected uncertainty refers to random stochasticity of rewards associated with a choice option around a constant mean value. By contrast, unexpected uncertainty refers to changes in the mean value of rewards associated with a choice option. Under expected uncertainty (i.e., reward stochasticity), individual rewards become less informative about their mean value as learning progresses, and agents can therefore tolerate low learning rates and little exploration. By contrast, under unexpected uncertainty (i.e., reward volatility), individual rewards are highly informative about changes in their mean value, and agents should therefore maintain high learning rates and frequent exploration of unobserved choice alternatives. There is ample experimental evidence that humans adjust their learning rates and exploration at short time scales depending on the dominant form of uncertainty in their environment (16–19). However, whether this regulation of learning rates and exploration is accompanied by a modulation of learning imprecisions remains unknown.

Here, we developed an experimental framework in which the effects of expected and unexpected uncertainty on decision variability can be compared in the context of the same reinforcement learning task (Fig. 1, A and B; see Materials and Methods). We decomposed the decision variability of two large samples of human participants (a "discovery" dataset and a "replication" dataset; see Materials and Methods) into exploration and learning imprecisions by fitting a noisy reinforcement learning model to their behavior. We found that participants choose more variably between options but learn more precisely their values in response to unexpected uncertainty. By studying individual differences in these opposite adjustments, we show that humans regulate the variability of their decisions based on not one but two cost-benefit trade-offs.

<footnote n="1">¹Laboratoire de Neurosciences Cognitives et Computationnelles, Institut National de la Santé et de la Recherche Médicale (Inserm), Paris, France. ²Département d'Études Cognitives, École Normale Supérieure, Université PSL, Paris, France. ³Institut du Psychotraumatisme de l'Enfant et de l'Adolescent, Conseil Départemental Yvelines et Hauts-de-Seine, Versailles, France.</footnote>

<footnote>*Corresponding author. jun.seok.lee@ens.fr (J.K.L.); valentin.wyart@inserm.fr (V.W.)</footnote>
A Task trial mechanics

<figure><img src="image_1.png" /></figure>

B Task reward schedule

<figure><img src="image_2.png" /></figure>

C Participant and optimal accuracy

<figure><img src="image_3.png" /></figure>

D Participant and optimal switch rate

<figure><img src="image_4.png" /><figcaption>**Fig. 1.** Experimental task properties and behavioral results. (A) Trial modalities. Two options are shown to the participant. Once a choice is made, the reward associated with the option on that trial is displayed. (B) Reward schedule of an option during a block. Colored shapes indicate the reward given for an option. The set of shape options are shown for each block. Darker colored shapes indicate the option chosen on a trial. Vertical lines demarcate the beginning and the end of a block in the reference (Ref) and the stochastic (S+) conditions and a reversal in the volatile (V+) condition. Top: Darker red horizontal lines show the generative mean for the option corresponding to the correct shape in the Ref and S+ conditions. Bottom: Darker lines indicate the generative reward mean for the best option drifting throughout the course of a block in the V+ condition. Shaded areas around lines represent the values of the probability density function from which a reward (shape) was drawn. (C) Left: Average accuracy within each condition. Colored dots represent individual participants' mean accuracy. White dots indicate the median accuracy. Error bars represent the first and third quartiles. Shaded areas indicate first and third quartiles of the optimal model's accuracy. Right: Accuracy over time within each condition. The vertical line represents the start of a new block or a reversal. Solid lines indicate the mean accuracy across participants. Dotted lines indicate the mean accuracy of the optimal model. Shaded areas correspond to the SEM. (D) Left: Average switching rate within each condition. The vertical line represents the start of a new block or a reversal. Colored dots represent individual participants' overall switch rate. White dots indicate the median switch rate. Error bars represent the first and third quartiles. Shaded areas indicate first and third quartiles of the optimal model's switch rate. Right: Proportion of switches over time within each condition. Solid lines indicate the mean switch rate across participants. Dotted lines indicate the mean accuracy of the optimal model. Shaded areas correspond to the SEM. \*\*\*P < 0.001.</figcaption></figure>

RESULTS

Two-armed bandit task design and performance

Adult participants (n = 200 per dataset; see Materials and Methods) played a two-armed bandit task in three conditions (Fig. 1B). The "reference" (Ref) condition consists of short rounds of trials using two choice options associated with reward distributions of fixed means and variances. The "stochastic" (S+) condition differs from the Ref condition in that reward distributions have larger variances, i.e., increased expected uncertainty. The "volatile" (V+) condition

differs from the Ref condition in that it consists of long rounds of trials with reward distributions of changing means, i.e., unexpected uncertainty. Participants were instructed of the structure of uncertainty and relative difficulty of each condition (see Materials and Methods). We describe below the analysis of the first "discovery" dataset of adult participants (n = 154 after exclusion; see Materials and Methods) and point toward the replication of observed effects in the second replication dataset (n= 142 after exclusion; see Materials and Methods).
As expected, participants were more accurate at choosing the option associated with the highest reward mean in the Ref condition than in the other two conditions with increased uncertainty (signed-rank test, S+ minus Ref: $z = -7.1, P < 0.001$; V+ minus Ref: $z = -10.2, P < 0.001$; Fig. 1C). Participants also switched more often between choice options in the S+ and V+ conditions than in the Ref condition (S+ minus Ref: $z = 3.9, P < 0.001$; V+ minus Ref: $z = 8.0, P < 0.001$; Fig. 1D). Comparing the two conditions with increased uncertainty showed that participants switched more often between choice options in the V+ condition (V+ minus S+: $z = 5.3, P < 0.001$). These differences between conditions were replicated in the second dataset (fig. S1).

## Computational model specification

We first sought to compare human decisions to those of an optimal learning agent in the same task conditions. For this purpose, we derived a Kalman filter whose parameters were set to the generative values of each task condition (see Materials and Methods). This optimal learning agent selected on each trial of each round the option associated with the highest estimated value (reward mean). Simulating the behavior of the optimal learning agent confirmed that human decisions were substantially less accurate and more variable than those of the optimal learning agent in all three conditions (Fig. 1, C and D). These differences between human and optimal decisions were replicated in the second dataset (fig. S1).

To capture the suboptimal variability of human decisions, we altered the optimal learning agent with four free parameters (Fig. 2, A and B; see Materials and Methods). First, a learning rate $\alpha$ controls how much estimated option values are updated following each obtained reward. Unlike reinforcement learning models, the learning rate $\alpha$ of a Kalman filter reflects the dominant form of uncertainty assumed by the learning agent, and the effective magnitude of updates varies from trial to trial as a function of uncertainty regarding the current value of the chosen option (see Supplementary Text). Second, a decay rate $\delta$ controls the rate at which the value of the unchosen option regresses toward its previous value, reflecting a decay of unchosen option values in working memory. Third, a learning noise term triggers imprecise updates of estimated option values, controlled by a scaling factor $\zeta$. As in recent work (6), we hypothesized that learning imprecisions follow Weber's law and scale with the magnitude of associated reward prediction errors. Fourth, a choice temperature $\tau$ generates exploration through a "softmax" choice policy (3, 4). Before analyzing fits of this suboptimal learning agent to human decisions, we performed model and parameter recovery analyses (20, 21) to validate that our fitting procedure was capable of identifying the source(s) of decision variability from choice behavior (fig. S2; see Materials and Methods). We also verified that the main findings are robust to the use of a reinforcement learning model instead of a Kalman filter to fit human decisions (see Supplementary Text).

## Model parameter fits in S+ and V+ conditions

We fitted the parameters of the suboptimal learning agent to each participant's behavior in each condition using approximate Bayesian inference (see Materials and Methods). We performed Bayesian model selection to compare the suboptimal learning agent including the two sources of decision variability (learning imprecisions and exploration) with variants including a single source (fig. S3). We found that both sources of decision variability are needed to

fit participants' behavior (exceedance of $P > 0.999$ in each condition). In line with recent work, we validated that learning imprecisions scale with the magnitude of updates following each outcome (comparison between flat and update-scaled imprecisions; exceedance of $P > 0.999$ in each condition). We also found evidence for a significant memory decay of unchosen option values in all three conditions (comparison between $\delta > 0$ and $\delta = 0$; exceedance of $P > 0.999$ in each condition).

As expected, participants had higher learning rates in the V+ condition than in the other two conditions (V+ against Ref: $z = 10.2, P < 0.001$; V+ against S+: $z = 9.8, P < 0.001$; Fig. 3A). Regarding decision variability, participants had lower learning noise in the V+ condition (V+ against Ref: $z = -2.8, P = 0.006$; V+ against S+: $z = -4.4, P < 0.001$), as well as higher choice temperature (V+ against Ref: $z = 4.4, P < 0.001$; V+ against S+: $z = 4.4, P < 0.001$). We found no statistically significant difference between the Ref and S+ conditions in terms of any model parameter (learning rate: $P = 0.437$; decay rate: $P = 0.479$; learning noise: $P = 0.100$; choice temperature: $P = 0.995$). Unlike its optimal counterpart, this suboptimal learning agent was able to capture participants' behavior in terms of accuracy and switch rate, as well as the trajectories of these quantities over the course of each round (Fig. 3, B and C). These effects were replicated in the second dataset (fig. S1).

Learning noise and choice temperature showed large individual differences in each condition. We took advantage of these differences to study how each parameter affects decision variability. We correlated each parameter with decision accuracy and switch rate across participants (Fig. 4A). Participants' accuracy showed significant negative correlations with learning imprecisions and exploration (rank correlation, learning noise: $\rho = -0.18, P < 0.001$; choice temperature: $\rho = -0.38, P < 0.001$). By contrast, participants' switch rates showed a negative relation with learning noise ($\rho = -0.14, P = 0.002$) but showed a strong positive relation with choice temperature ($\rho = 0.71, P < 0.001$). This means that, unlike exploration, learning imprecision did not make participants switch more between the two choice options. Median splits of participants' choice behavior as a function of learning noise (Fig. 4B) or choice temperature (Fig. 4C) confirmed these effects, which were replicated in the second dataset (fig. S3).

## Model parameter covariances across participants

We studied how the large individual differences in decision variability are shared between model parameters and across conditions. For this purpose, we computed the correlation matrix of model parameters across participants (12 parameters = 4 parameters per condition; see Materials and Methods). This matrix revealed significant covariances between model parameters and conditions (Fig. 5A). First, learning rate, learning noise, and choice temperature all showed significant within-parameter correlations between the Ref and the S+ or V+ conditions (Fig. 5C). Furthermore, learning rate correlated negatively with learning noise (rank correlation, $\rho = -0.36, P < 0.001$; Fig. 5D) but correlated positively with choice temperature ($\rho = 0.43, P < 0.001$). We verified that these between-parameter correlations remained significant within each condition and performed additional recovery analyses to validate that neither reflects spurious correlations arising from the fitting procedure (Fig. 5B; see Materials and Methods). These correlations were also replicated in the second dataset (figs. S4 and S5).
**A Model system diagram**

<figure><img src="image_5.png" /></figure>

**B Model dynamics**

<figure><img src="image_6.png" /><figcaption>**Fig. 2.** Details of the noisy Kalman filter (KF) model. (A) System diagram of the noisy KF model. The model tracks the means $x_t$ and variances $v_t$ of option values based on the sampling variance $v_{\text{sampling}}$ and drift variance $v_{\text{drift}}$ of observed rewards $r_t$. The model includes four free parameters, which control the update of option values (α; green box), the corruption of option values by learning noise (ζ; blue box), the decay of unchosen option values to their overall mean (δ; orange box), and the softmax choice policy (τ; purple box). Insets illustrate the effects of the learning noise and choice temperature parameters on the distributions of option value and decision probability, respectively. (B) Dynamics of the model within the task context (simulated on the V+ condition). Left: The trajectory of the estimated reward for each option over time. The chosen options are highlighted with a white border. Right: Detailed view of the model dynamics. Dots correspond to the tracked values in the left panel. Stars correspond to the reward feedback after choice. Smaller dots following the feedback correspond to potential learned values corrupted by learning noise. Distributions around dots correspond to the estimation uncertainty. Note that this estimation uncertainty increases for unchosen options as the tracked value itself decays over time.</figcaption></figure>
A Parameter fits

<figure><img src="image_7.png" /></figure>

B Model behavior - accuracy

<figure><img src="image_8.png" /></figure>

C Model behavior - switch rate

<figure><img src="image_9.png" /><figcaption>**Fig. 3. Summary of model fits to participant data.** (A) Colored dots represent individual participants' fits for each parameter. White dots indicate the median fitted parameter value. Error bars represent the first and third quartiles. (B) Left: Scatterplots of the average accuracy of participants to their best-fitting models in each condition. Right: Accuracy over time within each condition. The vertical line represents the start of a new block or a reversal. Solid lines indicate the mean accuracy across participants. Dashed lines indicate the mean accuracy of the best-fitting model. (C) Left: Scatterplots of the switch rate of participants to their best-fitting models in each condition. Right: Proportion of switches over time within each condition. The vertical line represents the start of a new block or a reversal. Solid lines indicate the mean switch rate across participants. Dotted lines indicate the mean accuracy of the best-fitting model. Shaded areas correspond to the SEM. n.s., not significant. \*\*P < 0.01; \*\*\*P < 0.001.</figcaption></figure>

The positive association of learning noise and learning rate and the negative association of choice temperature and learning rate across participants mirror their within-participant adjustments across task conditions. The decrease in learning noise and the increase in choice temperature are associated with higher learning rate in the V+ condition. However, are these simultaneous adjustments of decision variability distinct or tied to each other? To answer this question, we first measured the partial correlation between learning noise and choice temperature once the relation

of these two quantities with learning rate are partialled out. This analysis indicated no direct relation between the two quantities (partial rank correlation, $\rho = 0.03$, $P = 0.584$). We then measured the relation between adjustments of each source of variability with adjustments of learning rate between conditions. We found that the decrease in learning noise in the V+ condition (V+ minus Ref) correlates with the increase in learning rate ($\rho = -0.22$, $P = 0.007$) but not with the increase in choice temperature ($\rho = -0.03$, $P = 0.739$). Together, these results indicate that the
**A** Behavioral correlations with noise parameters
Learning noise

<figure><img src="image_10.png" /></figure>

<figure><img src="image_11.png" /></figure>

**B** Median split on learning noise

<figure><img src="image_12.png" /></figure>

<figure><img src="image_13.png" /></figure>

**C** Median split on choice temperature

<figure><img src="image_14.png" /></figure>

<figure><img src="image_15.png" /><figcaption>**Fig. 4. Behavioral expressions of learning and decision noise.** (A) Correlations of behavioral measures with learning noise and choice temperature. Learning noise and choice temperature axes are spaced with log scaling. Shaded areas are 95% confidence interval (CI). (B) Participant accuracy and switch rates with respect to learning noise (median split across participants' best-fitting learning noise in each of the three conditions separately). Statistical significance was determined from rank sum tests on median split participants. (C) Participant accuracy and switch rates with respect to choice temperature (median split across participants' best-fitting choice temperature in each of the three conditions separately). Error bars represent interquartile ranges. Upward triangles and solid lines signify the mean of the upper values of the measure on the median split of the parameter. Downward triangles and dashed lines signify the mean of the measure on the lower values of the median split of the parameter. \*P* < 0.05; \*\*P* < 0.01; \*\*\*P* < 0.001.</figcaption></figure>

opposite adjustments of learning noise and choice temperature to
volatility are independent of each other.

**Principal components analysis of model parameters**

To provide additional support for independent adjustments of
learning imprecision and exploration to volatility, we performed a
principal components analysis (PCA) of model parameters (see Ma-
terials and Methods). By construction, principal components (PCs)
are orthogonal to each other and reflect uncorrelated sources of var-
iability in model parameters. We therefore asked whether learning
noise and choice temperature project onto distinct PCs. This anal-
ysis revealed two PCs (PC1 and PC2) that capture covariance in
model parameters better than chance (Fig. 6A).

PC1 reflected the positive correlation between learning rate and
choice temperature and explained more variance in model param-
eters in the Ref and S+ conditions (Fig. 6B, top). PC1 explained sig-
nificantly more variance in choice temperature than learning noise
in all three conditions (Ref: bootstrapped *P* = 0.992, S+: boot-
strapped *P* = 0.983, V+: bootstrapped *P* = 0.963). Behaviorally,
high PC1 scores were associated with lower accuracy and higher
switch rates (see Supplementary Text). By contrast, PC2 (Fig. 6B,
bottom) reflected the negative correlation between learning rate
and learning noise and explained more variance in model parame-
ters in the V+ condition. PC2 explained significantly more variance
in learning noise than choice temperature in all conditions (Ref:
bootstrapped *P* = 0.988, S+: bootstrapped *P* = 0.934, V+: boot-
strapped *P* = 0.989). Behaviorally, PC2 showed a milder relation
A Correlations between parameters

<figure><img src="image_16.png" /></figure>

B Null correlation matrix

<figure><img src="image_17.png" /></figure>

C Covariations between conditions

<figure><img src="image_18.png" /></figure>

D Covariations between parameters

<figure><img src="image_19.png" /><figcaption>**Fig. 5. Model parameter covariations across participants.** (A) Correlation matrix of participants' fitted parameters. (B) Null correlation matrix (correlation structure destroyed) using shuffled parameters for simulations, then fitted using the same procedure. All *P* values in correlation matrices corrected for false discovery rates (α = 0.05). (C) Covariations of the learning rate, learning noise, and choice temperature between the Ref (R) to the S+ and the V+ conditions. Lines in scatterplots indicate the best-fitting regression line (blue: V+ to R; red: S+ to R). Shaded areas are 95% CI. Error bars on bar plots indicate 95% bootstrapped CI on the Spearman's *ρ* value. (D) Correlations of learning noise and choice temperature with learning rate. Regression lines fitted on aggregate data pooling all conditions. Shaded areas are 95% CI. Error bars on bar plots indicate 95% bootstrapped CI on the Spearman's *ρ* value. \* *P* < 0.05; \*\* *P* < 0.01; \*\*\* *P* < 0.001.</figcaption></figure>
**A** Percent variance explained

<figure><img src="image_20.png" /></figure>

**B** Two dimensions of adaptation

<figure><img src="image_21.png" /><figcaption>**Fig. 6. PCA of model parameters.** (A) Percent variance explained up to the first noncontributive PC. Colored dots correspond to the median value of the percent variance explained from the bootstrap procedure. Gray dots are median values of the percent variance explained from PCs of shuffled data. Statistical significance was calculated from one-tailed bootstrap significance tests. (B) Left: Percent variance explained by the first two PCs within each condition. The first PC, dominated by the variation of choice temperature, explains best the parameter adaptations in the nonvolatile conditions. The second PC, dominated by the variation of learning noise, explains best the parameter adaptations in the V+ condition. Top right: Ingredients and coefficients of the first two PCs. Bottom right: Coefficient of determination of each parameter for the PC. All error bars indicate 95% bootstrapped CI. \*\*P < 0.01.</figcaption></figure>

to accuracy and no significant relation to switch rate (see Supplementary Text). These different effects were replicated in the second dataset (fig. S6). The covariance structure of model parameters extracted using PCA confirms that the simultaneous adjustments of learning noise and choice temperature to volatility are largely independent of each other.

## Adaptive regulation of learning and choice variability to uncertainty

Participants choose more variably but learn less imprecisely in the V+ condition as compared to the other two conditions. However, do these opposite adjustments of decision variability reflect the changing costs of learning imprecisions and exploration across conditions? To address this question, we performed theoretical

simulations of noisy reinforcement learning agents to measure the reward costs of learning imprecisions and exploration in each condition.

We computed the marginal reward costs associated with each source of decision variability through simulations of the suboptimal learning agent. We selectively varied the associated model parameter (either the learning noise $\zeta$ or the choice temperature $\tau$) while holding all other model parameters constant and set to their best-fitting values (see Materials and Methods). We measured marginal reward cost $C_x$ as the fraction loss of reward excess (i.e., the difference between obtained and foregone rewards) compared to a learning agent without the corresponding source of variability $x$ ($x = \zeta$ or $\tau$) tested in the same condition (Ref, S+, or V+). This "relative" definition of reward cost $C$ was chosen to measure how much each
source of variability affects the reward that could have been obtained in each condition (from $C_x = 0$ for a cost-free source of variability to $C_x = 100\%$ for a purely random agent).

Marginal reward costs $C_ζ$ and $C_τ$ increased monotonically with learning noise and choice temperature in all conditions (Fig. 7A). However, the same amount of learning noise $ζ$ was associated with larger reward costs $C_ζ$ in the V+ condition than in the other two conditions. By contrast, the same choice temperature $τ$ was associated with smaller reward costs $C_τ$ in the V+ condition. This means that learning noise is effectively more costly in the V+ condition, whereas choice temperature is effectively less costly in the V+ condition.

On the basis of these theoretical simulations, we estimated the marginal reward costs incurred by learning noise ($C_ζ$) and choice temperature ($C_τ$) for each participant and each condition ($n = 296$ across the two datasets; Fig. 7B). We found that $C_ζ$ did not differ between the S+ and V+ conditions (S+: 11.8% [10.5 13.9]; V+: 12.5% [10.9 14.3], median [bootstrapped 95% confidence interval]; $z = 1.0, P = 0.307$). Similar to $C_ζ$, $C_τ$ did not differ between these two conditions (S+: 4.1% [3.2 5.6]; V+: 4.4% [3.4 5.6]; $z = 1.3, P = 0.190$). In other words, the decrease in learning noise and increase in choice temperature in the V+ condition compensate for the larger reward costs of learning noise and the smaller reward costs of choice temperature in this condition. Expressing reward costs in terms of

A Simulated excess loss

<figure><img src="image_22.png" /></figure>

B Excess loss comparison to participants

<figure><img src="image_23.png" /></figure>

<figure><img src="image_24.png" /></figure>

<figure><img src="image_25.png" /></figure>

**Fig. 7.** Adaptive adjustment of model parameters to uncertainty. (A) Marginal reward costs (reward loss, expressed as percentage of maximum obtainable reward rate) associated with each source of decision variability through simulations of the suboptimal learning agent, by varying selectively the associated model parameter (either the learning noise $ζ$ or the choice temperature $τ$) while holding all other model parameters constant and set to their best-fitting values. Top: The same amount of learning noise is associated with a larger marginal reward cost in the V+ condition. Bottom: The same choice temperature is associated with smaller marginal reward cost in the V+ condition. Inset: Marginal reward costs associated with observed amounts of learning noise (top) and choice temperatures (bottom) in each of the three conditions. (B) Simulated versus observed adjustments. Top: Amounts of learning noise associated with the same marginal reward cost (here, 10% reward loss) in the three conditions. Obtained values (left) match the observed learning noise estimates in participants' data (right). Bottom: Choice temperatures associated with the same marginal reward cost (here, 5% reward loss) in the three conditions. Obtained values (left) match the observed choice temperature estimates in participants' data (right).
"joint" reward costs of the two sources of variability $C_{\zeta,\tau}$ (compared to a learning agent without any variability) also results in similar costs across conditions (fig. S7C).

Conversely, the amounts of learning noise $\zeta$ and choice temperature $\tau$ associated with fixed marginal reward costs across conditions show a decrease in $\zeta$ and an increase in $\tau$ in the V+ condition, as in participants (Fig. 7B). In additional theoretical analyses (see Supplementary Text), we considered alternative definitions of reward costs and the possibility that participants optimize learning imprecisions and exploration in terms of a quantitative comparison between the costs (in terms of reward loss) and benefits (in terms of reduced cognitive resources for learning imprecisions and of lower uncertainty for exploration). Together, these theoretical considerations suggest that the opposite adjustments of learning imprecisions and exploration to unexpected uncertainty follow the opposite changes in their reward costs between conditions.

## DISCUSSION

Making variable decisions under uncertainty has not only clear benefits but also significant costs (9, 22). Previous research has provided compelling evidence that part of this variability reflects active and adaptive exploration aimed at reducing uncertainty about choice options (3–5). However, seeking information about uncertain options often means foregoing a rewarding option (10). Furthermore, recent work has shown that a substantial fraction of decision variability arises not only from exploration but also from learning imprecisions (6). Here, we studied how humans adapt these two sources of decision variability to different forms of uncertainty. We obtained converging evidence for an independent tuning of learning imprecisions and exploration to their specific costs and benefits. We discuss below the implications of these findings for existing theories of the explore-exploit trade-off that ignore learning imprecisions (23).

In agreement with existing theories (4, 10, 14, 22), we found that humans adjust the explore-exploit trade-off depending on the dominant form of uncertainty in their environment. Participants made more variable choices in the V+ condition where option values change over the course of a single game. Such "random" exploration is adaptive in this context to monitor possible changes in the values of recently unchosen options (3). Similar adaptations of the explore-exploit trade-off have been described across other task conditions. Humans make little to no exploratory choices when the outcomes of unchosen options are known (6), engage in "directed" exploration in conditions with imbalanced uncertainty across options (5, 22, 24), and use structured exploration in environments with spatially correlated option values (25).

Here, by measuring imprecisions in the reinforcement learning process that updates option values, we found that humans increase learning precision in the V+ condition. This within-participant adjustment of learning precision is resource-efficient: While outcomes from S+ options are weakly informative about their values, outcomes from V+ options are informative about changes in their values. Participants therefore require not only higher learning rates but also more precise updates. In the noisy Kalman filter that we fitted to participants' behavior, the learning rate $\alpha$ reflects the perceived volatility of choice options. We leveraged the large individual differences in learning rate in the V+ condition to validate a second prediction of our resource-efficient account: that individuals

with high learning rates (i.e., who perceive options that are more volatile) should (on average) learn option values more precisely than individuals with low learning rates. This pervasive relation between learning rate and learning precision reveals a second trade-off that shapes human learning and decision-making under uncertainty.

Adjustments of exploration and learning imprecisions are not only distinct in terms of their respective costs and benefits but they also correspond to fundamentally different types of adjustments in the decision process. Optimizing the explore-exploit trade-off consists in tuning the choice policy between available options (4), downstream from the reinforcement learning process which estimates option values based on obtained rewards. By contrast, optimizing the cost-benefit trade-off associated with imprecise computations consists in tempering with the estimation of option values themselves, upstream from the choice policy (6, 9). The observation of opposite adjustments of exploration and learning imprecisions in response to unexpected uncertainty provides empirical evidence that humans can simultaneously and independently regulate how they choose between options and how precisely they learn from choice outcomes.

The tuning of learning imprecisions to uncertainty cannot be described in terms of "efficient coding" theories (7, 26–28). These influential theories explain how noise-free computations can be tuned at long time scales to minimize the impact of external noise on performance. Instead, we show that humans rely on imprecise computations to learn option values and that they regulate this internal noise (6, 9) at short time scales as a function of task demands. This finding is in agreement with the idea that humans optimize the use of their limited cognitive resources in a flexible, context-dependent fashion as a function of task demands (29, 30). The fact that humans regulate learning imprecisions confirm theoretical considerations that describe precise computations as extremely costly in terms of neural resources (13). This finding is also consistent with the observation that reinforcement learning in multidimensional environments relies on selective attention mechanisms that focus learning resources on a subset of learnable option features (31–33).

The fact that humans adapt their exploration and learning imprecisions across conditions does not imply that participants are aware and in control of these adaptations. In our computational model, learning imprecisions corrupt value representations in an implicit fashion, i.e., trial-to-trial learning errors are not explicitly accounted for by increasing the uncertainty about option values. It is well known that humans exhibit partial blindness to internal sources of error (34, 35). In additional control analyses (see Supplementary Text), we have considered variants of our learning agent where the uncertainty triggered by learning imprecisions alters learning rates. The fact that these variants provide a poorer fit to human decisions suggests that learning imprecisions may be regulated in an implicit, nonintentional fashion. By contrast, exploration is thought to reflect a source of decision variability that can be intentionally regulated, particularly in conditions where sources of uncertainty are known (as it is the case in our study). In addition, in our model, exploration corresponds to explicit "non-greedy" decisions that do not maximize the expected value. Nevertheless, further work will be required to determine the extent to which exploration and learning imprecisions are regulated intentionally as a function of uncertainty.
We simulated the marginal reward costs of exploration and learning imprecisions in each condition, expressed in terms of relative loss compared to a learning agent without this source of variability. We found that unexpected uncertainty is associated with lower costs of exploration but higher costs of learning imprecisions than expected uncertainty. These opposite effects of unexpected uncertainty on the costs of exploration and learning imprecisions provide an adaptive account of their opposite adjustments. In agreement with this view, the marginal reward costs of participants' exploration and learning imprecisions were found to be constant across conditions. However, it remains unclear whether participants actively maintain fixed reward costs across conditions, or whether they optimize other cost-benefit trade-offs associated with these two sources of behavioral variability (see Supplementary Text).

If humans regulate (intentionally or not) the variability of their decisions in terms of two distinct cost-benefit trade-offs, then their neurophysiological substrates should be dissociable. Trial-to-trial decision variability has been associated with brain activity in frontopolar (3, 36) and anterior cingulate (37, 38) cortices and has been related to dopaminergic and noradrenergic pathways (39–42). These multiple effects have been linked to adjustments of the explore-exploit trade-off, which is the sole source of decision variability in state-of-the-art models relying on perfectly precise computations. Accounting for learning imprecisions has revealed that the variability of value updates correlates with blood oxygen level dependent (BOLD) activity in the anterior cingulate cortex and with pupil dilation (6). Pupil dilation correlates with the activity of the locus coeruleus (43, 44), a brainstem nucleus with norepinephrine-containing neurons that has bidirectional projections with the anterior cingulate cortex in the primate brain (45). Adjustments of exploration and learning imprecisions could be achieved either by dissociable brain regions and neuromodulatory pathways or by different signals in the same brain region (46). In particular, anterior cingulate activity has been related not only to exploration but also to the cost-benefit trade-off associated with cognitive control (12). Future research should study how exploration and learning imprecisions are simultaneously regulated in the human brain (47).

The delineation of opposite adjustments of exploration and learning imprecisions in response to volatility suggests that models lacking either source may draw incorrect interpretations. Our findings show that humans adjust not only how much they explore uncertain choice options, by arbitrating the explore-exploit trade-off, but also how precisely they learn their values through reinforcement learning. These adjustments may generalize to different tasks and computational models of behavior. Decisions in perceptual categorization tasks suffer from a similar source of imprecisions in the underlying inference problem (48). These categorization tasks can be embedded in V+ conditions (49) where statistical inference models of behavior include a parameter that controls learning rate and another parameter that controls inferential imprecisions (50, 51). In this context that differs widely from the two-armed bandit used in this study, the same negative relation between learning rate and inferential imprecisions may emerge. Together, these results indicate that the precision of cognitive computations acts as an important and flexible constraint on human cognition (9). Existing theories of cognitive control should therefore

be revised to account for imprecise computations and their associated costs and benefits.

## MATERIALS AND METHODS

### Experimental design

Three samples of $n = 200$ English-speaking adults were recruited on Prolific (prolific.co) for inclusion in the two datasets collected for this study (a discovery dataset and a replication dataset; total $n = 400$). We fixed this sample size of $n = 200$ per dataset a priori and calculated that it is sufficiently powered to detect a minimal between-subject correlation of 0.20 with 80% power at a significance threshold $\alpha = 0.05$. We excluded from analyses any participant whose accuracy did not significantly exceed chance-level performance at a one-tailed threshold $\alpha = 0.05$ in any of the three conditions (binomial test against 50% accuracy). This exclusion procedure left $n = 154$ participants (mean age = 29.6 ± 10.0 years; 53 females) in the first dataset and $n = 142$ participants (mean age = 25.4 ± 6.4 years; 72 females) in the second dataset. All participants provided informed consent regarding their participation in the study, which followed guidelines approved by the ethical review committee of the Institut National de la Santé et de la Recherche Médicale (IRB #00003888).

Upon gaining access to the task, participants were prompted to set the task to full-screen before being able to continue. Next, participants were introduced to the mechanics of their responses (e.g., keys for choosing options) and the presentation of the task as a slot machine game. They were instructed on the dynamics of the Ref, S+, and V+ conditions and the difference between these conditions via written prompts on the screen. Participants always played the Ref condition first, and the order of the other two conditions was counterbalanced across participants. Before every condition, participants played a "practice" block to familiarize them with the condition. At the end of each practice block, participants were presented with a debriefing of their performance and the condition with a visual illustration of the trajectory of their decisions and the rewards they obtained. The task was self-paced but had to be completed within 60 min to avoid disengagement. Participants were offered to take breaks between blocks.

### Task generation

The Ref and S+ conditions corresponded to two-armed bandits with fixed reward distributions, drawn between 1 and 99 points. The V+ condition corresponded to a restless two-armed bandit task where the means of the reward distributions associated with each option follow a random walk over time. In both conditions, the mean rewards $\bar{r}_o$ associated with the two options $o \in \{1, 2\}$ were symmetrical (i.e., $\bar{r}_2 = 100 - \bar{r}_1$). Reversals were defined as time points where the mean rewards associated with the two options cross each other.

Task difficulty was matched between the S+ and V+ conditions based on the simulated performance of an optimal learning agent. For each participant, we first generated 2000 random walks of mean rewards following a beta distribution of an initial mean of 50 points and a drifting SD of 5 points. All random walks that traveled below 10 points or above 90 points, or had steps larger than 10 points, were discarded. Excursions below or above 50 points of 8, 12, 16, 20, and 24 steps were chosen to populate the V+ condition. The average reward of each excursion was used as the static mean reward of a
round of trials for the Ref condition. We set the variance of sampled rewards such that 75% of sampled rewards from the better option in the Ref condition were higher than 50 points (and 25% lower than 50 points). We then computed the effective sampling variance ($\bar{v}_s$) and the effective drifting variance ($\bar{v}_d$) of rewards and simulated a standard Kalman filter with greedy choices in the V+ condition to obtain its accuracy (fraction of choices toward the option associated with the largest mean reward). To generate the last S+ condition, we incrementally regressed the static mean reward from the Ref condition toward 50 until the choice accuracy of a Kalman filter matched that of the V+ condition (fig. S8). Participants played two instances of each condition (two blocks of 80 trials for each condition).

Options were depicted as black shapes for the Ref and S+ conditions, and as colored discs in the V+ condition, to help participants distinguishing between stable (Ref and S+) and volatile (V+) conditions. To make sure that participants treated each new round of the Ref and V+ conditions as independent from the previous ones, we presented novel shapes for each new round. After each choice, the chosen option briefly appeared at the center of the screen and was then followed by the number of points obtained from the chosen option (Fig. 1A). Participants earned £3.30 upon successful completion of the task. Participants gained an extra £1.00 as a bonus if they performed significantly better than chance across the entire experiment (α = 0.05, meaning a choice accuracy above 53.75%). Participants who answered a battery of mental health questionnaires performed after the task (not analyzed for this study) earned another £3.00 upon completion.

The visuals and dynamics of the task (i.e., the frontend code) were implemented using jsPsych (version 6.3.0) (52). The backend server was handled using Node.js. Participants' responses to the task and questionnaires were stored in a MySQL database.

## Computational modeling

Following each outcome, the posterior value of the chosen option $x_t^c$ is updated according to a standard Kalman filter corrupted by additive random noise $\epsilon_t$ drawn from a normal distribution of zero mean and SD $\eta_t$

$$x_t^c = x_{t-1}^c + k_t (r_{t-1} - x_{t-1}^c) + \epsilon_t$$

where $k_t$ is the Kalman gain, which depends on the posterior variance of the chosen option $v_{t-1}^c$ and the sampling variance $v_s$ of rewards, which was set to its true effective value (0.0163 for rewards rescaled between 0 and 1) and used as the "scaling parameter" in the model

$$k_t^c = \frac{v_{t-1}^c}{v_{t-1}^c + v_s}$$

The posterior variances of the two options $o \in \{1,2\}$ were updated using the standard equation of the Kalman filter

$$v_t^o = (1 - k_t^o) v_{t-1}^o + v_d$$

where $v_d$ is the drifting variance assumed by the Kalman filter. We fitted $v_d$ in terms of its associated asymptotic Kalman gain $\alpha$ for an option that would be chosen on $n \to \infty$ trials. Following recent findings (6), the SD of the random noise corrupting the update of the posterior value of the chosen option scales as a constant fraction $\zeta$ of the update $|k_t(r_{t-1} - x_{t-1}^c)|$. Posterior variances of the two options

were initialized to the true variance of mean reward values across all conditions (0.0214 for rewards rescaled between 0 and 1).

The posterior value of the unchosen option $x_t^u$ decays toward the baseline value (50 points, i.e., 0.5 for values rescaled between 0 and 1). The rate of this decay is controlled by an exponential decay factor $\delta$

$$x_t^u = x_t^u + \delta (0.5 - x_{t-1}^u)$$

The probability of choosing option 1 $p_t^1$ based on the posterior values of the two options follows the standard softmax policy with choice temperature $\tau$

$$p_t^1 = [1 + e^{-(x_t^1 - x_t^u)/\tau}]^{-1}$$

where $\tau \to 0$ corresponds to a purely greedy argmax policy.

We used sequential Monte Carlo sampling methods to estimate the conditional likelihoods of the responses of each participant given the set of parameter values. Using Bayesian adaptive direct search with 10 random starting points for each parameter $\alpha, \zeta, \delta$, and $\tau$ (53), we first obtained point estimates of best-fitting parameter values. We then used these estimates as starting points for the estimation of their joint posterior distribution using Variational Bayes Monte Carlo (54, 55). Both model fitting algorithms accounted for noise in log-likelihood estimates because of the presence of random learning noise in the Kalman filter updates.

## Computational model validation

To ensure that the inclusion of learning noise (controlled by its Weber fraction $\zeta$) and soft choices (controlled by their temperature $\tau$) were both necessary to fit participants' choices, we performed random-effects Bayesian model selection based on its standard Dirichlet parameterization described in the literature (56). In practice, as in recent work (6), we compared the model including both sources of decision variability with two model variants: one without learning noise where $\zeta = 0$, and one with a purely greedy choice policy where $\tau = 0$. The full model outperformed the other two reduced models (model prevalence of 70% for the full model; exceedance of $P > 0.999$).

Critically, we performed standard model recovery to validate our model simulation and fitting procedure (20, 21). In particular, by simulating the different variances of our Kalman filter model using individual best-fitting parameters, we established that our three candidate models were distinguishable between each other (fig. S2B). Our fitting procedure was not biased in that we were able to successfully recover all three model variances by simulating decisions from each model and then fitted all three models again to this simulated behavior for which the ground-truth model structure is known.

## Statistical analyses

We used Wilcoxon signed-rank tests for the comparison of model-free behavioral variables across conditions (choice accuracy and switch rate). Wilcoxon signed-rank tests were also used for comparisons of best-fitting model parameters across conditions. Repeated-measures analysis of variance (ANOVA) was performed for comparing choice accuracy curves and switch rate curves across conditions. Spearman (rank-based) correlations were used unless noted otherwise. Split-plot tests were used to calculate the effects of median splits with respect to each model parameter on choice
accuracy and switch rate curves. We performed a PCA on standardized (zero mean and unit variance) model parameter values such that, by construction, (i) the variance explained by a PC in a given condition captures shared variance between model parameters and (ii) different PCs are orthogonal to each other and therefore reflect uncorrelated sources of variability in model parameters.

SEs on the Spearman's rank correlation coefficients were generated through bootstrapping by randomly sampled datasets from the original dataset (with identical sample size and replacement) 1000 times and calculating the correlation on the new datasets. The same procedure was done to obtain the SEs of PC coefficients and associated fractions of variance explained. Statistical significance of the fraction of variance explained was obtained by a one-sided comparison of the variance explained by bootstrapped PCs to the variance explained by bootstrapped PCs for shuffled data. A similar procedure was used to compare the fractions of variance of each PC explained by each model parameter.

## Supplementary Materials

**This PDF file includes:**

Supplementary Text

Figs. S1 to S9

References

View/request a protocol for this paper from Bio-protocol.

## REFERENCES AND NOTES

1. V. Wyart, E. Koechlin, Choice variability and suboptimality in uncertain environments. *Curr. Opin. Behav. Sci.* **11**, 109–115 (2016).

2. R. S. Sutton, A. G. Barto, *Reinforcement Learning: An Introduction* (MIT Press, 1998).

3. N. D. Daw, J. P. O'Doherty, P. Dayan, B. Seymour, R. J. Dolan, Cortical substrates for exploratory decisions in humans. *Nature* **441**, 876–879 (2006).

4. J. D. Cohen, S. M. McClure, A. J. Yu, Should I stay or should I go? How the human brain manages the trade-off between exploitation and exploration. *Philos. Trans. R. Soc. Lond. B Biol. Sci.* **362**, 933–942 (2007).

5. R. C. Wilson, A. Geana, J. M. White, E. A. Ludvig, J. D. Cohen, Humans use directed and random exploration to solve the explore-exploit dilemma. *J. Exp. Psychol. Gen.* **143**, 2074–2081 (2014).

6. C. Findling, V. Skvortsova, R. Dromelle, S. Palminteri, V. Wyart, Computational noise in reward-guided learning drives behavioral variability in volatile environments. *Nat. Neurosci.* **22**, 2066–2077 (2019).

7. R. Polanía, M. Woodford, C. C. Ruff, Efficient coding of subjective value. *Nat. Neurosci.* **22**, 134–142 (2019).

8. K. Juechems, J. Balaguer, B. Spitzer, C. Summerfield, Optimal utility and probability functions for agents with finite computational precision. *PNAS* **118**, e2002232118 (2021).

9. C. Findling, V. Wyart, Computation noise in human learning and decision-making: Origin, impact, function. *Curr. Opin. Behav. Sci.* **38**, 124–132 (2021).

10. B. B. Averbeck, Theory of choice in bandit, information sampling and foraging tasks. *PLoS Comput. Biol.* **11**, e1004164 (2015).

11. A. A. Faisal, L. P. J. Selen, D. M. Wolpert, Noise in the nervous system. *Nat. Rev. Neurosci.* **9**, 292–303 (2008).

12. A. Shenhav, M. M. Botvinick, J. D. Cohen, The expected value of control: An integrative theory of anterior cingulate cortex function. *Neuron* **79**, 217–240 (2013).

13. S. Denève, C. K. Machens, Efficient codes and balanced networks. *Nat. Neurosci.* **19**, 375–382 (2016).

14. A. Soltani, A. Izquierdo, Adaptive learning under expected and unexpected uncertainty. *Nat. Rev. Neurosci.* **20**, 635–644 (2019).

15. P. Piray, N. D. Daw, A model for learning based on the joint estimation of stochasticity and volatility. *Nat. Commun.* **12**, 6587 (2021).

16. T. E. J. Behrens, M. W. Woolrich, M. E. Walton, M. F. S. Rushworth, Learning the value of information in an uncertain world. *Nat. Neurosci.* **10**, 1214–1221 (2007).

17. M. Browning, T. E. Behrens, G. Jocham, J. X. O'Reilly, S. J. Bishop, Anxious individuals have difficulty learning the causal statistics of aversive environments. *Nat. Neurosci.* **18**, 590–596 (2015).

18. J. L. Cook, J. C. Swart, M. I. Frobóse, A. O. Diaconescu, D. E. Geurts, H. E. den Ouden, R. Cools, Catecholaminergic modulation of meta-learning. *eLife* **8**, e51439 (2019).

19. P. Piray, N. D. Daw, A simple model for learning in volatile environments. *PLoS Comput. Biol.* **16**, e1007963 (2020).

20. S. Palminteri, V. Wyart, E. Koechlin, The importance of falsification in computational cognitive modeling. *Trends Cogn. Sci.* **21**, 425–433 (2017).

21. R. C. Wilson, A. G. E. Collins, Ten simple rules for the computational modeling of behavioral data. *eLife* **8**, e49547 (2019).

22. R. C. Wilson, E. Bonawitz, V. D. Costa, R. B. Ebitz, Balancing exploration and exploitation with information and randomization. *Curr. Opin. Behav. Sci.* **38**, 49–56 (2021).

23. S. J. Gershman, Deconstructing the human algorithms for exploration. *Cognition* **173**, 34–42 (2018).

24. E. Schulz, R. Bhui, B. C. Love, B. Brier, M. T. Todd, S. J. Gershman, Structured, uncertainty-driven exploration in real-world consumer choice. *Proc. Natl. Acad. Sci. U.S.A.* **116**, 13903–13908 (2019).

25. C. M. Wu, E. Schulz, M. Speekenbrink, J. D. Nelson, B. Meder, Generalization guides human exploration in vast decision spaces. *Nat. Hum. Behav.* **2**, 915–924 (2018).

26. H. B. Barlow, in *Sensory Communication*, W. A. Rosenblith, Ed. (MIT Press, 1961), pp. 217–234.

27. B. B. Averbeck, P. E. Latham, A. Pouget, Neural correlations, population coding and computation. *Nat. Rev. Neurosci.* **7**, 358–366 (2006).

28. X.-X. Wei, A. A. Stocker, A Bayesian observer model constrained by efficient coding can explain "anti-Bayesian" percept. *Nat. Neurosci.* **18**, 1509–1517 (2015).

29. F. Lieder, T. L. Griffiths, Resource-rational analysis: Understanding human cognition as the optimal use of limited computational resources. *Behav. Brain Sci.* **43**, e1 (2019).

30. T. L. Griffiths, Understanding human intelligence through human limitations. *Trends Cogn. Sci.* **24**, 873–883 (2020).

31. Y. Niv, R. Daniel, A. Geana, S. J. Gershman, Y. C. Leong, A. Radulescu, R. C. Wilson, Reinforcement learning in multidimensional environments relies on attention mechanisms. *J. Neurosci.* **35**, 8145–8157 (2015).

32. Y. C. Leong, A. Radulescu, R. Daniel, V. DeWoskin, Y. Niv, Dynamic interaction between reinforcement learning and attention in multidimensional environments. *Neuron* **93**, 451–463 (2017).

33. A. Radulescu, Y. S. Shin, Y. Niv, Human representation learning. *Annu. Rev. Neurosci.* **44**, 253–273 (2021).

34. D. Rahnev, B. Maniscalco, T. Graves, E. Huang, F. P. de Lange, H. Lau, Attention induces conservative subjective biases in visual perception. *Nat. Neurosci.* **14**, 1513–1515 (2011).

35. S. Herce Castañón, R. Moran, J. Ding, T. Egner, D. Bang, C. Summerfield, Human noise blindness drives suboptimal cognitive inference. *Nat. Commun.* **10**, 1719 (2019).

36. A. Raja Beharelle, R. Polanía, T. A. Hare, C. C. Ruff, Transcranial stimulation over frontopolar cortex elucidates the choice attributes and neural mechanisms used to resolve exploration-exploitation trade-offs. *J. Neurosci.* **35**, 14544–14556 (2015).

37. M. Donoso, A. G. E. Collins, E. Koechlin, Foundations of human reasoning in the prefrontal cortex. *Science* **344**, 1481–1486 (2014).

38. T. C. Blanchard, S. J. Gershman, Pure correlates of exploration and exploitation in the human brain. *Cogn. Affect. Behav. Neurosci.* **18**, 117–126 (2018).

39. M. J. Frank, B. B. Doll, J. Oas-Terpstra, F. Moreno, Prefrontal and striatal dopaminergic genes predict individual differences in exploration and exploitation. *Nat. Neurosci.* **12**, 1062–1068 (2009).

40. M. Jepma, S. Nieuwenhuis, Pupil diameter predicts changes in the exploration-exploitation trade-off: Evidence for the adaptive gain theory. *J. Cogn. Neurosci.* **23**, 1587–1596 (2011).

41. M. Dubois, J. Habicht, J. Michely, R. Moran, R. J. Dolan, T. U. Hauser, Human complex exploration strategies are enriched by noradrenaline-modulated heuristics. *eLife* **10**, e59907 (2021).

42. T. Pfeffer, A. Ponce-Alvarez, K. Tsetsos, T. Meindertsma, C. J. Gahnström, R. L. van den Brink, G. Nolte, A. K. Engel, G. Deco, T. H. Donner, Circuit mechanisms for the chemical modulation of cortex-wide network interactions and behavioral variability. *Sci. Adv.* **7**, eabf5620 (2021).

43. M. Usher, J. D. Cohen, D. Servan-Schreiber, J. Rajkowski, G. Aston-Jones, The role of locus coeruleus in the regulation of cognitive performance. *Science* **283**, 549–554 (1999).

44. S. Joshi, Y. Li, R. M. Kalwani, J. I. Gold, Relationships between pupil diameter and neuronal activity in the locus coeruleus, colliculi, and cingulate cortex. *Neuron* **89**, 221–234 (2016).

45. S. Joshi, J. I. Gold, Pupil size as a window on neural substrates of cognition. *Trends Cogn. Sci.* **24**, 466–480 (2020).
46. S. Joshi, J. I. Gold, Context-dependent relationships between locus coeruleus firing patterns and coordinated neural activity in the anterior cingulate cortex. *eLife* **11**, e63490 (2022).

47. A. G. E. Collins, J. Cockburn, Beyond dichotomies in reinforcement learning. *Nat. Rev. Neurosci.* **21**, 576–586 (2020).

48. J. Drugowitsch, V. Wyart, A.-D. Devauchelle, E. Koechlin, Computational precision of mental inference as critical source of human choice suboptimality. *Neuron* **92**, 1398–1411 (2016).

49. C. M. Glaze, J. W. Kable, J. I. Gold, Normative evidence accumulation in unpredictable environments. *eLife* **4**, 08825 (2015).

50. A. Weiss, V. Chambon, J. K. Lee, J. Drugowitsch, V. Wyart, Interacting with volatile environments stabilizes hidden-state inference and its brain signatures. *Nat. Commun.* **12**, 2228 (2021).

51. M. Rouault, A. Weiss, J. K. Lee, J. Drugowitsch, V. Chambon, V. Wyart, Controllability boosts neural and cognitive signatures of changes-of-mind in uncertain environments. *eLife* **11**, e75038 (2022).

52. J. R. de Leeuw, jsPsych: A JavaScript library for creating behavioral experiments in a Web browser. *Behav. Res. Methods* **47**, 1–12 (2015).

53. L. Acerbi, W. J. Ma, Practical Bayesian optimization for model fitting with Bayesian adaptive direct search. arXiv:1705.04405 [stat.ML] (11 May 2017).

54. L. Acerbi, Variational Bayesian Monte Carlo, in *Proceedings of the 32nd Conference on Neural Information Processing Systems (NeurIPS)* (Curran Associates Inc., 2018), pp. 8213–8223.

55. L. Acerbi, Variational Bayesian Monte Carlo with noisy likelihoods. arXiv:2006.08655 [stat.ML] (15 June 2020).

56. K. E. Stephan, W. D. Penny, J. Daunizeau, R. J. Moran, K. J. Friston, Bayesian model selection for group studies. *Neuroimage* **46**, 1004–1017 (2009).

57. R. A. Rescorla, A. R. Wagner, A theory of Pavlovian conditioning: Variations in the effectiveness of reinforcement and nonreinforcement, in *Classical Conditioning II*, A. H. Black, W. F. Prokasy, Eds. (Appleton-Century-Crofts, 1972), pp. 64–99.

58. M. Speekenbrink, E. Konstantinidis, Uncertainty and exploration in a restless bandit problem. *Top. Cogn. Sci.* **7**, 351–367 (2015).

59. K. L. Louie, B. De Martino, The neurobiology of context-dependent valuation and choice, in *Neuroeconomics* (Academic Press, ed. 2, 2014), pp. 455–476.

60. S. Palminteri, M. Lebreton, Context-dependent outcome encoding in human reinforcement learning. *Curr. Opin. Behav. Sci.* **41**, 144–151 (2021).

## Acknowledgments

**Funding:** This work was supported by a starting grant from the European Research Council (ERC-StG759341) awarded to V.W. and by a department-wide grant from the Agence Nationale de la Recherche (ANR-17-EURE-0017, EUR FrontCog). M.R. is the beneficiary of a postdoctoral fellowship from the AXA Research Fund. M.R. is supported by La Fondation des Treilles. The funders had no role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript. **Author contributions:** J.K.L. designed the experiments, conducted the experiments, analyzed the data, and wrote the paper. M.R. provided feedback on data analyses and wrote the paper. V.W. designed the experiments, analyzed the data, wrote the paper, and supervised the study. **Competing interests:** The authors declare that they have no competing interests. **Data and materials availability:** All data needed to evaluate the conclusions in the paper are present in the paper and/or the Supplementary Materials. The datasets collected during the study, as well as the code developed to analyze the collected data, are available at https://doi.org/10.5281/zenodo.7624899.

Submitted 17 May 2022

Accepted 28 February 2023

Published 29 March 2023

10.1126/sciadv.add0501