# Compulsivity is linked to maladaptive choice variability but unaltered reinforcement learning under uncertainty

Junseok K Lee, Marion Rouault, Valentin Wyart

► To cite this version:

Junseok K Lee, Marion Rouault, Valentin Wyart. Compulsivity is linked to maladaptive choice variability but unaltered reinforcement learning under uncertainty. Nature Mental Health, 2025, 3 (2), pp.229-241. 10.1038/s44220-024-00364-5. hal-05379190

HAL Id: hal-05379190

https://hal.science/hal-05379190v1

Submitted on 24 Nov 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Compulsivity is linked to maladaptive choice variability but unaltered reinforcement learning under uncertainty

Junseok K. Lee<fnref n="1" />,<fnref n="2" />,*, Marion Rouault<fnref n="1" />,<fnref n="2" />,<fnref n="3" />, and Valentin Wyart<fnref n="1" />,<fnref n="2" />,<fnref n="4" />,∗

Compulsivity has been associated with variable behavior under uncertainty. However, previous work has not distinguished between two main sources of behavioral variability: the stochastic selection of choice options that do not maximize expected reward (choice variability), and random noise in the reinforcement learning process that updates option values from choice outcomes (learning variability). Here we studied the relation between dimensional compulsivity and behavioral variability, using a computational model which dissociates its two sources. We found that compulsivity is associated with more frequent switches between options, triggered by increased choice variability but no change in learning variability. This effect of compulsivity on the 'trait' component of choice variability is observed even in conditions where this source of behavioral variability yields no cognitive benefits. These findings indicate that compulsive individuals make variable and maladaptive choices under uncertainty, but do not hold degraded representations of option values.

Pages: 20; Figures: 6; Supplementary Figures: 5

Word count – main text: 2,914 (excluding abstract, methods, references and figure captions)

Word count – methods: 2,335

Keywords: learning; decision-making; uncertainty; compulsivity; volatility; computational modeling

<footnote n="1">¹ Département d'Études Cognitives, École Normale Supérieure, Université PSL, Paris, France</footnote>

<footnote n="2">² Laboratoire de Neurosciences Cognitives et Computationnelles, Institut National de la Santé et de la Recherche Médicale (Inserm), Paris, France</footnote>

<footnote n="3">³ Institut Jean Nicod, Centre National de la Recherche Scientifique (CNRS), Paris, France</footnote>

<footnote n="4">⁴ Institut du Psychotraumatisme de l'Enfant et de l'Adolescent, Conseil Départemental Yvelines et Hauts-de-Seine, Versailles, France</footnote>

<footnote>* corresponding authors (jun.seok.lee@ens.fr, valentin.wyart@inserm.fr)</footnote>
Compulsivity has been associated with shifts in decision-making under uncertainty¹. Patients with clinical obsessive-compulsive disorder (OCD) have been shown to need more sensory evidence to commit to a perceptual decision, reflected in increased decision thresholds but no changes in evidence accumulation²,³. Similarly, OCD patients show deficits in the reporting but not the learning of uncertain stimulus statistics⁴. These findings have recently been shown to extend to subclinical fluctuations of self-reported ‘compulsivity and intrusive thoughts’ symptoms in the general population⁵,⁶. OCD and schizophrenia, both of which are characterized by intrusive thoughts, are associated across studies with altered decision confidence⁷. During the learning of uncertain options values through trial-and-error – also known as reinforcement learning (RL), compulsivity has been linked to decision variability, but also the reliance on less complex decision policies⁸⁻¹¹.

However, these latter findings have been obtained using RL models that assign all decision variability to choice stochasticity. This standard assumption neglects an important source of decision variability: random noise in the RL process that updates option values from choice outcomes¹². This ‘learning noise’ accounts for a large fraction of decision variability otherwise incorrectly assigned to choice stochasticity¹³. It remains largely unclear whether compulsivity is specifically linked to choice stochasticity, learning noise, or both.

Furthermore, the decision variability observed during reinforcement learning can either be adaptive or maladaptive depending on the volatility of option values¹⁴,¹⁵. When option values change unpredictably over time, variable decisions trigger the exploration of recently unchosen options whose values have become more uncertain¹³,¹⁶. By contrast, when option values are stable, variable decisions result in unwarranted choices away from the highest rewarded option. Whether compulsivity is linked to decision variability selectively in volatile environments (where such variability is adaptive) or also in stable environments (where such variability is maladaptive) remains unknown.

Here, we studied human behavior in canonical reinforcement problems (two-armed bandits) involving either stable or volatile option values¹⁷ (Fig. 1a). We fitted a RL model incorporating both learning noise and choice stochasticity to human decisions to decompose participants’ decision variability into their learning- and choice-based sources¹³,¹⁷ (Fig. 2a). We also measured individual differences in a self-reported transdiagnostic dimension of compulsivity in the same participants⁶,⁸. Across two independent datasets, we obtained replicable evidence that compulsivity is associated with increased choice stochasticity without changes in learning noise. Using a test-retest reliability approach, we further quantified the effect of compulsivity on the ‘trait’ (test-retest reliable) component of choice stochasticity. This effect is present even when option values are stable and choice stochasticity yields no cognitive benefits in terms of exploration.
# Results

## Experimental task

We tested the behavior of participants in two-armed bandit problems involving choice options associated with uncertain values¹⁷ (Fig. 1a; see Methods). In the ‘reference’ (Ref) condition, participants played short rounds of trials with two choice options associated with stable reward distributions. In the ‘stochastic’ (S+) condition, reward distributions had larger variances than the ones used in the reference condition. In the ‘volatile’ (V+) condition, instead of playing short rounds of trials with stable reward distributions, participants played long rounds of trials with choice options associated with drifting reward distributions.

<figure><img src="image_2.png" /><figcaption>b task structure</figcaption></figure>

<figure><img src="image_3.png" /><figcaption>c task behavior</figcaption></figure>

<figure><img src="image_4.png" /></figure>

**Figure 1.** Experimental task and participants' behavior (first dataset: *N* = 137). (a) Trial description. On each trial, participants were asked to choose between two options. Their chosen option was highlighted, and were then presented with the reward associated with the chosen option in the center of the screen. (b) Task structure. Reward schedule of an option during a block. Colored shapes indicate the reward given for an option. The set of shape options are shown for each round. Bold colored shapes indicate the option chosen on a trial. Vertical lines demarcate the explicit beginning and the end of a round in the Reference (Ref) and the Stochastic (S+) conditions and a covert reversal in the Volatile (V+) condition. Upper: Bold colored horizontal lines show the generative mean for the option corresponding to the correct shape in the Reference and Stochastic conditions. Lower: Bold lines indicate the generative reward mean for the best option drifting throughout the course of a block in the Volatile condition. (c) Left: Average accuracy and average switch rate in each condition. Colored dots represent individual participants' mean accuracy and overall switch rate respectively (*N* = 137).
White dots indicate the median accuracy and median switch rate respectively. Error bars represent the 1st and 3rd quartiles. Right: Accuracy and proportion of switch decisions as a function of trial position in each condition. Solid lines indicate the mean accuracy and mean switch rate across participants. Shaded areas represent s.e.m. The vertical line represents the start of a new block (Ref, S+) or a reversal (V+).

Participants were less accurate at choosing the option associated with the highest reward mean in the S+ and V+ conditions compared to the Ref condition (Fig. 1b; signed-rank test, S+ vs Ref: $z = -6.5, p < 0.001$; V+ vs Ref: $z = -9.5, p < 0.001$). Participants also switched more between choice options in these two conditions (S+ vs Ref: $z = 3.8, p < 0.001$; V+ vs Ref: $z = 7.6, p < 0.001$), and particularly so in the volatile condition (V+ vs S+: $z = 5.1, p < 0.001$). These differences between conditions were replicated in the second dataset (Supplementary Fig. 1a).

a. sources of behavioral variability in the reinforcement learning model

<figure><img src="image_5.png" /></figure>

b. model comparison

<figure><img src="image_6.png" /></figure>

best-fitting model parameters

<figure><img src="image_7.png" /></figure>

**Figure 2. Computational model and participants’ best-fitting parameters.** (a) Sources of behavioral variability in the reinforcement learning model. Learning noise corrupts the precision with which participants updated the learnt option values. Choice temperature controls the stochasticity of the softmax choice policy. (b) Left: Model comparison favors a model with both learning noise and choice stochasticity (left bar), with high prevalence across tested participants. Models lacking either source of decision variability (middle and right bars) provide poorer fits to human behavior. Right: best-fitting individual model parameters in each of the three conditions. White dots indicate the median parameter value and error bars represent the 1st and 3rd quartiles. Circles display individual data points (first dataset, *N* = 137 participants). \***p<0.001, \*\*p<0.01, *n.s.*: not significant, Wilcoxon signed-rank tests.

Computational model of decision variability

We compared human behavior to a reward-maximizing agent (see Methods). Human behavior was substantially less accurate and more variable than the reward-maximizing agent, in all conditions (Supplementary Fig. 2). To characterize this variability, we fitted a RL model<sup>13,17</sup> decomposing variability into two sources (Fig. 2a): 1. variability arising from random noise in the update of option
values from choice outcomes (learning noise), and 2. variability arising from the stochastic selection of choice options which do not maximize value (choice temperature). We validated this model for the characterization of variability through ‘simulation-recovery’ analyses (see Methods, Supplementary Fig. 3).

Bayesian model selection (see Methods) indicated that both sources contributed to decision variability across conditions (Fig. 2b; exceedance *p* > 0.999). Participants used higher learning rates in the V+ (volatile) condition (Fig. 2c; signed-rank test, V+ vs Ref: *z* = 9.8, *p* < 0.001), but also lower learning noise (V+ vs Ref: *z* = -3.1, *p* < 0.001) and increased choice temperature (V+ vs Ref: *z* = 4.4, *p* < 0.001). By contrast, these three parameters did not differ between the S+ (stochastic) and the reference condition. These selective differences between conditions are adaptive in the sense that the V+ condition requires faster learning and more frequent exploration of choice options due to the drifting values of choice options.

## Dimensional analysis of mental health symptoms

To study the relation between compulsivity and task behavior, we also administered a battery of nine self-report questionnaires assessing a large range of mental health symptoms known to fluctuate across the general population<sup>6,8</sup> (first dataset: Fig. 3; second dataset: Supplementary Fig. 4). Participants’ mental health profiles varied within and across questionnaires, with most participants scoring within the normal range of symptoms (Fig. 3a).

Due to the strong correlations between questionnaire-specific scores, we employed a previously validated dimensional approach to the analysis of item-specific symptom scores<sup>8</sup>. Using a factor analysis (see Methods), we extracted three symptom dimensions from the covariance structure of responses to the 209 questionnaire items (Fig. 3b): an ‘anxious-depression’ (AD) dimension, a ‘compulsivity and intrusive thoughts’ (CIT) dimension, and a ‘social withdrawal’ (SW) dimension.

We also collected age and gender, along with a measure of general cognitive ability using the ICAR-16 test<sup>18,19</sup>, used as control variables in analyses of the relation between symptom dimensions and task behavior (Fig. 3c).

## Associations between symptom dimensions and decision variability

We first regressed accuracy and switch rate, averaged across task conditions, against the three symptom dimensions and control variables (Fig. 4a; see Methods). We found that accuracy was associated negatively with CIT (β = -0.255 ± 0.010, *p* = 0.011) and positively with ICAR-16 (β = 0.205 ± 0.088, *p* = 0.021). By contrast, switch rate was associated positively with CIT (β = 0.235 ± 0.099, *p* = 0.019) and negatively with ICAR-16 (β = -0.243 ± 0.087, *p* = 0.006). These results mean that CIT correlates negatively with accuracy (rank correlation, *ρ* = -0.230, *p* = 0.007) and positively with switch rate (*ρ* = 0.220, *p* = 0.010). These effects of CIT were replicated in the second dataset (Fig. 4b; accuracy: replication Bayes Factor (BF) = 9.44; switch rate: replication BF = 7.05). Neither accuracy nor switch rate was associated with AD or SW scores.
<figure><img src="image_8.png" /><figcaption>**b** factor loadings for symptom dimensions</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>**c** demographics</figcaption></figure>

<figure><img src="image_10.png" /></figure>

**Figure 3.** Inter-individual variability in symptom dimension scores and demographics variables (first dataset, *N* = 137 participants). (a) Histograms showing distributions across questionnaire scores in our sample. We evaluated depression using the Self-Rating Depression Scale (SDS), trait anxiety questionnaire using the State Trait Anxiety Inventory (STAI), obsessive-compulsive disorder (OCD) using the Obsessive-Compulsive Inventory-Revised (OCI-R), social anxiety using the Liebowitz Social Anxiety Scale (LSAS), impulsivity using the Barratt Impulsiveness Scale (BIS-11), schizotypy using the Short Scales for Measuring Schizotypy (SSMS), alcohol misuse using the Alcohol Use Disorders Identification Test (AUDIT), eating attitudes using the Eating Attitudes Test (EAT-26), and apathy using the Apathy Evaluation Scale (AES). We also estimated cognitive ability with the ICAR test (16 items). In all histograms, the horizontal axis is bounded between minimum and maximum score values for each questionnaire. (b) Factor loadings for each questionnaire item on each symptom dimension: Anxious-Depression (AD, top), Compulsive Behavior and Intrusive Thoughts (CIT, middle), and Social Withdrawal (SW, bottom). The right panels display the distribution of scores on each dimension across participants. (c) Participant demographics. Distributions of gender (top) and age (bottom).

We then regressed learning noise and choice temperature against the three symptom dimensions and control variables (Fig. 5a). We found no association between learning noise and any symptom dimensions, but a positive association between choice temperature and CIT ($\beta$ = 0.291 ± 0.096, *p*
= 0.003) and a negative association between choice temperature and ICAR-16 ($\beta$ = -0.202 ± 0.085, $p$ = 0.019). This result means that CIT correlates positively with choice temperature ($\rho$ = 0.276, $p$ = 0.001), but not with learning noise ($\rho$ = 0.108, $p$ = 0.208). This selective association between CIT and choice temperature was replicated in the second dataset (Fig. 5b; replication BF = 9.8).

<figure><img src="image_11.png" /><figcaption>a relation between behavioral measures and symptom dimensions in the first dataset</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>b relation between behavioral measures and symptom dimensions in the second dataset</figcaption></figure>

<figure><img src="image_13.png" /></figure>

<figure><img src="image_14.png" /></figure>

**Figure 4. Relation between behavioral measures and symptom dimension scores.** (a) First dataset: *N* = 137 participants. Top: regression analyses examining the associations between accuracy (left) and switch rate (right) as a function of AD, CIT, and SW scores along with a number of covariates among which cognitive ability (ICAR-16). Dots display regression coefficients and error bars are 95% CIs. \* *p*<0.05, \*\* *p*<0.01, statistical significance of regression coefficients. Bottom: correlation between CIT score and accuracy (left), and switch rate (right). Shaded area indicates the 95% CIs of the best fitting regression line. Circles indicate individual data points for each of the conditions (three data points per participant, green: Ref, red: S+, blue: V+). (b) Second dataset: *N* = 123 participants. Same analyses as in (a). Replication Bayes factor are shown for all effects that were significant in regression analyses in the first dataset.

As a control analysis, we regressed accuracy, switch rate and choice temperature against each of the nine questionnaire-specific scores (Supplementary Fig. 5; see Methods). Out of all questionnaires, we found that all these measures of variability were associated selectively with only the OCI-R score reflecting obsessive-compulsive symptoms²⁰ (accuracy: $p$ = 0.006; switch rate: $p$ = 0.011; choice temperature: $p$ = 0.001). Notably, no significant association was observed with the questionnaire-specific score measuring schizotypy in the general population²¹, despite the significant contribution of its items to the CIT dimension (Fig. 3b).

The positive association between CIT and choice temperature may either be adaptive if it is selective of the volatile condition where choice variability usefully drives exploration of choice options (account #1), or maladaptive if it is present even in the stochastic condition with stable option values (account #2). In line with account #2, we found that CIT correlated positively with choice temperature in both conditions (Fig. 5a; V+: $p$ = 0.208, $p$ = 0.015; S+: $p$ = 0.293, $p$ < 0.001). This pattern confers decisive evidence for a maladaptive account of this association (BF$_{v0}$ = 258.0; see Methods). This maladaptive association between CIT and choice temperature in the stochastic condition was replicated in the second dataset (Fig. 5b; BF$_{v0}$ = 31.9).
Although the positive association between CIT and choice temperature is maladaptive, we found no association between CIT and the magnitude of changes in learning rate, learning noise and choice temperature in the volatile condition (learning rate: *p* = 0.916, BF<sub>1/0</sub> = 0.184; learning noise: *p* = 0.481, BF<sub>1/0</sub> = 0.230; choice temperature: *p* = 0.748, BF<sub>1/0</sub> = 0.192). This means that high CIT scores were associated with increased choice temperature, but not with a decreased adaptation of learning noise or choice temperature to volatile options.

a relation between model parameters and symptom dimensions in the first dataset

<figure><img src="image_15.png" /></figure>

b relation between model parameters and symptom dimensions in the second dataset

<figure><img src="image_16.png" /></figure>

**Figure 5. Relation between model parameters and symptom dimension scores (a) First dataset: N = 137 participants. Top: regression analyses examining the associations between learning noise and choice temperature as a function of AD, CIT, and SW scores with a number of covariates among which cognitive ability (ICAR-16). Dots display regression coefficients and error bars are 95% CIs. \*p<0.05, \*\*p<0.01, \*\*\*p<0.001, statistical significance of regression coefficients. Middle: correlations between CIT score and learning noise (left), and choice temperature (right). Shaded area indicates the 95% CIs of the best-fitting regression line. Circles indicate individual data points for each of the conditions (three data points per participant, green: Ref, red: S+, blue: V+). Bottom: correlation between CIT and learning noise (left), and choice temperature (right), separately in each of the three conditions. Bar and error bars represent median Spearman coefficient and its 95% CI obtained using bootstrapping. \*\*\*p<0.001, \*p<0.05, n.s.: not significant. (b) Second dataset: N = 123 participants. Same analyses as in (a). Replication Bayes factors are shown for all effects that were significant in regression analyses in the first dataset.**
## Trait-state and mediation analyses of decision variability

Despite its statistical significance, the positive association between CIT and choice temperature only explains a limited fraction of individual differences in choice temperature (explained variance = 8.4%). However, the meaning of this association strongly depends on two considerations. The first consideration is the trait-state structure of choice temperature. The CIT dimension, like the self-report questionnaires from which it is constructed, is thought to reflect compulsive behaviors in everyday life – a dominantly ‘trait’ measure. By contrast, choice temperature has been observed to vary at much faster time scales (e.g., between conditions in our task), and is therefore likely to include a substantial ‘state’ component. This state component decreases by nature the observed association between CIT and choice temperature.

<figure><img src="image_17.png" /><figcaption>**b** trait-state decomposition</figcaption></figure>

<figure><img src="image_18.png" /></figure>

<figure><img src="image_19.png" /></figure>

**Figure 6.** Trait-state and mediation analyses of decision variability. (a) Test-retest reliabilities of the two behavioral measures (accuracy, switch rate) and the two model parameters (learning noise, choice temperature) evaluated in the same participants using intraclass correlations (ICC) (N = 100 participants from the first dataset). Top: reliability of accuracy, switch rate, learning noise and choice temperature (averaged across conditions) between the test (x-axis) and retest (y-axis) sessions. Bottom: breakdown of ICCs for each of the three conditions (S+, Ref, V+). The black dot indicates the ICC for condition-averaged behavioral
measures and model parameters. (b) Trait-state decomposition of learning noise and choice temperature. The trait component is defined as the variance in each model parameter that is shared across test and retest sessions. We assume that CIT is associated only with the trait component of each model parameter. (c) Mediation analysis comparing the indirect effect of CIT on switch rate through choice temperature (black paths a and b) against the direct effect of CIT on switch rate (grey path). Choice temperature significantly mediates the effect of CIT on switch rate.

To unpack individual differences in choice temperature into quantifiable trait and state components, we conducted a test-retest analysis in a subgroup of participants (N = 100 from the first dataset; see Methods). Accuracy, switch rate, learning noise and choice temperature all showed fair-to-good reliability (Fig. 6a; accuracy: intra-class correlation (ICC) = 0.441, *p* < 0.001; switch rate: ICC = 0.709, *p* < 0.001; learning noise: ICC = 0.468, *p* < 0.001; choice temperature: ICC = 0.718, *p* < 0.001). These results mean that the trait (i.e., test-retest reliable) component of learning noise explains about a fourth (22.2%) of its observed variance. In contrast, the trait component of choice temperature explains about half (52.0%) of its observed variance (Fig. 6b). Assuming that CIT reflects trait compulsivity and can therefore only explain the trait component of choice temperature, this means that CIT explains 8.4% of the overall variance in choice temperature, but a sixth (16.5%) of its trait component (Fig. 6b). By contrast, CIT explains less than 5% of the trait component of learning noise.

The second consideration that constrains the significance of the association between CIT and choice temperature is the extent to which this association mediates the relation between CIT and observed behavioral variability (i.e., the fraction of switches between the two choice options). We reasoned that the association between CIT and choice temperature is only meaningful if it explains a significant fraction of the observed increase in switch rate with CIT. A mediation analysis<sup>22,23</sup> (see Methods) confirmed this prediction. Indeed, we found a significant mediation effect of choice temperature on the relation between CIT and switch rate (Fig. 6c; *β* = 0.154 ± 0.053, *p* = 0.004), together with a weak direct effect of CIT on switch rate (*β* = 0.066 ± 0.035, *p* = 0.066) – down from a strong total effect (*β* = 0.219 ± 0.062, *p* < 0.001).
Discussion

Compulsivity has been linked to a broad range of shifts in decision-making under uncertainty¹. Compulsive individuals typically show indecisiveness when decision commitment can be postponed³,⁵, but also increased decision variability⁴ and impoverished decision policies in hierarchical environments¹⁰,¹¹. These effects may arise from shifts in how compulsive individuals choose between uncertain options. However, these effects may also result from upstream changes in the learning of option values from variable outcomes. Here, by decomposing behavioral variability into its learning and choice components, we found that under uncertainty, compulsive individuals make variable and maladaptive choices, but do not learn the values of choice options less precisely than others.

Our findings confirm that compulsive individuals make less accurate decisions and switch more often between uncertain choice options than other individuals. However, we show that compulsive individuals do not learn the values of these choice options less precisely than others. Rather, they differ from others in that they choose more variably than others – a finding which extends recent reports by accounting for the presence of learning imprecisions³,²⁴. This selective association between compulsivity and choice stochasticity is consistent with the observation that compulsivity is more strongly related to individual differences in executive than cognitive parameters⁶. Indeed, the stochasticity of participants’ choices is arguably more dependent on executive control²⁵ than on the learning and representation of option values during reinforcement learning. This pattern of findings is also in line with previous work showing that clinically-diagnosed OCD patients have difficulty with decision-making despite holding accurate representations of task variables and intact confidence estimates⁴.

Using a test-retest approach, we made use of the empirical reliability of model parameters to decompose learning noise and choice temperature into state and trait components. This analysis showed that individual differences in compulsivity alone explain about a sixth (16.5%) of the trait variance in choice temperature. This quantitative result is important, because most previous studies in computational psychiatry have (like us) attempted to relate personality or psychiatric traits – typically evaluated via self-report questionnaires – to model parameters measured in a specific task, such as choice temperature in our study. However, most of these studies typically do not measure the test-retest reliability of model parameters, and therefore typically assume a degree of stability of these parameters across time. This ‘trait’ component of model parameters, and task-specific variables in general, is often much lower than the ‘trait’ component of self-report questionnaire-specific scores²⁶,²⁷. Indeed, most behavioral tasks in cognitive science focus on the within-subject comparison of different experimental conditions²⁸. Quantifying the specific trait association between psychiatric symptoms and model parameters has critical implications for cognitive phenotyping and, in the longer term, for the possible identification of computational profiles for personalized psychiatry.

Here, we re-tested the same individuals in the same task conditions to decompose choice variability into state and trait components, in line with previous work which has quantified the trait and state components of risk-seeking behavior, or general intelligence²⁹,³⁰. For instance, preference for information seeking has been shown to be relatively stable²⁴, like certain aspects of preference for costly information, even in conditions of decreased immediate expected reward³¹. Decision parameters in general also show large interindividual differences that are relatively stable over time³². Here, we established that choice temperature (reflecting the stochasticity of the choice policy) has the largest trait component of all parameters of our reinforcement learning model (52%
of its overall variance). The other source of decision variability, arising from learning imprecisions, has a lower but still significant trait component (22% of its overall variance). It is important to note that having a higher trait component is not inherently 'better' than having a lower trait component. Indeed, the respective variances of state and trait components may reflect a flexibility-consistency trade-off in model parameters. For example, the state component may reflect online, moment-to-moment tuning of model parameters – as a function of available attentional/cognitive resources and environmental demands, to allow for adaptive behavior across changing task conditions. The fact that both learning noise and choice temperature differ between conditions in our study confirms that the state component does not reflect random noise in model parameters but rather an adaptation to specific task conditions.

Interestingly, we did not find any significant association between learning noise and any symptom dimension, despite its substantial interindividual differences and its significant (albeit moderate) trait component. Which personality or psychiatric trait(s) may explain individual differences in learning noise therefore remains to be identified. One possibility is that learning noise is a highly state-dependent, flexible cognitive parameter for which within-individual variations across tasks and contexts are much larger than between-individual variations. Given that learning noise (and more generally internal noise in cognitive computations) is an important source of decision variability under uncertainty¹²,¹³,³³, examining its trait-state structure and its relation to personality and psychopathology constitutes a major avenue for future research.

Our results show that compulsivity is associated with increased choice variability across task conditions, irrespective of whether this type of 'random' exploration is adaptive (when option values drift over time) or maladaptive (when option values are stable). This finding may potentially inform the development or refinement of cognitive behavioral therapies (CBTs), which aim at identifying which cognitive processes should be targeted to improve behavioral symptoms³⁴. By revealing a selective impact of compulsivity on choice variability without alterations in reinforcement learning, our study highlights executive control – rather than learning itself – as a possible target for treatment. Future work should validate this finding, obtained using subclinical differences in self-reported compulsivity in the general population, in patients diagnosed with OCD⁴,²⁵.
# Methods

## Experimental protocol

Our experimental design comprises three conditions, each divided into two-armed bandit tasks¹⁷. Rewards were drawn between 1 and 99 points. The Reference (Ref) and Stochastic (S+) conditions had fixed reward distributions. In contrast, in the Volatile (V+) condition the means of the reward distributions associated with each arm followed a random walk over time. In all conditions, the mean rewards associated with each option were mirrored relative to 50 points. In the V+ condition, we define a reversal as the moment where the mean rewards associated with each of the two options crossed each other. Participants underwent two blocks of 80 trials of each condition.

The two bandits were displayed as colored discs for the V+ condition (Fig. 1). By contrast, the two bandits were displayed as black shapes for the Ref and S+ conditions. In each new round, new shapes were presented, so as to avoid carry-over effects across rounds and ensure that participants reset and treat each new round separately from the previous one. We created these displays to ensure participants distinguish between stable (Ref and S+) vs. volatile (V+) conditions. After each decision, the chosen option was briefly displayed in the center of the screen, and followed by the points obtained on this trial (Fig. 1a). Upon completion of the task, participants earned £3.30, plus an additional £1.00 bonus if their accuracy was significantly better than chance level throughout the whole experiment.

Critically, we sought to achieve matched task difficulty between the S+ and V+ conditions. To achieve this, we used simulations from an optimal agent playing the task (see also Computational reinforcement learning model). For each participant, 2,000 random walks were generated with rewards drawn from a beta distribution with initial mean of 50 points and drifting standard deviation of 5 points. We subsequently discarded all walks that had steps larger than 10 points or travelled below 10 or above 90 points. To create blocks of trials for the V+ condition, we selected excursions below or above 50 points of 8, 12, 16, 20 and 24 steps. We employed the mean reward for each excursion as the static mean reward of a round of trials for the Ref condition. Furthermore, we set the variance of sampled rewards so that 75% of rewards from the better option were higher than 50 points in the Ref condition. Then, we computed the effective sampling variance and drifting variance of rewards; we simulated a Kalman filter with greedy decisions to obtain its accuracy in the V+ condition. Finally, we generated the S+ condition by incrementally regressing the static mean reward from the Ref condition towards 50, until the accuracy of the Kalman filter in this S+ condition matched that of the V+ condition.

After providing informed consent, participants had to be in full-screen mode to be able to carry on. A set of instructions introduced participants to the task mechanics (e.g., response keys). They were instructed about the three experimental conditions that were signaled via written prompts on the screen. The Ref condition was always performed first. Then, the order of S+ and V+ conditions was counterbalanced across participants. Before each condition, a practice block was provided so participants get familiar with the condition. After each practice block, we provided them with a debriefing about their performance on the condition, with an illustration of the trajectory of their decisions and rewards throughout the block. They were offered breaks in-between blocks. The experiment was self-paced, but had to be completed within 60 minutes.
## Self-report questionnaires assessing mental health symptoms

After completing the task, participants completed a set of nine questionnaires. We sought to sample variability in mental health symptoms naturally fluctuating in the general population. We evaluated depression using the Self-Rating Depression Scale (SDS)³⁵, trait anxiety using the State Trait Anxiety Inventory (STAI) Form Y-2, obsessive-compulsive disorder (OCD) using the Obsessive-Compulsive Inventory – Revised (OCI-R)²⁰, social anxiety using the Liebowitz Social Anxiety Scale (LSAS)³⁶, impulsivity using the Barratt Impulsiveness Scale (BIS-11)³⁷, schizotypy using the Short Scales for Measuring Schizotypy (SSMS)²¹, alcohol misuse using the Alcohol Use Disorders Identification Test (AUDIT), eating attitudes using the Eating Attitudes Test (EAT-26), and apathy using the Apathy Evaluation Scale (AES) (Fig. 3). We also estimated cognitive ability with the short International Cognitive Ability Resource (ICAR) test¹⁸,¹⁹ (16 items), which we used as a covariate alongside gender and age in subsequent regression analyses. In addition to the task bonus, participants who completed the questionnaire battery earned an extra £3.00.

## Participants

Three groups of 200 adult participants were initially recruited via the Prolific Academic platform (prolific.co). Here, we only use two of these datasets. The third one was not usable because ICAR scores were substantially (and significantly) lower than their expected distribution at the level of the group, thereby compromising the quality of questionnaire ratings. Sample size was determined a priori such that a between-participant correlation of 0.20 could be detected at standard levels of 80% power and 0.05 of significance threshold. Participants provided informed consent through a digital form regarding their participation in the study, as approved by the ethical review committee of the Institut National de la Santé et de la Recherche Médicale (IRB #00003888).

Exclusion criteria allowed us to maintain both task and questionnaire data quality. First, participants whose task accuracy did not exceed chance level were excluded (one-tailed threshold of 0.05 in any of the three conditions (binomial test against 50% accuracy)). This step led to a remaining N = 154 participants (mean age = 29.6 ± 10.0 years, 53 females) in the first dataset, and N = 142 participants (mean age = 25.4 ± 6.4 years, 72 females) in the second dataset. Second, we excluded participants who missed a catch item embedded into one of the questionnaires. The final sample comprised N = 137 participants (Fig. 3) for the first dataset, and N = 123 for the second dataset. To quantify the test-retest reliability of our behavioral variables and model parameters, out of the first dataset, N = 100 participants completed a second time the experiment after an average of two weeks from their first participation.

## Transdiagnostic factor analysis

To address the heterogeneity within-questionnaire and the correlations between questionnaires, we performed a factor analysis with an oblique rotation on the 209 questionnaire item data of 497 participants from our previous study⁶. The factor weights obtained were then applied on the questionnaire responses in our first and second datasets separately. Given the item-participant ratio here, instead of performing a de novo factor analysis, we leveraged the larger sample size obtained in our previous study. The factor analysis indicated that a three-factor solution best and most parsimoniously explained the data. In line with previous work⁶,⁸, these ‘transdiagnostic’ factors were labelled on the basis of questionnaire items loadings most strongly on each of these factors: ‘Anxious-Depression’ (AD), ‘Compulsive Behavior and Intrusive Thought’ (CIT), and ‘Social Withdrawal’ (SW). Consistent with previous work, we observed that AD and CIT factors were positively correlated (ρ = 0.357; *p* < 0.001).
# Computational reinforcement learning model

## Model description

After each reward, the posterior value of the chosen option $x_t^c$ is updated using a Kalman filter corrupted by random additive noise $\varepsilon_t$. $\varepsilon_t$ is drawn from a Gaussian distribution with a mean of zero and a standard deviation $\eta_t$:

$$x_t^c = x_{t-1}^c + k_t (r_{t-1} - x_{t-1}^c) + \varepsilon_t$$

where $k_t$ is the Kalman gain. It depends on the posterior variance of the chosen option $v_{t-1}^c$ and the sampling variance $v_s$ of outcomes. The latter was set to its true effective value (0.0163 for outcomes rescaled between 0 and 1) and used as a scaling parameter:

$$k_t^c = \frac{v_{t-1}^c}{v_{t-1}^c + v_s}$$

We updated the posterior variances of the two choice options $o \in \{1,2\}$ with the standard equation of the Kalman filter:

$$v_t^o = (1 - k_t^o) v_{t-1}^o + v_d$$

where $v_d$ is the drifting variance assumed by the Kalman filter. $v_d$ was estimated in terms of its associated asymptotic Kalman gain $\alpha$ for an option that would be selected on $n \to \infty$ trials. In line with recent results¹³, the standard deviation of the random noise corrupting the update of the posterior value of the chosen option scaled as a constant fraction $\zeta$ of the update $|k_t(r_{t-1} - x_{t-1}^c)|$. We initialized the posterior variances of the two options to the true variance of mean outcome values across all conditions (0.0214 for outcomes rescaled between 0 and 1).

The posterior value of the unchosen option $x_t^u$ regresses toward the overall average reward of 50 points across the two options (or 0.5 for rewards rescaled between 0 and 1). This regression-to-the-mean is controlled by an exponential decay factor $\delta$:

$$x_t^u = x_t^u + \delta (0.5 - x_{t-1}^u)$$

The probability of choosing option 1 $p_t^1$ based on the posterior values of the two options then follows a standard softmax policy with choice temperature $\tau$:

$$p_t^1 = [1 + e^{-(x_t^1 - x_t^2)/\tau}]^{-1}$$

## Model fitting

We applied Sequential Monte Carlo methods to estimate the conditional likelihoods of the decisions of each participant given the set of parameter values. We employed Bayesian Adaptive Direct Search (BADS) with 10 random starting points for each parameter $\alpha, \zeta, \delta$ and $\tau$ ³⁸ to first obtain point estimates of best-fitting parameter values. We then used these estimates as starting points for the estimation of their joint posterior distribution using Variational Bayes Monte Carlo (VBMC) ³⁹,⁴⁰.

## Model comparison

We performed a model comparison to examine whether the addition of learning noise (controlled by its Weber fraction $\zeta$) and soft choices (controlled by their temperature $\tau$) were both necessary to fit participants’ decisions. We ran a random-effects Bayesian Model Selection (BMS) using its standard Dirichlet parameterization $d$ ⁴¹. As in previous research ¹³, we compared the model including both sources of decision variability to two model variants. A first model variant had no
learning noise ($\zeta$ = 0). A second model variant had a greedy choice policy with $\tau$ = 0. The full model outperformed the other two model variants (model prevalence of 70% for the full model, exceedance $p$ > 0.999) (Fig. 2b).

## Model recovery

We performed model recovery to ensure that the models we would compare were both differentiable and recoverable in the fitting procedure. We simulated the model from the parameters obtained in the comparison procedure, and fit all three models on this simulated data. and performed model comparison for each of these three fits. The model prevalence and their protected exceedance probabilities showed each model as distinguishable and recoverable (see Supplementary Figure 3a).

## Parameter recovery

We performed parameter recovery analyses to ensure that there were no systematic relationships between the parameters generated by our model-fitting procedure. First, we destroyed any correlation between the best-fitting parameters via random permutation, and generated simulated data from these parameter sets. Then we fitted the model to this simulated data and calculated the resultant confusion matrix (Supplementary Figure 3b). The lack of significant correlations in the off-diagonal verifies that our fitting procedure does not generate any systematic correlations between model parameters. We then confirmed that the model parameter fits to participants' data are themselves recoverable via simulation and fitting of the generative model using these parameter values (Supplementary Figure 3c).

## Statistical analyses

We used Wilcoxon signed-rank tests for comparing (i) model-free variables, accuracy and switch rate; and (ii) best-fitting model parameters between conditions. We employed non-parametric Spearman correlations unless specified otherwise. We computed standard errors on Spearman's rank correlation coefficients (Fig. 5) using bootstrapping. We randomly sampled datasets from the original dataset (with identical sample size and replacement) 1,000 times and calculated the correlation on these new datasets.

To investigate the associations between mental health symptoms and task behavior, we conducted regression analyses (Fig. 4 and 5). We specified accuracy, switch rate, and model parameters in turn as dependent variables, in separate regressions. Predictors were scores on the three symptom factors, alongside age, gender and cognitive ability (ICAR) as additional control variables. Statistical significance was assessed using one-sample t-tests on regression coefficients. To directly illustrate the associations between task variables and symptom scores identified, we also examined Spearman correlations between CIT and accuracy, switch rate (Fig. 4), learning noise, and choice temperature (Fig. 5). To examine whether the identified associations replicated in our second dataset, we computed Bayes Factor (BF) tests for replication success⁴² answering the question “Is the effect from the replication attempt comparable to what was found before, or is it absent?”.

To examine the maladaptation of choice temperature in the stochastic condition (S+) with stable option values, we computed two Bayes Factors tests on the correlation coefficients between choice temperature and CIT. First, we calculated the relative evidence of finding a zero correlation in a null distribution over finding it in the distribution of S+ (Fig. 5; first dataset: $BF_{S0}$ = 448.1; second
dataset: BF<sub>S0</sub> = 35.1). We then calculated the relative evidence of finding the true S+ correlation
from the distribution of V+ over finding it in a null distribution (Fig. 5; first dataset: BF<sub>V0</sub> = 258.0;
second dataset: BF<sub>V0</sub> = 31.9). These Bayes Factor tests were obtained by calculating the likelihood
ratios of the marginal probability density functions derived from normal distributions generated by
values of Fisher z-transformations on Spearman's correlation coefficients.

To examine reliability between test and retest, we calculated intraclass correlations (ICC) for
accuracy, switch rate, and model parameters measured at test and retest sessions in the same
participants (Fig. 6a). We employed a two-way mixed-effects, absolute agreement model on single
measurements used in test-retest reliability studies<sup>43</sup>. All variables were log-transformed before
calculating ICC scores. Statistical significance was assessed via *F*-tests. Furthermore, we
performed a mediation analysis<sup>22,23</sup> to examine whether choice temperature mediates the link
between CIT and the measured behavioral variability (Fig 6b). We reasoned that the link between
CIT and choice temperature only matters if it explains to some degree the increase in switch rate
with increased CIT score. Hence, two pathways were specified. A direct pathway from CIT to
switch rate was compared against an indirect pathway from CIT to choice temperature to switch
rate.

Data and code availability statement

Anonymized behavioral and questionnaire data, together with MATLAB code for model fitting and
regression-based analyses of individual differences in task behavior, model parameters and mental
health symptoms, are available on a public repository:

https://github.com/jlexternal/RLVOLUNP_CIT_ana
**Author contributions**

Conceptualization: VW. Methodology: JKL, MR, VW. Investigation: JKL. Visualization: JKL, VW. Supervision: VW. Writing—original draft: JKL, MR, VW. Writing—review & editing: JKL, MR, VW.

**Conflicts of interest**

The authors declare no competing interests.

**Acknowledgements**

This work was supported by a starting grant from the European Research Council (ERC-StG759341) awarded to V.W., and by a department-wide grant from the Agence Nationale de la Recherche (ANR-17-EURE-0017, EUR FrontCog). M.R. is the beneficiary of a postdoctoral fellowship from the AXA Research Fund, and is supported by La Fondation des Treilles. The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript.
References

1. Robbins, T. W., Vaghi, M. M. & Banca, P. Obsessive-compulsive disorder: puzzles and prospects. *Neuron* **102**, 27–47 (2019).

2. Banca, P. et al. Evidence accumulation in obsessive-compulsive disorder: the role of uncertainty and monetary reward on perceptual decision-making thresholds. *Neuropsychopharmacology* **40**, 1192–1202 (2015).

3. Hauser, T. U., Moutoussis, M., NSPN Consortium, Dayan, P. & Dolan, R. J. Increased decision thresholds trigger extended information gathering across the compulsivity spectrum. *Transl Psychiatry* **7**, 1296 (2017).

4. Vaghi, M. M. et al. Compulsivity reveals a novel dissociation between action and confidence. *Neuron* **96**, 348-354.e4 (2017).

5. Hauser, T. U., Allen, M., NSPN Consortium, Rees, G. & Dolan, R. J. Metacognitive impairments extend perceptual decision making weaknesses in compulsivity. *Sci Rep* **7**, 6614 (2017).

6. Rouault, M., Seow, T., Gillan, C. M. & Fleming, S. M. Psychiatric symptom dimensions are associated with dissociable shifts in metacognition but not task performance. *Biol. Psychiatry* **84**, 443–451 (2018).

7. Hoven, M. et al. Abnormalities of confidence in psychiatry: an overview and future perspectives. *Transl Psychiatry* **9**, 268 (2019).

8. Gillan, C. M., Kosinski, M., Whelan, R., Phelps, E. A. & Daw, N. D. Characterizing a psychiatric symptom dimension related to deficits in goal-directed control. *eLife* **5**, e11305 (2016).

9. Voon, V., Reiter, A., Sebold, M. & Groman, S. Model-based control in dimensional psychiatry. *Biol Psychiatry* **82**, 391–400 (2017).

10. Patzelt, E. H., Kool, W., Millner, A. J. & Gershman, S. J. Incentives boost model-based control across a range of severity on several psychiatric constructs. *Biol Psychiatry* **85**, 425–433 (2019).

11. Seow, T. X. F. et al. Model-based planning deficits in compulsivity are linked to faulty neural representations of task structure. *J. Neurosci.* **41**, 6539–6550 (2021).

12. Findling, C. & Wyart, V. Computation noise in human learning and decision-making: origin, impact, function. *Curr. Opin. Behav. Sci.* **38**, 124–132 (2021).

13. Findling, C., Skvortsova, V., Dromnelle, R., Palminteri, S. & Wyart, V. Computational noise in reward-guided learning drives behavioral variability in volatile environments. *Nat. Neurosci.* **22**, 2066–2077 (2019).

14. Soltani, A. & Izquierdo, A. Adaptive learning under expected and unexpected uncertainty. *Nat. Rev. Neurosci.* **20**, 635–644 (2019).

15. Piray, P. & Daw, N. D. A model for learning based on the joint estimation of stochasticity and volatility. *Nat. Commun.* **12**, 6587 (2021).

16. Daw, N. D., O'Doherty, J. P., Dayan, P., Seymour, B. & Dolan, R. J. Cortical substrates for exploratory decisions in humans. *Nature* **441**, 876–879 (2006).

17. Lee, J. K., Rouault, M. & Wyart, V. Adaptive tuning of human learning and choice variability to unexpected uncertainty. 2022.12.16.520751 Preprint at https://doi.org/10.1101/2022.12.16.520751 (2022).

18. Condon, D. M. & Revelle, W. The international cognitive ability resource: development and initial validation of a public-domain measure. *Intelligence* **43**, 52–64 (2014).

19. Dworak, E. M., Revelle, W., Doebler, P. & Condon, D. M. Using the International Cognitive Ability Resource as an open source tool to explore individual differences in cognitive ability. *Personality and Individual Differences* **169**, 109906 (2021).

20. Foa, E. B. et al. The Obsessive-Compulsive Inventory: development and validation of a short version. *Psychol Assess* **14**, 485–496 (2002).
21. Mason, O., Linney, Y. & Claridge, G. Short scales for measuring schizotypy. *Schizophr Res* **78**, 293–296 (2005).

22. Shrout, P. E. & Bolger, N. Mediation in experimental and nonexperimental studies: New procedures and recommendations. *Psychological Methods* **7**, 422–445 (2002).

23. Kenny, D. A., Korchmaros, J. D. & Bolger, N. Lower level mediation in multilevel models. *Psychological Methods* **8**, 115–128 (2003).

24. Kelly, C. A. & Sharot, T. Individual differences in information-seeking. *Nat. Commun.* **12**, 7062 (2021).

25. Seow, T. X. F. & Gillan, C. M. Transdiagnostic phenotyping reveals a host of metacognitive deficits implicated in compulsivity. *Sci Rep* **10**, 2883 (2020).

26. Enkavi, A. Z. et al. Large-scale analysis of test-retest reliabilities of self-regulation measures. *Proc. Natl. Acad. Sci. U.S.A.* **116**, 5472–5477 (2019).

27. Loosen, A. M., Seow, T. & Hauser, T. U. Consistency within change: evaluating the psychometric properties of a widely-used predictive-inference task. Preprint at https://doi.org/10.31234/osf.io/qkf7j (2022).

28. Enkavi, A. Z. & Poldrack, R. A. Implications of the lacking relationship between cognitive task and self-report measures for psychiatry. *Biol Psychiatry Cogn Neurosci Neuroimaging* **6**, 670–672 (2021).

29. Frey, R., Pedroni, A., Mata, R., Rieskamp, J. & Hertwig, R. Risk preference shares the psychometric structure of major psychological traits. *Sci Adv* **3**, e1701381 (2017).

30. Pedroni, A. et al. The risk elicitation puzzle. *Nat Hum Behav* **1**, 803–809 (2017).

31. Bennett, D., Sutcliffe, K., Tan, N. P.-J., Smillie, L. D. & Bode, S. Anxious and obsessive-compulsive traits are independently associated with valuation of noninstrumental information. *Journal of Experimental Psychology: General* **150**, 739–755 (2021).

32. Moutoussis, M. et al. Decision-making ability, psychopathology, and brain connectivity. *Neuron* **109**, 2025-2040.e7 (2021).

33. Drugowitsch, J., Wyart, V., Devauchelle, A.-D. & Koechlin, E. Computational precision of mental inference as critical source of human choice suboptimality. *Neuron* **92**, 1398–1411 (2016).

34. Seow, T. X. F., Rouault, M., Gillan, C. M. & Fleming, S. M. How local and global metacognition shape mental health. *Biol Psychiatry* **90**, 436–446 (2021).

35. Zung, W. W. A self-rating depression scale. *Arch Gen Psychiatry* **12**, 63–70 (1965).

36. Liebowitz, M. R. Social phobia. *Mod Probl Pharmacopsychiatry* **22**, 141–173 (1987).

37. Patton, J. H., Stanford, M. S. & Barratt, E. S. Factor structure of the Barratt impulsiveness scale. *J Clin Psychol* **51**, 768–774 (1995).

38. Acerbi, L. & Ma, W. J. Practical Bayesian Optimization for model fitting with Bayesian Adaptive Direct Search. in *arXiv* 1705.04405 (2017).

39. Acerbi, L. Variational Bayesian Monte Carlo. in *Advances in Neural Information Processing Systems (NeurIPS)* (eds. Bengio, S. et al.) 8213–8223 (2018).

40. Acerbi, L. Variational Bayesian Monte Carlo with noisy likelihoods. in *arXiv* 2006.08655 (2020).

41. Stephan, K. E., Penny, W. D., Daunizeau, J., Moran, R. J. & Friston, K. J. Bayesian model selection for group studies. *NeuroImage* **46**, 1004–1017 (2009).

42. Verhagen, J. & Wagenmakers, E.-J. Bayesian tests to quantify the result of a replication attempt. *J Exp Psychol Gen* **143**, 1457–1475 (2014).

43. Koo, T. K. & Li, M. Y. A guideline of selecting and reporting Intraclass Correlation Coefficients for reliability research. *J Chiropr Med* **15**, 155–163 (2016).