Competing cognitive pressures on human exploration in the absence of trade-off with exploitation

Domain: neuroscience
Published: 2026-02-12
Editorial outcome: major_revision

Abstract
Exploring novel environments through sequential sampling is essential for efficient decision-making under uncertainty. In the laboratory, human exploration has been studied in situations where it is traded against reward maximisation. By design, these ‘explore-exploit’ dilemmas confound the behavioural characteristics of exploration with those of the trade-off itself. Here, we propose a sequential sampling task where exploration can be compared in the presence and absence of trade-off with exploitation. Detailed model-based analyses of choices reveal specific exploration patterns arising when information seeking is not traded against reward seeking or influenced by prospective value. Human choices are directed toward the most uncertain option available, but only after an initial sampling phase consisting of repeated choices from each novel option. These findings outline competing cognitive pressures on information seeking: the repeated sampling of the current option (local uncertainty minimisation), and the directed sampling of the most uncertain option available (global uncertainty minimisation).

Introduction
Making decisions often requires choosing between exploiting a familiar option whose rewarding value is known, and exploring an unfamiliar option whose value is not known1–3. Exploration is essential to discovering the structure of a novel environment (e.g. mapping possible itineraries from home to work in a new city), optimising behaviour (finding the fastest or most pleasant itinerary) or adapting to changes in the environment (looking for an alternative itinerary when the familiar one is unavailable). Exploration is thus adaptive to acquire information about available options, but also risky: choosing an unfamiliar option means foregoing a known reward for an uncertain one. When the known reward can be obtained without delay (e.g. choosing between two snacks at a vending machine), humans arbitrate between the costs and benefits of exploration4.

However, not all choices bring immediate rewards. For example, when browsing restaurant reviews online, exploring the reviews of different restaurants before ordering yields information at virtually no cost. In such situations, exploration is not traded against immediate reward maximisation, but used to acquire information about available options5. In some cases, like browsing restaurant reviews, the information acquired concerns the future rewarding values of available options, and can be readily used when choosing which restaurant to order from. But in other cases, like orienting oneself in a new city, the acquired information does not even concern value—whose definition depends on one’s particular goal at a later point in time (e.g. going to work on a weekday, or finding a new friend’s place on a Sunday).

In the laboratory, exploration is usually studied using bandit tasks in which agents are presented with two (or more) options to sample from6–8. Each option is associated with a specific reward distribution, and the agents are incentivized to maximise rewards through sequential sampling. Across successive choices, the agents learn to choose the most rewarding option—and thus trade exploration against immediate reward maximisation as we do when choosing snacks at the vending machine.

Variants of these tasks have dissociated information acquisition from reward acquisition in time, either by controlling the amount of information received (e.g. through forced choices9,10), or by delaying the acquisition of rewards (e.g. in the ‘sampling paradigm’ from Decisions From Experience (DFE) research5,11–13, where participants first sample the environment for information before they have to make a rewarded decision). However, even in these tasks, the information acquired during the sampling phase immediately concerns the ultimate rewarding values of options. Outcomes do not immediately contribute to participants’ final earnings, but since they are expressed in points, they immediately reveal which option will ultimately be rewarding, potentially biasing choices—this is similar to browsing restaurant reviews. This mean that the exploration patterns displayed in such delayed-reward tasks are still constrained by the prospective rewarding values of available options.

Here, we sought to fully dissociate exploration from reward-guided decision making by designing a two-armed bandit task where the outcomes of the two options are shades of colour instead of points or monetary rewards. In themselves, outcomes thus provide information about the structure of the task (the option-colour associations), but carry no intrinsic value (Fig. 1a). We contrasted three conditions in a within-subject design. In one condition, we rewarded participants for drawing shapes matching a particular colour, which was their target colour. This instruction does confer a rewarding value to each outcome and requires participants to trade exploration against exploitation to choose the option associated with the instructed colour—it mimics regular bandit tasks. By contrast, in a second condition, the target colour was only revealed at the end of the sampling sequence. Participants freely sampled the options to learn the option-colour pairing until a final two-alternative forced choice asked them to indicate an option of a specific colour (this colour was only revealed at the end of the sequence, as the final two-alternative forced choice probe). In this condition, participants were not rewarded on a sample-by-sample basis, but only based on their ability to indicate correct option-outcome associations after the end of the sequence. This instruction thus fully dissociates exploration from any form of immediate or prospective reward maximisation. Finally, we added a third, hybrid condition where participants were asked to find an option of one particular colour, which they only had to indicate once, at the end of the sequence of several free choices. Again, participants were rewarded based on their ability to indicate correct option-outcome associations after the end of the sequence, but in this condition, each outcome directly informed them about the options’ future value—similar to a classic delayed reward task. Except for these differences in instructions, the three conditions were matched in every possible aspect so that sampling patterns could be directly compared (Figs. 1 and 5; see ‘Methods’).

We collected a first discovery dataset and derived task-based variables to compare the different conditions, including a computational model of human exploration patterns. We observed differences in human exploration and we validated the existence of these effects in two additional datasets, whose data were analysed using the exact same procedures (see ‘Methods’—the statistics reproduced in the text are from the confirmatory datasets). Together, the findings reveal that when exploration is released from an immediate trade-off with exploitation, participants not only engage in directed exploration (focusing their exploration efforts on the most uncertain option), but they initially sample in chunks of the same option (focusing their exploration effort on a single option at a time). We show that this tendency is reduced as soon as outcomes convey information about the future rewarding value of options, even if the real reward itself is delayed. The two strategies expose competing forms of information seeking: a local form of uncertainty minimisation that aims at gathering evidence about the statistics of the current option though repetitive sampling, and a global form of uncertainty minimisation that aims at gathering evidence about the task structure through directed sampling of the most uncertain option available.

Results
Differences in human exploration patterns between conditions
We started by assessing participants’ performance in each condition. In the condition where participants were asked to match a target colour (MATCH), we assessed performance by plotting the fraction of sequences where participants chose the option truly associated with the target colour—focusing on sequences where only one option was associated with the target colour (Fig. 2a and Supplementary Fig. 1a, and Supplementary Table 1 for the descriptive statistics in the discovery sample). As expected, participants (N=27 after exclusion in the discovery and confirmatory datasets; see ‘Methods’) chose the option associated with the target colour more often than the other option on the last choice of each sequence (confirmatory two-sided signed-rank test against chance, z=3.9, p<0.001, r=0.75, 95% CI=[0.51: 0.88]).

When participants were asked to guess the colour associated with each option (condition GUESS), we assessed performance as the fraction of correct option-outcome associations provided at the two-alternative forced choice at the end of each sequence (Fig. 1a). As expected, participants guessed better than chance the correct option-outcome associations across all sequences (confirmatory two-sided signed-rank test against chance, z=4.4, p<0.001, r=0.86, 95% CI=[0.71: 0.93]).

We then studied participants’ choice patterns in the two conditions. First, we computed the fraction of repeated choices. The average fraction of repeat decisions was similar in the two conditions (Fig. 2c and Supplementary Fig. 1c): even though the GUESS condition incentivised participants to maximise knowledge about both options at any time, it did not lead them to switching more between options overall—see also Supplementary Table 2 for the descriptive statistics in the discovery sample. Then we computed the fraction of repeat decisions as a function of the number of samples previously acquired from this option—in other words, the fraction of repeat decisions, as a function of participants’ familiarity with the option before the current choice (Fig. 2c and Supplementary Fig. 1c). In the GUESS condition, participants had a higher tendency to re-sample the option for which few samples had been acquired. This pattern was absent from the MATCH condition, where participants appeared to progressively converge on a preferred option (confirmatory repeated-measures ANOVA, interaction between number of acquired samples from the current option and condition, F =11.10, p<0.001, η2=0:048, see Supplementary Table 3). In other words, only in the GUESS condition, where participants were incentivised to maximise information about both options at any time, did they seem to repeatedly sample from the same option when they first encountered it. This pattern was confirmed by ‘choice-similarity’ matrices—the average similarity of a choice to all later choices in the same sequence (Fig. 2d, Supplementary Fig. 1d, and Supplementary Table 4 for the descriptive statistics in the discovery sample). We confirmed that participants converged progressively on a preferred option in the MATCH condition. In the GUESS condition, on the other hand, choice similarity matrices showed that participants not only sampled repeatedly from the same option in the first trials (choice similarity among trials 1–4, confirmatory two-sided signed-rank test against random sampling, z=3.9, p<0.001, r=0.75, 95% CI=[0.52: 0.88]), but also sampled preferentially from the other option in subsequent trials (choice similarity between trials 1–4 and trials 8–11, confirmatory two-sided signed-rank test against random sampling, z=−2.2, p=0.024, r=−0.43, 95% CI=[−0.70, −0.07]). Both conditions also showed an elevated similarity between consecutive choices.

Finally, we quantified how target-driven and/or how uncertainty-driven the choices of the participants were. In this task, the distributions of colours were designed such that summing the shades of the colour outcomes over consecutive trials was equivalent to accumulating information about the probability of the two distributions given the outcomes observed (Fig. 1b). This allowed us to identify, at each choice, based on what the participant had observed so far, the objectively most target-like option (the option with the highest accumulated value in direction of the target colour category), and the objectively most uncertain option (the option with the lowest absolute accumulated value). At each trial, we could then categorise the choices of participants depending on whether they were directed toward the target option (only possible in condition MATCH), and whether they were directed toward the most uncertain option (in both conditions)—Fig. 2b and Supplementary Fig. 1b. As expected, a majority of participants’ choices were target-directed in the MATCH condition (two-sided t-tests against 50%, confirmatory: peak t =8.83, cluster-level p<0.001), and choices were uncertainty-directed in the GUESS condition more than in the MATCH condition (confirmatory two-sided paired t-tests between conditions: peak t =5.01; cluster-level p<0.001, trials 7–19). But this was not true in the first trials of each GUESS sequence, where participants’ choices were more strongly biased toward the more certain option in the GUESS than in the MATCH condition (confirmatory two-sided paired t-test between conditions: peak t =3.95; cluster-level p=0.010, trials 2–5).

The specific choice pattern observed in the GUESS condition—where outcomes have no rewarding nor prospective value—differs sharply from the behaviour of an optimal sampling agent who chooses the option whose outcome is most uncertain on each trial of each sequence (see ‘Methods’). Unlike participants, this optimal sampling agent does not sample repeatedly from one option, and then from the other option in the first trials of each sequence (Supplementary Fig. 2).

Modelling differences in human exploration patterns between conditions
Based on the specific features of human exploration patterns, we designed a process model of sampling choices derived from the optimal sampling agent (Fig. 3). In the MATCH condition, based on the perfectly accumulated values of the colour outcomes, the optimal sampling agent always selects the option with highest accumulated value in direction of the target colour (the most target-like option—Fig. 3d and Supplementary Fig. 2). Whereas in the GUESS condition, the optimal strategy is to always choose the option with lowest absolute accumulated value (the most uncertain option—Fig. 3d and Supplementary Fig. 2b). This allows the agent to maximise information about both shapes at any trial—thus maximising its probability of responding correctly to the final question, whose timing was not disclosed to participants. Like the optimal sampling agent, the choices of the model are sensitive to the difference in uncertainty between the two options (sensitivity parameter β_unc), and attracted toward the option most likely to be associated with the target colour in the MATCH condition (sensitivity parameter β_tar). In addition, and unlike the optimal sampling agent, the model features a repetition bias parameter b_rep, and an optional, initial sampling phase. In the beginning of the sequence, instead of relying on a decision variable computed from the three choice parameters (β_unc, β_tar and b_rep), the model can acquire, in turn, a number of samples from each option. This initial sampling strategy is controlled by a threshold θ, governing how much evidence the agent wants to initially acquire about the current option before switching to the other (see ‘Methods’ and Fig. 3). We fitted this model to human sampling patterns in the MATCH and GUESS conditions, and confirmed the recoverability of the model and its parameters (Supplementary Figs. 5 and 6).

Best-fitting parameter estimates obtained through maximum likelihood estimation (see Methods) confirmed behavioural observations (Fig. 4). First, participants’ choices were attracted toward the option most likely to be associated with the target colour in the MATCH condition (Fig. 4a, b; sensitivity parameter β_tar, confirmatory two-sided t-test against zero: t =5.7, p<0.001, d=1.10, 95% CI=[0.61: 1.57]). Second, participants’ choices were sensitive to the difference in information acquired from the two options, but only in the GUESS condition (Fig. 4a, b, confirmatory two-sided t-test against 0: t =2.7, p=0.013, d=0.51, 95% CI=[0.11: 0.91]). This mean that participants’ choices were attracted toward the option for which the associated colour is most uncertain only in the GUESS condition—where participants do not aim at maximising the immediate reward of each choice. There was no significant evidence for such directed exploration in the MATCH, reward maximising condition (two-sided paired t-test for β_unc between conditions in the confirmatory sample: t =3.6, p=0.001, d=0.69, CI=[0.26: 1.10] two-sided t-test for β_unc in the MATCH condition against chance t =−0.2; p=0.875; d=−0.03, 95% CI=[−0.41: 0.35]). Participants displayed significant repetition biases b_rep in both conditions (Fig. 4a, b, confirmatory two-sided t-tests against chance: MATCH t =6.0, p<0.001, d=1.16, 95% CI=[0.66: 1.64]; GUESS t =6.9, p<0.001, d=1.32, 95% CI=[0.79: 1.83]), with no significant difference between conditions. The descriptive statistics for the discovery sample are reproduced in Supplementary Tables 5 and 6.

Finally, we compared variants of the model with vs. without the initial sampling phase using Bayesian model selection on cross-validated measures of model likelihood (Fig. 4a, b; see ‘Methods’). In the MATCH condition, the model with the initial sampling phase was much less prevalent than the model without the initial sampling phase (Fig. 4a, b, confirmatory protected exceedance probability = 0.421). By contrast, the inclusion of an initial sampling phase improved model fits for a majority of participants in the GUESS condition (confirmatory protected exceedance probability = 0.992, probability that the model with initial phase was more prevalent in the GUESS than in the DRAW condition = 0.990). Additionally, in the GUESS condition, the threshold level on the amount of evidence to initially accumulate for an option before switching was equivalent to a 0.75 (median, IQR=[0.54: 0.95]) probability of the underlying category. This level was significantly less in the MATCH condition (confirmatory two-sided signed-rank test between conditions: z=3.2, p=0.002, r=0.61, 95% CI=[0.30: 0.80]). The strength of this threshold (ω) was comparably high in both conditions and both datasets.

To validate the notion that the initial sampling phase was required to reproduce the specific features of human exploration patterns in the GUESS condition, we performed a ‘knock-out’ simulation procedure in which we tested the effects of each aspect of the model on behavioural measures in the two conditions (Supplementary Figs. 3 and 4; see ‘Methods’). The inclusion of an initial sampling phase was necessary to reproduce—both qualitatively and quantitatively—the sampling patterns observed in the GUESS condition. A repetition bias was necessary to explain participants’ overall tendency to sample repeatedly from the same choice option in both conditions.

To summarise, participants’ choices deviated from those of the optimal agent in two ways: quantitatively participants had lower sensitivity to condition relevant variables (β_unc and β_tar <1, reflecting some randomness in choices), and qualitatively they featured repetition biases in both conditions (positive repetition bias), and a distinctive repetitive sampling strategy at the beginning of GUESS sequences only.

Dissociating the effects of delayed exploitation and outcome valuation on exploration
The GUESS condition, in which participants sample freely the two options to learn option-outcome associations, differs from the MATCH condition, in which the same participants seek to sample a rewarded outcome, in two ways. First, in the MATCH condition, as in traditional bandits, participants are rewarded for the outcomes of each individual choice in the sequence, whereas in the GUESS condition rewards are delayed to a final, single two-alternative forced choice at the end of each sequence. Second, in the MATCH condition, outcomes are associated with a positive or negative value depending on the target colour (which option is good or bad), whereas in the GUESS condition, they only convey information about the structure of the task, but cannot be immediately related to the future rewarding value of options (which option is blue or orange). While it was the goal of our manipulation to completely release exploration from the influence of rewards, it follows that the two dimensions—delayed exploitation and outcome valuation—were confounded, and it was impossible to discern their relative contributions to the specific exploration patterns observed in the GUESS condition.

To address this issue, we designed a third, intermediate condition (FIND). Participants were asked to identify an option associated with a target colour at the end of the sequence (delayed rewards), but they were informed of the target colour from the beginning of the sequence (prospective outcome valuation—Fig. 5a). In this third condition, as in the GUESS condition, participants were not rewarded based on the outcomes they were sampling during the sequence, but rather based on their answer to a single two-alternative forced choice, at the end of the sequence. However, as in the MATCH condition, each outcome was associated with a positive or negative value depending on the target colour. This situation was akin to browsing restaurant reviews: choices yielded no immediate reward, but outcomes were still valued with respect to an end goal. It was also closer to existing tasks, like the Horizon task9 or the ‘sampling paradigm’ from Decisions From Experience studies12 Both paradigms disentangle information search from reward seeking—either by separating them in time, as in the ‘sampling paradigm’, or by controlling for the amount of information available to participants, as in the Horizon task. However, in both cases the immediate goal of each sampling decision is to identify which option will be most rewarding ultimately, and the outcomes convey information as a numerical value in the same unit as the reward. In other words, it is expected that a prospective reward value already taints the outcomes and biases choice patterns—i.e. participants will seek options associated to higher rewards in the future. While the original GUESS condition removed this concern, the addition of the FIND condition eliminates a gap in the paradigm. In a nutshell, the MATCH condition features immediate rewards, the FIND condition delays rewards but allows for prospective valuation, and the GUESS condition delays rewards and prevents prospective outcome valuation. The contrast between the FIND and MATCH conditions reflects the difference between delayed and immediate exploitation, whereas the contrast between the FIND and GUESS conditions reflects the difference between the sampling of outcomes with and without rewarding value.

In this third dataset, participants (N=31 after exclusion; see ‘Methods’) replicated the behavioural patterns previously described in the MATCH and GUESS conditions. They reached similar performance in both of the original conditions (Fig. 6a), and in the GUESS, but not in the MATCH condition, they initially sampled in chunks of repeated decisions for each option in turn. This was visible both in behavioural indices (Fig. 6) and in the distributions of parameter estimates (Fig. 7a). The statistics for these conditions are reproduced in Supplementary Tables 1–6. We focus the remainder of this section on the third, FIND condition.

In the FIND condition, participants responded accurately to the final question (Fig. 6a, median=93.8%, IQR=[87.5% 100%], two-sided signed-rank test against 50% z=4.9, p<0.001, r=0.88, 95% CI=[0.76: 0.94]). They also displayed a preference for sampling the target-coloured option during the sequence even though it was explicitly stated that this did not contribute to performance (Fig. 6a, median at the last sample=62.5%, IQR=[50.0% 75.0%]). However, this tendency was less pronounced than in the MATCH condition (two-sided signed-ranks test between conditions: z=4.2, p<0.001, r=0.75, 95% CI=[0.54: 0.87]), and the fact that participants selected the option associated with the target colour much more often on the final choice (93.8%) than on the last sampling choice of the sequence (62.5%) shows that they effectively delayed exploitation until the final choice.

Comparably to the original datasets, there was no significant difference between conditions in the average fraction of repeat decisions, (Fig. 6b) but only in the fraction of repeat decisions as a function of the number of previous samples acquired from this option (Fig. 6b; interaction between number of acquired samples from the current option and condition, F =6.2, p<0.001, η2=0.027). As in the GUESS condition, participants tended to re-sample more from options for which few samples had been acquired in the FIND condition than in the MATCH condition, whereas the converse was true for options for which several samples had been acquired. On the other hand, choice similarity matrices in the FIND condition only partially resembled those of the GUESS condition. Participants did sample repeatedly from the same option in the early trials—similar to the GUESS condition (Fig. 6d, choice similarity among trials 1–4, two-sided signed-rank test against random sampling, z=3.9, p<0.001, r=0.71, 95% CI=[0.47: 0.85]). But unlike in the GUESS condition, there was no evidence in choice similarity matrices that they sampled preferentially from the other option in subsequent trials (similarity between trials 1–4 and trials 8–11, two-sided signed-rank test against random sampling in FIND, z=−0.4, p=0.681, r=−0.07, 95% CI=[−0.42, 0.29]; two-sided signed-rank test between conditions GUESS and FIND: z=−3.1, p=0.002, r=−0.56, 95% CI=[−0.76, −0.25]). Choice direction in the FIND condition also displayed features from both original conditions. As in the GUESS condition, participants made more uncertainty-directed choices in the FIND condition than in the MATCH condition (Fig. 6c, two-sided paired t-tests between the FIND vs. MATCH condition peak t =5.53, cluster-level p=0.023; no significant difference between the FIND and GUESS conditions). And, as in the MATCH condition, participants’ choices were biased towards the more target-coloured option. This, however, was less pronounced than in the original MATCH condition (two-sided t-test against chance: peak t =6.43, cluster-level p<0.001; two-sided paired t-tests between the FIND and MATCH conditions: peak t =10.9, cluster-level p<0.001).

Model fits corroborated these observations. First, the sensitivity of participants’ choices to target outcomes, β_tar, was lower in the FIND than in the MATCH condition (Fig. 7a; one-sided t-test against zero: t =5.8, p<0.001, d=1.03, 95% CI = [0.59, 1.47]; one-sided paired t-test between conditions: t =5.6, p<0.001, d=1.00, 95% CI=[0.56, 1.43]). Participants’ choices were also directed toward the more uncertain option in the FIND condition, as in the GUESS condition (Fig. 7a.; sensitivity parameter β_unc, one-sided t-test against zero in FIND: t =2.6, p=0.008, d=0.46, 95% CI=[0.08: 0.83]; no significant difference between the FIND and GUESS conditions), but unlike the MATCH condition (one-sided paired t-test between FIND and MATCH: t =−1.9, p=0.033, d=−0.34, 95% CI=[−0.70, 0.02]). As in previous datasets, participants’ choices showed similar repetition biases, b_rep, across all three conditions (Fig. 7a; one-sided t-tests against zero: p<0.001, d>0.69 in all conditions, no significant difference between conditions).

In the FIND condition, the presence of an initial sampling phase only improved the fits for about a third of tested participants (Fig. 7a, mean=35%, s.d.=8.2%, protected exceedance probability = 0.332)—more than in the MATCH condition (mean=12%, s.d.=5.5%, protected exceedance probability = 0.027), but less than in the GUESS condition (mean=81%, s.d.=6.7%, protected exceedance probability = 0.953—see Fig. 7a for comparisons.). The level of evidence which participants initially wanted to secure about the current option before switching (θ) was lower across participants in the FIND than in the GUESS condition (one-sided signed-rank test between the FIND and GUESS conditions: z=−2.5, p=0.006, r=−0.45, 95% CI=[−0.70, −0.12]). The strength of the threshold was similarly high across all conditions. Finally, the inclusion of an initial threshold was necessary to reproduce the early sampling patterns observed in the GUESS and FIND conditions only. By contrast, as in previous datasets, a repetition bias was necessary to explain participants’ overall tendency to sample repeatedly from the same option in all three conditions (Supplementary Fig. 7).

In summary, this additional dataset confirmed that it was only when rewards were delayed to the end of the sequence that participants engaged in uncertainty driven exploration and in initial repetitive sampling (in the FIND and GUESS conditions, but not in the MATCH condition). However, the prevalence of initial repetitive sampling was diminished when outcomes conveyed direct information about the future rewarding value of options (in the FIND condition: which option is the target, similar to the usual delayed rewards tasks9,12), compared to when they were only informative about the structure of the task (in the GUESS condition: which option is orange or blue).

Positive covariations between exploration patterns across individuals
Human exploration patterns in the GUESS condition show two specific features absent from the MATCH condition. First, participants preferred sampling the option whose associated outcome is currently more uncertain (feature 1). But they also displayed initial sampling phases where sampling proceeds in chunks of repeated choices from the same option until a certain level of information is reached (feature 2). The first feature reflects a more global form of information seeking and is statistically optimal in this task: an optimal sampling agent would always choose the more uncertain option to maximise global knowledge and response accuracy during the final choice (Fig. 3, Supplementary Fig. 2 and ‘Methods’). The second feature, on the other hand is a more local form of information gathering (reducing uncertainty about a single option at a time), and is statistically suboptimal (Supplementary Fig. 2b). From a rational standpoint, we would thus expect initial repetitive sampling to be detrimental to performance, and to co-occur with less choices directed toward the most uncertain option (more statistically suboptimal behaviour). But it is also possible that these two features of exploration are not traded-off, and rather reflect two different exploration strategies which can be co-expressed in the same individuals.

To address this issue, we studied covariations in exploration patterns in a between-subject manner by pooling the three datasets together (total N=85). We split participants based on which of the model with (ini+ group, N=67) or the model without an initial evidence threshold (ini- group, N=18) best fit their behaviour, but to ensure a fair comparison between groups, we then compared parameter estimates obtained in the most specified version of the model (the version with initial threshold, ini+). As expected from this splitting criterion, in the GUESS condition, participants in the first group (ini+) showed a higher tendency to initially resample options (Fig. 8a, b, d and Supplementary Fig. 9b).

From this post-hoc analysis, the two exploration strategies (features 1 and 2) appeared to co-occur. Participants in the ini+ group, who displayed a stronger tendency to initially repeat choices also showed a higher tendency to later prefer the most uncertain of options. This was visible on the choice direction curves (Fig. 8b) and in higher estimates for the sensitivity to uncertainty (Fig. 8e; β_unc, mixed-effects ANOVA, main effect of group (ini+ or ini-): F =6.82, 1,83 p=0.011, η2=0.055; interaction between group and condition: F =8.49, 1,83 p=0.005, η2=0.030). In other words, the suboptimal, initially local information-seeking strategy was expressed along with the more optimal feature of global information-seeking strategies later on in the sequences.

Additionally, we observed that this combination of optimal and suboptimal exploratory features was associated to repetition bias and reward drive. Participants in the ini+ group also displayed stronger repetition biases, which is another statistically suboptimal feature. Their choice similarity matrices displayed thicker diagonals (Fig. 8d) and model fits showed higher b_rep estimates (Fig. 8e; mixed-effects ANOVA; main effect of group: F =4.01, 1,83 p<0.05, η2=0.041, no significant interaction), which were strongly correlated across conditions (N=85, linear correlation: r2=0.76, 95% confidence interval=[0.63: 0.84]). However, ini+ participants also had a more pronounced tendency to prefer the target-coloured options in the MATCH condition, which is optimal in this condition (choice direction curves on Fig. 8b; higher β_tar sensitivity parameter on Fig. 8e; two-sided two-sample t-test between groups of participants: t =2.20, 83 p=0.031, d=0.58, bootstrapped 95% CI=[0.03: 1.10]). This strategic mixture of normatively optimal and suboptimal features was also associated to higher performance at the final question for ini+ participants (Fig. 8c; two-sided, two-sample rank-sum test between groups of participants in GUESS: z=2.1, p=0.004, probability of superiority=0.66, bootstrapped 95% CI=[0.49, 0.82]), suggesting that the presence of suboptimal tendencies to repeat choices, both during the initial sampling phase and later throughout the sequence, was not due to mere task disengagement or inattention.

Taken together, these analyses suggest positive covariations between exploration patterns across tested participants and conditions. Stronger initial repetitive sampling could be associated to stronger sustained repetition bias (two statistically suboptimal features), but also to higher sensitivity to condition-relevant variables (β_tar and β_unc, two statistically optimal features), and consequently higher performance.

Methods
Participants
For the first (discovery) dataset, we recruited N=30 adult participants (15 females, mean age=25 years, s.d.=4.3 years). For the discovery dataset, we had no specific expectation regarding effect sizes: no statistical method was used to pre-determine sample size, and it was determined to match or exceed earlier studies of human exploration patterns in the literature. Participants were recruited through a local, open mailing list advertising psychology and neuroscience experiments in the Paris area. Participant compensation was 22 euros in cash for 90 min of testing. Recruitment criteria included age between 18 and 35 years, right-handedness, normal or correct-to-normal vision, normal colour perception, and no history of psychiatric or neurological disorder.

For the second (replication) dataset, we also recruited N=30 adult participants (15 females, mean age=25 years, s.d.=3.7 years). Participants were recruited through the local mailing list used for the first dataset, and through a national website (L’Etudiant) advertising short-term jobs for adult students. The other recruitment criteria were identical to those used for the first dataset. Participants recruited through the local mailing list are typically students (in the age range we targeted) and typically accustomed to psychology experiments. Participants recruited through L’Etudiant were also a majority of students, but they had never taken part in psychology studies. Participant compensation was 22 euros in cash for 90 min of testing.

For the third (three-condition) dataset, we balanced the lower number of sequences per condition by aiming to reach N=30 adult participants after exclusion. We recruited a total of N=38 participants, exclusively on the website l’Etudiant (21 females, mean age=24 years, s.d.=4.4 years). The other recruitment criteria were identical to those used for the first two datasets. Participant compensation was 20 euros in cash for 75 min of testing.

All participants gave written informed consent before taking part in the study. Ethics approval was obtained from the relevant authorities (Comité de Protection des Personnes Ile-de-France VI, IDRCB: 2007-A01125-48, 2017-A01778-45), which covers all three datasets. Informed consent was obtained for all three datasets.

We recruited participants to achieve a roughly equal number of self-reported males and females in each sample. However, we had no theoretically motivated reason to perform disaggregated sex or gender-based analyses. We did not collect consent from participants to share their individual self-reported sex data.

We excluded participants whose sampling strategy was stereotyped or did not depend on the presented outcomes. We computed the number of stereotyped sequences for each participant: sequences where the participant systematically alternated between left and right responses, or between the two options, and sequences where the participant sampled a single option throughout the whole sequence. We excluded participants who produced more than 50% of such stereotyped sequences in any condition (N=3 participants in the discovery dataset, N=3 participants in the replication dataset, and N=6 participants in the third dataset). In the third experiment, one additional participant was sent home before taking part in the study due to COVID-19 symptoms. After exclusion, the final datasets comprised: 27 participants in the discovery dataset (12 females, mean age=25, age standard deviation=4), 27 participants in the replication dataset (13 females, mean age=25, age standard deviation=4), 31 participants in the third dataset (16 females, mean age=24, age standard deviation=4).

The experiment was within-participants: all participants in the discovery and replication datasets saw the two conditions, and all the participants of the third dataset saw all three conditions. There was no randomisation and no blinding involved.

Experiment
Experimental design. Participants made sequences of choices between two options depicted by shapes, and each choice yielded an outcome. Outcomes were colour mixtures, drawn from a continuum between blue and orange (through grey in the middle of the range). The outcomes of each option were drawn from one of two probability distributions: an ‘orange’ distribution with 67% of dominantly orange outcomes (and 33% of dominantly blue outcomes), and a ‘blue’ distribution with 67% of dominantly blue outcomes (and 33% of dominantly orange outcomes)—Fig. 1b. The dominant colour associated with an option was independent of the dominant colour associated with the other option, such that participants needed to sample each individual option to learn its associated colour (Fig. 1c). Each sequence consisted of a series of choices (trials) whose number varied pseudo-randomly and uniformly across conditions from 8, 12, 16, or 20 choices (Fig. 1a.).

Each sequence was preceded by condition-specific instructions. In the MATCH condition, participants were asked to draw a maximum of outcomes matching a target colour (orange or blue, counterbalanced across sequences). Participants were informed that the colour intensity of each outcome was translated into points: the more the outcome was of the target colour, the higher the number of points. This task feature rendered each outcome intrinsically rewarding. In this first condition, the sequence ended with the last choice: No final question was asked and participants immediately moved on to the instructions screen for the next sequence. In the GUESS condition, participants were asked to guess the dominant colour associated with each of the two options. Each sequence ended with a final question, where participants were asked about the dominant colour associated with one of the two options. In this second condition, participants were rewarded solely based on the accuracy of their responses to this final question. Each outcome was not intrinsically rewarding, but it was informative regarding the dominant colour associated with the sampled option. In the FIND condition, participants were asked to find an option associated with a target colour (orange or blue, counterbalanced across sequences). As in the GUESS condition, participants were rewarded solely based on the accuracy of their responses to a final question, which in this case asked them to select an option associated with the target colour at the end of the sequence. To further enhance the symmetry between the FIND and GUESS conditions in the third experiment, we changed the final question of the GUESS sequences to match that of the FIND sequences. Instead of asking the colour of a given option, we asked to identify an option of a given colour (orange or blue, counterbalanced across sequences). The single difference between the two conditions is that the question (i.e., the target colour to identify at the end of the sequence) was known in advance in the FIND condition, but unknown in the GUESS condition.

Eight geometric shapes were used to instantiate options, arranged into twenty-eight possible pairs. Shape-colour associations changed pseudo-randomly between sequences in order to minimise shape repetition. The colour intensities of outcomes were pre-defined in advance. Colour intensity indexes the amount of information provided by the outcome about its associated probability distribution: a more intense colour is more informative than a less intense one. This advanced pre-definition of colour intensities allowed us to make sure that, for each choice, the amount of information that could be acquired from the two options was exactly the same. Because the colour associated with an option could not be identified from the colour associated with the other option, different sequences used options associated either with the same or with different colours. This means that in the MATCH condition, some sequences had one single target option, whereas other sequences had either zero, or two target option(s). We included only sequences with a single target option when measuring performance. Each unique sequence was duplicated to be used once in each condition, so that each condition was tightly matched with the other(s)—except for the instructions provided to the participants. In particular, the uncertainty associated with each condition was therefore matched across conditions.

Experimental procedures. Participants were instructed on how to perform the task by a step-by-step, self-paced tutorial, followed by a practice session of eight sequences. Participants were then asked to describe the instructions to the experimenter, and it was emphasised again for all participants that: 1. the colour intensity of each outcome was relevant, 2. each option would not always draw the exact same shade, so the task was to identify its dominant colour, 3. there were only two possible fixed distributions (blue and orange), 4. the two options were not always associated with different colours (they could be both associated with orange, or blue), 5. each sequence would end after a variable, unpredictable number of choices, and 6. performance was measured by the colours of all sampled outcomes in the MATCH condition, and by the final question in the GUESS and FIND conditions.

For the discovery dataset, the experiment (excluding practice sequences) consisted of 48 sequences of each condition (96 in total, divided in 8 blocks of 12 sequences). For the replication dataset, the experiment consisted of 64 sequences of each condition (128 in total, divided in 8 blocks of 16 sequences). For the third dataset (with the additional FIND condition), the experiment consisted of 32 sequences of each condition (96 in total, divided in 4 blocks of 24 sequences). Conditions were interleaved pseudo-randomly within blocks, with self-paced breaks between blocks. Conditions were pseudo-randomised to balance the number of sequences of each condition within each block, avoid too many (>3) consecutive repetitions of the same condition, and avoid immediate repetitions of the same shapes or the same response sides.

The task was run using the Psychtoolbox-3 toolbox (version 3.0.14) for MATLAB 2017b (The Mathworks). The testing conditions were slightly different across datasets. For the first (discovery) dataset, eye position was recorded during the task using an EyeLink-1000 Plus eye-tracker system (SR Research), and participants’ head movements were constrained by a chin rest. For the second (replication) and third datasets, eye position was not recorded and no chin rest was used. Participants were seated with their eyes at approximately 70 cm from a 24-in. LCD screen with a resolution of 1920×1080 pixels and a refresh rate of 60 Hz.

The trial structure is depicted in Fig. 1a. After reading the instructions and initiating a sequence by pressing the space bar, each sampling decision began with the two shapes appearing on the left and right sides of a fixation circle - sides were pseudo-randomly assigned. This choice screen was displayed until participants selected an option by pressing the corresponding [A] (left shape) or [P] (right shape) key on an azerty keyboard, after which the outcome was displayed for 500 ms. The outcome corresponds to the filling of the selected shape with the sampled colour mixture. A fixation circle was then displayed alone for 500 ms before the trial began. In the MATCH condition, each sequence ended as soon as all the sampling choices were made (8, 12, 16, or 20) and their outcomes seen. The instructions for the following sequence then appeared on screen. In the GUESS (and FIND) conditions, each sequence ended with a final question screen. In the first two datasets, one shape (pre-selected at random) was displayed and participants were asked to report its associated (dominant) colour. In the third dataset, one colour (pseudo-randomly selected) was displayed on screen and participants were asked to identify an option associated to the demanded colour. Participants selected a colour or option with a key press, and a selection box was displayed around the selected colour or option for 400 ms. Feedback on the accuracy of this final response was provided by filling the selected response with its associated colour during 500 ms. The instructions for the following sequence then appeared on screen.

Computational model
Description of the model. The computational model was updated during revisions: again, we first developed the model on the discovery dataset before testing it in the confirmatory samples. The computational model of sampling behaviour featured the same learning module across all three conditions. In practice, the colour distributions were designed such that the log-evidence for the categories (x) was proportional to the colour shades c (ranging from −1 pure blue to +1 pure orange—see Fig. 1).

x = log (p(c|blue) / p(c|orange)), where c = 1/2(c_i) (1)

At each trial, the model could then accumulate evidence for each option (x^_option,t) by summing the colour outcomes c observed from each option up to the current trial:

x^_option,t = sum(c_i) = log (p(c_1...c_t|blue) / p(c_1...c_t|orange)) (2)

The sum of evidence observed for this option x^_option,t contained information about which colour distribution was more likely to have produced the observed samples (sign(x^_option,t)), but also about how much information had been accumulated (|x^_option,t|). Based on these quantities, the decision variable DV_t was computed at each trial t, as the sum of three components:

DV_t = β_tar * (x^_1,t - x^_2,t) * c_tar + β_unc * (|x^_2,t| - |x^_1,t|) + b_rep * r_t-1 (3)

The first component was the difference in accumulated values between the two options, signed by the target colour: (x^_1,t - x^_2,t) * c_tar, where c_tar was participants’ target colour (+1 for orange, −1 for blue, and 0 when no colour was announced at the beginning of the sequence). In practice, this first component was positive when the first option was closer to the target colour, and negative when the second option was closer to the target colour. This component was weighted by a sensitivity parameter β_tar, controlling the sensitivity of the model choices to the target colour. For sequences when participants played the guessing game, no target colour was given, so c_tar was set to zero, and this first component was null. The second component was the difference in the amount of information accumulated for each option: |x^_2,t| - |x^_1,t|. This was also scaled by a sensitivity parameter β_unc, controlling the sensitivity of the model choices to the amount of information already acquired for each option. This component was defined in all conditions. The third component was also defined in all conditions for all choices except the first choice of each sequence, and corresponds to an additive bias in the direction of the previous choice, controlled by a repetition bias parameter b_rep. Its value was +1 when the previous choice (r_t-1) was option 1, and −1 when the previous choice was option 2, effectively biasing the decision variable towards choosing option 1. The choice process itself could take two different forms. By default, it was modelled by a standard softmax process on the decision variable DV_t. But the ini+ model also included an initial sampling phase, where, as long as the level of evidence acquired for an option was below a given threshold (θ), choices were not controlled by the decision variable described above, but the model repeated the previous decision with probability ε. This initial sampling phase of each option (initialPhase_o) was therefore controlled by the threshold θ on the amount of information already accumulated for the option, and by the threshold hardness ε, governing how much the model will repeat the previous decision while under the θ threshold. If the threshold hardness ε is at 1, the model systematically repeats the previous choices as long as it is below the θ threshold, whereas it the hardness is at 0 the model does not implement the initial repeated sampling and always relies on the decision variable DV_t. Similarly, if the level of the threshold θ is at 0, the model also does not implement the initial sampling phase:

if initialPhase_o = true & |x^_o,t| < θ p(r_t = r_t-1) = ε (4)

else p(r_t = r_t-1) = 1 / (1 + e^-DV_t) (5)

if r_t ≠ r_t-1 initialPhase_o = false (6)

To avoid numerical discontinuities in the likelihood function of the model due to the presence of a hard bound θ, we assume that θ follows a normal distribution with a small standard deviation of 0.1 in log LR units (which is considered as small given that a single colour sample can provide up to 1.4 in log LR units).

For plotting and statistical testing, θ was transformed from log LR units into probabilities (and therefore treated as a bounded parameter). It can be understood as a certain probability (of the current shape being associated to either the blue or the orange distribution) that the agent wants to reach before terminating the initial repetitive sampling phase.

Note that the computational model described above nests the optimal sampling agent for the MATCH and GUESS conditions. The optimal agent does not include an initial sampling phase or a repetition bias in either condition. In the MATCH condition, the optimal sampler chooses based solely on the first component: the difference between the mean colour mixture estimates signed by the target colour, with β_tar → 1 (i.e. an argmax choice process). In the GUESS condition, the optimal sampler chooses based solely on the difference between the information already acquired from each option, with β_unc → 1. The optimal sampling agent was simulated for each participant and each condition to compare its behavioural measures to those of tested participants (Supplementary Fig. 2).

Fitting procedures. Best-fitting parameter estimates for the model, which can either include (ini+) or not include an initial sampling threshold (ini-), were obtained through maximum likelihood estimation (MLE) in two ways. First, we used Matlab’s fmincon function to cover a wider range of possible parameter values and avoid local minima (the decision variable parameters were initialised at random points in the range [−2:2] for β_tar and β_unc and [−4:4] for b_rep, while the threshold value was set at a hundred different values in the interval [0.05:5]). Second, we used the best-fitting estimates obtained from this first fmincon pass as starting points for the Bayesian Adaptive Direct Search algorithm39 (version 1.0.8). This second pass was two-fold: we used leave-one-out cross-validation on the sequences used to obtain model likelihood estimates. These were used in Bayesian model selection to compare variants of the model with vs. without the initial sampling phase. BMS was conducted using a random-effects approach, which assumes that different participants may rely on different models. It consists of estimating the distribution over models that participants are drawn from. We used the Dirichlet parameterisation of the random-effects approach implemented in SPM12 (http://www.fil.ion.ucl.ac.uk/spm-version6906), with default (uniform) priors. We also used the BADS algorithm on all the sequences at once to obtain final parameter estimates39.

Knock-out simulations. To test the relative effects of the two suboptimal aspects of the model (the initial sampling phase and the repetition bias) on behavioural measures in the different conditions, we used a ‘knock-out’ simulation procedure: starting from the ‘full’ model, separate model simulations were conducted after inactivating the initial sampling phase and the repetition bias independently of each other (by setting θ=0, or b_rep=0, none, or both).

Model recovery. To assess whether our fitting procedure could distinguish in principle between the behaviour of an agent with and without initial sampling phase, we performed model recovery (Supplementary Fig. 5). In each condition, we simulated the behaviour of agents with or without an initial sampling phase, and used Bayesian model selection (BMS40 (http://www.fil.ion.ucl.ac.uk/spm)) to measure which of the model with, or without initial sampling phase best fit the simulated data. During simulations, the general choice sensitivity parameters were set to the average of best-fitting values in each condition. For the values of the initial sampling phase parameters, in the GUESS condition, they were either set to the average of best-fitting values (ini+ agent) or to zero (ini− agent). In the MATCH condition, they were set to either the average of best-fitting values of the GUESS condition (ini+ agent), or to zero (ini− agent). This is because best-fitting parameters in the MATCH condition do not reflect the presence of an initial sampling phase. We then used the spm_BMS toolbox40 (http://www.fil.ion.ucl.ac.uk/spm) to compute (a) the average frequency of both fit models for each simulated agent (confusion matrix), and (b) the probability of the simulated agents given the fit of each model (inversion matrix).

Parameter recovery. To evaluate our fitting procedure, we simulated the behaviour of artificial agents on each participant’s true stimulus sequences, and fit those artificial choices using our initial fitting procedure (see Fitting Procedure, and Supplementary Fig. 5). We first permuted the values of each best-fitting parameter to remove correlations between parameters (<0.07 absolute Spearman’s r between parameters - this preserved parameter distributions while selectively removing between-parameters correlations). We then simulated choices for agents with initial sampling phases, under these decorrelated parameters (simulated parameters) and fit the model to the simulated choices using the same fitting procedure we had used for participants’ choices (yielding the recovered parameters). The fitting procedure was evaluated within-parameter, based on the correlation between simulated and recovered values (Spearman correlations, weighted by the goodness of the original fit for each parameter value), and we used correlations between recovered parameters to assess whether artificial, between-parameter correlations were introduced during fitting (Spearman correlations, weighted by the goodness of the original fit for each parameter value). To confirm that the presence of an initial sampling phase in the model didn’t substantially bias the estimation of the later phase parameters (sensitivity to the target, sensitivity to uncertainty, and repetition bias), we also performed recoveries. We simulated choice behaviour for agents either with or without an initial sampling phase and recovered the parameters using a model with initial phase parameters (see Supplementary Fig. 6). We then assessed the bias of the model (the difference between recovered and simulated parameter values) for agents with or without an initial sampling phase.

Principal component analysis. To check that our model captured not only the mean behaviour of tested participants but also individual differences in the data, we ran a principal component analysis (PCA—Supplementary Fig. 11). We used choice similarity matrices as a summary snapshot of behaviour in the task, and we contrasted the behaviour of original participants with the behaviour of agents simulated under the best-fitting parameters of the participants (n=1000 simulations). We derived the principal components (PCs) explaining the behaviour of the original participants on the one hand, and the simulated behaviour on the other hand.

Stability of estimated parameters throughout the task. To test for a change in exploration strategies throughout the task we compared the parameter estimates obtained from the first vs. the second half of the task, in the pooled datasets (Supplementary Fig. 12). The fitting procedure was the same as for the individual datasets (cf. above section ‘Fitting Procedures’).

Statistical analyses
Unless noted otherwise, statistical analyses of differences between unbounded metrics relied on two-sided parametric tests (single sample and paired t-tests), for bounded metrics we used two-sided non-parametric tests (Wilcoxon signed-rank test). Given our sample sizes, these statistical tests were applied outside the small-sample regime. Data were not explicitly tested for normality. p values were not corrected for multiple comparisons.

Performance. In all conditions, performance is reported in single-target sequences only, as computing performance in target-absent or both-target sequences was not interpretable. Because performance is typically not distributed normally across participants, we performed Wilcoxon signed-rank tests to compare participants’ performance to chance level (50%), or between conditions. Effect sizes are reported as r= z/sqrt(n) with 95% confidence intervals. We used a cluster-level permutation-based analysis to identify time periods where the fraction of target-directed choices was significantly different from chance performance. We first identified clusters of contiguous trials where choice direction was significantly different from chance using a two-sided p-value of 0.05 at the trial-wise level (t-tests against 50%). We extracted the statistics of each identified cluster as its sum of t-values. We then computed ‘null’ distributions for the cluster statistic by extracting the highest absolute cluster statistic for simulated random choices (N=1000 simulations). We then computed the empirical cluster-level p-value as the fraction of ‘null’ cluster statistics exceeding the observed cluster statistic.

Fraction of re-sampling the same option. We performed repeated-measures ANOVAs41 on the fraction of re-sampling from the same option as a function of the number of previous samples acquired from this option. We tested the interaction between the condition (MATCH and GUESS in the discovery and replication datasets; MATCH, GUESS and FIND in the third dataset) and the number of previous samples acquired from the current option (from 1 to 10). Effect sizes are reported as η2= SS_effect / (SS_effect + SS_residuals), where SS corresponds to the sum of squares of the effect (interaction) of interest, and SS_residuals is the aggregate sum of squares of the residual errors42. To identify for which numbers of acquired samples the fraction of resampling choices was significantly different between conditions, we used cluster-level permutation-based tests. First, we identified clusters of contiguous numbers of previous trials where the fraction resample was significantly different between conditions (p-value obtained from a paired-sample t-test < 0.05). We extracted the statistics of each identified cluster as its sum of t-values. Then we computed the ‘null’ distribution for the cluster sum of t-value by shuffling the condition labels of individual sequences within participants. For each participant, we re-assigned the sequence choices to each condition randomly and recomputed the cluster statistic in the same way (N=1000 permutations). We then computed the empirical cluster-level p-value as the fraction of ‘null’ cluster statistics exceeding the observed cluster statistic.

Choice similarity. At the choice level, choice similarity is defined as 1 if the participant chose the same option, and 0 if they chose a different option. Each choice in each sequence was compared to all the other choices in the same sequence (choice 1 was compared to choices 2:end, choice 2 to choices 3:end, etc), resulting in a similarity score for each pair of choice positions within each sequence. This was averaged across sequences, resulting in an average similarity score (between 0 and 1) for each position in the sequence. Because choice similarity is typically not distributed normally across participants, we performed Wilcoxon signed-rank tests to compare participants’ choice similarity to chance level (50%).

Choice direction. We used a cluster-level permutation-based analysis to identify time periods where choices were significantly target- or uncertainty-directed. We first identified clusters of contiguous trials where choice direction was significantly different from chance using a two-sided p-value of 0.05 at the trial-wise level (t-tests against 50%). We extracted the statistic of each identified cluster as its sum of t-values. We then computed ‘null’ distributions for the cluster statistic by extracting the highest absolute cluster statistic for simulated random choices (N=1000 simulations). We then computed the empirical cluster-level p-value as the fraction of ‘null’ cluster statistics exceeding the observed cluster statistic. We used a similar procedure to compare the fractions of uncertainty-directed choices between conditions. In this case, we computed ‘null’ distributions by permuting randomly the condition labels of individual sequences and extracting cluster statistics for these permuted conditions.

Model parameter estimates. For the initial statistics in the confirmatory dataset, all tests were two-sided tests: t-tests against zero and paired t-tests between conditions for unbounded, sampling parameters β_tar and β_unc, and for the repetition bias b_rep; and signed-rank tests between conditions for the bounded, initial sampling phase parameters θ and ε. Effect sizes are reported as Cohen’s d with 95% confidence intervals, and as r= z/sqrt(n) for Wilcoxon’s tests, with 95% confidence intervals. For the third dataset, we did one-sided tests where we had a clear prediction of the direction of the effect. We tested unbounded sensitivity parameters β_tar and β_unc with one-sided t-tests against zero and paired between conditions. The presence of a repetition bias was assessed through one-sided t-tests against zero, but two-sided paired t-tests to compare conditions. Similarly, we used one-sided signed-rank tests to compare θ values between conditions but two-sided signed-rank tests to compare ε values, as we had no directional prediction. The exact test used is reported either in the Results or on the corresponding Figure legend and in Supplementary Tables. Effect sizes are reported as Cohen’s d with 95% confidence intervals for t-tests, and as r= z/sqrt(n) for Wilcoxon’s tests, with 95% confidence intervals.

Inter-individual analyses. For this analysis, we split participants into groups of different sizes based on whether a model with or without the initial sampling phase best fit their behaviour in the GUESS condition. The groups were of unequal sizes. In this analysis, all tests were post-hoc and exploratory. We formally tested for equal variance between groups using the Levene test from Matlab. Differences in performance were tested with two-sided sum-rank tests (groups of unequal size and unequal variance), and effect sizes are reported as the probability of superiority along with bootstrapped 95% CIs. For differences in sensitivity to the target colour (β_tar) between the two groups (ini+ and ini− participants), no evidence was detected for a difference in variance between groups, and we compared the conditions with two-sided, two-sample t-tests, effect size was computed as Cohen’s d along with bootstrapped 95% CI. To test for differences in sensitivity to uncertainty (β_unc) between conditions and between groups of participants (ini+ vs. ini−) we ran a mixed effects anova as implemented by Laurent Caplette41 for Matlab. Effect sizes are reported as η2= SS_effect / (SS_effect + SS_residuals), where SS_effect corresponds to the sum of squares of the effect of interest, and SS_residuals is the aggregate sum of squares of the residual errors42. For post-hoc tests, the Levene test detected a difference in variance between groups and between conditions. We ran two-sided, two-sample t-tests for unequal variance with MATLAB’s ttest2 function (which uses Welch’s t-test and Satterthwaite approximation for degrees of freedom). Effect sizes were computed as Cohen’s d using the average of the variance of the two groups/conditions, along with bootstrapped 95% confidence intervals. We used the same approach for differences in repetition biases, except the Levene test did not detect evidence of a difference in variance, so we used the regular implementation of Matlab’s ttest2 function for two-sided, two-sample t-tests. Effect sizes were computed as Cohen’s d using the pooled variance, along with bootstrapped 95% confidence intervals.

Peer review rounds
[{"round": 1, "reviewer_comments": {"reviewer_1": "In this paper, Almeras and colleagues study human exploration strategies under different experimental conditions. When\npeople are asked to solve a prediction problem (guessing the outcome or finding the option that produces a particular\noption), they repeatedly sample each option until they have sufficiently reduced their uncertainty ('local' information seeking).\nIn contrast, when people are asked to solve a reward maximization problem (picking the rewarding option), they selectively\nsample the most uncertain option ('global' information seeking). These patterns are not captured by a 'rational' model. The\nauthors develop a process model that gives an excellent fit to the data.\nI have somewhat mixed feelings about this paper, though on the whole I like it. On the positive side, both the experimental\nand modeling parts are rigorous, the figures are beautiful, the topic is interesting and timely. I also commend the authors for\nreplicating their results, which yields strong confidence in their results. On the negative side, I felt that the paper is\noverburdened by lots of analyses that provide partially redundant information (e.g., multiple metrics that illustrate the same\nchoice pattern). The paper could be made more readable by condensing these results and perhaps moving some of them to\nthe supplement. A more substantive issue is that I'm not sure how big a conceptual leap the results offer. Some of these\nfindings (e.g., directed exploration of uncertain options) have been shown before. Repeated sampling of an option to reduce\nits uncertainty is, I believe, novel, though there are possibly other explanations for this kind of \"streaky\" behavior (see\nbelow).\nSpecific comments:\nInformation selection based on uncertainty reduction has been widely studied in cognitive psychology. So I was surprised\nthat references to this literature were missing. See Nelson (2005) for a useful review. More recently, there has been relevant\nwork by Bramley et al. (2015), Coenen et al. (2015), Meder et al. (2019), to name a few.\nIt is stated that the tendency to sample in streaks is suboptimal (p. 13). But as the authors acknowledge in the Discussion (p.\n17), one determinant of streaks might be the avoidance of switch costs. If we take this into consideration, perhaps streaks\naren't suboptimal after all? That is, people sample in streaks because switching is in some way costly.\nAnother argument for streaks might come from consideration of policy complexity, whereby perseveration arises from a limit\non the number of bits available to encode a policy (Gershman, 2020).\nYet another perspective comes from studies of concept learning, which have documented a kind of \"stickiness\" in\nhypothesis testing behavior. Restle (1962) proposed that subjects sample one hypothesis at a time, only switching when\nthey receive disconfirming evidence (see also Bower & Trabasso, 1963). A more recent stochastic version of this idea was\nintroduced by Bonawitz et al. (2014).\nMinor comments:\nLine 160: I think this should be a reference to Supplementary Figure 2?\nThere appears to be a mismatch between the panel labels and the caption for Figure 6.\nWere subjects told about the relationship between color intensity and informativeness?", "reviewer_2": "Review of Competing cognitive pressures on human exploration in the absence of trade-off with exploitation\nSummary\nThis paper presents three studies on human exploration in a two-option\ntask. Participants are presented repeatedly with two options, and\nchoosing an option provides an outcome on a continuous colour scale from\nblue to orange. Different conditions set different goals: In the DRAW\ncondition, participants were instructed to obtain outcomes closest to\none of the colours (this is like a standard two-armed bandit task). In\nthe GUESS condition, they could sample freely from the options, and then\nwere asked to determine the dominant colour for each of the options (Exp\n1 and 2) or to pick an option with a specified dominant colour (Exp 3).\nThe FIND condition was like the GUESS condition in Exp 3, but\nparticipants were informed beforehand of the colour their final choice\nshould correspond to. Results show that participants allocate their\nchoices differently to the options in the different conditions. In the\nDRAW condition, participants gradually choose more often the option\nwhich delivers more of the rewarding colour. In the GUESS condition,\nchoice patterns show “streaks” where participants choose one option for\nsome time (regardless of outcome) and then the other. The FIND condition\nappears to be a mix of these two. These patterns are eluded in a variety\nof analyses, including a computational analysis.\nEvaluation\nWhilst I think there are interesting aspects to the studies, and the\nfindings show potentially interesting patterns, I don’t believe the\npaper in its present form is ready for publication. The key finding\nabout initial streaks in choices in the GUESS/FIND conditions may be of\ninterest to select readers, but needs a more convincing explanation than\nprovided. It also does not warrant a paper of this length. I will detail\nmy major concerns are detailed below.\nMajor issues\n1. The framing of the paper seems a little off to me. The main aim is\nto study exploration with and without an exploitation trade-off. The\nFIND condition seems a more appropriate comparison to the DRAW\ncondition for this then the GUESS condition, as the latter changes\nthe goal of information seeking. In the DRAW and FIND conditions,\nthe aim is to find the option which dominates in producing more\noutcomes of a given colour. In the GUESS condition, the aim is to learn the\ndominant colour of each options. That choice patterns\ndiffer when given a different goal is not that striking. One goal\ninvolves a direct comparison between options (which option offers\nmore of something), whilst the other option does not (need to learn\ncharacteristics of each option). Differences between the DRAW and\nFIND condition are more informative about the influence of the\nexplore-exploit tradeoff on exploration. So, to me, the key\nconditions are DRAW and FIND, and the GUESS condition is more of an\nadd-on. But the paper focuses mostly on the DRAW and GUESS\nconditions, and then introduces the FIND condition as an additional\ncheck. I suggest focusing on the third experiment, and then perhaps\nshowing that the results of two conditions replicate in two other\nexperiments. This could offer a more focused and shorter paper (see\nalso below).\n2. The main finding is that in the GUESS/FIND conditions, participants\nshow initial “streaks” in choices, repeatedly sampling the same\noption in a row. The explanation for this is vague, and also not\nclearly addressed in their model, which just includes this as a\ncomponent without being able to base this on e.g. aspects of\nexperience or the goal. The explanation provided (“a ‘local’ form of\ninformation seeking that aims at generating and testing a hypothesis\nabout the currently sampled option” which contrasts to a “global”\nform of information seeking through directed sampling of the most\nuncertain option) may be true, but is entirely speculative. Even the\nmodel provided does not include any notion of hypothesis testing,\n3. The “process model”, as I see it, is not really a process model, but\nmore of a data-analytic model. No reason is provided for turning the\nthe “initial sampling” component “on” or “off”. A data-analytic\nmodel is fine, and it succeeds in highlighting the differences in\nsearch patterns. But then there are claims that certain parameter\nsettings for this model correspond to rational models, yet it is\nentirely unclear what criterion of rationality is alluded to. If I\nunderstand the task correctly, the purported “rational models” would\nfail at the task completely! The model includes mean colour estimates for\neach option, as well as what I presume are the absolute values of these estimates. For the “rational”\nDRAW model, choices are based on the difference in mean colour estimates between\nthe options, i.e. P(choose 1) = F(β_(tar) × (x̂_(t, 1) − x̂_(t, 2))),\nwith F(x) = 1/(1 + exp ( − x)). This is a simple model for a\ntwo-armed bandit, but by no means optimal. Given the short horizon,\nan optimal model might be possible through defining the task as a\nPOMDP (accounting also for the variability in horizons). So the\nmodel is not optimal for the task, which is usually what is meant by\na rational model. Furthermore, the authors claim that the rational\nmodel would set β_(tar) → ∞. This implies a pure exploit strategy,\nwithout exploration. This model always chooses option 1 if\nx̂_(t, 1) − x̂_(t, 2) > 0 and option 2 if x̂_(t, 1) − x̂_(t, 2) < 0. If\nboth options have a positive mean, but option 2 a higher mean, then\nit could be the case that after sampling option 1, option 2 is never\nsampled, so the model would not find the best option. In the present\nexperiment, there are only two possible means for each option, so\nthis is not a practical concern, but were participants aware of this\ndesign constraint? For the GUESS condition, the “rational” model\nbases its choices (if I understand this correctly, the information\nprovided is unclear) on\nP(choose 1) = F(β_(inf) × (|x̂_(t, 2)| − |x̂_(t, 1)|)). I don’t follow\nhow the absolute mean value estimates encode “information already\nacquired”. It would make more sense to me to relate this to the\nnumber of times an option has been sampled, or a value encoding the\nuncertainty associated with the value estimate (which will also\ndepend on the number of samples). Again, the “rational model” sets\nβ_(inf) → ∞, which, if |x̂_(t, 2)| − |x̂_(t, 1)| encodes information\nobtained, might make sense here. But to what extent is this\nrational? Does following these choice strategy maximise the\nprobability of getting the dominant colour for both options right?\nThe model should be specified in much more detail (I had to guess\ncertain parts) and it should be considered how this model provides\noptimal (or at least very good) performance on the tasks.\n4. Whilst the highly detailed analyses have merits, I struggled to “see\nthe forest for the trees”. A much more focused and shorted paper\nwith key analyses presented in the main paper and supporting ones in\nsupplementary material, would work better, I believe. For choice\npatterns, we get a range of analyses (sampling imbalance, option\nre-sampling, choice-choice similarity) which all indicate the same\nthing (early streaks vs gradual increase to preferred option). I\nsuggest focusing on one of these and including other analyses as\nsupplementary material. I also suggest to mainly focus on experiment\n3, and report the other two as replicating effects.\n5. There is a large literature on contrasting “Decisions from\ndescription” to “Decisions from experience”. The latter is either\noperationalised as a standard multi-armed bandit, or via a “sampling\nparadigm” where participants can freely sample from the options,\nuntil making a final (rewarded) choice. These two\noperationalisations have also been contrasted (see e.g. Wulf et al\n2017, “A Meta-Analytic Review of Two Modes of Learning and the\nDescription‒Experience Gap”). This literature is highly relevant but\nhas been (almost) completely ignored in this paper.\n6. The paper is unclear/does not provide crucial information in various\nplaces. Key aspects (such as that this was a within-subjects\ndesign), should be reported in the introduction or results (this was\nonly evident later when it turned out to be possible to study\ncovariation over conditions). Crucial parts of the design (e.g., is\nthe order of conditions randomised or counterbalanced, and if so\nhow?) are not detailed. The model is described in vague terms,\nrather than a clear formula of its workings and how parameters are\ninitialized. It also seems that the GUESS condition differs between\nExp3 and Exp 1&2 (i the last experiment, participants are asked\nwhich of two options is blue or orange, whilst in the first two they\nare asked about the dominant colour of one of the options), but this\nis not clearly stated. How is choice similarity defined? In Fig 7.\nIn the top plots, it is unclear that subplots compare DRAW to GUESS.\nEtc."}, "author_response": "We thank both reviewers for their helpful comments. We have done our best to address the different\nsuggestions of the reviewers, we hope they agree that this has improved the manuscript.\nOverall, we have condensed the presentation of the results, which we hope has increased the clarity of the\npaper. Thanks to the suggestions reviewer #2, we have changed the process model we use: evidence\naccumulation now directly reflects the optimal strategy, and the initial sampling phase no longer depends\non a number of samples but rather on the level of evidence acquired about the current option.\nWe address the specific comments of the reviewers below; our responses are in green and we highlighted\nquotes from the revised manuscript in blue.\nReviewer #1 (Remarks to the Author):\nIn this paper, Almeras and colleagues study human exploration strategies under different\nexperimental conditions. When people are asked to solve a prediction problem (guessing the\noutcome or finding the option that produces a particular option), they repeatedly sample each option\nuntil they have sufficiently reduced their uncertainty ('local' information seeking). In contrast, when\npeople are asked to solve a reward maximization problem (picking the rewarding option), they\nselectively sample the most uncertain option ('global' information seeking). These patterns are not\ncaptured by a 'rational' model. The authors develop a process model that gives an excellent fit to\nthe data.\nI have somewhat mixed feelings about this paper, though on the whole I like it. On the positive side,\nboth the experimental and modeling parts are rigorous, the figures are beautiful, the topic is\ninteresting and timely. I also commend the authors for replicating their results, which yields strong\nconfidence in their results. On the negative side, I felt that the paper is overburdened by lots of\nanalyses that provide partially redundant information (e.g., multiple metrics that illustrate the same\nchoice pattern). The paper could be made more readable by condensing these results and perhaps\nmoving some of them to the supplement. A more substantive issue is that I'm not sure how big a\nconceptual leap the results offer. Some of these findings (e.g., directed exploration of uncertain\noptions) have been shown before. Repeated sampling of an option to reduce its uncertainty is, I\nbelieve, novel, though there are possibly other explanations for this kind of \"streaky\" behavior (see\nbelow).\nWe thank the reviewer for their careful assessment of our manuscript, their helpful comments, and literature\nsuggestions. We have trimmed the presentation of the results and we hope this helps highlight the specific\ncontribution of the paper. We answered specific comments below.\nSpecific comments:\nInformation selection based on uncertainty reduction has been widely studied in cognitive\npsychology. So I was surprised that references to this literature were missing. See Nelson (2005)\nfor a useful review. More recently, there has been relevant work by Bramley et al. (2015), Coenen et\nal. (2015), Meder et al. (2019), to name a few.\nWe thank the reviewer for pointing out these relevant references: we agree that sampling patterns aiming to\nreduce uncertainty have been documented in other tasks. We believe that the contribution of our\nmanuscript is to propose very comparable conditions, where sampling patterns can be directly contrasted\nunder different constraints/goals. This contrast allows us to show how the immediate goal of accumulating\nrewards (in the canonical studies of exploration in bandit tasks) distorts sampling patterns. We have now\nincluded these references to the discussion of directed exploration:\nDiscussion page 15: “Targeted sampling based on uncertainty reduction has been documented in various\ninformation gathering tasks13–16. However, when rewards are involved, previous work has shown that such\n1\ntargeted selection of uncertain options critically depends on lengthening the temporal ‘horizon’ of the task\n(i.e. delaying rewards): such targeted sampling is often described as ‘directed’ exploration in contrast to\n‘random’ exploration9,10. Consistently, in our dataset, directed exploration was evident whenever rewards\nwere delayed: when participants were rewarded for their response at the final question. In addition,\nbecause the paradigm allowed to contrast different delayed rewards situations, we observed that directed\nexploration appeared to only be sensitive to delaying rewards, and not to whether outcomes were\nimmediately informative about the ultimate rewarding value of options (i.e., it reached comparable levels in\nthe GUESS and FIND conditions).”\nIt is stated that the tendency to sample in streaks is suboptimal (p. 13). But as the authors\nacknowledge in the Discussion (p. 17), one determinant of streaks might be the avoidance of switch\ncosts. If we take this into consideration, perhaps streaks aren't suboptimal after all? That is, people\nsample in streaks because switching is in some way costly.\nThank you for helping us realise this wasn’t clear. Repetitive sampling is statistically suboptimal (i.e. it is not\nthe optimal strategy): in the GUESS condition, the best strategy is to always sample from the option for\nwhich the least information has been accumulated, as this will maximise information gain. We have made\nthis clearer on Figure 3 and Supplementary Figure 2, as well as in the description of the model page 6:\n“Whereas in the GUESS condition, the optimal strategy is to always chose the option with lowest absolute\naccumulated value (the most uncertain option – Figure 3.d., Supplementary Figure 2.b.). This allows the\nagent to maximise information about both shapes at any trial – thus maximising its probability of responding\ncorrectly to the final question, whose timing was not disclosed to participants.”\nThe strategy of the participants (repeatedly sampling the same option) is thus suboptimal, and will produce\nobjectively degraded evidence. We propose that this strategy results from cognitive limitations: because\nswitching is costly, participants preferred to sample in streak. Yet, we observed that the participants who\nadopted such repeated sampling strategies obtained higher performance. This suggests that repetitive\nsampling could be cognitively more efficient: that observing evidence in streaks makes accumulation by\nhuman brains easier. We speculate that reducing switch costs, for instance, may have allowed participants\nto make better use of the evidence they sampled, however statistically degraded. In other words, even\nthough sampling streaks are statistically suboptimal, they may be cognitively optimal: they overcome\ncognitive limitations (the switch costs) to produce more efficient evidence and thus better performance.\nIt is unclear at this point whether this works by improving the (e.g. sensory) quality of the evidence sampled\n(akin to an attentional effect), or by reducing the probability of forgetting all or part of the information about\nthe option during a switch (a memory retrieval effect), or through any other mechanism.\nWe thank the reviewer for helping us clarify this point, we have amended the discussion to highlight the\ndifference between statistical and cognitive suboptimality.\nDiscussion page 17: “[…] Participants who displayed patterned sampling had higher repetition biases, but\nwere also more accurate at identifying the overall structure of each sequence (i.e. the option outcome\nassociations), and were more sensitive to task-relevant decision variables, even in conditions where they\ndid not display the patterned sampling (in the DRAW condition, where sequences required to maximise\nimmediate reward). These results are reminiscent of work from Hills and colleagues5, using the sampling\nparadigm in Decisions From Experience tasks (where participants who sampled for longer also produced\nlonger streaks for each option), and of work from Abir and colleagues38, using a two-step task (in which\nparticipants who obtained higher performance also had higher repetition bias and more strategic sensitivity\nto uncertainty). The positive relationship we document between initial repetitive sampling (a suboptimal\nstrategy in normative terms) and task performance is important, because it suggests that patterned\nsampling reflects an active cognitive strategy, rather than mere task disengagement or inattention. It\nsuggests that statistically suboptimal strategies could be more efficient for human subjects, which\noperate under specific biological constraints. Even though they stem from cognitive limitations in\nthe first place, strategies which produce objectively degraded evidence could be more cognitively\noptimal, under the specific biological constraints of the human brain (see 18,39,40 for comparable\n2\nfindings).\nThe interpretation of this positive relation between initial repeated sampling and task performance raises\ntwo questions. First, the question of the mechanisms through which repetitive sampling leads to\nhigher performance despite degraded evidence. Repetitive sampling reduces switch costs24,25 and\ncould thus alleviate the cognitive effort required; but it is unclear at this point whether this\nimproves the (e.g. sensory) quality of the evidence sampled (akin to an attentional effect), or\nreduces the probability of forgetting all or part of the information about the option during a switch\n(a memory retrieval effect), or facilitates processing through any other mechanism.”\nAnother argument for streaks might come from consideration of policy complexity, whereby\nperseveration arises from a limit on the number of bits available to encode a policy (Gershman,\n2020).\nWe thank the reviewer for pointing out this relevant reference which shows that policy perseveration can\narise from memory limitations leading to policy simplification/compression. In our case, participants were\nonly ever choosing between two options, which limits the complexity of their potential policies. For this\nreason, we don’t expect complexity reduction to be the prime pressure shaping participants’ choices.\nIn addition, in our specific case of initial repetitions, and considering the revised version of the model\nwherein repeating initial decisions depends on whether the level of accumulated evidence for the current\noption has surpassed a given information threshold, we propose that streaking is not mere perseveration,\nand actually makes the policy more complex. Repetition here is not merely motor, as the options’ symbols\ncould switch sides at any time – repeating therefore required active executive control. But repetition of a\nsymbol is also not blind: it requires the agent to compare the current level of evidence to the threshold\n(parameter θ in the model), and to decide whether to repeat or not, based on this comparison. In these\nterms, we believe that initial streaking is a more complex policy than simply always choosing the most\nuncertain option (the optimal strategy). While streaking seems behaviourally simple, and while it possibly\nstems from cognitive limitations (the costs of switching), it does not necessarily manifest in simpler policies.\nWhile we argue that streaking might lead to more efficient, perhaps easier, evidence accumulation, it is\nindeed a more complex sampling strategy.\nWe have added a paragraph to the discussion to address this comment pages 16-17: “Similarly, the\nevidence-dependent repetitive sampling in the beginning of sequences is not reducible to mere\nperseveration. While previous studies have shown that policy perseveration could arise from memory\nlimitations leading to policy simplification33,34, the initial repetitions we describe are hardly a simplification of\nthe policy. Repetitions here are not blind: participants only repeat initial choices as long as the level of\nevidence acquired on the preceding choices is below a certain threshold. In other words, this conditional\npolicy requires to compare the current level of evidence for the option to the threshold level (parameter θ in\nthe model), and based on this comparison, to decide whether to repeat or to compute the decision variable.\nAs such, initial repetitions are a more complex policy than simply always computing the decision variable.\nIncidentally, in this task, participants are only choosing between two options: This restrains the complexity\nof potential policies, and makes it unlikely that policy compression be the prime driver of the smart\nrepetition strategies we observed.”\nYet another perspective comes from studies of concept learning, which have documented a kind of\n\"stickiness\" in hypothesis testing behavior. Restle (1962) proposed that subjects sample one\nhypothesis at a time, only switching when they receive disconfirming evidence (see also Bower &\nTrabasso, 1963). A more recent stochastic version of this idea was introduced by Bonawitz et al.\n(2014).\nAgain, we thank the reviewer for suggesting interesting references. In the tasks mentioned by the reviewer,\nparticipants are presented with stimuli, and the task is to correctly categorise them. In these tasks, adults\nand children are shown to stick to their current hypothesis (about category boundaries compatible with their\n3\nobservations), as long as they don’t receive critically disconfirmatory information. In our task as well,\nparticipants had to categorise each bag as either orange or blue. But, in contrast to the tasks mentioned\nabove, our participants also had to actively select which option to sample from: the repetitive sampling\npatterns we describe don’t refer to participants’ sampling into the hypothesis space, but between the two\noptions. It is thus very well possible that participants stuck to a hypothesis (“this bag is blue / orange”)\nunless critically disconfirmed (like the participants of Restle, Bower and Trabasso, and Bonawitz and\ncolleagues), but this alone doesn’t explain why they should want to re-sample from the same option -\nespecially since they are incentivised to maximise information about both options. We could speculate that\nthey repeated their choice until they received sufficiently disconfirmatory information. But the updated\nversion of the model we propose suggests, on the contrary, that participants actually initially switched\noptions only when their category hypothesis about the current option had been sufficiently confirmed (i.e.\nwhen they reached a specific level of evidence in favour of the current hypothesis). Note that the two\npossibilities are not mutually exclusive: it is possible that participants only switched hypotheses about an\noption when sufficiently disconfirmed, but our results suggest that they also wanted to resample the same\noption until they were x% confident in their hypothesis.\nWe have added a discussion of this point to the discussion on page 16: “It is important to distinguish\nbetween the sampling of options and the sampling of the hypotheses. Previous studies of categorisation or\nconcept learning have shown that when participants were gradually presented with evidence, they didn’t\ntend to resample the best-fitting hypothesis with each piece of new incoming evidence, but rather stuck to\nthe same hypothesis until it was critically disconfirmed (e.g. 29–31). Our results pertain to a different level of\ndescription: stickiness about the hypotheses doesn’t explain why participants would want to resample the\nsame option. It is very well possible that our participants were also conservative about hypotheses\nand only changed their minds in the face of sufficiently contradictory evidence, but these results\nalso suggest that they wanted to resample the same option until they were sufficiently convinced of\ntheir current hypothesis – i.e. until they reached a certain level of evidence in favour of said\nhypothesis.”\nMinor comments:\nLine 160: I think this should be a reference to Supplementary Figure 2?\nThere appears to be a mismatch between the panel labels and the caption for Figure 6.\nThank you for spotting this, we have fixed these imprecisions.\nWere subjects told about the relationship between color intensity and informativeness?\nYes, indeed. We apologise for the ambiguity; we have amended the Methods to make this clearer,\nthank\nyou. Page 20: “Participants were instructed on how to perform the task by a step-by-step, self-paced\ntutorial, followed by a practice session of eight sequences. Participants were then asked to describe the\ninstructions to the experimenter, and it was emphasised again for all participants that: 1. the colour\nintensity of each outcome was relevant, 2. each option would not always draw the exact same shade, so\nthe task was to identify its dominant colour, 3. there were only two possible fixed distributions (blue and\norange), 4. the two options were not always associated with different colours (they could be both\nassociated with orange, or blue), 5. each sequence would end after a variable, unpredictable number of\nchoices, and 6. performance was measured by the colours of all sampled outcomes in the DRAW\ncondition, and by the final question in the GUESS and FIND conditions.”\nWe are grateful to reviewer #1 for their valuable challenges and suggestions. We hope that our responses\ncan convince the reviewer that this set of experiments documents genuinely novel contrasts in sampling\npatterns, which could not be evidenced in existing tasks and cannot be directly explained by existing data.\nReview of Competing cognitive pressures on human exploration in the absence of trade-off with\nexploitation\nSummary\nThis paper presents three studies on human exploration in a two-option task. Participants are\npresented repeatedly with two options, and choosing an option provides an outcome on a\ncontinuous colour scale from blue to orange. Different conditions set different goals: In the DRAW\ncondition, participants were instructed to obtain outcomes closest to one of the colours (this is like a\nstandard two-armed bandit task). In the GUESS condition, they could sample freely from the\noptions, and then were asked to determine the dominant colour for each of the options (Exp 1 and\n2) or to pick an option with a specified dominant colour (Exp 3). The FIND condition was like the\nGUESS condition in Exp 3, but participants were informed beforehand of the colour their final choice\nshould correspond to. Results show that participants allocate their choices differently to the options\nin the different conditions. In the DRAW condition, participants gradually choose more often the\noption which delivers more of the rewarding colour. In the GUESS condition, choice patterns show\n“streaks” where participants choose one option for some time (regardless of outcome) and then the\nother. The FIND condition appears to be a mix of these two. These patterns are eluded in a variety\nof analyses, including a computational analysis.\nEvaluation\nWhilst I think there are interesting aspects to the studies, and the\nfindings show potentially interesting patterns, I don’t believe the\npaper in its present form is ready for publication. The key finding\nabout initial streaks in choices in the GUESS/FIND conditions may be of\ninterest to select readers, but needs a more convincing explanation than\nprovided. It also does not warrant a paper of this length. I will detail\nmy major concerns are detailed below.\nWe thank the reviewer for taking the time to evaluate our paper, and for raising several points which\nallowed us to improve the computational analyses and presentation of the results. We hope this revised\npaper will convince the reviewer of the strength of our findings and their relevance in light of the existing\nliterature on human exploration patterns.\nMajor issues\n1. The framing of the paper seems a little off to me. The main aim is to study exploration with and\nwithout an exploitation trade-off. The FIND condition seems a more appropriate comparison to the\nDRAW condition for this then the GUESS condition, as the latter changes the goal of information\nseeking. In the DRAW and FIND conditions, the aim is to find the option which dominates in\nproducing more outcomes of a given colour. In the GUESS condition, the aim is to learn the\ndominant colour of each options. That choice patterns differ when given a different goal is not that\nstriking. One goal involves a direct comparison between options (which option offers more of\nsomething), whilst the other option does not (need to learn characteristics of each option).\nDifferences between the DRAW and FIND condition are more informative about the influence of the\nexplore-exploit trade-off on exploration. So, to me, the key conditions are DRAW and FIND, and the\nGUESS condition is more of an add-on. But the paper focuses mostly on the DRAW and GUESS\nconditions, and then introduces the FIND condition as an additional check. I suggest focusing on the\nthird experiment, and then perhaps showing that the results of two conditions replicate in two other\nexperiments. This could offer a more focused and shorter paper (see also below).\nWe would like to thank the reviewer for giving us the opportunity to clarify the framing of the paper. We\nagree that the FIND condition makes our results more directly comparable to the existing literature, and that\nthis condition was necessary to disentangle the effects of delaying rewards from those of the absence of a\ntarget colour in the GUESS condition. The FIND condition relates more directly to existing delayed-rewards\ntask: notably, the Horizon task (e.g. Wilson2014) and the Sampling Paradigm of Decisions From\nExperience tasks (e.g. Hills and Hertwig 2010, Wulff et al. 2015). In both these cases, participants are\nallowed to sample freely during a certain period before they can make an incentivised choice (either at a\nforced time in the Horizon task or self-paced in the DFE tasks). These tasks have shown that delaying\nrewards allows directed exploration (horizon task), and that participants who explore longer also alternate\nless, effectively exploring in streaks (Hills & Hertwig 2010).\nWe have included these references in the introduction of the third dataset page 9: “This new condition was\nakin to browsing restaurant reviews: choices yielded no immediate reward, but outcomes were still valued\nwith respect to an end goal. It was also closer to existing delayed-reward tasks, like the Horizon task9 or in\nthe “sampling paradigm” from Decisions From Experience studies12. In these paradigms, rewards are\nonly granted after a sequence of so-called “free choices” (as in the GUESS condition), but (in contrast to\nthe GUESS condition), choice outcomes are numerical values, and therefore directly inform the participants\nof the option’s future rewarding value. In these tasks, it is thus possible that a future reward value already\ntaints the outcomes and biases choice patterns. While the original GUESS condition was necessary to\neliminates this concern, the addition of the FIND condition eliminate a gap in the paradigm. Here, the\ncontrast between the FIND and DRAW conditions reflects the difference between delayed and immediate\nexploitation, whereas the contrast between the FIND and GUESS conditions reflects the difference\nbetween the sampling of outcomes with and without rewarding value.”\nWe have also related Hills and Hertwig’s key observations to our results in the discussion page 17: “These\nresults are reminiscent of work from Hills and colleagues5, using the sampling paradigm in Decisions\nFrom Experience tasks (where participants who sampled for longer also produced longer streaks\nfor each option), and of work from Abir and colleagues38, using a two-step task (in which\nparticipants who obtained higher performance also had higher repetition bias and more strategic sensitivity\nto uncertainty).”.\nNevertheless, in all these tasks, even though outcomes don’t directly contribute to participants’ earnings,\nthey are expressed in a format which is directly informative about future rewards. It is thus possible that\noutcomes already acquire a prospective reward value, potentially skewing the sampling patterns. In the\nHorizon task or the sampling paradigm, this can happen because outcomes are directly expressed in\npoints, and in the FIND condition, this can happen because participants already know the target colour (on\nwhich they will be rewarded) before the start of the sequence. In this condition, participants did delay\nproper exploitation to the end of the sequences: the lower tendency to seek the target colour in the FIND\nthan in the DRAW condition, and the difference in target-oriented choices between the last sampling\ndecision and the final question in the FIND condition both attest that the delayed-reward manipulation did\nwork. However, even though they were explicitly not rewarded at each decision, participants did have a\nsignificant tendency to sample the target coloured option (visible on the choice direction curves and on the\nestimates of their sensitivity to the target β ). In other words, sampling patterns during the sequence were\ntar\naffected by participants’ knowledge of the target colour, which tainted outcomes with prospective reward\nvalue, even though actual rewards were delayed.\nAs an alternative, it is possible that target sensitivity in the FIND condition reflected the fact that knowing\nthe target colour in advance channelled the attention of participants onto a single option at a time. Because\nwe did not ask participants to report “the bluest option”, but simply a blue option, it was sufficient for\nperformance to quickly identify the bluest option in the first couple of trials (possibly through streaks of\nrepeated decisions), and then to only sample the bluest option unless sufficiently disconfirmed. In other\n6\nwords, knowing the target colour in advance narrowed the focus of the task to a single symbol at a time: it\nwas sufficient to attend to a single shape. Such a strategy would also manifest as a preference for the\ntarget colour, and our analysis wasn’t designed to disentangle these two interpretations. Yet, the correlation\nof target sensitivities in the DRAW and FIND conditions (Supp. Fig. 7) suggests that target sensitivity in the\nFIND condition could reflect outcome valuation– i.e. that outcome valuation would at least partially drive\ntarget preference in the FIND condition. Fully disentangling the two interpretations would require additional\nwork, including new task conditions. Regardless of the interpretation, behaviour shows that exploration in\nconditions of delayed rewards, like FIND, is still not completely free from the attraction of exploitation.\nFor these reasons, in order to study exploration when it is completely separated from exploitation, we\nneeded a task in which outcomes were not immediately translated into a prospective reward value. Using\ncolour shades gave us this flexibility. Through different instructions, we could manipulate the information\nand value they carried. In the GUESS condition, we only revealed the rewarding colour at the end of the\nsequence: during sampling, outcomes were thus completely free of any intrinsic reward value – i.e. it was\nnot any better to see orange or blue outcomes. This condition was thus crucial to study sampling patterns\noutside of the influence of rewards. And indeed, while directed exploration (the fact that choices were\nsensitive to the uncertainty of options) was present whenever rewards were delayed (FIND and GUESS\nconditions), repetitive sampling was most prevalent when rewards were delayed and outcomes conveyed\nno immediate information about the ultimate reward value of options (GUESS). It was sensibly decreased\nin the FIND condition (where the colour outcomes were immediately translatable into a prospective reward\nvalue because participants were given the target colour at the beginning of the sequence). Completely\nremoving the immediate influence of rewards (the GUESS condition) was thus necessary for the novel\nsampling patterns to manifest. We hope that this helps convince reviewer #2 that the DRAW vs. GUESS\ncontrast is relevant as the main focus of this paper.\nWe have tried to clarify this in the general introduction p 2-3: Variants of these tasks have\ndissociated exploration from reward maximization by delaying the acquisition of rewards. In such tasks,\nparticipants either receive information (e.g. 9,10 through forced choices) or sample the environment for\ninformation (e.g. in the sampling paradigm from Decisions From Experience research5,11,12) before they\nhave to make a rewarding decision. However, in these tasks, the information acquired during the sampling\nphase immediately concerns the ultimate rewarding values of options. Outcomes do not immediately\ncontribute to participants’ final earnings, but since they are expressed in points, they immediately reveal\nwhich option will ultimately be rewarding, which potentially biases choices – this is similar to browsing\nrestaurant reviews. This observation means that the exploration patterns observed in such delayed-\nrewards tasks are still constrained by the prospective rewarding values of available options.\nHere we sought to fully dissociate exploration from reward-guided decision-making by designing a\nnovel two-armed bandit task where the outcomes of the two options are shades of colour instead of\npoints or monetary rewards. In themselves, outcomes thus provide information about the structure\nof the task (the option-colour associations), but carry no intrinsic value (Figure 1.a.). In one\ncondition, we rewarded participants for drawing one particular colour. This instruction does confer a\nrewarding value to each outcome, and requires participants to trade exploration against exploitation to\nchoose the option associated with the instructed colour – as in regular bandit tasks. By contrast, in a\nsecond condition, the same participants (a within subject design) were told that they were sampling to find\nan option of one particular colour, which they only had to indicate once, at the end of the sequence of\nseveral free choices. Participants were rewarded based on their ability to indicate correct option-outcome\nassociations after the end of the sequence, but each outcome directly informed them about the options’\nfuture value – similar to a classic delayed reward task. In a third, novel condition, the target colour was\nonly revealed at the end of the sampling sequence. Participants sampled the options to learn the\noption-colour pairing and a final two-alternative forced choice asked them to indicate an option of a\nspecific colour (only revealed then). Because participants could not foresee the ultimate rewarding\nvalue of options, this instruction fully dissociates exploration from any form of immediate or\n7\nprospective reward maximisation. Except for this difference, the three conditions were matched in every\npossible aspect so that sampling patterns could be directly compared between them (Figure 1 & 5; see\nMethods).”\nWe have also highlighted this difference when introduction the third dataset (condition FIND), on page 9:\n“The GUESS condition (in which participants sample freely the two options to learn option-outcome\nassociations), differs from the DRAW condition (in which the same participants seek to sample a rewarded\noutcome) in two important ways. First, in the DRAW condition, as in traditional bandits, participants are\nrewarded for the outcomes of each individual choice in the sequence, whereas in the GUESS condition\nrewards are delayed to a final, single two-alternative forced choice at the end of each sequence. Second, in\nthe DRAW condition, outcomes are associated with a positive or negative value depending on the target\ncolour (they are directly associated to points), whereas in the GUESS condition, they only convey\ninformation about the structure of the task (which option is blue or orange), but cannot immediately be\nrelated to the future rewarding value of option. While it was the goal of our manipulation to completely\nrelease exploration from the influence of rewards, it means that the two factors (delayed exploitation and\noutcome valuation) were confounded and it was impossible to know the extent to which each of them\ncontributed to the specific exploration patterns observed in the GUESS condition.\nTo address this issue, we designed a third condition (FIND) where participants were asked to identify, at\nthe end of the sequence, an option associated with a target colour instructed at the beginning of the\nsequence (Figure 5.a.). In this new condition, as in the DRAW condition, each outcome was associated\nwith a positive or negative value depending on the target colour. However, as in the GUESS condition,\nparticipants were not rewarded based on the outcomes they were sampling, but based on their answers to\na final two-alternative forced choice at the end of each sequence (where they had to select an option\nassociated with the target colour). This new condition was akin to browsing restaurant reviews: choices\nyielded no immediate reward, but outcomes were still valued with respect to an end goal. It was also closer\nto existing delayed-reward tasks, like the Horizon task9 or in the “sampling paradigm” from Decisions From\nExperience studies12. In these paradigms, rewards are only granted after a sequence of so-called “free\nchoices” (as in the GUESS condition), but (in contrast to the GUESS condition), choice outcomes are\nnumerical values, and therefore directly inform the participants of the option’s ultimate rewarding value. In\nthese tasks, it is thus possible that a future reward value already taints the outcomes and biases choice\npatterns. While the original GUESS condition was necessary to eliminates this concern, the addition of the\nFIND condition eliminate a gap in the paradigm. Here, the contrast between the FIND and DRAW\nconditions reflects the difference between delayed and immediate exploitation, whereas the contrast\nbetween the FIND and GUESS conditions reflects the difference between the sampling of\noutcomes with and without rewarding value.”\nAnd finally tried to highlight the specific contribution of the GUESS condition at the end of the discussion,\non page 18: “Taken together, our findings delineate specific features of human exploration patterns in the\nabsence of trade-off with immediate reward maximisation. While random exploration and repetition biases\nwere present in all cases, we only found evidence for directed exploration in cases where rewards were\ndelayed to the end of the sequence. Additionally, we showed that when outcomes only provide\ninformation about the structure of the world (as is often the case in the real world), rather than\ndirectly about the option’s ultimate value, participants preferred to initially explore each option in\nturn before they engaged in directed exploration. Removing the prospective-value bias reveals\ncompeting cognitive pressures on information seeking that are neither described nor present in\nconditions eliciting explore-exploit dilemmas: 1. a ‘local’ form of information seeking that aims at\ngenerating and testing a hypothesis about the currently sampled option, and 2. a ‘global’ form of\ninformation seeking that aims at reducing the overall uncertainty about the task structure. These results call\nfor further research into the neural mechanisms of these exploration patterns outside of any explore-exploit\ndilemma, and into their ecological/psychiatric validity. A general conclusion of our study is that humans\ndisplay specific exploration patterns when they can freely acquire information that is not associated with\nreward. Our findings thus suggest that restricting the study of exploration to explore-exploit dilemmas may\nhave taught us more about how humans arbitrate this trade-off – and its neural correlates, than about\nexploration itself32.”\n2. The main finding is that in the GUESS/FIND conditions, participants show initial “streaks” in choices,\nrepeatedly sampling the same option in a row. The explanation for this is vague, and also not clearly\naddressed in their model, which just includes this as a component without being able to base this on\ne.g. aspects of experience or the goal. The explanation provided (“a ‘local’ form of information seeking\nthat aims at generating and testing a hypothesis about the currently sampled option” which contrasts to\na “global” form of information seeking through directed sampling of the most uncertain option) may be\ntrue, but is entirely speculative. Even the model provided does not include any notion of hypothesis\ntesting,\nWe understand the reviewer’s point. We have updated the model to a more cognitively motivated version\n(see below, our response to point (3)), in a way that directly addresses the reviewer’s comment. The\nduration of the initial streak of repeated choices is now dependent on how much information the agent has\ngathered about the current option. More specifically, in this model, the agent will initially repeat its choice\nuntil it has gathered a specific, agent-dependent level of information about the option: It will repeat its\nchoice until it has reached a certain level of confidence about its hypothesis. This revised model yields\ncomparable results to the initial version, but provides a more motivated account of participants’ behaviour.\nWe speculate that participants initially need to form a sufficiently convincing representation of the current\noption before they switch: in the updated model, how clear this representation needs to be is governed by\nthe θ threshold on accumulated information. Participants first explored locally, by repeatedly sampling the\nsame option in order to form a consistent idea of its associated colour, before they started switching more\nbetween options. In the second phase of GUESS sequences, their choices were driven by their sensitivity\nto the level of information acquired for each option (along with random exploration and a repetition bias),\nleading them to explore more globally, by targeting the most uncertain option.\n3. The “process model”, as I see it, is not really a process model, but more of a data-analytic model.\nNo reason is provided for turning the the “initial sampling” component “on” or “off”. A data-analytic\nmodel is fine, and it succeeds in highlighting the differences in search patterns. But then there are\nclaims that certain parameter settings for this model correspond to rational models, yet it is entirely\nunclear what criterion of rationality is alluded to. If I understand the task correctly, the purported\n“rational models” would fail at the task completely! The model includes mean colour estimates for\neach option, as well as what I presume are the absolute values of these estimates. For the “rational”\nDRAW model, choices are based on the difference in mean colour estimates between the options,\ni.e. P(choose 1) = F(β_(tar) × (x̂_(t, 1) − x̂_(t, 2))), with F(x) = 1/(1 + exp ( − x)). This is a simple\nmodel for a two-armed bandit, but by no means optimal. Given the short horizon, an optimal model\nmight be possible through defining the task as a POMDP (accounting also for the variability in\nhorizons). So the model is not optimal for the task, which is usually what is meant by a rational\nmodel. Furthermore, the authors claim that the rational model would set β_(tar) → ∞. This implies a\npure exploit strategy, without exploration. This model always chooses option 1 if x̂_(t, 1) −\nx̂_(t, 2) > 0 and option 2 if x̂_(t, 1) − x̂_(t, 2) < 0. If both options have a positive mean, but option 2 a\nhigher mean, then it could be the case that after sampling option 1, option 2 is never sampled, so\nthe model would not find the best option. In the present experiment, there are only two possible\nmeans for each option, so this is not a practical concern, but were participants aware of this\ndesign constraint? For the GUESS condition, the “rational” model bases its choices (if I understand this\ncorrectly, the information provided is unclear) on P(choose 1) = F(β_(inf) × (|x̂_(t, 2)| − |x̂_(t, 1)|)). I\ndon’t follow how the absolute mean value estimates encode “information already acquired”. It would\nmake more sense to me to relate this to the number of times an option has been sampled, or a\nvalue encoding the uncertainty associated with the value estimate (which will also\ndepend on the number of samples). Again, the “rational model” sets β_(inf) → ∞, which, if |x̂_(t, 2)| − |x̂_(t, 1)|\nencodes information obtained, might make sense here. But to what extent is this rational? Does\nfollowing these choice strategy maximise the probability of getting the dominant colour for both\noptions right? The model should be specified in much more detail (I had to guess certain parts) and\nit should be considered how this model provides optimal (or at least very good) performance on the\ntasks.\nThis is an important point, and we thank the reviewer for raising it. Based on the remarks of the reviewers,\nwe revised several aspects of the computational analyses: we updated two features of the model (the\n“learning” component and the threshold on the initial sampling phase), and consequently we changed our\nfitting procedure. We will detail some changes when addressing the specific points of the reviewer but we\nhave also added more details about the model’s specifics in the methods.\nWe did use the word “rational” in the sense of “optimal” in the task, and the reviewer was right that the initial\nversion of the model was not rational in this sense. This is because the initial version of the model was\nbased on averaging the values of successive outcomes instead of summing them. We have changed this in\nthe revised version: the model now “learns” by perfectly summing the outcomes observed (no leak). In this\nsense, the revised version of the learning module is Bayes-optimal. Because the colour distributions were\ndesigned such that the colour shades were proportional to the log odds ratio of the colour distributions (i.e.,\neach shade carried a specific amount of information about its generative distribution), summing shades\nobserved over successive trials was equivalent to accumulating evidence about the option’s underlying\ncategory (see figure a. below).\n' \np 𝑏𝑙𝑢𝑒 𝑜𝑢𝑡𝑐𝑜𝑚𝑒 …𝑜𝑢𝑡𝑐𝑜𝑚𝑒) \n𝑜𝑢𝑡𝑐𝑜𝑚𝑒 =log ( ( * ) \n' 𝑝 𝑜𝑟𝑎𝑛𝑔𝑒 𝑜𝑢𝑡𝑐𝑜𝑚𝑒 …𝑜𝑢𝑡𝑐𝑜𝑚𝑒) \n( * \n(:* \nBased on this quantity, at each trial, we could simply identify the option with most evidence in favour of the\ntarget colour distribution (target option), and the option with the lowest evidence in favour of any category\n(the most uncertain option). \nThe reviewer first raises the question of whether it is optimal in the task to then set 𝛽 → ∞ in the DRAW\n*;< \ncondition. As they point out, our task is a specific version of bandits, with only two possible fixed means.\nParticipants know this, and they are familiarised with the distributions first hand during training. Under these\ncircumstances, the agent doesn’t risk fixating on a “less blue” option, and setting 𝛽 → ∞ (pure exploitation)\n*;< \ndoes maximise the expected “target-ness” of the next outcome (i.e. is optimal). We thank the reviewer for\nletting us realise this was unclear, we have made this explicit in the Experimental Procedures section of the\nMethods, page 20: “[…] it was emphasised again for all participants that: 1. the colour intensity of each\noutcome was relevant, 2. each option would not always draw the exact same shade, so the task was to\nidentify its dominant colour, 3. there were only two possible fixed distributions (blue and orange), 4.\nthe two options were not always associated with different colours (they could be both\nassociated with orange, or blue), 5. each sequence would end after a variable, unpredictable number of\nchoices, and 6. performance was measured by the colours of all sampled outcomes in the DRAW\ncondition, and by the final question in the GUESS and FIND conditions.”\nSecond, regarding the optimality of the model in the GUESS condition. The reviewer is correct that\nabsolute mean values did not index the information acquired about the option. As we explained earlier, in\nthe revised version of the model, |𝑥 − 𝑥 | truly encodes the relative uncertainty about options. From\nA,* (,* \nthere, the reviewer questions whether it is optimal to set 𝛽 → ∞ in the GUESS condition. Because of the\nDEF \nway we designed the distributions, the probability of correctly identifying the underlying category of any\noption increased as a function of the level of evidence acquired with the following relation: (\n(GHIJKLMJNOJ \n(Supplementary Figure 2.a-b.). Because there are only two, symmetrical, distributions, the evidence\nincrease that an agent could expect from sampling an outcome from any option was constant (see an\narbitrary example on Supplementary Figure 2.b.). However, because of the convex relationship between\nevidence and the probability of correctly categorising the option, the same increment in evidence did not\n10\nalways lead to same gain in performance. More specifically, for the same increment in evidence, the\nexpected performance gain was higher when the initial level of evidence was lower – see figure b. below.\nFor the agent whose goal is to maximise the probability of correctly answering the final question about any\noption at any time (and provided that the expected evidence boost is a priori equal between options), this\nmeans that they can expect a larger performance boost from sampling the option with lowest current level\nof evidence. In other words, the optimal strategy to maximise performance at the final question is to always\nsample the option whose current level of evidence is lower. This corresponds to setting 𝛽 →∞.\nDEF \nA final point raised by the reviewer is whether our model qualifies as a process or data analytic model. We\nbelieve that the revised version of the model is a more cognitively motivated account of participants’\nbehaviour than the initially version. The revised model accumulates evidence in a Bayesian fashion and\nwhile the original model was simply fitting the number of choices in the initial phase, in the revised version\ninitial repetitions are conditioned on a threshold over accumulated evidence. We believe the revisions make\nthis more of a process model and we thank the reviewer for this comment.\nAs the reviewer requested, we have added more accurate details about the model specifics in the methods\nsections “computational model” and “fitting procedures” (p21-23), and we added figures illustrating the\nmodel and its mechanics in more details (Figure 3, Supplementary Figure 2.a-b.).\nMethods, pages 21-22: “Computational model: The computational model was updated during revisions:\nagain, we first developed the model on the discovery dataset before testing it in the confirmatory samples.\nThe computational model of sampling behaviour featured the same learning module across all three\nconditions. In practice, the colour distributions were designed such that the log-evidence for the categories\n(𝑥) was proportional to the colour shades 𝑐 (ranging from -1 pure blue to +1 pure orange – see Fig.1.).\n𝑥 = 𝑙𝑜𝑔\nP FL QRDH ) \n∝ 𝑐 , where 𝑐 = [−1 ∶ 1]\n' ' ' \nP FL S<;ETH ) \nAt each trial, the model could then accumulate evidence for each option (𝑥) by summing the colour\noutcomes 𝑐 observed from each option up to the current trial:\n'\n𝑝 𝑏𝑙𝑢𝑒 𝑐 …𝑐 ) \n( * \n𝑥 = 𝑐 = 𝑙𝑜𝑔 ( ) \nSP*'SE,* ' 𝑝 𝑜𝑟𝑎𝑛𝑔𝑒 𝑐 …𝑐 ) \n( * \n(:* \nThe sum of evidence observed for this option 𝑥 , contained information about which colour\nSP*'SE,* \ndistribution was more likely to have produced the observed samples ( 𝑠𝑖𝑔𝑛(𝑥 )), but also about how\nSP*'SE,* \nmuch information had been accumulated ( |𝑥 | ). Based on these quantities, the decision variable 𝐷𝑉𝑡 \nSP*'SE,* \nwas computed at each trial 𝑡, as the sum of three components:\n𝐷𝑉𝑡 = 𝛽 ∗ 𝑥 − 𝑥 ∗ 𝑡𝑐 + 𝛽 ∗ 𝑥 − 𝑥 + 𝑏 ∗ 𝑝𝑐 \n*;< (,* A,* DEF A,* (,* <HP \n𝑡𝑐 ∈ { −1:𝑏𝑙𝑢𝑒,\n+1:𝑜𝑟𝑎𝑛𝑔𝑒,\n0: 𝑖𝑛 GUESS 𝑠𝑒𝑞𝑢𝑒𝑛𝑐𝑒𝑠 }\n𝑝𝑐 ∈ { +1: 𝑖𝑓 𝑟𝑒𝑠𝑝 = 1\n*f( \n−1: 𝑖𝑓 𝑟𝑒𝑠𝑝 = 2 }\n*f( \nThe first component was the difference in accumulated values between the two options, signed by the\ntarget colour: 𝑥 − 𝑥 ∗ 𝑡𝑐. In practice, this was positive when the first option was closer to the target\n(,* A,* \ncolour, and negative when the second option was closer to the target colour. This component was weighted\nby a sensitivity parameter 𝛽 , controlling the sensitivity of the model choices to the target colour. For\n*;< \nsequences when participants played the guessing game, no target colour was given so 𝑡𝑐 was set to zero\nand this first component was null. The second component was the difference in the amount of information\naccumulated for each option: 𝑥 − 𝑥 . This was also scaled by a sensitivity parameter 𝛽 , controlling\nA,* (,* DEF \n11\nthe sensitivity of the model choices to the amount of information already acquired for each option. This\ncomponent was defined in all conditions. The third component was also defined in all conditions for all\nchoices except the first choice of each sequence, and corresponds to an additive bias in direction of the\nprevious choice, controlled by a repetition bias parameter 𝑏 . Its value was +1 when the previous choice\n<HP \nwas option 1, and -1 when the previous choice was option 2, effectively biasing the decision variable\ntowards choosing option 1. The choice process itself could take two different forms. By default, it was\nmodelled by a standard 𝑠𝑜𝑓𝑡𝑚𝑎𝑥 process on the decision variable 𝐷𝑉𝑡. But the ini+ model also included an\ninitial sampling phase, where, as long as the level of evidence acquired for an option was below a given\nthreshold (𝜃), choices were not controlled by the decision variable described above, but the model repeated\nthe previous decision with probability 𝜀. This initial sampling phase was therefore controlled by the\nthreshold 𝜃 on the amount of information already accumulated for the option, and by the threshold\nhardness 𝜀, governing how much the model will repeat the previous decision while under the 𝜃 threshold. If\nthe threshold hardness 𝜀 is at 1, the model systematically repeats the previous choices as long as it is\nbelow the 𝜃 threshold, whereas it the hardness is at 0 the model does not implement the initial repeated\nsampling and always relies on the decision variable 𝐷𝑉𝑡. Similarly, if the level of the threshold 𝜃 is at 0, the\nmodel also does not implement the initial sampling phase.\n𝐼𝑓 𝐼𝑛𝑖𝑡𝑖𝑎𝑙𝑃ℎ𝑎𝑠𝑒 = 𝑡𝑟𝑢𝑒 & |𝑥 | < 𝜃 𝑝 𝑟𝑒𝑠𝑝 = 𝑟𝑒𝑠𝑝 = 𝜀\nm SP*'SE,* * *f( \n1\n𝐸𝑙𝑠𝑒 𝑝 𝑟𝑒𝑠𝑝 = 𝑟𝑒𝑠𝑝 = \n* *f( 1+𝑒fqr* \n𝐼𝑓 𝑟𝑒𝑠𝑝 ≠ 𝑟𝑒𝑠𝑝 𝐼𝑛𝑖𝑡𝑖𝑎𝑙𝑃ℎ𝑎𝑠𝑒 = 𝑓𝑎𝑙𝑠𝑒\n* *f( m \nTo avoid numerical discontinuities in the likelihood function of the model due to the presence of a hard\nbound 𝜃, we assume that 𝜃 follows a normal distribution with a small standard deviation of 0.1 in logLR\nunits (which is considered as small given that a single colour sample can provide up to 1.4 in logLR units).\nFor plotting and statistical testing, 𝜃 was transformed from logLR units into probabilities (and therefore\ntreated as a bounded parameter). It can be understood as a certain probability (of the current shape being\nassociated to either the blue or the orange distribution) that the agent wants to reach before terminating the\ninitial repetitive sampling phase.\nNote that the computational model described above nests the optimal sampling agent for the DRAW and\nGUESS conditions. The optimal agent does not include an initial sampling phase or a repetition bias in\neither condition. In the DRAW condition, the optimal sampler chooses based solely on the first component:\nthe difference between the mean colour mixture estimates signed by the target colour, with 𝛽 → ∞ (i.e.,\n*;< \nan argmax choice process). In the GUESS condition, the optimal sampler chooses based solely on the\ndifference between the information already acquired from each option, with 𝛽 → ∞. The optimal\nDEF \nsampling agent was simulated for each participant and each condition to compare its behavioural measures\nto those of tested participants (Supplementary Figure 2).\nFitting Procedures: Best-fitting parameter estimates for the model, which can either include (ini+) or not\ninclude an initial sampling threshold (ini-), were obtained through maximum likelihood estimation (MLE) in\ntwo ways. First we used Matlab’s fmincon function to cover a wider range of possible parameter values and\navoid local minima (the decision variable parameters were initialised at random points in the range [-2:2] for\n𝛽 and 𝛽 and [-4:4] for 𝑏 while the threshold value was set at a hundred different values in the\n*;< DEF <HP \ninterval [0.05:5]). Second, we used the best-fitting estimates obtained from this first fmincon pass as\nstarting points for the Bayesian Adaptive Direct Search algorithm45. This second pass was two-fold: we\nused leave-one-out cross validation on the sequences used to obtain model likelihood estimates. These\nwere used in Bayesian model selection to compare variants of the model with vs. without the initial\nsampling phase. BMS was conducted using a random-effects approach, which assumes that different\nparticipants may rely on different models. It consists in estimating the distribution over models that\n12\nparticipants are drawn from. We used the Dirichlet parameterization of the random-effects approach\nimplemented in SPM12 (http://www.fil.ion.ucl.ac.uk/spm), with default (uniform) priors. We also used the\nBADS algorithm on all the sequences at once to obtain final parameter estimates45.”\n4. Whilst the highly detailed analyses have merits, I struggled to “see the forest for the trees”. A\nmuch more focused and shorted paper with key analyses presented in the main paper and\nsupporting ones in supplementary material, would work better, I believe. For choice patterns, we get\na range of analyses (sampling imbalance, option re-sampling, choice-choice similarity) which all\nindicate the same thing (early streaks vs gradual increase to preferred option). I suggest focusing\non one of these and including other analyses as supplementary material. I also suggest to mainly\nfocus on experiment 3, and report the other two as replicating effects.\nWe thank the reviewer for helping us trim the presentation of the results: we have removed some\nbehavioural indices and moved some of the replication data to the supplementary material. We have\ncondensed the presentation of the replication results in the third dataset to a single paragraph to shift the\nfocus to the novel (third, FIND) condition, on page 10: “In this new dataset, participants (N = 31 after\nexclusion; see Methods) replicated the behavioural patterns previously described in the DRAW and\nGUESS conditions. They reached similar performance in both of the original conditions (Figure 6.a.), and in\nthe GUESS, but not in the DRAW condition, they initially sampled in streaks of repeated decisions for each\noption in turn. This was visible both in behavioural indices (Figure 6), and in the distributions of parameters\nestimates (Figure 7.a.). The statistics for these conditions are reproduced in Supplementary tables 1\nto 6. We focus the remainder of this section on the new, FIND condition.”\nWe did keep two indices of choice patterns (choice similarity triangles and the fraction of repeat decisions),\nbecause they provide specific insights. Choice similarity triangles are a more global, and rawer visualisation\nof the effect: they allow to clearly display the behaviour of interest at the scale of full sequences, while still\nbeing a relatively less processed characterisation of the choices. The curves of the fraction of repeat\ndecisions are more remote from the raw choice data but allow to directly contrast early sampling behaviour\nin a model-free approach – they reveal a cross-over interaction. We believe both indices are necessary to\nconvey a good understanding of the effect we describe.\nWe hope that the reviewer accepts our response to point 1 as justification for keeping the DRAW vs.\nGUESS contrast (experiment 1 and 2) as the main focus of this paper.\n5. There is a large literature on contrasting “Decisions from description” to “Decisions from\nexperience”. The latter is either operationalised as a standard multi-armed bandit, or via a “sampling\nparadigm” where participants can freely sample from the options, until making a final (rewarded)\nchoice. These two operationalisations have also been contrasted (see e.g. Wulf et al 2017, “A Meta-\nAnalytic Review of Two Modes of Learning and the Description‒Experience Gap”). This literature is\nhighly relevant but has been (almost) completely ignored in this paper.\nThis is very relevant indeed and we are grateful to reviewer #2 for suggesting to dive further into this\nliterature. Two papers using the “sampling paradigm” of DFEs (Hills & Hertwig, 2010 and Wulff et al. 2015)\nare particularly relevant to this study as they directly analysed the exploration patterns during the sampling\nphase (while, to our knowledge, other DFE papers were more interested in contrasting the final decisions\nmade after experience – akin to our final question –, against those made after receiving descriptive\ninformation). We have added these papers to the general introduction page 2: Variants of these tasks have\ndissociated exploration from reward maximization by delaying the acquisition of rewards. In such tasks,\nparticipants either receive information (e.g. 9,10 through forced choices) or sample the environment for\ninformation (e.g. in the sampling paradigm from Decisions From Experience research5,11,12) before\nthey have to make a rewarding decision. However, in these tasks, the information acquired during the\nsampling phase immediately concerns the ultimate rewarding values of options. Outcomes do not\nimmediately contribute to participants’ final earnings, but since they are expressed in points, they\n13\nimmediately reveal which option will ultimately be rewarding, which potentially biases choices – this is\nsimilar to browsing restaurant reviews. This observation means that the exploration patterns observed in\nsuch delayed-rewards tasks are still constrained by the prospective rewarding values of available options.”\nWe also discuss them in the introduction of the third dataset, as they relate more specifically to the FIND\ncondition we introduced in this dataset, page 9: “This new condition was akin to browsing restaurant\nreviews: choices yielded no immediate reward, but outcomes were still valued with respect to an end goal.\nIt was also closer to existing delayed-reward tasks, like the Horizon task9 or in the “sampling paradigm”\nfrom Decisions From Experience studies12. In these paradigms, rewards are only granted after a\nsequence of so-called “free choices” (as in the GUESS condition), but (in contrast to the GUESS condition),\nchoice outcomes are numerical values, and therefore directly inform the participants of the option’s future\nrewarding value. In these tasks, it is thus possible that a future reward value already\ntaints the outcomes and biases choice patterns. While the original GUESS condition was necessary to\neliminates this concern, the addition of the FIND condition eliminate a gap in the paradigm. Here, the\ncontrast between the FIND and DRAW conditions reflects the difference between delayed and immediate\nexploitation, whereas the contrast between the FIND and GUESS conditions reflects the difference\nbetween the sampling of outcomes with and without rewarding value.”\nHowever, as we mentioned in our response to the reviewer’s first point, the “sampling paradigm” is closer to\nthe FIND condition: exploitation is delayed to the end of a free exploration phase, but the outcomes of\noptions are numbers, and therefore directly informative about the option’s ultimate rewarding value. The\nresults we observed in the FIND condition (namely that participants’ choices were driven towards the most\ntarget option even though this was not increasing their earnings), suggest that outcomes which can be\ndirectly related to rewards already orient choice patterns towards the ultimately rewarding option. In this\npaper, using coloured outcomes instead of numerical points allowed us to introduce the GUESS condition,\ncompletely removing the intrinsic value of options: we showed that this reveals specific sampling patterns.\n6. The paper is unclear/does not provide crucial information in various places. Key aspects (such as\nthat this was a within-subjects design), should be reported in the introduction or results (this was\nonly evident later when it turned out to be possible to study covariation over conditions).\nWe have added this detail more explicitly in the general introduction, on pages 2-3: “Here we sought to fully\ndissociate exploration from reward-guided decision-making by designing a novel two-armed bandit task\nwhere the outcomes of the two options are shades of colour instead of points or monetary rewards. In\nthemselves, outcomes thus provide information about the structure of the task (the option-colour\nassociations), but bear no intrinsic value (Figure 1.a.). In one condition, we rewarded participants for\ndrawing one particular colour. This instruction does confer a rewarding value to each outcome, and\nrequires participants to trade exploration against exploitation to choose the option associated with the\ninstructed colour – as in regular bandit tasks. By contrast, in a second condition, the same participants\n(a within subject design) were told that they were sampling to find an option of one particular\ncolour, which they only had to indicate once, at the end of the sequence of several free choices.\nParticipants were rewarded based on their ability to indicate correct option-outcome associations after the\nend of the sequence, but each outcome directly informed them about the options’ future value – similar to a\nclassic delayed reward task. In a third, novel condition, the target colour was only revealed at the end of the\nsampling sequence. Participants sampled the options to learn the option-colour pairing and a final two-\nalternative forced choice asked them to indicate an option of a specific colour (only revealed then).\nBecause participants could not foresee the ultimate rewarding value of options, this instruction fully\ndissociates exploration from any form of immediate or prospective reward maximisation. Except for this\ndifference, the three conditions were matched in every possible aspect so that sampling patterns could be\ndirectly compared between them (Figure 1 & 5; see Methods).”\nhow?) are not detailed. The model is described in vague terms, rather than a clear formula of its\nworkings and how parameters are initialized. \nWe have added more details about the counterbalancing in the Methods section on page 20: “Conditions\nwere interleaved pseudo-randomly within blocks, with self-paced breaks between blocks. Conditions were\npseudo-randomised to balance the number of sequences of each condition within each block, avoid too\nmany (>3) consecutive repetitions of the same condition, and avoid immediate repetitions of the same\nshapes or the same response sides.”\nIn response to the reviewer’s third point, we have updated the computational model and we provide more\ndetails about its workings in the Methods on pages 21-22 (this section is reproduced in response to the\nreviewer’s third point). We have also added figures illustrating the model and its mechanics (see Figure 3,\nSupplementary Figure 2.a-b.), and we clarified the presentation of the model in the results section. Results,\npage 6: “Based on the specific features of human exploration patterns, we designed a process model of\nsampling choices derived from the optimal sampling agent (Figure 3). In the DRAW condition, based on the\nperfectly accumulated values of the colour outcomes, the optimal sampling agent always selects the option\nwith highest accumulated value in direction of the target colour (the most target-like option - Figure 3.d.,\nSupplementary Figure 2). Whereas in the GUESS condition, the optimal strategy is to always chose the\noption with lowest absolute accumulated value (the most uncertain option – Figure 3.d., Supplementary\nFigure 2.b.). This allows the agent to maximise information about both shapes at any trial – thus\nmaximising its probability of responding correctly to the final question, whose timing was not disclosed to\nparticipants. Like the optimal sampling agent, the choices of the model are sensitive to the difference in\nuncertainty between the two options (sensitivity parameter 𝛽 ), and attracted toward the option most likely\nDEF\nto be associated with the target colour in the DRAW condition (sensitivity parameter 𝛽 ). In addition, and\n*;<\nunlike the optimal sampling agent, the model features a repetition bias parameter 𝑏 , and an optional initial\n<HP\nsampling phase. In the beginning of the sequence, instead of relying on a decision variable computed from\nthe three choice parameters (𝛽 , 𝛽 and 𝑏 ), the model can acquire, in turn, a number of samples from\nDEF *;< <HP\neach option. This initial sampling strategy is controlled by a threshold 𝜃, governing how much evidence the\nagent wants to initially acquire about the current option before switching to the other (see Methods, Figure\n3). We fitted this model to human sampling patterns in the DRAW and GUESS conditions, and confirmed\nthe recoverability of the model and its parameters (Supplementary Figure 5).”\nRegarding parameters initialisation, the fitting procedures for the revised model is now described in more\ndetails in the Methods section, on page 22: “Fitting Procedures: Best-fitting parameter estimates for the\nmodel, which can either include (ini+) or not include an initial sampling threshold (ini-), were obtained\nthrough maximum likelihood estimation (MLE) in two ways. First we used Matlab’s fmincon function to\ncover a wider range of possible parameter values and avoid local minima (the decision variable parameters\nwere initialised at random points in the range [-2:2] for 𝛽 and 𝛽 and [-4:4] for 𝑏 while the threshold\n*;< DEF <HP\nvalue was set at a hundred different values in the interval [0.05:5]). Second, we used the best-fitting\nestimates obtained from this first fmincon pass as starting points for the Bayesian Adaptive Direct Search\nalgorithm45. This second pass was two-fold: we used leave-one-out cross validation on the sequences used\nto obtain model likelihood estimates. These were used in Bayesian model selection to compare variants of\nthe model with vs. without the initial sampling phase. BMS was conducted using a random-effects\napproach, which assumes that different participants may rely on different models. It consists in estimating\nthe distribution over models that participants are drawn from. We used the Dirichlet parameterization of the\nrandom-effects approach implemented in SPM12 (http://www.fil.ion.ucl.ac.uk/spm), with default (uniform)\npriors. We also used the BADS algorithm on all the sequences at once to obtain final parameter\nestimates45.”\nIt also seems that the GUESS condition differs between Exp3 and Exp 1&2 (in the last experiment,\nparticipants are asked which of two options is blue or orange, whilst in the first two they are asked\nabout the dominant colour of one of the options), but this is not clearly stated.\nGUESS and FIND condition. We have added more details about this in the experimental design and\nprocedures section page 21, we thank the reviewer for raising this point. “In the first two datasets, one\nshape (pre-selected at random) was displayed and participants were asked to report its associated\n(dominant) colour. In the third dataset, one colour (pseudo-randomly selected) was displayed on screen\nand participants were asked to identify an option associated to the demanded colour.”\nHow is choice similarity defined?\nAt the choice level, choice similarity is defined as 1 if the participant chose the same option, and 0 if they\nchose a different option. Each choice of a sequence was compared to all the other choices in the sequence\n(so choice 1 was compared to choices 2:end, choice 2 to choices 3:end, etc…). This was averaged across\nsequences, resulting in an average similarity score (between 0 and 1) for each position in the sequence.\nWe have clarified this definition in the Methods section (reproduced below) and have detailed the\ndescription in the figure captions (e.g. Figure 2).\nMethods, page 24, section Statistical analyses, paragraph Choice similarity: “Choice similarity At the choice\nlevel, choice similarity is defined as 1 if the participant chose the same option, and 0 if they chose a\ndifferent option. Each choice in each sequence was compared to all the other choices in the same\nsequence (choice 1 was compared to choices 2:end, choice 2 to choices 3:end, etc…), resulting in a\nsimilarity score for each pair of choice positions within each sequence. This was averaged across\nsequences, resulting in an average similarity score (between 0 and 1) for each position in the sequence.\nBecause choice similarity is typically not distributed normally across participants, we performed Wilcoxon\nsigned-rank tests to compare participants’ choice similarity to chance level (50%).”\nIn Fig 7. In the top plots, it is unclear that subplots compare DRAW to GUESS.\nEtc.\nWe have amended this figure to be more specific (it is now Figure 8).\nWe thank the second reviewer once again for helping us improve the accuracy and clarity of the analyses,\nand for challenging us to spell out more clearly the exact contributions of this set of experiments. We hope\nthat this revised version of the paper will convince the reviewer.\nmanuscript. We are very pleased to read that Reviewer #1 is satisfied with the\nrevisions, and that Reviewer #2 found the revised manuscript more focused and the\nelling improved. We address below the remaining comments of Reviewer #2 in\ngreen. Quotes from the new revisions to the manuscript are highlighted in blue.\nReviewer #1 (Remarks to the Author):\nI'm satisfied with the revision. One small typo in the abstract: \"patterns arising\nin when\" -> \"patterns arising when\"\nWe thank the reviewer for spotting this typo. We have corrected it in the revised\nmanuscript. We are happy that the first reviewer is satisfied with the revised manuscript\nand we thank them again for their insights.\nReviewer #2 (Remarks to the Author):\nThe authors did a good job responding to the issues raised in the reviews of\nthe previous paper. The presentation is more focused, and the modelling has\nimproved. That said, I still have doubts about the theoretical significance of the\nfindings (initial streaks in the GUESS condition), due to the different goals\nbetween the conditions. The focus should be on comparing the two options in\nthe DRAW and FIND conditions (i.e. which option has more of the target\ncolour), but in the GUESS condition, the focus needs to be on identifying the\nnature of each option, which can be done in isolation and does not require\ncontrasting the options. To determine the nature of a single option, \"streaky\"\nexploration may be subject to less memory demands than more evenly\ndistributed exploration. A heuristic for the other conditions is to pitch the\noptions against each other, for instance sampling one and then the other and\njust determining which one had the most of the target colour in each\nconsecutive pairwise comparison. This seems quite reasonable to me and\nwould account for the results without many of the processes relied upon in the\ndiscussion. That said, the finding of a difference in exploration behaviour is\nsupported well by the analyses and the \"streaky\" exploration is suboptimal\nwhen the length of a sequence is unknown and a trial can end at any time. But\nI do think the discussion can be shortened and streamlined further.\nWe are very happy that the reviewer found our responses to their previous comments\nconvincing, and the revised manuscript more focused with an improved modelling of\nhuman behaviour.\nWe thank the reviewer for allowing us to further clarify the theoretical significance of\nour findings. The reviewer proposes that participants sample options in streaks in the\nGUESS condition because they were free to identify the dominant colour associated\nwith each option in isolation. The reviewer suggests that participants do not use this\nstrategy in the DRAW condition because they were pushed to compare options by\nsampling one after the other to immediately compare the pairs of outcomes.\nWe believe that this interpretation cannot explain our findings. First, we do not observe\nbasic alternated sampling in the DRAW and FIND conditions, but rather a focus on the\nPage 1 of 6\ntarget-coloured option. The model results also suggest that participants were\naccumulating information over several samples, rather than comparing the pair sample\nby sample. In addition, repetition biases are positive (meaning that participants do not\ntend to alternate between options in any condition), comparable and correlated across\nall three conditions (Figure 7.a. and Supplementary Figure 7.b.), as well as the\nsensitivity to uncertainty across the GUESS and FIND conditions (Figure 7.a. and\nadditional figure a. below). Given such consistency in parameter estimates across\nconditions (including repetition biases which would be negative if participants indeed\nrelied on the strategy suggested by the reviewer), it is very unlikely that participants\nresorted to wildly different heuristics in the different conditions.\nFrom a theoretical perspective, in the GUESS condition the incentive really was on\nidentifying the colour of both options as fast as possible (since the game ended after\na variable number of choices), yet this is the condition where participants did sample\nin streaks – i.e. they were taking the time to explore a single option at the expense of\nuncertainty regarding the other option. We fully agree with the reviewer that such\n“streaky sampling” is likely cognitively easier for participants – e.g. in terms of memory\ndemands – and this is the focus of a dedicated paragraph in the Discussion section.\nImportantly, there was also no actual need to compare options in the DRAW and FIND\nconditions. Participants had learnt during the training (both by explicit instructions and\nby sampling during training games) that the dominant colour of each option was\nindependent of the dominant colour of the other option. This means that comparing\noptions would have been a bad strategy for half of the games where the dominant of\nthe two options was the same. And indeed, as stressed above, we did not find any\nevidence that participants tended to alternate between options in the DRAW and FIND\nconditions.\nFinally, if sampling in streaks is cognitively easier, and if there was no actual benefit to\nalternate / compare options in the DRAW (and FIND) condition(s), it is difficult to\nexplain why participants did not produce streaks in this condition but only in the GUESS\ncondition, where they were incentivised to sample from both options in a single game.\nThis means that the presence of rewarding value of outcomes across conditions\n(present in the DRAW and FIND conditions, absent in the GUESS condition)\nconstitutes a more parsimonious explanation for the different exploration patterns\nobserved in the three conditions.\nWe have included a discussion of the reviewer’s point in the discussion on page 16:\n“An alternative explanation is to propose that participants produced streaks in the\nGUESS condition because they were free to study each option in isolation (probably\nbecause repetitive sampling was cognitively easier than alternating), whereas in the\nDRAW and FIND conditions they were pushed to compare options by sampling one\nafter the other to immediately compare the pairs of outcomes. However, we did not\nfind any evidence that participants tended to alternate between options in the DRAW\nand FIND conditions. Rather, repetition biases were positive (meaning that participants\ndid not tend to alternate between options in any condition), comparable and correlated\nacross all three conditions (Figure 7.a. and Supplementary Figure 7.b.), and so was\nthe sensitivity to uncertainty across the GUESS and FIND conditions (Figure 7.a.).\nGiven such consistency in parameter estimates across conditions, it is very unlikely\nthat participants resorted to wildly different heuristics in the different conditions. In\naddition, participants had learnt during training that the dominant colour of each option\nPage 2 of 6\nwas independent of the dominant colour of the other option. Thus, comparing options\nwould have been a bad strategy for half of the games where the dominant of the two\noptions was the same. If sampling in series is cognitively easier, and if there was no\nactual benefit to alternate / compare options in the DRAW and FIND conditions, it is\ndifficult to explain why participants did not produce streaks in this condition but only in\nthe GUESS condition, where they were incentivised to sample from both options in a\nsingle game. The presence of (immediate or prospective) reward value appears to be\nthe main determinant of participants’ early sampling strategies.”\nAdditional figure a. Sensitivity to uncertainty correlation. Correlation of sensitivity to uncertainty\nestimates (𝛽 ) obtained in the GUESS and FIND conditions. Spearman’s correlation.\n\"#$\nTo follow the reviewer’s recommendations, we have edited short parts of the\ndiscussion for clarity, but we believe all sections are relevant to discussing the\nsignificance of the results. The discussion is longer with the previous suggestions from\nthe reviewers, but we believe these were relevant, sometimes necessary, additions.\nOne major issue I have with the current version relates to the \"Positive\ncovariations between exploration patterns across individuals\" section, where\nparticipants are split into two groups based on whether the model with initial\n\"streaky\" sampling fits better than a model without. Part of the analysis here\nfocuses on the remaining parameters of the model. If I understand the model\ncorrectly, then – because the initial choices are governed strongly by the initial\nsampling policy in the INI+ group, the estimates of the remaining parameters\nwill be mostly determined by later choices (after the initial sampling phase is\nover). In the INI- group, the remaining parameters likely cover the whole\nsequence (or at least most of the sequence). Results such as that the\nsensitivity to the reward maximising option is higher in the INI+ group than the\nINI- group are then not overly surprising, as one would expect that during the\nlater trials participants would have determined the best option. So, these\ncomparisons are biased due to the parameters in the INI+ group being\noptimized for the later parts of the sequences, whilst this is less the case for\nthe INI- group. In my view, then, comparisons between the INI+ and INI-\ngroup, in terms of model parameters, are not overly meaningful. If such a\ncomparison is of interest, I suggest to only focus on the behavioural results\n(e.g. proportion of maximising choices, repeat choices, etc.)\nWe agree with the reviewer that comparing parameters obtained in two different\nversions of the model would be biased in principle. This is why, for this analysis, we\nsplit the participants based on which of the models best fit their behaviour in the\nGUESS condition, but then compared parameters obtained in the most specified\nversion of the model (the ini+ model). This ensures that the “later phase parameters”\nPage 3 of 6\n(repetition bias and sensitivity parameters) were immediately comparable. The\nreviewer is right that this was not explicitly described in the text, and we have therefore\nadded the following clarification on page 13 in the revised manuscript: “We split\nparticipants based on which of the model with (ini+ group, N = 67) or the model without\nan initial evidence threshold (ini- group, N = 18) best fit their behaviour, but to ensure\na fair comparison between groups we then compared parameter estimates obtained in\nthe most specified version of the model (the version with initial threshold, ini+).”. We\nthank the reviewer for spotting this imprecision.\nHowever, we would like to point out that the presence of an initial sampling phase of\nthe model does not bias the estimation of “later phase parameters” (repetition bias and\nsensitivity parameters).\nRegarding the sensitivity to the target specifically: in the DRAW condition (where\nsensitivity to the target is estimated), threshold values estimated from the ini+ were\nvery close to 0.5 for the large majority of participants, resulting in virtually ini- models\n(i.e. even when taking the ini+ model, the estimated threshold values are so low that it\nresults in a threshold free, ini- behaviour – see additional Figure b. below).\nAdditionally, we performed recovery analyses showing that all “later phase\nparameters” (repetition bias and sensitivity parameters) were actually equally well\nestimated by an ini+ for agents with or without initial sampling phase. We simulated\ndata from agents with or without the initial sampling phase, and then fitted this data\nwith an ini+ model (so a model with a free θ parameter, allowing for an initial sampling\nphase). As illustrated on a new Supplementary Figure 6, we observed that regardless\nof whether the agent’s behaviour truly displayed an initial sampling phase or an\nabsence of initial sampling phase (simulated as ini+ or ini-), “later phase parameters”\nwere equally well recovered with an ini+ model. In other words, the additional initial\nphase parameters featured in an ini+ model did not introduce a bias in the estimation\nof later phase parameters (repetition bias and sensitivity to uncertainty).\nWe hope that this additional analysis helps convince the reviewer that the presence of\nan initial sampling phase in the model does not negatively impact the estimation of\nother parameters, and therefore, that splitting participants based on the\npresence/absence of an initial phase in the best-fitting model for their behaviour does\nnot preclude further analysis of the remaining parameters.\nAdditional figure b. Threshold level. Threshold levels (θ) obtained in the ini+ model in the DRAW\ncondition, for participants assigned to ini+ or ini- group based on the fits obtained in the GUESS\ncondition.\nAs the reviewer suggests, we have emphasised the model-free differences between\nthe two groups. The proportion of reward maximising and the proportion of repeat\nchoices are provided, e.g. on page 13: “However, ini+ participants also had a stronger\nPage 4 of 6\ntendency to prefer the target-coloured options in the DRAW condition, which is optimal\nin this condition (choice direction curves on Figure 8.b.; higher sensitivity parameter\n𝛽\n%&'\non Figure 8.e.; two-sample t-test between groups of participants: t = 2.20, p < 0.05,\n83\nd = 0.58).” and “As expected from this splitting criterion, in the GUESS condition,\nparticipants in the first group (ini+) showed a higher tendency to initially resample\noptions (Figure 8.a.,b.,d.).”\nBecause we split participants based on which of the model with or without initial\nsampling phase best fit their behaviour, differences in the proportion of repeat\ndecisions were expected and merely confirm the basis of the split (i.e. that participants\nwhose behaviour is best represented by the ini+ model display more repeated choices\nat the beginning of sequences). Thus, we focussed on differences in performance and\nsensitivity to the target and to uncertainty because these are independent from our\nsplitting criterion. We agree with the reviewer that it is particularly striking that\nparticipants assigned ini+ based on their behaviour in the GUESS condition also have\nhigher performance in both conditions. These results suggest that suboptimal\nexploratory behaviours (initial repetitive sampling) co-occur with more optimal\nstrategies (target-seeking and directed exploration in the DRAW and GUESS & FIND\nconditions respectively).\nA more minor issue is that I think the Horizon task is misrepresented on page\n9: \"This new condition was akin to browsing restaurant reviews: choices\nyielded no immediate reward, but outcomes were still valued with respect to\nan end goal. It was also closer to existing delayed-reward tasks, like the\nHorizon task9 or in the “sampling paradigm” from Decisions From Experience\nstudies12. In these paradigms, rewards are only granted after a sequence of\nso-called “free choices” (as in the GUESS condition), but (in contrast to the\nGUESS condition), choice outcomes are numerical values, and therefore\ndirectly inform the participants of the option’s ultimate rewarding value.\" This\nseems to imply delayed rewards for free choices in the Horizon task, but - as\nfar as I'm aware - rewards are not delayed in the Horizon task, beyond an\ninitial sequence of forced choices which manipulate the amount of information\ngathered. So I think this is a misrepresentation of the Horizon task.\nThe reviewer is right that the wording was confusing, we have stated the difference\nexplicitly. p9: ”In these paradigms, as in the GUESS condition, rewards are only\ngranted after a sequence of “non-incentivised choices” (after a predefined sequence\nof forced choices in the Horizon task, or after a self-terminated sequence of free but\nnon-rewarded choices in the “sampling paradigm”). However, in both cases choice\noutcomes are numerical values, and therefore directly inform the participants of the\noption’s ultimate rewarding value, in contrast to the GUESS condition.”\nReviewer #2 (Remarks on code availability):\nI do not have access to Matlab, on which the code relies. I did have a look and\nthe repository exists. But the contents are from 2 years ago and as the\nmodelling has changed since the last manuscript, I think it needs updating.\nPage 5 of 6\nThe reviewer is correct that we had not yet made the revised code public. It is now\nupdated and publically available at the same link: https://gitlab.com/cle-a/colrep\nPage 6 of 6\n2\n4 information search from choice outcomes to different extents. Across three experiments, evidence is\n5 presented that suggests different information search patterns, with conditions with delayed or fully\n6 suspended rewards producing behavior directed more strongly at reducing uncertainty. To account for\n7 this, the authors suggest a computational model. Overall, the presented results are interesting and\n8 suitable for Nature Communications. I am inclined to recommend publication, but I believe the authors\n9 should first address the following comments, which may help clarify the exposition and interpretation of\n10 their findings. This can be achieved without conducting additional analyses or studies.\n11\n13 inclined to recommend publication. We have amended the manuscript in line with the reviewer’s\n14 recommendations and we hope they will be fully satisfied with the revised submission; we thank\n15 them again for their help in improving the final paper.\n16\n18 the full complexity of information search situations, which includes prior expectations and types of\n19 uncertainty. In our recent work forthcoming in PNAS (Spektor & Wulff, 2024), we attempted to reveal\n20 additional details of the full complexity of human information search by means of analyzing a large\n21 database of the sampling paradigm. This revealed that people enter the task with ecological (structural)\n22 knowledge, which they update across the task and use to steer their information search behavior. It\n23 further revealed that individuals reduce multiple types of uncertainties and differ not only in the details\n24 of a common computational strategy but likely recruit entirely different strategies. Finally, it also showed\n25 evidence of strategy shifts during the task, which may be related to the local-global strategy shift\n26 described here. Not all aspects can be studied with the current paradigm, which is fine. However, I think\n27 it is important to emphasize more strongly and discuss in greater length the limitations of both the\n28 paradigm and the computational account with respect to capturing the full complexity of human\n29 information search behavior.\n30\n32 and Wulff focusses on a meta-analysis of the sampling paradigm; it demonstrates that\n33 participants’ information search strategies tackle different types of uncertainty (structural\n34 uncertainty first then estimation uncertainty, while also minimising computational uncertainty),\n35 and that such strategies are informed by prior expectations which are then updated throughout\n36 the task, as well as by perceived difficulty. We appreciate the contribution of this large-scale\n37 study to the understanding of information search outside of explore-exploit situations, and have\n38 added its insights to the following paragraph of the Discussion (p. 16):\n39\n40 “However, while our findings highlight a novel balance between local and global\n41 uncertainty minimisation, they also relate to recent work showing evidence for strategy\n42 shifts over the course of a single sampling sequence. In a three-alternative version of the\n44 heuristic in addition to more complex exploration computations27. Similarly, Spektor and\n46 experiments, participants prioritised observing all possible outcomes first, only\n1\n47 switching to targeting the most uncertain options once all unique outcomes had been\n48 observed. Our results also document how participants flexibly adjust exploration\n49 strategies within a single sampling sequence.”\n50\n52 is to study exploration in situations where the sampled information is not even prospectively or\n53 conditionally rewarding. In the sampling paradigm, numerical outcomes can be treated as\n54 prospective rewards. Our control, FIND condition, which mimics the sampling paradigm,\n55 demonstrates that they are. Forewarning participants about which colour will be associated with\n56 rewards in the future (after the end of the free sampling) already shapes their sampling\n57 decisions during the sampling phase. This is evident in the computational results (participants’\n58 sensitivity to the target-value of options is positive – Figure 7.a), but also in the behaviour\n59 (Figure 6.a). For this reason, the sampling paradigm provides a situation where information\n60 search is released from immediate reward maximisation, but not from the prospective influence\n61 of rewards. To observe exploration outside of the reward constraint, we also need for outcomes\n62 to be as neutral as possible regarding the future rewarding value of options. We used colours as\n63 outcomes because they are fully neutral in isolation, unlike numerical outcomes. In our study,\n64 the GUESS condition is the only one to fully disentangle information search from reward\n65 acquisition. To our knowledge, no other paradigm has afforded a contrast between different\n66 conditions displaying the effect of immediate, prospective or delayed rewards on information\n67 search.\n68\n70 investigated with any single paradigm, but we do think that our main conclusions are warranted\n71 in light of the different points mentioned by the reviewer.\n72\n73 1/ Regarding participants’ expectations. Individual differences in expectations within each\n74 condition were not our focus of our study, and we therefore focused our main analyses on mean\n75 group-level differences between conditions of our paradigm. We would of course be interested\n76 to study whether and how individual differences in expectations may influence exploration\n77 patterns in the GUESS condition where information search is divorced from reward seeking, but\n78 this is not the aim of the present study.\n79\n80 2/ Regarding variability in information search strategies. There is evidence in the data for\n81 between-participant variation in information search strategies (visible in the spread of\n82 behavioural measures and parameters estimates - Figures 2 & 4, 6 & 7). This is the topic of the\n83 last Results section. But the reviewer also suggests that participants could recruit completely\n84 different computational strategies to what we describe and implement in the model. The goal of\n85 our computational account was not to exhaust the full range of possible exploration strategies,\n86 and indeed, we agree with the reviewer that this is always a possibility. Still, our model is\n87 derived from behavioural observations, and it provides a good fit and a plausible explanation for\n88 behaviour in the task. To check that it captures not only the mean behavior of tested\n89 participants but also individual differences in the data, we turned to a principal component\n90 analysis (PCA). We used choice similarity matrices as a summary snapshot of behaviour in the\n91 task, and we contrasted the behaviour of original participants with simulated behaviour using\n92 the best-fitting parameters (N = 1,000 simulations). We derived the principal components (PCs)\n93 explaining the behaviour of the original participants on the one hand and the simulated\n2\n94 behaviour on the other hand. Supplementary Figure 11 (reproduced on p. 4 of this point-by-\n95 point response to comments) shows that our computational model accurately captures\n96 individual differences in behaviour, and thus provides a relevant description of participants\n97 behaviour.\n98\n100 assessed the similarity of parameter estimates between the first and the second half of the task\n101 in a new analysis (Supplementary Figure 12, reproduced below on p. 5 of this response).\n103 uncertainty was slightly increased in the second half of the experiment, but with a small effect\n104 size (paired t-test in the pooled datasets, first vs. second half t = 2.11, p = 0.04, d = 0.23). This\n81\n105 may reflect that participants improved with experience in doing the task, but is not convincing\n106 evidence for a full strategy shift at the scale of the task. We do, however, document a strategy\n107 shift at the scale of the single sequence, and this is similar to what is described in Spektor &\n109 information seeking (initial repetitive sampling) and later turned to more global search (directed\n110 towards the most uncertain option) – we return to this in response to the reviewer’s next point.\n112 the Spektor & Wulff 2024 paper (Discussion, p. 16):\n113\n114 “Similarly, Spektor and Wulff13 have shown that in the sampling paradigm of Decisions\n115 from Experience (DfE) experiments, participants prioritised observing all possible\n116 outcomes first, only switching to targeting the most uncertain options once all unique\n117 outcomes had been observed. Our results also document how participants flexibly adjust\n118 exploration strategies within a single sampling sequence.”\n119\n120\n121\n3\nThe three datasets were pooled for this analysis (N = 85). a. Variance explained in participants’ behaviour. Percentage\nof variance explained by the first five principal components (PCs) of the PCA on the participants’ behaviour (left), and on the\nsimulated behaviour (right), in the DRAW and the GUESS conditions. All three datasets were pooled for this analysis, using\nonly the DRAW and GUESS conditions. The total variance explained by the first five principal components is annotated, plain\nline represents the cumulative explained variance, and horizontal black lines on the simulation plots (right) reproduce the\nvariance explained in the participants’ PCA. b. First component coefficients and scores. Comparison of the coefficients\nand scores of the first component of the PCA done on participants’ behaviour v. on simulated behaviour. Bar plots show the\ncosine similarity of the first component coefficients in each PCA. Scatter plots show the correlation of the individual scores\n(weights) in the first component of the participants v. the simulations PCAs. Dotted lines are the identity line, solid lines are\nregression lines. Pearson’s ρ correlation coefficients and bootstrapped 95% confidence intervals. c. Median split on the\nscores of the first component in participants’ behaviour. Choice similarity matrices plotted for the participants’ behaviour,\nfor participants whose first PC was above and below the median (left). And for simulated behaviour based on the best-fitting\nparameter values of participants whose first PC scores were above or below the median (right).\n4\nfor this analysis (N = 85). a. Distribution of parameter estimates. For the DRAW (top row) and the GUESS condition\n(bottom row): distribution of participant’s parameter values estimated in the first v. the second half of the task – group means\n± 95% CI plotted in colours, and group medians ± inter-quartile ranges for the bounded parameters plotted in grey. Data\npoints outside of the first and third quartile by 1.5 inter-quartile ranges were considered outliers and excluded to avoid\ninsensitive testing between first and second halves (Tuckey’s method).\n5\n125 full complexity of information search behaviour. We have thus emphasised the limitations of our\n126 paradigm in the following Discussion paragraph (p. 17):\n127\n128 “It would be interesting for future research to test our findings in more naturalistic\n129 environments. Our study also focussed on a specific feature of sampling patterns (their\n130 sensitivity to immediate or prospective rewards). Future research should investigate how\n131 this feature interacts with dimensions of choice behaviour – for example participants’\n132 prior expectations about option statistics, the types of uncertainties displayed13, or the\n133 underlying structure of the option space38. Another constraint of the laboratory-based\n134 setting of the study is the sample size available (three datasets of about 30 participants\n135 each). This sample size was sufficient to replicate the effects we had observed in the\n136 discovery dataset, which we replicated twice, but it would be interesting for future\n137 research to replicate our findings again, in a more representative dataset.”\n138\n140 discussion of our main findings.\n141\n143 sense to map it strictly to the reliance on uncertainty minimization. Local search also results in the\n144 acquisition of information and thus can be used to reduce one or the other type of uncertainty.\n146 unsure on what basis the authors infer that participants engage in hypothesis testing during local\n147 search. I recommend that the authors clarify the relationship between local and global search and\n148 uncertainty minimization, as well as potential evidence of hypothesis testing.\n149\n151 have updated the manuscript to make sure that we are consistent in the concepts we use, and\n152 we explain our rationale below.\n153\n155 that do not maximise immediate reward (Sutton and Barto, 1998), including stochastic (e.g.,\n156 softmax, epsilon-greedy) policies. We use ‘information search’ to denote non-random\n157 exploration: this encompasses directed exploration, but also possibly initial repetitive sampling\n158 in the delayed reward conditions. We agree with the reviewer that both of these, directed\n159 exploration (global search) and initial repetitive sampling (local search) result in uncertainty\n160 minimisation. But they differ in: 1/ the type of uncertainty they target (global uncertainty in the\n161 first case, local uncertainty in the second case), and 2/ their efficiency at increasing\n162 performance in the delayed reward conditions.\n163\n165 performance relied on maximising knowledge about both options’ statistics at each choice.\n167 search (see Supplementary Figure 2.b). On the other hand, local search does minimise local\n168 uncertainty, by improving the estimates for the sampled option only – at the expense of global\n169 uncertainty. Thus, while both reduce uncertainty, global uncertainty minimisation offers higher\n170 performance improvements in the GUESS condition, and it is the only strategy expected from an\n6\n171 optimally behaving agent. Therefore, the distinction between local and global search is not so\n172 much about whether one type of search minimizes uncertainty more than the other: it is rather\n173 about the type of uncertainty (local or global) that is being minimised. It is striking that\n174 participants choose to prioritise local uncertainty minimisation at the beginning of each game,\n175 in the condition where they are incentivised to minimise global uncertainty.\n176\n178 very first sample from the option, and go on to test it on the following samples until they were\n179 sufficiently convinced – at which point they could switch options. We phrased this as a\n180 suggestion, but we agree with the reviewer that this concept may confuse readers if it is used\n181 throughout the paper. We have therefore revised the manuscript to restrict the mention of\n182 hypothesis testing to a single paragraph of the Supplementary Discussion section. Importantly,\n183 we now use only “local uncertainty minimisation” to qualify initial repetitive sampling. We\n184 believe this reflects participants trying to form a sufficiently accurate estimation of the current\n185 option before switching to the other option and engaging in global uncertainty minimisation –\n186 what they deem sufficient being reflected in their threshold value. This results in a two-step\n187 strategy, where participants prioritise different forms of uncertainty minimisation over time. This\n188 may be related to the findings described in Spektor and Wulff (2024), where participants’\n189 behaviour was initially best accounted for by a model seeking to observe all possible outcomes\n190 from the current option, before relying on “estimation-driven search” (targeting the most\n191 uncertain option, also known as “directed exploration”, leading to global uncertainty reduction).\n194 consistent in the concepts we use to interpret our findings.\n195\n197 dissipate any confusion in future readers of the paper.\n198\n200 limitations in terms of representativeness.\n201\n203 sample size of our discovery dataset was set to 30 based on existing research performed in\n204 similar (laboratory-based) conditions. While this is still an average sample size for laboratory-\n205 based tasks, it is a relatively small N. On the other hand, the sample size for subsequent\n206 samples was defined through a power calculation on the between-condition differences in\n207 parameter values estimated in the discovery dataset. With an alpha set at 0.05 and a power of\n208 90%, the power analysis called for a sample size of N < 20. Considering that replication effect\n209 sizes are often smaller, we collected about 30 participants in subsequent samples (27 in the first\n210 replication, 31 in the second replication with an additional condition) – both of which therefore\n211 exceeded the theoretically needed sample size to replicate our finding with 90% power.\n212\n214 an important variable often not considered when thinking about sample sizes. Participants\n215 played 96, 128 and 96 sequences in the different datasets (equivalent to what Spektor & Wulff\n216 called “choice problems”), which comprised a total of 1,344 to 1,792 sampling decisions per\n217 participant – this is higher than is typically the case in delayed rewards paradigms (e.g., 404\n7\n218 sampling decisions on average in Spektor & Wulff, 2024). Recent methodological research in\n219 psychology has documented a trade-off between the number of participants and the number of\n220 trials per participant: a higher ratio of trials by participant provides better intra-individual\n221 estimation and lowering the estimates of the sample’s variance (Baker et al., 2021, Smith &\n223 results in this manuscript. Therefore, while we took a somewhat smaller-N approach, we are\n224 confident that the findings we describe are genuine. We have already replicated them in\n225 additional samples for subsequent studies.\n226\n228 the reviewer’s comment on sample size and representativeness in the following section of the\n230\n231 “Another constraint of the laboratory-based setting of the study is the sample size\n232 available (three datasets of about 30 participants each). This sample size was sufficient\n233 to replicate the effects we had observed in the discovery dataset, which we replicated\n234 twice, but it would be interesting for future research to replicate our findings again, in a\n235 more representative dataset.”\n236\n238 authors place too much trust in their computational model. It is unclear whether it represents a\n239 sufficiently accurate and unbiased account of the empirical data to draw such strong inferences. Even if\n240 recovery is demonstrated using simulations, it is always wise not to place too much trust in any\n241 computational model as the data-generating process could be entirely different. In fact, given the\n242 research highlighted in my review, this is to be expected. To address this, I recommend that the authors\n243 present a separate descriptive analysis of switching behavior and individual differences to investigate\n244 the hypothesis expressed in the first comment of reviewer two and demonstrate the existence of\n245 individual differences questioned in the second comment of reviewer two.\n246\n248 hope our response below will dissipate any lingering concerns.\n249\n251 with the reviewer that all models are approximations. Our approach does not exhaustively cover\n252 the full range of possible exploration strategies (i.e., we do not claim that this is the only and\n253 definitive account), nor does it provide the full detail of how computations enfold in participants’\n254 brains (i.e. we are not describing the neural data-generating process). But this was not our goal.\n256 to derive cognitively-relevant variables, as a way to qualify and quantify behaviour more\n257 accurately. The model implements some strategies well established in the literature (reward\n258 seeking, random and directed exploration, and repetition bias – e.g. Sutton & Barto 1998, Wilson\n259 et al., 2014 & 2021, Wu et al. 2018, Dubois et al., 2021), with the addition of a specific early\n260 sampling feature, added as a plausible approximation of the cognitive process underlying early\n261 repetitive sampling. This behaviour had not been documented in related tasks before, so our\n262 study was rooted in behavioural observation, and a discovery-replication approach.\n263\n8\n265 of evidence to accumulate before a switch rather than a mere count). This model fulfills its goal.\n267 when inspecting simulations of the best-fitting model and knock-out simulations\n268 (Supplementary Figures 3, 4 and 7), it captures both average behaviour and inter-individual\n269 differences (Figure 8, Supplementary Figure 11), and it is also well recoverable (Supplementary\n271 words, we hope to convince the reviewer that while we do not trust our model to account for\n272 each and every aspect of sampling, the approach is methodologically sound, and it is well\n273 suited for its purpose. The model allows us to qualify and quantify participants’ behaviour, and\n274 it formulates a cognitively motivated account of the underlying mechanisms.\n275\n277 previous reviewer, and their recommendation for additional descriptive analyses.\n278\n280 difference between the DRAW (immediate rewards) and GUESS (delayed rewards no target)\n281 conditions, based on the different goals of the conditions. We quote Reviewer #2 again here, as\n282 a basis for our response:\n283\n285 conditions (i.e. which option has more of the target colour), but in the GUESS condition, the\n286 focus needs to be on identifying the nature of each option, which can be done in isolation and\n287 does not require contrasting the options. [In the GUESS condition] “streaky” exploration may be\n288 subject to less memory demands than more evenly distributed exploration. A heuristic for the\n289 other conditions [DRAW and FIND] is to pitch the options against each other, for instance\n290 sampling one and then the other and just determining which one had the most of the target\n291 colour in each consecutive pairwise comparison”\n292\n294 analyses, but Reviewer #1 and #2 pointed out that presenting several indices demonstrating the\n295 same behaviour was redundant and obscured the message of the paper. The current version of\n296 the manuscript still features the important behavioural indices on Figure 2, Figure 6 and Figure\n297 8a-d. Thus, to address Reviewer #3’s recommendations for additional behavioural analyses of\n298 switching behaviour without crowding the manuscript, we provide additional behavioural\n299 analyses for the pooled dataset as Supplementary Information (Supplementary Figure 9), but we\n300 refrain from discussing them at length in the main text.\n301\n303 a sequence. They argue that the behavioural difference is not due to the reward and the target\n304 colour manipulations as much as it is due to “the different goals between the conditions”. Their\n305 proposal is that participants produce repetitive streaks in the GUESS condition because they\n306 learn the option colours in isolation, whereas in the DRAW condition they alternate because\n307 they want to compare option values. Even in the early choices, in all conditions, participants\n308 repeated their choices more than they switched (Figures 2.c, Supp.1.c, 6.b, Supp.10). So even in\n309 the DRAW condition, they were not doing alternate sampling and pitting options against each\n310 other, as Reviewer #2 suggests.\n311\n313 (Supplementary Figure 10.a) – this is the main effect we report. If this reflected a systematic\n314 comparison strategy to learn option values in the early choices of the DRAW condition, we\n315 would expect (a) that participants switched regardless of the outcome they observed, and (b)\n316 that switching behaviour would change after the initial choices, once participants have gathered\n317 information about option values. To assess this, we plotted the switch rate at choice t, as a\n318 function of the rewarding value of the outcome observed at choice t-1 (binned into 8 levels –\n320 (always < 0.5), and depended critically on how rewarding the preceding outcome was. We also\n321 observed no evidence for a shift in switching behaviour between early and later choices. In\n322 other words, in the DRAW condition we found no evidence that participants alternated in order\n323 to initially compare options – rather, their decisions to switch appeared to depend primarily on\n324 the rewards obtained (as predicted by our computational model - Supplementary Figure 10.b),\n325 and this was sustained throughout the sequence.\n326\n328 were rewarded for blue and the first option they sampled was blue, they did not need to\n329 investigate the colour of the second option. And indeed, when participants played a DRAW\n330 game in which both options were associated with the target colour, they switched less between\n331 options from early in the game (Supplementary Figure 10.c). On the other hand, there was a\n332 strong incentive to explore both options in the GUESS condition: participants needed to obtain\n333 as much information as possible about both options as soon as possible, before the end of the\n334 game. Yet it is only in the GUESS condition that participants chose to learn in streaks of\n335 repeated decisions. While we agree with Reviewer #2 that repetitive sampling is likely\n336 cognitively easier, it is striking that participants only did it when they were incentivised to\n337 minimise global uncertainty about the task structure. We discuss the tension between cognitive\n338 costs and information gathering on p. 16:\n339\n340 “Thus, it is striking that participants chose to prioritise local uncertainty minimisation\n341 precisely in conditions where they were incentivised to minimise global uncertainty. This\n342 likely reflects a trade-off between information search and cognitive costs: while directed\n343 sampling of uncertain option was most efficient in reducing global uncertainty, it was\n344 also cognitively costly to update beliefs about several options in parallel – e.g. ‘switch\n345 costs’, which have been associated with mental effort24,25 and even subjective pain26.\n347 likely alleviated these costs. This strategy, however, was only used by tested participants\n348 when rewards were delayed.”\n349\n352\n353 “Another alternative explanation for the difference in sampling patterns between\n354 conditions is to propose that participants relied on different heuristics because the\n355 instructions gave them different goals. One could propose that participants repeated\n356 choices in the delayed rewards (GUESS) condition because they were free to study each\n357 option in isolation, whereas in the DRAW and FIND conditions they were pushed to\n10\n358 compare options by sampling one after the other, juxtaposing pairs of outcomes. We did\n359 not find any evidence that participants tended to alternate between options in the DRAW\n360 and FIND sequences (Supplementary Figure 10). Rather, repetition biases were positive\n361 (meaning that participants did not tend to alternate between options in any condition),\n362 comparable and correlated across all three conditions (Figure 7.a. and Supplementary\n364 conditions (Figure 7.a.). Given such consistency in parameter estimates across\n365 conditions, it is very unlikely that participants resorted to wildly different heuristics in the\n366 different conditions. In addition, in the DRAW condition, even in the initial choices,\n367 participants’ decisions to switch option appeared to depend mainly on the reward value\n368 of the preceding outcome (Supplementary Figure 10), rather than demonstrating\n369 systematic alternation.”\n370\n372 hypothesis from Reviewer #2 cannot not explain the behavioural effects we find and report.\nAlternation behaviour. Average switch rate at each choice position in the sequence (group mean ± 95% within participant\nCI). b. Alternation rate by previous outcome. In the DRAW condition, alternation rate in participants’ decisions as a\nfunction of the rewarding value of the preceding outcome (binned into 8 bins), in the first four, following four, and remaining\ntwelve choices . Group means (dots) and 95% CI (confidence bars). c. Choice to choice similarity. Average similarity of\neach choice to the other choices in the same sequence. In the DRAW condition, for sequences where a both, a single, or\nnone of the options were associated to the target colour. In the GUESS condition, for sequences where the options were\nassociated to the same, or different colours.\n11\n374\n375\n377 patterns across individuals\" section. Reviewer #2 was concerned that the inter-individual\n378 differences in parameter values we report were based on a biased estimation of those\n379 parameters, and reflected differences in model fitting rather than genuine differences in\n380 behaviour.\n381\n383 model best fit their behaviour in the GUESS condition. Although participants were split based\n384 on which model (INI+ or INI-) best fit their behaviour, we compared parameters using the most\n385 parameterized version of the model (INI+). This approach allowed for a fair comparison between\n386 parameters, and for participants whose best-fit was the INI- model, threshold parameters in the\n388 accounting for the additional parameter. Based on this split, we reported group differences both\n389 in behavioural indices (e.g. higher performance in both conditions for the INI+ group) and in\n390 parameter estimates (higher estimates for the later phase parameters in the INI+ group:\n391 sensitivity to the target, to uncertainty, and repetition bias).\n392\n394 groups due to not being optimised on the same choices (i.e. estimated on the full sequence in\n395 the INI- participants vs. only after the initial phase was over in the INI+ participants). In other\n396 words, Reviewer #2 suspected that the inter-individual differences in parameter estimates were\n397 a trivial by-product of the model fitting, rather than genuine differences in sampling strategies\n398 between participants.\n399\n401 through recovery that later phase parameters were equally well estimated regardless of the\n402 presence or absence of an initial sampling phase (Supplementary Figure 6). It was not the case\n403 that later phase parameters were mechanically biased by the presence or absence of an initial\n404 phase. Differences in parameter estimates between the INI+ and INI- groups do not trivially\n405 surface from the model fitting pipeline, but rather, the difference in parameters suggests\n406 underlying differences in sampling strategies. More specifically, that seemingly contradictory\n407 sampling strategies co-occur in participants (i.e. suboptimal initial repetitive sampling and\n408 optimal directed exploration), and that statistically suboptimal strategies can be associated with\n409 higher performance in participants (higher evidence for the initial sampling phase in the delayed\n410 rewards condition was associated to higher performance in both conditions). We hope that this\n411 clarification will convince Reviewer #3 that the model-based analyses are actually well suited for\n412 measuring the inter-individual differences questioned by Reviewer #2. We would like to reiterate\n413 that we acknowledge that, like all models, our model is ultimately an approximation of cognitive\n414 strategies. Yet the strict methodology we applied makes it valuable in revealing genuine\n415 differences in behaviour.\n416\n418 them of the existence of individual differences between INI+ and INI- participants. Because we\n419 split participants based on which of the INI+ or INI- model best fit their behaviour, some\n420 analyses of repetition rate are circular. For example, the analysis of repetition rate by choice\n12\n421 position, repetition rate by familiarity with the option, or choice imbalance are trivially expected\n422 to differ between groups (Figure 8.a and Supplementary Figure 9.b). We mention this in the\n423 corresponding subsection of the Results on p. 13:\n424\n425 “As expected from this splitting criterion [emphasis added], in the GUESS condition,\n426 participants in the first group (INI+) showed a higher tendency to initially resample\n427 options (Figure 8.a., b., d.).”\n428\n430 performance (Figure 8.b-c.). Both of these are behavioural indices, and they are not trivially\n431 expected to differ from our splitting criterion. It is particularly striking that participants who\n432 displayed stronger evidence for suboptimal initial repetitive sampling in the GUESS condition\n433 (the INI+ group) also achieved higher performance in both tasks – a purely model-free measure\n434 (Figure 8.c.). Additionally, the INI+ participants displayed more condition-relevant choices in\n435 both conditions (Figure 8.b.). Such co-occurrence is surprising because initial repetitive\n436 sampling is suboptimal yet associated with task-optimal choices, and with higher performance.\n438 sampling, or whether it is merely dependent on otherwise higher task-optimal behaviour in the\n439 same participants (higher reward-drive and higher sensitivity to uncertainty) – but this was\n440 beyond the scope of this initial study.\n441\n443 our focus on performance and choice direction will convince Reviewer #3 of the existence of the\n444 individual differences questioned in the second comment of Reviewer #2.\nthree datasets were pooled and participants were split in two groups based on whether a model with (ini+, N = 67) or without\n(ini-, N = 18) an initial sampling phase better accounted for their exploration patterns in GUESS sequences. a. Overall\nfraction of repeat decisions. Fraction of repeat decisions in each condition, averaged per participant for all the participants\n(left), and for the ini+ and ini- participants separately (right) – median ± inter-quartile range. b. Choice imbalance. Difference\nbetween the number of choices for the most and the least chosen option at each choice throughout the games. Solid lines\ndisplay the average for participants whose best-fitting model in the GUESS condition included an initial sampling phase (ini+,\nN = 67), dotted lines show the average of participants whose best-fitting model did not include an initial sampling phase (ini-\n, N = 18) – group means ± 95% CI.\n13\n446\n447\n449 the authors model whether a sampling decision is repeated as a function of the decision variable DV,\n450 independent of which option was sampled last. Consequently, I would expect that the probability of a\n451 repeat should be based on the absolute differences in the amount and direction of evidence for the two\n452 options rather than their signed differences (i.e., \\hat{x_{1,t}} - \\hat{x_{2,t}} and \\hat{\\lvert x_{1,t} \\rvert} -\n453 \\hat{\\lvert x_{2,t} \\rvert}), but the definition of DV suggests the opposite. The result is that only\n454 differences in one direction (e.g., \\hat{x_{1,t}} > \\hat{x_{2,t}} but not \\hat{x_{1,t}} < \\hat{x_{2,t}}) can\n455 trigger a switch to the other option. I don’t understand how this makes sense.\n456\n458 much like to dissipate in the final version of the manuscript. Outside of the initial sampling\n459 phase, the decision variable (DV) used by the model to choose between option 1 and option 2 –\n460 using a standard softmax policy – consists of three terms: 1/ the difference in the log-odds that\n461 each option is associated with the target colour, controlled by a target sensitivity parameter\n462 𝜷 , 2/ the difference in the absolute log-odds that each option is associated with a colour (i.e.,\n𝒕𝒂𝒓\n463 the certainty of the association between each option and a colour), controlled by an uncertainty\n464 sensitivity parameter 𝜷 , and 3/ a bias toward sampling the same option as in the previous\n𝒕𝒂𝒓\n465 trial, controlled by a repetition bias parameter 𝒃 .\n𝒓𝒆𝒑\n466\n468 second term reflects directed exploration (as in Wilson et al., 2014), where the model will\n469 preferentially sample (when 𝜷 > 0) the option whose value is most uncertain (as in e.g.\n𝒖𝒏𝒄\n471 which option was sampled on the previous trial, with the inclusion of the third term – the\n472 repetition bias. But to be perfectly explicit, our model computes the probability of choosing\n473 option 1 over option 2, not the probability of re-sampling the previous option. See below the\n474 pseudo-code adapted from https://gitlab.com/cle-a/COLPUB:\n475\n476\n477\n478 % accumulate information about each option\n479 x(t,1) = x(t-1,1) + previous_outcome; % update option 1 (chosen)\n480 x(t,2) = x(t-1,2); % do not update option 2 (not chosen)\n481\n482 % at each sample, compute the decision variable DV\n483 dv = beta_tar * (x(t,1)-x(t,2)) * target_color; % first term\n484 dv = dv + beta_unc * (abs(x(t,2))–abs(x(t,1))); % second term\n485 dv = dv + beta_rep * previous_choice; % third term\n486\n487 % compute the choice probability of each option\n488 choice_prob(1) = 1/(1+exp(-dv)); % p(choose option 1)\n489 choice_prob(2) = 1-choice_prob(1); % p(choose option 2)\n490\n491\n14\n𝒕𝒂𝒓\n493 option (it is only defined when the target is given in the beginning of the sequence - in DRAW).\n𝒖𝒏𝒄\n495 accumulated evidence). Finally, a higher 𝒃 pushes choices towards re-sampling the same\n𝒓𝒆𝒑\n496 option as in the previous trial. Note that in the online code, 𝜷 was positive when the model\n𝒖𝒏𝒄\n497 was attracted toward the most certain option (which is the opposite of directed exploration),\n498 such that negative values were observed in human participants. We have above (before the\n499 original submission, but after putting the first version of the model code online) flipped 𝜷 to\n𝒖𝒏𝒄\n500 be positive when the model is attracted toward the most uncertain option. We hope this clarifies\n501 any lingering ambiguity about the implementation of the model.\n502\n503\n505 responses to Reviewer #2. We are pleased to read that they are inclined to recommend\n506 publication, and we hope that our detailed responses to each of their different suggestions have\n507 addressed all remaining concerns. Please find references and copies of the figures that are\n508 useful when reading this response in the remaining pages below.\n- Baker, D. H., Vilidaite, G., Lygo, F. A., Smith, A. K., Flack, T. R., Gouws, A. D., & Andrews, T. J.\nPower contours: Optimising sample size and precision in experimental psychology and human\nneuroscience. Psychological methods, 26(3), 295 (2021).\n- Dubois, M., & Hauser, T. U. Value-free random exploration is linked to impulsivity. Nature\nCommunications, 13(1), 4542 (2022).\n- Smith, P. L., & Little, D. R. Small is beautiful: In defense of the small-N design. Psychonomic\nbulletin & review, 25, 2083-2101 (2018).\n- Spektor, M. S., & Wulff, D. U. Predecisional information search adaptively reduces three types\nof uncertainty. Proceedings of the National Academy of Sciences, 121(47), e2311714121\n(2024).\n- Sutton, R. S., & Barto, A. G. Reinforcement learning: An introduction (Vol. 1, No. 1, pp. 9-11).\nCambridge: MIT press (1998).\n- Wilson, R. C., Geana, A., White, J. M., Ludvig, E. A., & Cohen, J. D. Humans use directed and\nrandom exploration to solve the explore–exploit dilemma. Journal of experimental psychology:\nGeneral, 143(6), 2074 (2014).\n- Wilson, R. C., Bonawitz, E., Costa, V. D., & Ebitz, R. B. Balancing exploration and exploitation\nwith information and randomization. Current opinion in behavioral sciences, 38, 49-56 (2021).\n- Wu, C. M., Schulz, E., Speekenbrink, M., Nelson, J. D., & Meder, B. Generalization guides\nhuman exploration in vast decision spaces. Nature human behaviour, 2(12), 915-924 (2018).\nthree datasets were pooled and participants were split in two groups based on whether a model with (ini+, N = 67) or without\n(ini-, N = 18) an initial sampling phase better accounted for their exploration patterns in GUESS sequences. a. Overall\nfraction of repeat decisions. Fraction of repeat decisions in each condition, averaged per participant for all the participants\n(left), and for the ini+ and ini- participants separately (right) – median ± inter-quartile range. b. Choice imbalance. Difference\nbetween the number of choices for the most and the least chosen option at each choice throughout the games. Solid lines\ndisplay the average for participants whose best-fitting model in the GUESS condition included an initial sampling phase (ini+,\nN = 67), dotted lines show the average of participants whose best-fitting model did not include an initial sampling phase (ini-\n, N = 18) – group means ± 95% CI.\nAlternation behaviour. Average switch rate at each choice position in the sequence (group mean ± 95% within participant\nCI). b. Alternation rate by previous outcome. In the DRAW condition, alternation rate in participants’ decisions as a\nfunction of the rewarding value of the preceding outcome (binned into 8 bins), in the first four, following four, and remaining\ntwelve choices . Group means (dots) and 95% CI (confidence bars). c. Choice to choice similarity. Average similarity of\neach choice to the other choices in the same sequence. In the DRAW condition, for sequences where a both, a single, or\nnone of the options were associated to the target colour. In the GUESS condition, for sequences where the options were\nassociated to the same, or different colours.\nThe three datasets were pooled for this analysis (N = 85). a. Variance explained in participants’ behaviour. Percentage\nof variance explained by the first five principal components (PCs) of the PCA on the participants’ behaviour (left), and on the\nsimulated behaviour (right), in the DRAW and the GUESS conditions. All three datasets were pooled for this analysis, using\nonly the DRAW and GUESS conditions. The total variance explained by the first five principal components is annotated, plain\nline represents the cumulative explained variance, and horizontal black lines on the simulation plots (right) reproduce the\nvariance explained in the participants’ PCA. b. First component coefficients and scores. Comparison of the coefficients\nand scores of the first component of the PCA done on participants’ behaviour v. on simulated behaviour. Bar plots show the\ncosine similarity of the first component coefficients in each PCA. Scatter plots show the correlation of the individual scores\n(weights) in the first component of the participants v. the simulations PCAs. Dotted lines are the identity line, solid lines are\nregression lines. Pearson’s ρ correlation coefficients and bootstrapped 95% confidence intervals. c. Median split on the\nscores of the first component in participants’ behaviour. Choice similarity matrices plotted for the participants’ behaviour,\nfor participants whose first PC was above and below the median (left). And for simulated behaviour based on the best-fitting\nparameter values of participants whose first PC scores were above or below the median (right).\nfor this analysis (N = 85). a. Distribution of parameter estimates. For the DRAW (top row) and the GUESS condition\n(bottom row): distribution of participant’s parameter values estimated in the first v. the second half of the task – group means\n± 95% CI plotted in colours, and group medians ± inter-quartile ranges for the bounded parameters plotted in grey. Data\npoints outside of the first and third quartile by 1.5 inter-quartile ranges were considered outliers and excluded to avoid\ninsensitive testing between first and second halves (Tuckey’s method).\nReviewer #4\nHaving reviewed R3’s comments and the author’s responses to them, I believe that the authors have\ndone a great job addressing all the comments thoroughly and convincingly, substantially improving the\nconceptual clarity of the paper and adding helpful model-independent analyses.\nWe thank the reviewer for taking the time to evaluate our response to Reviewer 3, and we are\npleased to read that they are satisfied with our revisions.\n1"}, {"round": 2, "reviewer_comments": {"reviewer_1": "I'm satisfied with the revision. One small typo in the abstract: \"patterns arising in when\" -> \"patterns arising when\"", "reviewer_2": "The authors did a good job responding to the issues raised in the reviews of the previous paper. The presentation is more\nfocused, and the modelling has improved. That said, I still have doubts about the theoretical significance of the findings\n(initial streaks in the GUESS condition), due to the different goals between the conditions. The focus should be on\ncomparing the two options in the DRAW and FIND conditions (i.e. which option has more of the target colour), but in the\nGUESS condition, the focus needs to be on identifying the nature of each option, which can be done in isolation and does\nnot require contrasting the options. To determine the nature of a single option, \"streaky\" exploration may be subject to less\nmemory demands than more evenly distributed exploration. A heuristic for the other conditions is to pitch the options against\neach other, for instance sampling one and then the other and just determining which one had the most of the target colour in\neach consecutive pairwise comparison. This seems quite reasonable to me and would account for the results without many of\nthe processes relied upon in the discussion. That said, the finding of a difference in exploration behaviour is supported\nwell by the analyses and the \"streaky\" exploration is suboptimal when the length of a sequence is unknown and a trial can\nend at any time. But I do think the discussion can be shortened and streamlined further.\nOne major issue I have with the current version relates to the \"Positive covariations between exploration patterns across\nindividuals\" section, where participants are split into two groups based on whether the model with initial \"streaky\" sampling\nfits better than a model without. Part of the analysis here focuses on the remaining parameters of the model. If I understand\nthe model correctly, then – because the initial choices are governed strongly by the initial sampling policy in the INI+ group,\nthe estimates of the remaining parameters will be mostly determined by later choices (after the initial sampling phase is\nover). In the INI- group, the remaining parameters likely cover the whole sequence (or at least most of the sequence).\nResults such as that the sensitivity to the reward maximising option is higher in the INI+ group than the INI- group are then\nnot overly surprising, as one would expect that during the later trials participants would have determined the best option. So,\nthese comparisons are biased due to the parameters in the INI+ group being optimized for the later parts of the sequences,\nwhilst this is less the case for the INI- group. In my view, then, comparisons between the INI+ and INI- group, in terms of\nmodel parameters, are not overly meaningful. If such a comparison is of interest, I suggest to only focus on the behavioural\nresults (e.g. proportion of maximising choices, repeat choices, etc.)\nA more minor issue is that I think the Horizon task is misrepresented on page 9: \"This new condition was akin to browsing\nrestaurant reviews: choices yielded no immediate reward, but outcomes were still valued with respect to an end goal. It was\nalso closer to existing delayed-reward tasks, like the Horizon task9 or in the “sampling paradigm” from Decisions From\nExperience studies12. In these paradigms, rewards are only granted after a sequence of so-called “free choices” (as in the\nGUESS condition), but (in contrast to the GUESS condition), choice outcomes are numerical values, and therefore directly\ninform the participants of the option’s ultimate rewarding value.\" This seems to imply delayed rewards for free choices in the\nHorizon task, but - as far as I'm aware - rewards are not delayed in the Horizon task, beyond an initial sequence of forced\nchoices which manipulate the amount of information gathered. So I think this is a misrepresentation of the Horizon task."}, "author_response": "We thank the reviewers for taking the time again to assess the revised version of our\nmanuscript. We are very pleased to read that Reviewer #1 is satisfied with the\nrevisions, and that Reviewer #2 found the revised manuscript more focused and the\nmodelling improved. We address below the remaining comments of Reviewer #2 in\ngreen. Quotes from the new revisions to the manuscript are highlighted in blue.\nReviewer #1 (Remarks to the Author):\nI'm satisfied with the revision. One small typo in the abstract: \"patterns arising\nin when\" -> \"patterns arising when\"\nWe thank the reviewer for spotting this typo. We have corrected it in the revised\nmanuscript. We are happy that the first reviewer is satisfied with the revised manuscript\nand we thank them again for their insights.\nReviewer #2 (Remarks to the Author):\nThe authors did a good job responding to the issues raised in the reviews of\nthe previous paper. The presentation is more focused, and the modelling has\nimproved. That said, I still have doubts about the theoretical significance of the\nfindings (initial streaks in the GUESS condition), due to the different goals\nbetween the conditions. The focus should be on comparing the two options in\nthe DRAW and FIND conditions (i.e. which option has more of the target\ncolour), but in the GUESS condition, the focus needs to be on identifying the\nnature of each option, which can be done in isolation and does not require\ncontrasting the options. To determine the nature of a single option, \"streaky\"\nexploration may be subject to less memory demands than more evenly\ndistributed exploration. A heuristic for the other conditions is to pitch the\noptions against each other, for instance sampling one and then the other and\njust determining which one had the most of the target colour in each\nconsecutive pairwise comparison. This seems quite reasonable to me and\nwould account for the results without many of the processes relied upon in the\ndiscussion. That said, the finding of a difference in exploration behaviour is\nsupported well by the analyses and the \"streaky\" exploration is suboptimal\nwhen the length of a sequence is unknown and a trial can end at any time. But\nI do think the discussion can be shortened and streamlined further.\nWe are very happy that the reviewer found our responses to their previous comments\nconvincing, and the revised manuscript more focused with an improved modelling of\nhuman behaviour.\nWe thank the reviewer for allowing us to further clarify the theoretical significance of\nour findings. The reviewer proposes that participants sample options in streaks in the\nGUESS condition because they were free to identify the dominant colour associated\nwith each option in isolation. The reviewer suggests that participants do not use this\nstrategy in the DRAW condition because they were pushed to compare options by\nsampling one after the other to immediately compare the pairs of outcomes.\nWe believe that this interpretation cannot explain our findings. First, we do not observe\nbasic alternated sampling in the DRAW and FIND conditions, but rather a focus on the\nPage 1 of 6\ntarget-coloured option. The model results also suggest that participants were\naccumulating information over several samples, rather than comparing the pair sample\nby sample. In addition, repetition biases are positive (meaning that participants do not\ntend to alternate between options in any condition), comparable and correlated across\nall three conditions (Figure 7.a. and Supplementary Figure 7.b.), as well as the\nsensitivity to uncertainty across the GUESS and FIND conditions (Figure 7.a. and\nadditional figure a. below). Given such consistency in parameter estimates across\nconditions (including repetition biases which would be negative if participants indeed\nrelied on the strategy suggested by the reviewer), it is very unlikely that participants\nresorted to wildly different heuristics in the different conditions.\nFrom a theoretical perspective, in the GUESS condition the incentive really was on\nidentifying the colour of both options as fast as possible (since the game ended after\na variable number of choices), yet this is the condition where participants did sample\nin streaks – i.e. they were taking the time to explore a single option at the expense of\nuncertainty regarding the other option. We fully agree with the reviewer that such\n“streaky sampling” is likely cognitively easier for participants – e.g. in terms of memory\ndemands – and this is the focus of a dedicated paragraph in the Discussion section.\nImportantly, there was also no actual need to compare options in the DRAW and FIND\nconditions. Participants had learnt during the training (both by explicit instructions and\nby sampling during training games) that the dominant colour of each option was\nindependent of the dominant colour of the other option. This means that comparing\noptions would have been a bad strategy for half of the games where the dominant of\nthe two options was the same. And indeed, as stressed above, we did not find any\nevidence that participants tended to alternate between options in the DRAW and FIND\nconditions.\nFinally, if sampling in streaks is cognitively easier, and if there was no actual benefit to\nalternate / compare options in the DRAW (and FIND) condition(s), it is difficult to\nexplain why participants did not produce streaks in this condition but only in the GUESS\ncondition, where they were incentivised to sample from both options in a single game.\nThis means that the presence of rewarding value of outcomes across conditions\n(present in the DRAW and FIND conditions, absent in the GUESS condition)\nconstitutes a more parsimonious explanation for the different exploration patterns\nobserved in the three conditions.\nWe have included a discussion of the reviewer’s point in the discussion on page 16:\n“An alternative explanation is to propose that participants produced streaks in the\nGUESS condition because they were free to study each option in isolation (probably\nbecause repetitive sampling was cognitively easier than alternating), whereas in the\nDRAW and FIND conditions they were pushed to compare options by sampling one\nafter the other to immediately compare the pairs of outcomes. However, we did not\nfind any evidence that participants tended to alternate between options in the DRAW\nand FIND conditions. Rather, repetition biases were positive (meaning that participants\ndid not tend to alternate between options in any condition), comparable and correlated\nacross all three conditions (Figure 7.a. and Supplementary Figure 7.b.), and so was\nthe sensitivity to uncertainty across the GUESS and FIND conditions (Figure 7.a.).\nGiven such consistency in parameter estimates across conditions, it is very unlikely\nthat participants resorted to wildly different heuristics in the different conditions. In\naddition, participants had learnt during training that the dominant colour of each option\nPage 2 of 6\nwas independent of the dominant colour of the other option. Thus, comparing options\nwould have been a bad strategy for half of the games where the dominant of the two\noptions was the same. If sampling in series is cognitively easier, and if there was no\nactual benefit to alternate / compare options in the DRAW and FIND conditions, it is\ndifficult to explain why participants did not produce streaks in this condition but only in\nthe GUESS condition, where they were incentivised to sample from both options in a\nsingle game. The presence of (immediate or prospective) reward value appears to be\nthe main determinant of participants’ early sampling strategies.”\nAdditional figure a. Sensitivity to uncertainty correlation. Correlation of sensitivity to uncertainty\nestimates (𝛽 ) obtained in the GUESS and FIND conditions. Spearman’s correlation.\n\"#$\nTo follow the reviewer’s recommendations, we have edited short parts of the\ndiscussion for clarity, but we believe all sections are relevant to discussing the\nsignificance of the results. The discussion is longer with the previous suggestions from\nthe reviewers, but we believe these were relevant, sometimes necessary, additions.\nOne major issue I have with the current version relates to the \"Positive\ncovariations between exploration patterns across individuals\" section, where\nparticipants are split into two groups based on whether the model with initial\n\"streaky\" sampling fits better than a model without. Part of the analysis here\nfocuses on the remaining parameters of the model. If I understand the model\ncorrectly, then – because the initial choices are governed strongly by the initial\nsampling policy in the INI+ group, the estimates of the remaining parameters\nwill be mostly determined by later choices (after the initial sampling phase is\nover). In the INI- group, the remaining parameters likely cover the whole\nsequence (or at least most of the sequence). Results such as that the\nsensitivity to the reward maximising option is higher in the INI+ group than the\nINI- group are then not overly surprising, as one would expect that during the\nlater trials participants would have determined the best option. So, these\ncomparisons are biased due to the parameters in the INI+ group being\noptimized for the later parts of the sequences, whilst this is less the case for\nthe INI- group. In my view, then, comparisons between the INI+ and INI-\ngroup, in terms of model parameters, are not overly meaningful. If such a\ncomparison is of interest, I suggest to only focus on the behavioural results\n(e.g. proportion of maximising choices, repeat choices, etc.)\nWe agree with the reviewer that comparing parameters obtained in two different\nversions of the model would be biased in principle. This is why, for this analysis, we\nsplit the participants based on which of the models best fit their behaviour in the\nGUESS condition, but then compared parameters obtained in the most specified\nversion of the model (the ini+ model). This ensures that the “later phase parameters”\nPage 3 of 6\n(repetition bias and sensitivity parameters) were immediately comparable. The\nreviewer is right that this was not explicitly described in the text, and we have therefore\nadded the following clarification on page 13 in the revised manuscript: “We split\nparticipants based on which of the model with (ini+ group, N = 67) or the model without\nan initial evidence threshold (ini- group, N = 18) best fit their behaviour, but to ensure\na fair comparison between groups we then compared parameter estimates obtained in\nthe most specified version of the model (the version with initial threshold, ini+).”. We\nthank the reviewer for spotting this imprecision.\nHowever, we would like to point out that the presence of an initial sampling phase of\nthe model does not bias the estimation of “later phase parameters” (repetition bias and\nsensitivity parameters).\nRegarding the sensitivity to the target specifically: in the DRAW condition (where\nsensitivity to the target is estimated), threshold values estimated from the ini+ were\nvery close to 0.5 for the large majority of participants, resulting in virtually ini- models\n(i.e. even when taking the ini+ model, the estimated threshold values are so low that it\nresults in a threshold free, ini- behaviour – see additional Figure b. below).\nAdditionally, we performed recovery analyses showing that all “later phase\nparameters” (repetition bias and sensitivity parameters) were actually equally well\nestimated by an ini+ for agents with or without initial sampling phase. We simulated\ndata from agents with or without the initial sampling phase, and then fitted this data\nwith an ini+ model (so a model with a free θ parameter, allowing for an initial sampling\nphase). As illustrated on a new Supplementary Figure 6, we observed that regardless\nof whether the agent’s behaviour truly displayed an initial sampling phase or an\nabsence of initial sampling phase (simulated as ini+ or ini-), “later phase parameters”\nwere equally well recovered with an ini+ model. In other words, the additional initial\nphase parameters featured in an ini+ model did not introduce a bias in the estimation\nof later phase parameters (repetition bias and sensitivity to uncertainty).\nWe hope that this additional analysis helps convince the reviewer that the presence of\nan initial sampling phase in the model does not negatively impact the estimation of\nother parameters, and therefore, that splitting participants based on the\npresence/absence of an initial phase in the best-fitting model for their behaviour does\nnot preclude further analysis of the remaining parameters.\nAdditional figure b. Threshold level. Threshold levels (θ) obtained in the ini+ model in the DRAW\ncondition, for participants assigned to ini+ or ini- group based on the fits obtained in the GUESS\ncondition.\nAs the reviewer suggests, we have emphasised the model-free differences between\nthe two groups. The proportion of reward maximising and the proportion of repeat\nchoices are provided, e.g. on page 13: “However, ini+ participants also had a stronger\nPage 4 of 6\ntendency to prefer the target-coloured options in the DRAW condition, which is optimal\nin this condition (choice direction curves on Figure 8.b.; higher sensitivity parameter\n𝛽\n%&'\non Figure 8.e.; two-sample t-test between groups of participants: t = 2.20, p < 0.05,\n83\nd = 0.58).” and “As expected from this splitting criterion, in the GUESS condition,\nparticipants in the first group (ini+) showed a higher tendency to initially resample\noptions (Figure 8.a.,b.,d.).”\nBecause we split participants based on which of the model with or without initial\nsampling phase best fit their behaviour, differences in the proportion of repeat\ndecisions were expected and merely confirm the basis of the split (i.e. that participants\nwhose behaviour is best represented by the ini+ model display more repeated choices\nat the beginning of sequences). Thus, we focussed on differences in performance and\nsensitivity to the target and to uncertainty because these are independent from our\nsplitting criterion. We agree with the reviewer that it is particularly striking that\nparticipants assigned ini+ based on their behaviour in the GUESS condition also have\nhigher performance in both conditions. These results suggest that suboptimal\nexploratory behaviours (initial repetitive sampling) co-occur with more optimal\nstrategies (target-seeking and directed exploration in the DRAW and GUESS & FIND\nconditions respectively).\nA more minor issue is that I think the Horizon task is misrepresented on page\n9: \"This new condition was akin to browsing restaurant reviews: choices\nyielded no immediate reward, but outcomes were still valued with respect to\nan end goal. It was also closer to existing delayed-reward tasks, like the\nHorizon task9 or in the “sampling paradigm” from Decisions From Experience\nstudies12. In these paradigms, rewards are only granted after a sequence of\nso-called “free choices” (as in the GUESS condition), but (in contrast to the\nGUESS condition), choice outcomes are numerical values, and therefore\ndirectly inform the participants of the option’s ultimate rewarding value.\" This\nseems to imply delayed rewards for free choices in the Horizon task, but - as\nfar as I'm aware - rewards are not delayed in the Horizon task, beyond an\ninitial sequence of forced choices which manipulate the amount of information\ngathered. So I think this is a misrepresentation of the Horizon task.\nThe reviewer is right that the wording was confusing, we have stated the difference\nexplicitly. p9: ”In these paradigms, as in the GUESS condition, rewards are only\ngranted after a sequence of “non-incentivised choices” (after a predefined sequence\nof forced choices in the Horizon task, or after a self-terminated sequence of free but\nnon-rewarded choices in the “sampling paradigm”). However, in both cases choice\noutcomes are numerical values, and therefore directly inform the participants of the\noption’s ultimate rewarding value, in contrast to the GUESS condition.”\nReviewer #2 (Remarks on code availability):\nI do not have access to Matlab, on which the code relies. I did have a look and\nthe repository exists. But the contents are from 2 years ago and as the\nmodelling has changed since the last manuscript, I think it needs updating.\nPage 5 of 6\nThe reviewer is correct that we had not yet made the revised code public. It is now\nupdated and publically available at the same link: https://gitlab.com/cle-a/colrep\nPage 6 of 6"}, {"round": 3, "reviewer_comments": {"reviewer_3": "The current manuscript investigates information search patterns in three conditions, divorcing information search from choice outcomes to different extents. Across three experiments, evidence is presented that suggests different information search patterns, with conditions with delayed or fully suspended rewards producing behavior directed more strongly at reducing uncertainty. To account for this, the authors suggest a computational model. Overall, the presented results are interesting and suitable for Nature Communications. I am inclined to recommend publication, but I believe the authors should first address the following comments, which may help clarify the exposition and interpretation of their findings. This can be achieved without conducting additional analyses or studies.\nThe simplicity of the behavioral paradigm is intriguing but also limiting. It does not afford the study of the full complexity of information search situations, which includes prior expectations and types of uncertainty. In our recent work forthcoming in PNAS (Spektor & Wulff, 2024), we attempted to reveal additional details of the full complexity of human information search by means of analyzing a large database of the sampling paradigm. This revealed that people enter the task with ecological (structural) knowledge, which they update across the task and use to steer their information search behavior. It further revealed that individuals reduce multiple types of uncertainties and differ not only in the details of a common computational strategy but likely recruit entirely different strategies. Finally, it also showed evidence of strategy shifts during the task, which may be related to the local-global strategy shift described here. Not all aspects can be studied with the current paradigm, which is fine. However, I think it is important to emphasize more strongly and discuss in greater length the limitations of both the paradigm and the computational account with respect to capturing the full complexity of human information search behavior.\nI believe the distinction between local and global search is interesting. However, I am not sure it makes sense to map it strictly to the reliance on uncertainty minimization. Local search also results in the acquisition of information and thus can be used to reduce one or the other type of uncertainty. Relatedly, it is important to distinguish between exploration and information search. Moreover, I am unsure on what basis the authors infer that participants engage in hypothesis testing during local search. I recommend that the authors clarify the relationship between local and global search and uncertainty minimization, as well as potential evidence of hypothesis testing.\nI believe the authors should discuss the relatively small sample sizes of their studies and potential limitations in terms of representativeness.\nConcerning the responses to past reviewers, which I was asked to evaluate, my impression is that the authors place too much trust in their computational model. It is unclear whether it represents a sufficiently accurate and unbiased account of the empirical data to draw such strong inferences. Even if recovery is demonstrated using simulations, it is always wise not to place too much trust in any computational model as the data-generating process could be entirely different. In fact, given the research highlighted in my review, this is to be expected. To address this, I recommend that the authors present a separate descriptive analysis of switching behavior and individual differences to investigate the hypothesis expressed in the first comment of reviewer two and demonstrate the existence of individual differences questioned in the second comment of reviewer two.\nFinally, I would appreciate it if the authors could clarify aspects of their model. If I understand correctly, the authors model whether a sampling decision is repeated as a function of the decision variable DV, independent of which option was sampled last. Consequently, I would expect that the probability of a repeat should be based on the absolute differences in the amount and direction of evidence for the two options rather than their signed differences (i.e., \\hat{x_{1,t}} - \\hat{x_{2,t}} and \\hat{\\lvert x_{1,t} \\rvert} - \\hat{\\lvert x_{2,t} \\rvert}), but the definition of DV suggests the opposite. The result is that only differences in one direction (e.g., \\hat{x_{1,t}} > \\hat{x_{2,t}} but not \\hat{x_{1,t}} < \\hat{x_{2,t}}) can trigger a switch to the other option. I don’t understand how this makes sense."}, "author_response": "1 Reviewer #3\n2\n4 information search from choice outcomes to different extents. Across three experiments, evidence is\n5 presented that suggests different information search patterns, with conditions with delayed or fully\n6 suspended rewards producing behavior directed more strongly at reducing uncertainty. To account for\n7 this, the authors suggest a computational model. Overall, the presented results are interesting and\n8 suitable for Nature Communications. I am inclined to recommend publication, but I believe the authors\n9 should first address the following comments, which may help clarify the exposition and interpretation of\n10 their findings. This can be achieved without conducting additional analyses or studies.\n11\n13 inclined to recommend publication. We have amended the manuscript in line with the reviewer’s\n14 recommendations and we hope they will be fully satisfied with the revised submission; we thank\n15 them again for their help in improving the final paper.\n16\n18 the full complexity of information search situations, which includes prior expectations and types of\n19 uncertainty. In our recent work forthcoming in PNAS (Spektor & Wulff, 2024), we attempted to reveal\n20 additional details of the full complexity of human information search by means of analyzing a large\n21 database of the sampling paradigm. This revealed that people enter the task with ecological (structural)\n22 knowledge, which they update across the task and use to steer their information search behavior. It\n23 further revealed that individuals reduce multiple types of uncertainties and differ not only in the details\n24 of a common computational strategy but likely recruit entirely different strategies. Finally, it also showed\n25 evidence of strategy shifts during the task, which may be related to the local-global strategy shift\n26 described here. Not all aspects can be studied with the current paradigm, which is fine. However, I think\n27 it is important to emphasize more strongly and discuss in greater length the limitations of both the\n28 paradigm and the computational account with respect to capturing the full complexity of human\n29 information search behavior.\n30\n32 and Wulff focusses on a meta-analysis of the sampling paradigm; it demonstrates that\n33 participants’ information search strategies tackle different types of uncertainty (structural\n34 uncertainty first then estimation uncertainty, while also minimising computational uncertainty),\n35 and that such strategies are informed by prior expectations which are then updated throughout\n36 the task, as well as by perceived difficulty. We appreciate the contribution of this large-scale\n37 study to the understanding of information search outside of explore-exploit situations, and have\n38 added its insights to the following paragraph of the Discussion (p. 16):\n39\n40 “However, while our findings highlight a novel balance between local and global\n41 uncertainty minimisation, they also relate to recent work showing evidence for strategy\n42 shifts over the course of a single sampling sequence. In a three-alternative version of the\n44 heuristic in addition to more complex exploration computations27. Similarly, Spektor and\n46 experiments, participants prioritised observing all possible outcomes first, only\n1\n47 switching to targeting the most uncertain options once all unique outcomes had been\n48 observed. Our results also document how participants flexibly adjust exploration\n49 strategies within a single sampling sequence.”\n50\n52 is to study exploration in situations where the sampled information is not even prospectively or\n53 conditionally rewarding. In the sampling paradigm, numerical outcomes can be treated as\n54 prospective rewards. Our control, FIND condition, which mimics the sampling paradigm,\n55 demonstrates that they are. Forewarning participants about which colour will be associated with\n56 rewards in the future (after the end of the free sampling) already shapes their sampling\n57 decisions during the sampling phase. This is evident in the computational results (participants’\n58 sensitivity to the target-value of options is positive – Figure 7.a), but also in the behaviour\n59 (Figure 6.a). For this reason, the sampling paradigm provides a situation where information\n60 search is released from immediate reward maximisation, but not from the prospective influence\n61 of rewards. To observe exploration outside of the reward constraint, we also need for outcomes\n62 to be as neutral as possible regarding the future rewarding value of options. We used colours as\n63 outcomes because they are fully neutral in isolation, unlike numerical outcomes. In our study,\n64 the GUESS condition is the only one to fully disentangle information search from reward\n65 acquisition. To our knowledge, no other paradigm has afforded a contrast between different\n66 conditions displaying the effect of immediate, prospective or delayed rewards on information\n67 search.\n68\n70 investigated with any single paradigm, but we do think that our main conclusions are warranted\n71 in light of the different points mentioned by the reviewer.\n72\n73 1/ Regarding participants’ expectations. Individual differences in expectations within each\n74 condition were not our focus of our study, and we therefore focused our main analyses on mean\n75 group-level differences between conditions of our paradigm. We would of course be interested\n76 to study whether and how individual differences in expectations may influence exploration\n77 patterns in the GUESS condition where information search is divorced from reward seeking, but\n78 this is not the aim of the present study.\n79\n80 2/ Regarding variability in information search strategies. There is evidence in the data for\n81 between-participant variation in information search strategies (visible in the spread of\n82 behavioural measures and parameters estimates - Figures 2 & 4, 6 & 7). This is the topic of the\n83 last Results section. But the reviewer also suggests that participants could recruit completely\n84 different computational strategies to what we describe and implement in the model. The goal of\n85 our computational account was not to exhaust the full range of possible exploration strategies,\n86 and indeed, we agree with the reviewer that this is always a possibility. Still, our model is\n87 derived from behavioural observations, and it provides a good fit and a plausible explanation for\n88 behaviour in the task. To check that it captures not only the mean behavior of tested\n89 participants but also individual differences in the data, we turned to a principal component\n90 analysis (PCA). We used choice similarity matrices as a summary snapshot of behaviour in the\n91 task, and we contrasted the behaviour of original participants with simulated behaviour using\n92 the best-fitting parameters (N = 1,000 simulations). We derived the principal components (PCs)\n93 explaining the behaviour of the original participants on the one hand and the simulated\n2\n94 behaviour on the other hand. Supplementary Figure 11 (reproduced on p. 4 of this point-by-\n95 point response to comments) shows that our computational model accurately captures\n96 individual differences in behaviour, and thus provides a relevant description of participants\n97 behaviour.\n98\n100 assessed the similarity of parameter estimates between the first and the second half of the task\n101 in a new analysis (Supplementary Figure 12, reproduced below on p. 5 of this response).\n103 uncertainty was slightly increased in the second half of the experiment, but with a small effect\n104 size (paired t-test in the pooled datasets, first vs. second half t = 2.11, p = 0.04, d = 0.23). This\n81\n105 may reflect that participants improved with experience in doing the task, but is not convincing\n106 evidence for a full strategy shift at the scale of the task. We do, however, document a strategy\n107 shift at the scale of the single sequence, and this is similar to what is described in Spektor &\n109 information seeking (initial repetitive sampling) and later turned to more global search (directed\n110 towards the most uncertain option) – we return to this in response to the reviewer’s next point.\n112 the Spektor & Wulff 2024 paper (Discussion, p. 16):\n113\n114 “Similarly, Spektor and Wulff13 have shown that in the sampling paradigm of Decisions\n115 from Experience (DfE) experiments, participants prioritised observing all possible\n116 outcomes first, only switching to targeting the most uncertain options once all unique\n117 outcomes had been observed. Our results also document how participants flexibly adjust\n118 exploration strategies within a single sampling sequence.”\n119\n120\n121\n3\nThe three datasets were pooled for this analysis (N = 85). a. Variance explained in participants’ behaviour. Percentage\nof variance explained by the first five principal components (PCs) of the PCA on the participants’ behaviour (left), and on the\nsimulated behaviour (right), in the DRAW and the GUESS conditions. All three datasets were pooled for this analysis, using\nonly the DRAW and GUESS conditions. The total variance explained by the first five principal components is annotated, plain\nline represents the cumulative explained variance, and horizontal black lines on the simulation plots (right) reproduce the\nvariance explained in the participants’ PCA. b. First component coefficients and scores. Comparison of the coefficients\nand scores of the first component of the PCA done on participants’ behaviour v. on simulated behaviour. Bar plots show the\ncosine similarity of the first component coefficients in each PCA. Scatter plots show the correlation of the individual scores\n(weights) in the first component of the participants v. the simulations PCAs. Dotted lines are the identity line, solid lines are\nregression lines. Pearson’s ρ correlation coefficients and bootstrapped 95% confidence intervals. c. Median split on the\nscores of the first component in participants’ behaviour. Choice similarity matrices plotted for the participants’ behaviour,\nfor participants whose first PC was above and below the median (left). And for simulated behaviour based on the best-fitting\nparameter values of participants whose first PC scores were above or below the median (right).\n4\nfor this analysis (N = 85). a. Distribution of parameter estimates. For the DRAW (top row) and the GUESS condition\n(bottom row): distribution of participant’s parameter values estimated in the first v. the second half of the task – group means\n± 95% CI plotted in colours, and group medians ± inter-quartile ranges for the bounded parameters plotted in grey. Data\npoints outside of the first and third quartile by 1.5 inter-quartile ranges were considered outliers and excluded to avoid\ninsensitive testing between first and second halves (Tuckey’s method).\n5\n125 full complexity of information search behaviour. We have thus emphasised the limitations of our\n126 paradigm in the following Discussion paragraph (p. 17):\n127\n128 “It would be interesting for future research to test our findings in more naturalistic\n129 environments. Our study also focussed on a specific feature of sampling patterns (their\n130 sensitivity to immediate or prospective rewards). Future research should investigate how\n131 this feature interacts with dimensions of choice behaviour – for example participants’\n132 prior expectations about option statistics, the types of uncertainties displayed13, or the\n133 underlying structure of the option space38. Another constraint of the laboratory-based\n134 setting of the study is the sample size available (three datasets of about 30 participants\n135 each). This sample size was sufficient to replicate the effects we had observed in the\n136 discovery dataset, which we replicated twice, but it would be interesting for future\n137 research to replicate our findings again, in a more representative dataset.”\n138\n140 discussion of our main findings.\n141\n143 sense to map it strictly to the reliance on uncertainty minimization. Local search also results in the\n144 acquisition of information and thus can be used to reduce one or the other type of uncertainty.\n146 unsure on what basis the authors infer that participants engage in hypothesis testing during local\n147 search. I recommend that the authors clarify the relationship between local and global search and\n148 uncertainty minimization, as well as potential evidence of hypothesis testing.\n149\n151 have updated the manuscript to make sure that we are consistent in the concepts we use, and\n152 we explain our rationale below.\n153\n155 that do not maximise immediate reward (Sutton and Barto, 1998), including stochastic (e.g.,\n156 softmax, epsilon-greedy) policies. We use ‘information search’ to denote non-random\n157 exploration: this encompasses directed exploration, but also possibly initial repetitive sampling\n158 in the delayed reward conditions. We agree with the reviewer that both of these, directed\n159 exploration (global search) and initial repetitive sampling (local search) result in uncertainty\n160 minimisation. But they differ in: 1/ the type of uncertainty they target (global uncertainty in the\n161 first case, local uncertainty in the second case), and 2/ their efficiency at increasing\n162 performance in the delayed reward conditions.\n163\n165 performance relied on maximising knowledge about both options’ statistics at each choice.\n167 search (see Supplementary Figure 2.b). On the other hand, local search does minimise local\n168 uncertainty, by improving the estimates for the sampled option only – at the expense of global\n169 uncertainty. Thus, while both reduce uncertainty, global uncertainty minimisation offers higher\n170 performance improvements in the GUESS condition, and it is the only strategy expected from an\n6\n171 optimally behaving agent. Therefore, the distinction between local and global search is not so\n172 much about whether one type of search minimizes uncertainty more than the other: it is rather\n173 about the type of uncertainty (local or global) that is being minimised. It is striking that\n174 participants choose to prioritise local uncertainty minimisation at the beginning of each game,\n175 in the condition where they are incentivised to minimise global uncertainty.\n176\n178 very first sample from the option, and go on to test it on the following samples until they were\n179 sufficiently convinced – at which point they could switch options. We phrased this as a\n180 suggestion, but we agree with the reviewer that this concept may confuse readers if it is used\n181 throughout the paper. We have therefore revised the manuscript to restrict the mention of\n182 hypothesis testing to a single paragraph of the Supplementary Discussion section. Importantly,\n183 we now use only “local uncertainty minimisation” to qualify initial repetitive sampling. We\n184 believe this reflects participants trying to form a sufficiently accurate estimation of the current\n185 option before switching to the other option and engaging in global uncertainty minimisation –\n186 what they deem sufficient being reflected in their threshold value. This results in a two-step\n187 strategy, where participants prioritise different forms of uncertainty minimisation over time. This\n188 may be related to the findings described in Spektor and Wulff (2024), where participants’\n189 behaviour was initially best accounted for by a model seeking to observe all possible outcomes\n190 from the current option, before relying on “estimation-driven search” (targeting the most\n191 uncertain option, also known as “directed exploration”, leading to global uncertainty reduction).\n194 consistent in the concepts we use to interpret our findings.\n195\n197 dissipate any confusion in future readers of the paper.\n198\n200 limitations in terms of representativeness.\n201\n203 sample size of our discovery dataset was set to 30 based on existing research performed in\n204 similar (laboratory-based) conditions. While this is still an average sample size for laboratory-\n205 based tasks, it is a relatively small N. On the other hand, the sample size for subsequent\n206 samples was defined through a power calculation on the between-condition differences in\n207 parameter values estimated in the discovery dataset. With an alpha set at 0.05 and a power of\n208 90%, the power analysis called for a sample size of N < 20. Considering that replication effect\n209 sizes are often smaller, we collected about 30 participants in subsequent samples (27 in the first\n210 replication, 31 in the second replication with an additional condition) – both of which therefore\n211 exceeded the theoretically needed sample size to replicate our finding with 90% power.\n212\n214 an important variable often not considered when thinking about sample sizes. Participants\n215 played 96, 128 and 96 sequences in the different datasets (equivalent to what Spektor & Wulff\n216 called “choice problems”), which comprised a total of 1,344 to 1,792 sampling decisions per\n217 participant – this is higher than is typically the case in delayed rewards paradigms (e.g., 404\n7\n218 sampling decisions on average in Spektor & Wulff, 2024). Recent methodological research in\n219 psychology has documented a trade-off between the number of participants and the number of\n220 trials per participant: a higher ratio of trials by participant provides better intra-individual\n221 estimation and lowering the estimates of the sample’s variance (Baker et al., 2021, Smith &\n223 results in this manuscript. Therefore, while we took a somewhat smaller-N approach, we are\n224 confident that the findings we describe are genuine. We have already replicated them in\n225 additional samples for subsequent studies.\n226\n228 the reviewer’s comment on sample size and representativeness in the following section of the\n230\n231 “Another constraint of the laboratory-based setting of the study is the sample size\n232 available (three datasets of about 30 participants each). This sample size was sufficient\n233 to replicate the effects we had observed in the discovery dataset, which we replicated\n234 twice, but it would be interesting for future research to replicate our findings again, in a\n235 more representative dataset.”\n236\n238 authors place too much trust in their computational model. It is unclear whether it represents a\n239 sufficiently accurate and unbiased account of the empirical data to draw such strong inferences. Even if\n240 recovery is demonstrated using simulations, it is always wise not to place too much trust in any\n241 computational model as the data-generating process could be entirely different. In fact, given the\n242 research highlighted in my review, this is to be expected. To address this, I recommend that the authors\n243 present a separate descriptive analysis of switching behavior and individual differences to investigate\n244 the hypothesis expressed in the first comment of reviewer two and demonstrate the existence of\n245 individual differences questioned in the second comment of reviewer two.\n246\n248 hope our response below will dissipate any lingering concerns.\n249\n251 with the reviewer that all models are approximations. Our approach does not exhaustively cover\n252 the full range of possible exploration strategies (i.e., we do not claim that this is the only and\n253 definitive account), nor does it provide the full detail of how computations enfold in participants’\n254 brains (i.e. we are not describing the neural data-generating process). But this was not our goal.\n256 to derive cognitively-relevant variables, as a way to qualify and quantify behaviour more\n257 accurately. The model implements some strategies well established in the literature (reward\n258 seeking, random and directed exploration, and repetition bias – e.g. Sutton & Barto 1998, Wilson\n259 et al., 2014 & 2021, Wu et al. 2018, Dubois et al., 2021), with the addition of a specific early\n260 sampling feature, added as a plausible approximation of the cognitive process underlying early\n261 repetitive sampling. This behaviour had not been documented in related tasks before, so our\n262 study was rooted in behavioural observation, and a discovery-replication approach.\n263\n8\n265 of evidence to accumulate before a switch rather than a mere count). This model fulfills its goal.\n267 when inspecting simulations of the best-fitting model and knock-out simulations\n268 (Supplementary Figures 3, 4 and 7), it captures both average behaviour and inter-individual\n269 differences (Figure 8, Supplementary Figure 11), and it is also well recoverable (Supplementary\n271 words, we hope to convince the reviewer that while we do not trust our model to account for\n272 each and every aspect of sampling, the approach is methodologically sound, and it is well\n273 suited for its purpose. The model allows us to qualify and quantify participants’ behaviour, and\n274 it formulates a cognitively motivated account of the underlying mechanisms.\n275\n277 previous reviewer, and their recommendation for additional descriptive analyses.\n278\n280 difference between the DRAW (immediate rewards) and GUESS (delayed rewards no target)\n281 conditions, based on the different goals of the conditions. We quote Reviewer #2 again here, as\n282 a basis for our response:\n283\n285 conditions (i.e. which option has more of the target colour), but in the GUESS condition, the\n286 focus needs to be on identifying the nature of each option, which can be done in isolation and\n287 does not require contrasting the options. [In the GUESS condition] “streaky” exploration may be\n288 subject to less memory demands than more evenly distributed exploration. A heuristic for the\n289 other conditions [DRAW and FIND] is to pitch the options against each other, for instance\n290 sampling one and then the other and just determining which one had the most of the target\n291 colour in each consecutive pairwise comparison”\n292\n294 analyses, but Reviewer #1 and #2 pointed out that presenting several indices demonstrating the\n295 same behaviour was redundant and obscured the message of the paper. The current version of\n296 the manuscript still features the important behavioural indices on Figure 2, Figure 6 and Figure\n297 8a-d. Thus, to address Reviewer #3’s recommendations for additional behavioural analyses of\n298 switching behaviour without crowding the manuscript, we provide additional behavioural\n299 analyses for the pooled dataset as Supplementary Information (Supplementary Figure 9), but we\n300 refrain from discussing them at length in the main text.\n301\n303 a sequence. They argue that the behavioural difference is not due to the reward and the target\n304 colour manipulations as much as it is due to “the different goals between the conditions”. Their\n305 proposal is that participants produce repetitive streaks in the GUESS condition because they\n306 learn the option colours in isolation, whereas in the DRAW condition they alternate because\n307 they want to compare option values. Even in the early choices, in all conditions, participants\n308 repeated their choices more than they switched (Figures 2.c, Supp.1.c, 6.b, Supp.10). So even in\n309 the DRAW condition, they were not doing alternate sampling and pitting options against each\n310 other, as Reviewer #2 suggests.\n311\n313 (Supplementary Figure 10.a) – this is the main effect we report. If this reflected a systematic\n314 comparison strategy to learn option values in the early choices of the DRAW condition, we\n315 would expect (a) that participants switched regardless of the outcome they observed, and (b)\n316 that switching behaviour would change after the initial choices, once participants have gathered\n317 information about option values. To assess this, we plotted the switch rate at choice t, as a\n318 function of the rewarding value of the outcome observed at choice t-1 (binned into 8 levels –\n320 (always < 0.5), and depended critically on how rewarding the preceding outcome was. We also\n321 observed no evidence for a shift in switching behaviour between early and later choices. In\n322 other words, in the DRAW condition we found no evidence that participants alternated in order\n323 to initially compare options – rather, their decisions to switch appeared to depend primarily on\n324 the rewards obtained (as predicted by our computational model - Supplementary Figure 10.b),\n325 and this was sustained throughout the sequence.\n326\n328 were rewarded for blue and the first option they sampled was blue, they did not need to\n329 investigate the colour of the second option. And indeed, when participants played a DRAW\n330 game in which both options were associated with the target colour, they switched less between\n331 options from early in the game (Supplementary Figure 10.c). On the other hand, there was a\n332 strong incentive to explore both options in the GUESS condition: participants needed to obtain\n333 as much information as possible about both options as soon as possible, before the end of the\n334 game. Yet it is only in the GUESS condition that participants chose to learn in streaks of\n335 repeated decisions. While we agree with Reviewer #2 that repetitive sampling is likely\n336 cognitively easier, it is striking that participants only did it when they were incentivised to\n337 minimise global uncertainty about the task structure. We discuss the tension between cognitive\n338 costs and information gathering on p. 16:\n339\n340 “Thus, it is striking that participants chose to prioritise local uncertainty minimisation\n341 precisely in conditions where they were incentivised to minimise global uncertainty. This\n342 likely reflects a trade-off between information search and cognitive costs: while directed\n343 sampling of uncertain option was most efficient in reducing global uncertainty, it was\n344 also cognitively costly to update beliefs about several options in parallel – e.g. ‘switch\n345 costs’, which have been associated with mental effort24,25 and even subjective pain26.\n347 likely alleviated these costs. This strategy, however, was only used by tested participants\n348 when rewards were delayed.”\n349\n352\n353 “Another alternative explanation for the difference in sampling patterns between\n354 conditions is to propose that participants relied on different heuristics because the\n355 instructions gave them different goals. One could propose that participants repeated\n356 choices in the delayed rewards (GUESS) condition because they were free to study each\n357 option in isolation, whereas in the DRAW and FIND conditions they were pushed to\n10\n358 compare options by sampling one after the other, juxtaposing pairs of outcomes. We did\n359 not find any evidence that participants tended to alternate between options in the DRAW\n360 and FIND sequences (Supplementary Figure 10). Rather, repetition biases were positive\n361 (meaning that participants did not tend to alternate between options in any condition),\n362 comparable and correlated across all three conditions (Figure 7.a. and Supplementary\n364 conditions (Figure 7.a.). Given such consistency in parameter estimates across\n365 conditions, it is very unlikely that participants resorted to wildly different heuristics in the\n366 different conditions. In addition, in the DRAW condition, even in the initial choices,\n367 participants’ decisions to switch option appeared to depend mainly on the reward value\n368 of the preceding outcome (Supplementary Figure 10), rather than demonstrating\n369 systematic alternation.”\n370\n372 hypothesis from Reviewer #2 cannot not explain the behavioural effects we find and report.\n11\n374\n375\n377 patterns across individuals\" section. Reviewer #2 was concerned that the inter-individual\n378 differences in parameter values we report were based on a biased estimation of those\n379 parameters, and reflected differences in model fitting rather than genuine differences in\n380 behaviour.\n381\n383 model best fit their behaviour in the GUESS condition. Although participants were split based\n384 on which model (INI+ or INI-) best fit their behaviour, we compared parameters using the most\n385 parameterized version of the model (INI+). This approach allowed for a fair comparison between\n386 parameters, and for participants whose best-fit was the INI- model, threshold parameters in the\n388 accounting for the additional parameter. Based on this split, we reported group differences both\n389 in behavioural indices (e.g. higher performance in both conditions for the INI+ group) and in\n390 parameter estimates (higher estimates for the later phase parameters in the INI+ group:\n391 sensitivity to the target, to uncertainty, and repetition bias).\n392\n394 groups due to not being optimised on the same choices (i.e. estimated on the full sequence in\n395 the INI- participants vs. only after the initial phase was over in the INI+ participants). In other\n396 words, Reviewer #2 suspected that the inter-individual differences in parameter estimates were\n397 a trivial by-product of the model fitting, rather than genuine differences in sampling strategies\n398 between participants.\n399\n401 through recovery that later phase parameters were equally well estimated regardless of the\n402 presence or absence of an initial sampling phase (Supplementary Figure 6). It was not the case\n403 that later phase parameters were mechanically biased by the presence or absence of an initial\n404 phase. Differences in parameter estimates between the INI+ and INI- groups do not trivially\n405 surface from the model fitting pipeline, but rather, the difference in parameters suggests\n406 underlying differences in sampling strategies. More specifically, that seemingly contradictory\n407 sampling strategies co-occur in participants (i.e. suboptimal initial repetitive sampling and\n408 optimal directed exploration), and that statistically suboptimal strategies can be associated with\n409 higher performance in participants (higher evidence for the initial sampling phase in the delayed\n410 rewards condition was associated to higher performance in both conditions). We hope that this\n411 clarification will convince Reviewer #3 that the model-based analyses are actually well suited for\n412 measuring the inter-individual differences questioned by Reviewer #2. We would like to reiterate\n413 that we acknowledge that, like all models, our model is ultimately an approximation of cognitive\n414 strategies. Yet the strict methodology we applied makes it valuable in revealing genuine\n415 differences in behaviour.\n416\n418 them of the existence of individual differences between INI+ and INI- participants. Because we\n419 split participants based on which of the INI+ or INI- model best fit their behaviour, some\n420 analyses of repetition rate are circular. For example, the analysis of repetition rate by choice\n12\n421 position, repetition rate by familiarity with the option, or choice imbalance are trivially expected\n422 to differ between groups (Figure 8.a and Supplementary Figure 9.b). We mention this in the\n423 corresponding subsection of the Results on p. 13:\n424\n425 “As expected from this splitting criterion [emphasis added], in the GUESS condition,\n426 participants in the first group (INI+) showed a higher tendency to initially resample\n427 options (Figure 8.a., b., d.).”\n428\n430 performance (Figure 8.b-c.). Both of these are behavioural indices, and they are not trivially\n431 expected to differ from our splitting criterion. It is particularly striking that participants who\n432 displayed stronger evidence for suboptimal initial repetitive sampling in the GUESS condition\n433 (the INI+ group) also achieved higher performance in both tasks – a purely model-free measure\n434 (Figure 8.c.). Additionally, the INI+ participants displayed more condition-relevant choices in\n435 both conditions (Figure 8.b.). Such co-occurrence is surprising because initial repetitive\n436 sampling is suboptimal yet associated with task-optimal choices, and with higher performance.\n438 sampling, or whether it is merely dependent on otherwise higher task-optimal behaviour in the\n439 same participants (higher reward-drive and higher sensitivity to uncertainty) – but this was\n440 beyond the scope of this initial study.\n441\n443 our focus on performance and choice direction will convince Reviewer #3 of the existence of the\n444 individual differences questioned in the second comment of Reviewer #2.\nthree datasets were pooled and participants were split in two groups based on whether a model with (ini+, N = 67) or without\n(ini-, N = 18) an initial sampling phase better accounted for their exploration patterns in GUESS sequences. a. Overall\nfraction of repeat decisions. Fraction of repeat decisions in each condition, averaged per participant for all the participants\n(left), and for the ini+ and ini- participants separately (right) – median ± inter-quartile range. b. Choice imbalance. Difference\nbetween the number of choices for the most and the least chosen option at each choice throughout the games. Solid lines\ndisplay the average for participants whose best-fitting model in the GUESS condition included an initial sampling phase (ini+,\nN = 67), dotted lines show the average of participants whose best-fitting model did not include an initial sampling phase (ini-\n, N = 18) – group means ± 95% CI.\n13\n446\n447\n449 the authors model whether a sampling decision is repeated as a function of the decision variable DV,\n450 independent of which option was sampled last. Consequently, I would expect that the probability of a\n451 repeat should be based on the absolute differences in the amount and direction of evidence for the two\n452 options rather than their signed differences (i.e., \\hat{x_{1,t}} - \\hat{x_{2,t}} and \\hat{\\lvert x_{1,t} \\rvert} -\n453 \\hat{\\lvert x_{2,t} \\rvert}), but the definition of DV suggests the opposite. The result is that only\n454 differences in one direction (e.g., \\hat{x_{1,t}} > \\hat{x_{2,t}} but not \\hat{x_{1,t}} < \\hat{x_{2,t}}) can\n455 trigger a switch to the other option. I don’t understand how this makes sense.\n456\n458 much like to dissipate in the final version of the manuscript. Outside of the initial sampling\n459 phase, the decision variable (DV) used by the model to choose between option 1 and option 2 –\n460 using a standard softmax policy – consists of three terms: 1/ the difference in the log-odds that\n461 each option is associated with the target colour, controlled by a target sensitivity parameter\n462 𝜷 , 2/ the difference in the absolute log-odds that each option is associated with a colour (i.e.,\n𝒕𝒂𝒓\n463 the certainty of the association between each option and a colour), controlled by an uncertainty\n464 sensitivity parameter 𝜷 , and 3/ a bias toward sampling the same option as in the previous\n𝒕𝒂𝒓\n465 trial, controlled by a repetition bias parameter 𝒃 .\n𝒓𝒆𝒑\n466\n468 second term reflects directed exploration (as in Wilson et al., 2014), where the model will\n469 preferentially sample (when 𝜷 > 0) the option whose value is most uncertain (as in e.g.\n𝒖𝒏𝒄\n471 which option was sampled on the previous trial, with the inclusion of the third term – the\n472 repetition bias. But to be perfectly explicit, our model computes the probability of choosing\n473 option 1 over option 2, not the probability of re-sampling the previous option. See below the\n474 pseudo-code adapted from https://gitlab.com/cle-a/COLPUB:\n475\n476\n477\n478 % accumulate information about each option\n479 x(t,1) = x(t-1,1) + previous_outcome; % update option 1 (chosen)\n480 x(t,2) = x(t-1,2); % do not update option 2 (not chosen)\n481\n482 % at each sample, compute the decision variable DV\n483 dv = beta_tar * (x(t,1)-x(t,2)) * target_color; % first term\n484 dv = dv + beta_unc * (abs(x(t,2))–abs(x(t,1))); % second term\n485 dv = dv + beta_rep * previous_choice; % third term\n486\n487 % compute the choice probability of each option\n488 choice_prob(1) = 1/(1+exp(-dv)); % p(choose option 1)\n489 choice_prob(2) = 1-choice_prob(1); % p(choose option 2)\n490\n491\n14\n𝒕𝒂𝒓\n493 option (it is only defined when the target is given in the beginning of the sequence - in DRAW).\n𝒖𝒏𝒄\n495 accumulated evidence). Finally, a higher 𝒃 pushes choices towards re-sampling the same\n𝒓𝒆𝒑\n496 option as in the previous trial. Note that in the online code, 𝜷 was positive when the model\n𝒖𝒏𝒄\n497 was attracted toward the most certain option (which is the opposite of directed exploration),\n498 such that negative values were observed in human participants. We have above (before the\n499 original submission, but after putting the first version of the model code online) flipped 𝜷 to\n𝒖𝒏𝒄\n500 be positive when the model is attracted toward the most uncertain option. We hope this clarifies\n501 any lingering ambiguity about the implementation of the model.\n502\n503\n505 responses to Reviewer #2. We are pleased to read that they are inclined to recommend\n506 publication, and we hope that our detailed responses to each of their different suggestions have\n507 addressed all remaining concerns. Please find references and copies of the figures that are\n508 useful when reading this response in the remaining pages below.\n- Baker, D. H., Vilidaite, G., Lygo, F. A., Smith, A. K., Flack, T. R., Gouws, A. D., & Andrews, T. J.\nPower contours: Optimising sample size and precision in experimental psychology and human\nneuroscience. Psychological methods, 26(3), 295 (2021).\n- Dubois, M., & Hauser, T. U. Value-free random exploration is linked to impulsivity. Nature\nCommunications, 13(1), 4542 (2022).\n- Smith, P. L., & Little, D. R. Small is beautiful: In defense of the small-N design. Psychonomic\nbulletin & review, 25, 2083-2101 (2018).\n- Spektor, M. S., & Wulff, D. U. Predecisional information search adaptively reduces three types\nof uncertainty. Proceedings of the National Academy of Sciences, 121(47), e2311714121\n(2024).\n- Sutton, R. S., & Barto, A. G. Reinforcement learning: An introduction (Vol. 1, No. 1, pp. 9-11).\nCambridge: MIT press (1998).\n- Wilson, R. C., Geana, A., White, J. M., Ludvig, E. A., & Cohen, J. D. Humans use directed and\nrandom exploration to solve the explore–exploit dilemma. Journal of experimental psychology:\nGeneral, 143(6), 2074 (2014).\n- Wilson, R. C., Bonawitz, E., Costa, V. D., & Ebitz, R. B. Balancing exploration and exploitation\nwith information and randomization. Current opinion in behavioral sciences, 38, 49-56 (2021).\n- Wu, C. M., Schulz, E., Speekenbrink, M., Nelson, J. D., & Meder, B. Generalization guides\nhuman exploration in vast decision spaces. Nature human behaviour, 2(12), 915-924 (2018).\nthree datasets were pooled and participants were split in two groups based on whether a model with (ini+, N = 67) or without\n(ini-, N = 18) an initial sampling phase better accounted for their exploration patterns in GUESS sequences. a. Overall\nfraction of repeat decisions. Fraction of repeat decisions in each condition, averaged per participant for all the participants\n(left), and for the ini+ and ini- participants separately (right) – median ± inter-quartile range. b. Choice imbalance. Difference\nbetween the number of choices for the most and the least chosen option at each choice throughout the games. Solid lines\ndisplay the average for participants whose best-fitting model in the GUESS condition included an initial sampling phase (ini+,\nN = 67), dotted lines show the average of participants whose best-fitting model did not include an initial sampling phase (ini-\n, N = 18) – group means ± 95% CI.\nAlternation behaviour. Average switch rate at each choice position in the sequence (group mean ± 95% within participant\nCI). b. Alternation rate by previous outcome. In the DRAW condition, alternation rate in participants’ decisions as a\nfunction of the rewarding value of the preceding outcome (binned into 8 bins), in the first four, following four, and remaining\ntwelve choices . Group means (dots) and 95% CI (confidence bars). c. Choice to choice similarity. Average similarity of\neach choice to the other choices in the same sequence. In the DRAW condition, for sequences where a both, a single, or\nnone of the options were associated to the target colour. In the GUESS condition, for sequences where the options were\nassociated to the same, or different colours.\nThe three datasets were pooled for this analysis (N = 85). a. Variance explained in participants’ behaviour. Percentage\nof variance explained by the first five principal components (PCs) of the PCA on the participants’ behaviour (left), and on the\nsimulated behaviour (right), in the DRAW and the GUESS conditions. All three datasets were pooled for this analysis, using\nonly the DRAW and GUESS conditions. The total variance explained by the first five principal components is annotated, plain\nline represents the cumulative explained variance, and horizontal black lines on the simulation plots (right) reproduce the\nvariance explained in the participants’ PCA. b. First component coefficients and scores. Comparison of the coefficients\nand scores of the first component of the PCA done on participants’ behaviour v. on simulated behaviour. Bar plots show the\ncosine similarity of the first component coefficients in each PCA. Scatter plots show the correlation of the individual scores\n(weights) in the first component of the participants v. the simulations PCAs. Dotted lines are the identity line, solid lines are\nregression lines. Pearson’s ρ correlation coefficients and bootstrapped 95% confidence intervals. c. Median split on the\nscores of the first component in participants’ behaviour. Choice similarity matrices plotted for the participants’ behaviour,\nfor participants whose first PC was above and below the median (left). And for simulated behaviour based on the best-fitting\nparameter values of participants whose first PC scores were above or below the median (right).\nfor this analysis (N = 85). a. Distribution of parameter estimates. For the DRAW (top row) and the GUESS condition\n(bottom row): distribution of participant’s parameter values estimated in the first v. the second half of the task – group means\n± 95% CI plotted in colours, and group medians ± inter-quartile ranges for the bounded parameters plotted in grey. Data\npoints outside of the first and third quartile by 1.5 inter-quartile ranges were considered outliers and excluded to avoid\ninsensitive testing between first and second halves (Tuckey’s method).\nReviewer #4\nHaving reviewed R3’s comments and the author’s responses to them, I believe that the authors have\ndone a great job addressing all the comments thoroughly and convincingly, substantially improving the\nconceptual clarity of the paper and adding helpful model-independent analyses.\nWe thank the reviewer for taking the time to evaluate our response to Reviewer 3, and we are\npleased to read that they are satisfied with our revisions.\n1"}, {"round": 4, "reviewer_comments": {"reviewer_4": "Having reviewed R3's comments and the authors' responses to them, I believe that the authors have done a great job addressing all the comments thoroughly and convincingly, substantially improving the conceptual clarity of the paper and adding helpful model-independent analyses."}, "author_response": ""}]

Decision letter