Developmental Trajectories
People are not born with fully-developed cognitive abilities but instead develop them during their lifetime. In this section, we tested whether increasing the description length of RR-RL 2 matches the behavioral trajectories of people as they grow up. To test this hypothesis, we reanalyzed data collected by Somerville et al. ( 2017 ) , who studied changes in exploration behavior between early adolescence and adulthood.

Experimental Design: In their study, Somerville et al. ( 2017 ) made use of an experimental paradigm known as the horizon task (Wilson et al., 2014 ) . Each task was based on a two-armed bandit problem and involved four forced-choice trials, followed by either one or six free-choice trials. Participants were aware of the number of remaining choices and could use this information to guide their behavior. The mean reward of one of the arms was drawn randomly from \{40,60\} , while the mean reward for the other was determined by sampling the difference to the first arm from \{4,8,12,20,30\} . The arrangement of arms as well as the sign of their difference was randomized. In each time-step, the observed reward was sampled from a normal distribution with the corresponding mean value and a standard deviation of 8 . The addition of forced-choice trials allowed to control the amount of information that was available to participants. They either provided an equal amount of information for both arms (i.e., two observations each) or an unequal amount of information (i.e., a single observation from one arm, three from the other). In total, Somerville et al. ( 2017 ) collected data for 147 participants between the ages of 12.08 and 28 , completing 160 bandit tasks each.

Analysis: Following Somerville et al. ( 2017 ) , we used the decision in the first free-choice trial to distinguish between different types of exploration. In the unequal information condition, a choice was classified as directed exploration if it corresponded to the option that was observed fewer times during the forced-choice trials. In the equal information condition, a choice was classified as random exploration if it corresponded to the option with the lower estimated mean. We refer to an exploration behavior as strategic if it occurs more frequently in the long horizon tasks compared to the short horizon tasks. Somerville et al. ( 2017 ) found – as shown in Figure 4 (a) – that strategic directed exploration emerges during adolescence, whereas strategic random exploration is age-invariant.

To quantify these effects, they fitted two independent linear regression models, using the probability of engaging in directed and random exploration as dependent variables. Both models used age, the corresponding horizon, and the interaction between the two as regressors. They found a significant effect of horizon in both conditions, indicating that participants engaged more in both directed and random exploration in tasks with a longer horizon. Furthermore, they found a significant interaction effect between horizon and age for directed exploration but not for random exploration, confirming that strategic directed exploration increases during cognitive development, while strategic random exploration remains constant over time.

Results: We trained RR-RL 2 with a targeted description length of \{100,200,\ldots,10000\} nats on the same distribution used in the original experimental study. Figure 4 (b) visualizes how strategic directed and random exploration change as the description length of RR-RL 2 increases. Matching the main result of the experimental study, we find that strategic directed exploration increases with description length, while strategic random exploration remains unaffected. We repeated the previously described regression analysis on data simulated by RR-RL 2 to quantify this conclusion (replacing age as a regressor with description length). The outcome of this analysis mirrored the results of the original study. We found a significant effect of horizon on both directed ( F_{1,194}=56.50,p<0.001,\eta^{2}=0.20 ) and random exploration ( F_{1,194}=6.80,p=0.01,\eta^{2}=0.03 ). This means that RR-RL 2 made more exploratory decisions of both types if it was beneficial to do so. We also found a significant interaction effect between horizon and description length on directed exploration ( F_{1,194}=17.48,p<0.001,\eta^{2}=0.06 ) but not on random exploration ( F_{1,194}=0.02,p=0.89 ). These results confirm that description length and age have comparable qualitative effects on the development of strategic exploration.

However, when comparing the effect sizes of our analysis to those from the experimental study, we find that the interaction effect between description length and horizon on directed exploration only amounts to around half of the effect between age and horizon. We speculated that part of this difference comes from a mismatch between the distribution used to train our models and what kind of tasks people expect in the experiment. People, for instance, might assume that task rewards are noisier than they are, which would require more exploratory choices, and, in turn, lead to stronger effects. We tested this hypothesis by retraining RR-RL 2 on the same distribution but with the standard deviation of the reward noise increased by 50\% . While this modification increased the effect size of the interaction effect on directed exploration, it did not close the gap entirely, suggesting that there are additional – currently undiscovered – factors that contribute to the development of strategic directed exploration during adolescence. Furthermore, we found that, while strategic random exploration was unaffected by both age and description length, humans had a higher base rate of engaging in random exploration than our models. We, however, believe that this issue could be resolved by adding an \varepsilon -greedy error model to our models as doing so would naturally increase the base rate of random exploration.