Latent Learning Progress Drives Autonomous Goal Selection in Human Reinforcement Learning

Authors: gaia  molinaro
pierre-yves  oudeyer
anne g e collins
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 21:44:30

Abstract: Humans are autotelic agents who learn by setting and pursuing their own goals. However, the precise mechanisms guiding human goal selection remain unclear. Learning progress, typically measured as the observed change in performance, can provide a valuable signal for goal selection in both humans and artificial agents. We hypothesize that human choices of goals may also be driven by _latent learning progress_, which humans can estimate through knowledge of their actions and the environment – even without experiencing immediate changes in performance. To test this hypothesis, we designed a hierarchical reinforcement learning task in which human participants (N = 175) repeatedly chose their own goals and learned goal-conditioned policies. Our behavioral and computational modeling results confirm the influence of latent learning progress on goal selection and uncover inter-individual differences, partially mediated by recognition of the task's hierarchical structure. By investigating the role of latent learning progress in human goal selection, we pave the way for more effective and personalized learning experiences as well as the advancement of more human-like autotelic machines.

Main paper: 
 introduction: 
 Animals often spontaneously explore their environment in the absence of immediate extrinsic payoffs [1] [2] [3] [4] [5] [6] [7] [8] . While seemingly wasteful, such intrinsically motivated behavior is argued to adaptively guide learning and exploration, paying off across developmental and evolutionary timescales [9] [10] [11] . As a result, researchers and engineers started endowing artificial systems with intrinsic reward signals that aim to reproduce aspects of human curiosity-driven learning [11] [12] [13] [14] [15] [16] [17] . A striking aspect of human intrinsic motivation is autotelic exploration, wherein people self-generate goals, and reward themselves for achieving them [18] [19] [20] . How people decide which goals to pursue is thus an outstanding issue in the cognitive sciences [21] . Identifying the key variables affecting goal selection may in turn enable the development of artificial agents that intelligently set their own objectives, rather than being optimized for predefined ones [22] .Alongside factors such as performance and novelty, learning progress (LP) -the derivative of performance with respect to time -has proven a useful signal for goal selection, steering both humans [23, 24] and artificial agents [25] [26] [27] [28] away from tasks that are either too simple or too hard. Most existing approaches approximate LP using past observations (e.g., differences in recent and earlier performance [23] ), such that LP is greatest when performance changes rapidly ( [29] , but see [30] ). However, there are often situations where no external change is visible, yet some other form of progress toward the desired outcome is made. Imagine being tasked with identifying the correct sequence of numbers to open a combination lock, which you might attempt through trial and error. Throughout most of this scenario, repeated failures would yield no difference in performance, hence no empirical learning progress (as typically defined). Nonetheless, provided that the lock has a limited number of slots and numbers and that you can avoid repeating incorrect combinations, every attempt is a step toward the solution. Based on this observation, we propose extending the definition of LP to include a latent variant, distinct from its manifest counterpart in that it relies on an internal model of the environment and one's behavior in addition to performance. Following this example, we formulate latent learning progress (LLP) as a measure of progress an agent infers by leveraging knowledge about the environment and its interactions with it, e.g., through reasoning. For example, LLP could be tracking the proportion of hypotheses tested over the entire solution space, thus changing with every new set of actions taken and dropping upon identifying the correct one. Unlike LP, LLP does not require observing performance improvements to provide informative signals about progress. At least in certain settings, LLP could be a more valuable signal for goal selection than the classically defined LP, as it proactively estimates future progress without waiting for performance improvements.Here, we hypothesize that LLP is a signal humans use for goal selection, and that it can be considered separately from LP. Specifically, we test the role of LLP in human goal selection by introducing a novel learning paradigm with goal choices as the key dependent variable. Several features of this environment (detailed in Section 3.1) make it possible to distinguish between LP and LLP and test the latter's contributions to autotelic exploration in a sample of human participants. Among these, a hierarchical component is introduced in the learning environment to further target LLP, which might be sensitive to hierarchy-based inferences leading to sharp changes in the agents' knowledge of the environment. These features make our environment suited for probing LLP and testing its contributions to autotelic exploration in a sample of human participants. By combining behavioral findings and computational modeling of human decisions in the experiment, we find support for our hypothesis that LLP captures human goal choices better than standard LP in this hierarchical reinforcement learning task. We also find inter-individual differences in goal selection strategies, warranting further studies on the factors underlying such idiosyncrasies. In our discussion of the results, we argue that adopting the notion of LLP may foster the development of autotelic machines. 
related work: 
 Intrinsic motivation The various forms of intrinsic motivation can broadly be classified as either knowledge-based (KB) or competence-based (CB) [12, 28, 29, 31, 32] . KB intrinsic motivations derive from comparisons between pre-existing and newly acquired information and are well-documented in the animal world (e.g., [1, 3, 5, 6, 8, [33] [34] [35] [36] [37] ). In artificial agents, KB signals of novelty, diversity, or prediction error broaden exploration when provided alongside extrinsic rewards [13, 15, 16, 38, 39] or even on their own ( [11, 14] , see [40] for review). However, human curiosity is often driven towards stimuli of intermediate complexity [7, 41, 42] , rather than extremes. Accordingly, some artificial models aim to maximize an "intermediate" difference between predictions and observations [30, 43] . Intrinsic motivations based on learning progress (LP) can avoid arbitrary cutoffs for what signifies intermediate complexity. LP is particularly useful as a CB motivation, prompting agents to improve their performance on self-determined goals [29, 32, 44] . Indeed, humans often find pleasure in activities whose difficulty matches their abilities [45, 46] . CB intrinsic motivations highlight the autotelic property of human learning, which is often driven by self-defined objectives. Two concurrent challenges in cognitive science and artificial intelligence involve identifying the signals people use to develop their learning curricula [19, 21] and adapting those principles to machines [22] .Goal selection in humans Goals, defined as "a cognitive representation of a future object that the organism is committed to approach" [47] , have long been a key concept in psychology (e.g., [48] [49] [50] [51] ), with a rich literature on how various types of goals interact with performance and motivation [52, 53] . In cognitive neuroscience, goals are typically imparted by the experimenter and are not the main focus of the analyses [54] -though recent work illustrated the impact of goals on core aspects of neural, cognitive, and behavioral responses ([18, 20, 21, 55-59] ; see [19] for review). Nevertheless, among the principles thought to guide human goal selection is the tendency to choose activities that balance the expected outcome's desirability and the probability of attaining it [60] . When free to allocate their time to various tasks or choosing activities for pure enjoyment, people naturally tend to match the activity's level of challenge to their current abilities [45, 46, 61] , which might be optimal for learning [62, 63] . However, most studies fail to provide precise computational details on the mechanisms of goal selection. Although not explicitly addressing goals, two recent experiments attempted to fill this gap. In particular, [23] tasked participants with repeatedly choosing which of four learning activities (each with a different difficulty level) to engage with, and found that performance and LP jointly contribute to activity selection. Similarly, [24] found that people use novelty and LP when exploring various activities based on their own curiosity. Here, we propose that in deciding which goals to set, humans may rely not only on manifest signals of performance but also on an internally tracked, latent form of learning progress.Goal selection in artificial agents Enabling agents to learn autonomously has been a longstanding challenge in artificial intelligence and a central tenet of many successes in the field [43, [64] [65] [66] [67] [68] . However, most existing models are optimized for a predetermined objective. Overcoming this limitation, autotelic agents are designed to autonomously select goals and learn to achieve them [22, 25, 69] , begging the question of how to optimally set goals for efficient learning. Even random goal sampling [70] [71] [72] or entropy-maximization objectives [73, 74] present some advantages [75] , but agents can be programmed to adjust their goal selection to currently owned skills and further benefit from autotelic properties. One such approach involves a two-part adversarial system, wherein a "teacher" proposes challenging tasks and a "student" learns to maximize performance over them [44, [76] [77] [78] . Algorithmic implementations of LP also help agents avoid wasting time and resources on objectives that are too far-reaching or even impossible (e.g., due to inherent noise in the system; [26] [27] [28] [79] [80] [81] [82] [83] [84] [85] ). These approaches calculate LP relative to performance, not considering changes in the agent's goal-conditioned competence that might occur latently. Thus, to our knowledge, LLP-mediated goal selection in artificial agents is an unexplored avenue.3 Experiment design and modeling approach 
experimental setup: 
 Most reinforcement learning problems consider a single task to master. Similarly, experimental psychologists usually instruct participants to pursue a specific objective. By contrast, we developed an environment in which goal selection was the main dependent variable, enabling us to study key determinants of goal setting in humans. The experiment was implemented in the Unity game engine and presented as an interactive online game, in which healthy human participants (N = 175; see Appendix A for details) played the role of alchemists and could learn to make different potions from sets of ingredients. Below we refer to potions as "goals" and ingredients as "actions". The paradigm included training, learning, and testing phases. Learning phase trials (144) were composed of a goal selection stage, an action selection stage, and a feedback stage (Figure 1 , top; Appendix B). In the goal selection stage, participants viewed colored potions available for selection, with icons indicating the number of ingredients required and the four different ingredients available to make them. In the action selection stage, participants chose which ingredients to add to the cauldron. In the feedback stage, participants received binary feedback as a result of their actions (the empty flask in the foreground either filled up or remained empty based on whether participants chose the correct ingredients in order). In the 12 training trials preceding the learning phase, goal selection was forced to ensure participants experienced each goal at least twice. In the final testing trials, participants were prompted to try and make each potion 4 times without feedback.Potions could either be made of basic ingredients ("simple" and "complex" goals G1-G4) or potions of different colors ("compound" goals G5-G6) and required either two (simple and compound goals G1-G2, G5-G6) or four actions to complete (complex goals G3-G4; Figure 1 , bottom). Two potions could be made from either basic ingredients or other potions (potions for G3-G6). One complex goal ("hierarchical", G3) was made by combining the ingredients of the two simple potions (G1, G2), whereas the other complex goal was not ("non-hierarchical", G4). One complex goal ("hierarchical", G5) was made by flasks of the two simple potions (G1, G2), whereas the other complex goal was not ("non-hierarchical", G6). Thus, G3 and G4, and G5 and G6, were matched in terms of actions required to create the corresponding potions but differed in whether knowledge from other potions could be used to find the corresponding solutions faster. Two additional, simple goals made of the first two and last two ingredients of G4 were presented to participants in the testing phase (G7 and G8; Appendix B). Unless otherwise specified, we focus on goals G1-G6, which participants engaged with in the training and learning phases. Further details about the goal space are provided in Appendix B. The information displayed in Figure 1 was never shown to participants. The correspondence between goal identity, color, position on the screen, and solution was pseudo-randomized across participants.Various features make the experimental design well-suited to study the contributions of LLP to goal selection by helping decorrelate LP from LLP. First, the correct response was static and signaled through deterministic feedback. While the standard LP measure only flattens upon repeated subsequent attempts, deterministic feedback enables LLP to reach its limit after a single correct response. Second, the action space is limited and solutions are unique, so negative feedback can be used to exclude the attempted action sequence from the set of possible solutions. This feature was especially meaningful for studying LLP, which can be immediately updated following failed attempts with a novel action sequence. Third, the choice space is vast enough that rewards are sparse early on in learning, which further decorrelates LP and LLP since LP is 0 until the first successful attempt, while LLP is updated at the first trial no matter the outcome. Fourth, hierarchical relationships exist among some, but not all goals. Upon becoming aware of hierarchical relationships, participants could take shortcuts to identify the correct recipe for the goals involved, leading to sharp changes in LLP even with small differences in performance. This feature additionally enables us to test for hierarchy effects on goal selection beyond the indirect effects of hierarchy on learning through LLP.On each trial, participants first selected a goal and then a sequence of actions. Unless otherwise specified, we only modeled goal selection, conditioning on true participants' actions within each trial after goal selection. However, action selection influences performance, LP, and LLP. As in existing studies [23] , we model human goal selection as a multi-arm bandit problem, where the probability of selecting a goal depends on its subjective value relative to other goals. Multiple factors f may jointly contribute to the overall subjective value of a goal [23, 24] , which is thus calculated as a weighted sum of the goal's value relative to each factor V f times its weight β f . The probability P t (g * ) of choosing goal g * among possible goals G on trial t is obtained through a softmax function over the goal values (as is standard in the decision-making literature [86] ):P t (g * ) = exp( f β f • V t f (g * )) g∈G exp( f β f • V t f (g))Subjective goal values are updated as a function of experience through the delta rule [87] :V t+1 f (g * ) = V t f (g * ) + α • δ t f (g * )where α is a learning rate for value updates and δ f is factor-dependent. Models differed in which factor, or combination of factors, composed the overall goal value. Below we detail the individual factors we considered.Performance On each trial, the utility of the selected goal with respect to performance is updated based on the goal-contingent feedback received on that trial r t (1 for positive feedback, 0 for negative feedback): δ t performance (g * ) = r t -V t performance (g * ) Performance estimates were initialized as V 0 performance (g * ) = 1 6 where 6 is the number of goals.Learning progress Learning progress (LP) was updated through the change in reward prediction error for the current goal after initializing V 0 LP (g * ) = 0 (see [24] for a similar approach) . "Reward", here refers to the feedback r participants received on each trial. However, note that no external rewards were delivered for successfully obtaining the desired potion (Appendix C). On each trial, the reward prediction error was calculated as the difference between the current performance value estimate and the feedback obtained. The reward prediction error on trial t was compared to the reward prediction error on the previous trial t -1:δ t LP (g * ) = [(r t -V t performance (g * )) -(r t-1 -V t-1 performance (g * ))] -V t LP (g * )Latent learning progress Goal utilities with respect to latent learning progress (LLP) were updated based on the difference between the estimate at trial t and the current trial's LLP:δ t LLP (g * ) = LLP t g * -V t LLP (g * ) with V 0 LLP (g * ) = 1.Until the goal was learned (when LLP immediately became 0) subjective (unobservable) estimates of LLP were approximated as LLP t g * = 1 -Naction sequences testedNpossible action sequences . The number of possible action sequences is 12 for 2-action goals and 24 for 4-action goals. The number of action sequences tested is the sum of unique action sequences the agent has tried up until trial t. A goal is considered learned when the subject has obtained positive feedback for the current goal at least once (but see Appendix D for alternative heuristics and approximations). Therefore, this formulation of LLP captures the size of the space left to explore to find the correct sequence of actions for the current goal, i.e. the "distance" from the solution. As a result, negative β LLP values indicate a high willingness to pursue activities one is close to solving. This choice of formulation -rather than its opposite, i.e. one where high values indicate high levels of progress toward perfect performancewas made to maintain continuity in the LLP function, which is 0 after a solution is found.Hierarchy Hierarchical relationships between goals could impact goal selection in at least two ways. First, indirectly, by enabling inferences about the solution for goals involved in the hierarchy, thus speeding up exploration and changes in LLP. Second, directly, by acting on biases people might have for selecting goals that share common structures. To account for the latter, we also included a hierarchy factor, whose estimates were updated via: δ t hierarchy (g * ) = H g * -V t hierarchy (g * ) with V 0 hierarchy (g * ) = 0 and H equal to 1 for hierarchical (G3, G5), -1 for non-hierarchical (G4, G6), and 0 for simple goals (G1, G2; but see Appendix D for an alternative scheme and update rule).Model space Because performance is a powerful motivator in people's decisions about time allocation [23, 45] , we include it in all our models. In addition to a performance-only model, we fit and compared models with additional factors for LP, LLP, hierarchy, and both LLP and hierarchy. Thus, fit parameters included a shared α across factors and weighting parameters β f for each factor in the model. Adding a control variable for choice perseveration improved the likelihood of goal choices but did not affect modeling results (Appendix D), so we removed it to facilitate interpretation.Model fitting and comparison Models were fit and compared through hierarchical Bayesian inference (HBI; [88] ) based on the likelihood of participants' goal selection choices [18, 58] . We compare models through protected exceedance probability (PXP), i.e., the probability that each model is the most frequently expressed in the studied population after accounting for chance [89] , and average responsibility (R), which captures how well each model explains participants behavior. Models and parameters for the winning model were recoverable [90] (Appendix E). Simulating participants' behavior [91] also required modeling the learning process, which we assumed followed a reinforcement learning architecture with options [92] (Appendix F).Participants learn successful goal-conditioned policies Learning phase performance was above chance for each goal (Wilcoxon tests were used in place of parametric t-tests since the data was not normally distributed according to Shapiro-Wilk tests of normality; chance levels were 0.083 for 2-action and 0.042 for 4-action goals; all average performances ≤ 0.29, all W(174) ≥ 160, all p ≤ 0.004; Figure 2A ). As expected, performance was higher for 2-action (M = 0.51 ± 0.02) than 4-action goals (M = 0.33 ± 0.02, W(174) = 1207, p < 0.001, r = 0.51). Similar patterns for goals G1-G6 were observed in the testing phase, where performance was not contingent on goal selection (Figure 2B ).Hierarchical inference supports learning If individuals recognized hierarchical relationships during learning, they could use them to draw inferences about the correct recipes for the potions involved. For instance, upon discovering that G5 could be made from the potions created in G1 and G2, participants may correctly infer that G3 is achieved by chaining the recipes for G1 and G2 (Figure 1 , bottom). Consistent with the hypothesis that at least some participants exploited the environment's hierarchical structure, learning performance was higher for hierarchical goals compared to their non-hierarchical counterparts (complex: hierarchical (G3) M = 0.37 ± 0.02, non-hierarchical (G4) M = 0.29 ± 0.02, W(174) = 4168.5, p = 0.004, r = 0.16; compound: hierarchical (G5) M = 0.53 ± 0.02, non-hierarchical (G6) M = 0.49 ± 0.02, W(174) = 6036.5, p = 0.032, r = 0.11; Figure 2A ). Similarly, participants took fewer attempts to learn solutions to hierarchical than non-hierarchical goals (complex: hierarchical (G3) M = 16.66 ± 1.19, non-hierarchical (G4) M = 22.1 ± 1.23, W(120) = 1884.5, p = 0.009, r = 0.17; compound: hierarchical (G5): M = 11.66 ± 0.58, non-hierarchical (G6) M = 13.21 ± 0.53, W(155) = 3889.5, p = 0.03, r = 0.12; Figure 2C ). However, hierarchy effects were not present in the testing phase, likely due to ceiling effects (complex: hierarchical (G3) M = 0.56 ± 0.03, non-hierarchical (G4) M = 0.52 ± 0.03, W(174) = 1592.5, p = 0.223, r = 0.07; compound: hierarchical (G5) M = 0.73 ± 0.03, non-hierarchical (G6) M = 0.73 ± 0.03, W(174) = 1158.5, p = 0.765, r = 0.02; Figure 2B ).Goal selection is sensitive to performance and hierarchy In assessing whether difficulty levels and hierarchy affected participants' goal selection, we found it displayed opposite patterns to learning performance (Figure 2D ). The average probability of selecting a goal was lower for 2-action (M = 0.15 ± 0.00) than 4-action goals (M = 0.19 ± 0.01; W(174) = 4214, p < 0.001, r = 0.27). Overall, participants chose the hierarchical (G3; M = 0.18 ± 0.01) less than the non-hierarchical complex goal (G4; M = 0.21 ± 0.01, W(174) = 5231.5, p = 0.002, r = 0.17). Differences between hierarchical (G5; M = 0.15 ± 0.01) and non-hierarchical compound goal selection were not significant (G6; M = 0.16 ± 0.00, W(174) = 6444, p = 0.128, r = 0.08). At the individual level, the greater the difference in performance between 2-action and 4-action goals, the lower the probability of selecting a 2-action goal (Spearman's ρ = -0.35, p < 0.001), and the greater the performance difference between hierarchical and non-hierarchical goals, the lower the probability of selecting a hierarchical goal (complex: ρ = -0.68, p < 0.001; compound: ρ = -0.51, < 0.001). These results indicate that goal selection depends on performance and hierarchical relationships between goals. We next turn to computational modeling to detail the contributions of performance, hierarchy, LP, and LLP to goal selection.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] Justification: No theoretical results are presented. Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
experimental result reproducibility: 
 Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: All necessary information to reproduce the main experimental results is provided in the main text and/or the appendices. Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
modeling results: 
 We that predicted LLP supports human goal selection, thus expecting models relying on this signal to fit participants' goal selection data better than models that do not.The model with performance and LLP utility factors was the most strongly represented across participants (PXP = 1; R = 0.63) and replicated the main goal selection patterns seen in participants' data (Figure 2E ). Some participants were also well-captured by a model that additionally integrated hierarchy into goal values (R = 0.22). We note that hierarchy likely plays a more direct role in learning (thereby impacting LLP) than in goal selection, given that hierarchy effects in goal selection and performance could be recovered after eliminating hierarchical components from goal selection, but not learning (Appendix G). The model selecting goals based on performance and LP did not account well for participants' behavior (R = 0.12). Models with just performance (R = 0.00) or performance and hierarchy factors (R = 0.04) were marginally represented. The winning model's best-fitting β performance and β LLP were on average negative, suggesting most individuals focused on activities where performance was low -indicating the solution had yet to be found -and where LLP was low (Figure 2F ) -indicating learning potential was high (Section 5). However, more nuanced patterns and temporal dynamics may exist at the individual level. 
inter-individual differences in learning and goal selection: 
 Several interesting strategies emerged upon inspecting individual behaviors (Figure 3 ). We first exemplify them before using a data-driven approach to classify participants according to their approach to the game, revealing differences in how hierarchy impacts learning and thus goal selection. Finally, we assess whether LLP supports goal selection across groups.People express a variety of goal and action selection strategies Many participants repeatedly selected the same goal -often testing possible solutions in a systematic order (Figure 3A ) -only switching to a new goal after learning the current one (Figure 3A, B ). After learning to make all potions, some participants began alternating goals, as if testing or training their ability to retain multiple solutions (Figure 3A, B ). Although most participants showed interest in learning all goals (Figure 3A, B ), a small subset of individuals preferred setting few objectives, or even a single goal, in which they excelled (Figure 3C ), while disregarding others. Lastly, some participants moved across goals and ingredients in a seemingly unprincipled manner, presumably because they were confused or unmotivated (Figure 3D ). A subset of the participants seemed to be aware of adopted strategies (see Appendix H for example verbal reports). We next clustered participants to explore how idiosyncrasies in learning related to goal selection. Note that participants in Figure 3 were chosen as examples and do not directly relate to the clustering analyses presented below. Clustering of individual strategies reveals differences in hierarchical learning We used Kmeans clustering [93] to identify 4 groups of participants, where K was determined through the elbow method (Appendix I). The behavioral metrics used to identify the clusters (e.g., performance and goal selection entropy and distributions) can be found in Appendix I. To avoid statistical doubledipping [94] , we limit ourselves to qualitative observations on cluster-based behavior. To better illustrate salient characteristics of each cluster, we also identified "archetypal" participants through archetypal analysis -a method, often used to profile video game players [95] , which detects data points that can reproduce the original population when linearly recombined [96] . We then matched each archetypal feature with the best-aligned cluster (Appendix I) and identified the participant closest to the corresponding archetype as an extreme example of the cluster it belonged to (Figure 4M-P ).Participants in cluster 1 (N = 39) had below-average performance at learning (M = 0.18 ± 0.03) and testing (M = 0.27 ± 0.06) and favored 2-action goal. They often failed to learn or retain solutions and switched goals frequently (Figure 4M ). In the largest cluster 2 (N = 75; Figure 4B ), performance was excellent at learning (M 0.59 ± 0.0) and testing (M = 0.86 ± 0.03). This group leveraged hierarchical structures to speed up learning (Figure 4B ,F) -which also impacted goal selection. Non-hierarchical goals were favored (Figure 4J ) and often learned later than hierarchical ones (Figure 4N ). Cluster 3 participants (N = 53) showed relatively high performance at learning (M = 0.26 ± 0.04) and testing (M = 0.62 ± 0.06; (Figure 4C ), but seemed to struggle with all 4-action goals -performing worse in them but choosing them more -and took less advantage of hierarchical relationships across goals (Figure 4G, K, O ). Given its small size (N = 8), we invite caution in interpreting the results for cluster 4. However, this group showed an interesting, strong tendency to repeatedly select the same, small set of goals at which they succeeded (often preferring 2-action goals; Figure 4L, P ). As a result, their performance was excellent in the learning phase, where goals were freely chosen (M = 0.67 ± 0.13, Figure 4D ,L) but dropped in the test phase, where goals were imposed (M = 0.46 ± 0.16).Despite such distinct patterns of behaviors, LLP was a central component of goal selection in all clusters (Figure 4Q -T), suggesting LLP is a flexible signal that adapts to idiosyncrasies in people's learning while similarly guiding goal selection (but see Appendix G for further discussion of how learning interacts with goal selection). At the same time, the best-fit parameters of each cluster captured specific properties of the different groups (Figure 4Q -T insets). The efficient learners of cluster 2 relied more strongly on LLP (cluster 2 M βLLP = -3.17 ± 0.18, other clusters M βLLP -2.46 ± 0.16, Mann-Whitney U(173) = 2696, p = 0.001, r = 0.24) and updated both performance-and LLP-based goal values faster than other groups (cluster 2 α = 0.73 ± 0.03, cluster 2 α = 0.6 ± 0.03, U(173) = 4979, p < 0.001, r = 0.28). Cluster 3, which seemed less likely to rely on hierarchy, was more attuned to performance when selecting goals (cluster 3 M βperformance = -4.63 ± 0.22, other clusters M βperformance = -3.67 ± 0.16, U(173) = 2274, p = 0.002, r = 0.24)). Cluster 1, which lacked precise strategies, was less sensitive to both performance (cluster 1 M βperformance = -2.93 ± 0.26, other clusters M βperformance -4.25 ± 0.14, U(173) = 3778, p < 0.001, r = 0.31) and LLP (cluster 1 M βperformance = -1.74 ± 0.16, other clusters M βperformance -3.06 ± 0.14, U(173) = 3980, p < 0.001, r = 0.36). 
discussion: 
 Human behavior is often motivated by self-defined challenges. Despite the centrality of goals in human learning [19] , how people select their own objectives without external guidance or incentives remains to be specified. Following advances in artificial open-ended skill learning [26] [27] [28] , learning progress (LP) has emerged as a useful signal for human goal selection [23, 24] . Here, we introduced a "latent" form of LP (LLP), which is informed by an agent's model of the environment and memory of the actions pursued in it, in addition to the manifest changes in performance tracked by standard LP. In a purposefully developed hierarchical reinforcement learning task, we find that people's goal selection is driven by LLP, calling for an extension of the LP notion to include its latent variants.Inter-goal relationships in our test environment enabled us to check whether hierarchy additionally drives goal selection, as intuition suggests (e.g., one might decide to become competent in baking by first relying on ready-made dough and only later setting the lower-level goal of making dough from scratch). We find that hierarchy impacts goal selection indirectly by enabling inferences and thus affecting LLP. That is, participants' learning is sensitive to hierarchical structures, leading to changes in LLP which, in turn, affect goal selection. We also gathered partial evidence for a standalone role of hierarchy in curriculum development, which further studies may address. We found marked inter-individual differences in strategies for goal setting and pursuit. However, our sample was restricted to a relatively homogeneous group of undergraduate university students. Future studies and computational models may extend participation to a broader subject pool and specifically address individual differences in goal selection and achievement, as well as their relationship with demographics and cultural background, cognitive abilities, and psychopathology (cf. [97] ), which might inform personalized teaching strategies and productivity tools. As our knowledge of human goal setting becomes more precise, however, ethical concerns regarding the use of behavioral sciences in marketing and management should be addressed, particularly in cases where highly personalized methods of influence could transform advertising into manipulation.Elucidating the computational mechanisms driving human goal selection may propel advances in open-ended machine learning. Our suggestion, wherever relevant, is to incorporate LLP in artificial autotelic agents' goal selection. Compared to classic LP, LLP is sensitive to latent changes in an agent's abilities -which LP would only be able to capture after repeatedly observing positive feedback. Using LLP could result in a faster adaptation of the agent's goal selection, ultimately speeding up its learning by enabling it to detect progress before receiving the first reward and letting it proceed to more challenging tasks sooner. Some limitations of the present study may need to be addressed before other fields can fully benefit from the results we presented. In our task, people chose goals from a predefined menu of options. Although this facilitates the study of goal setting, people often invent their own goals by combining observations and imagination [98] . For simplicity, our modeling focused on goal selection. Future research, however, should jointly model goal choices and goal-conditioned action selection, since -as our data suggests -learning participates in complex interactions with the goal selection process. While we provide a simple and task-specific formalization of LLP, a generalized definition is necessary to understand the differences between LLP and other intrinsic motivation signals and ease the implementation of LLP-based goal selection in autotelic machines. By providing initial evidence for LLP, we hope to inspire the establishment of even more precise signals for goal selection in both humans and artificial agents. G. The role of hierarchy in learning and goal selection: Results and discussion of some evidence for a role of hierarchy learning in the context of the present experimental paradigm, and how it relates to goal selection.H. Verbal reports of example participants' strategies: Example verbal reports from participants about their adopted learning strategies.I. Clustering methods: Details about the clustering methods.One hundred and seventy-nine participants were recruited through the University of California, Berkeley, Research Participation Program pool of participants and received course credit as compensation for their time. Participants completed the experiment online from their own devices. We excluded four participants who either responded affirmatively when directly asked whether they took written notes about the correct recipe for each potion, or mentioned writing down solutions on their phone or a piece of paper when reporting their strategies in a follow-up questionnaire. Thus, 175 participants (76% female, age: range = 18-47, M = 21.21 ± 3.75, 94% right-handed) were included in the final data set. All were native or fluent English speakers with no medical history of brain injury, mental/psychiatric illness, or drug/alcohol abuse. The Institutional Review Board at the University of California, Berkeley, approved the study.The paradigm was divided into training, learning, and testing phases. Each learning phase trial encompassed a goal selection, an action selection, and a feedback stage. In the goal selection stage, participants viewed a blackboard depicting six "goal" potions, characterized by their color. Next to each potion were either two or four empty slots -indicating the number of ingredients required to make the potion -and illustrations of the four different ingredients available for that potion. Potions could either be made of basic ingredients (a frog, a bunch of mushrooms, a horseshoe, and a butterfly), or potions of different colors (e.g., teal, orange, red, and blue-colored potions). Two potions appeared twice on the left side of the board, as they could be made from either basic ingredients or other potions. Participants were prompted to select one of the available potions on the left side of the blackboard with a mouse click (Figure 1 ). At the beginning of the action selection stage, the blackboard contents changed to depict the goal potion and either two or four empty slots, which indicated the number of ingredients required. Four different ingredients appeared on separate shelves on the right side of the screen. When a participant clicked on an ingredient, it moved from the shelves to a cauldron depicted in the center of the screen and appeared in the corresponding slot on the blackboard. An empty flask appeared on the table next to the cauldron for the duration of the action selection stage. Participants were prompted to select ingredients, one at a time, until all slots were filled (Figure 1 , top). During the feedback stage, the flask either remained empty and shook left and right, signaling failure to make the goal potion, or became filled with the same color as the goal potion and wiggled up and down, signaling that the chosen ingredients were added to the cauldron in the correct order. No points or otherwise explicit feedback messages were provided.The learning phase (144 trials) was divided into 6 blocks, at the beginning of which participants were allowed to take a break if needed. The training phase (12 trials, 2 per goal) served to help participants get familiar with the game flow and was similar to the learning phase, except that goal selection was forced. This section comprised 12 trials (two per goal), and was intended to help participants get familiar with the game flow. The testing phase was similar to the learning phase, except that goal selection and feedback stages were skipped. Here, participants were instead prompted to make a given potion without receiving information about whether the selected order and ingredients were correct. A set of instructions preceded the gameplay (Appendix C). Importantly, participants were not told to try and learn all the correct mappings between actions and goal achievement. We did not disburse performance bonuses, nor were there any external incentives for successful goal completion. Instead, participants were free to explore the goal space as they pleased for a fixed number of trials.At the end of the experiments, participants completed a short demographics questionnaire and optionally answered the questions "Did you try to use any strategy while playing the game? If so, what was it?", and "Did you notice any structure in the game? If so, what did you notice?". A subset of participants were also asked, "Did you have specific aims or objectives while playing? If so, what were they?". Participants were also invited to leave written feedback.The six different goals (potions) available for selection were constructed as illustrated in Figure 1 , bottom. While we present them in an orderly fashion to ease readability, participants were not told about the underlying structure of the set of goals. The relative position on the screen of goals with different features was pseudorandomized across participants. Below is a description of the six available goals from the training and learning phases:• G1: simple goal 1. This potion was made of two basic ingredients.• G2: simple goal 2. This potion was made of two basic ingredients, none of which were used to make G1. • G3: hierarchical complex goal. This potion was made of four basic ingredients, in an order that reflected the recipe for G1 followed by that for G2. Thus, participants who became aware of the structure could leverage knowledge of G1 and G2 to learn G3, or vice-versa. • G4: non-hierarchical complex goal. Like G3, this potion was made of four basic ingredients.However, the order in which they had to be added to the cauldron did not reflect G1 or G2 in any way. Thus, learning G1 or G2 did not offer any learning advantage for G4, or vice-versa. • G5: hierarchical compound goal. This potion had the same color as G3. However, potions of different colors were available for making G5, as opposed to the basic ingredients available for G4. The potions required to make G5 correctly were G1 followed by G2. Learning this potion could, in theory, be used to gain insight into how G3 is made, or vice-versa. • G6: non-hierarchical compound goal. This potion had the same color as G4 but, similar to G5, was made of two other potions rather than basic ingredients. These were potions that were not available as goals, such that learning G6 could not leverage or provide insight into other potions' recipes. However, participants could infer that the recipes for the first and second potions used to make G6 were the first two and last two ingredients used in G4.Two additional goals were presented to participants in the testing phase:• G7: simple goal 3. This potion was made of the first two basic ingredients of G4.• G8: simple goal 4. This potion was made of the last two basic ingredients of G4.We report the instructions provided to participants verbatim below. Each item corresponds to a screen presented to participants, who paced their own reading by moving to the next screen upon pressing the spacebar key on their keyboard. They were instructed to do so by the message "Press SPACEBAR to continue" appearing at the bottom of each screen. Wherever applicable, images accompanying each item are references in the following list. 1. Welcome to our study! Thank you very much for deciding to participate. 2. This experiment requires your full attention. We will only be able to use your data if you participate fully in the experiment, which should only take about 35 minutes when you are fully engaged. 3. Before we begin, please maximize your browser window and turn off all notifications (Facebook, Instagram, texts, etc.) on both your phone and computer. 4. Please do not disengage (for example, by switching tabs in your browser or navigating away from the task) while you take part in the experiment, including during breaks. 5. If you're not engaged, your data are not valid and cannot be used for the experiment, and therefore you may not get credit. Thank you for understanding. 6. In this game, you will play the role of an alchemist and try to make potions from a set of ingredients. 7. On each trial, you will see a blackboard that shows all the potions you can try to make.[ We performed several checks to verify our modeling results were robust to small variations in the adopted methods. To save time and resources in these additional analyses, we compare models through exceedance probability (XP) rather than the more computationally expensive PXP [89] .Since adding a perseveration term can benefit models of human choices [99] , we also ran a version of the model comparison pipeline in which each model included such a feature in the goal valuation process. The utility V perseveration grew each time a goal was selected to the extent determined by α.Adding perseveration to each model did not change the results (winning model XP = 1; R: model with perseveration only = 0.00, perseveration and performance = 0.00, perseveration, performance, and LP = 0.12, perseveration, performance, and LLP = 0.70, perseveration, performance, and hierarchy = 0.06, perseveration, performance, LLP, and hierarchy = 0.17). For simplicity and ease of interpretation, we thus opted for models without perseveration.Our implementation of LLP drops to 0 after a participant has learned the solution to a goal, as (assuming perfect memory) no further learning progress can be made once the solution is found. Since we did not have access to the true cutoff for when a goal was "learned", we placed an arbitrary cutoff after the first time the solution was identified. However, participants may require more than a single trial to retain a goal's correct action sequence. Therefore, we also developed models where LLP dropped to 0 after 3 successful attempts. Using this heuristic instead of the 1 successful attempt heuristic did not change the results (winning model XP = 1; R: model with performance = 0.03, performance and LP = 0.18, performance and LLP = 0.66, performance and hierarchy = 0.00, performance, LLP, and hierarchy = 0.13).To test the reliability of a heuristic method to decide after how many trials a goal was "learned" for the purpose of updating LLP, we also used a Bayesian cutoff to set a more precise threshold. This was calculated by computing confidence intervals over each subject's learning curve and determining the trial at which performance surpassed chance levels with 95% confidence (see [100] for details on methods). Using the Bayesian threshold for learning did not change the results (winning model XP = 1; R: model with performance = 0.00, performance and LP = 0.14, performance and LLP = 0.62, performance and hierarchy = 0.04, performance, LLP, and hierarchy = 0.21).Within the specifics of our task, LLP decreased every time a participant chose a new action sequence for the active goal, eventually reaching 0 once the participant found the solution. This ensured there were no discontinuities in LLP. However, another intuitive way to model LLP is by incrementing it every time a participant attempted a new action sequence, until they found the solution:LLP t G =Naction sequences tested Npossible action sequences if G is not learned 0, otherwise Since this formulation did not significantly change the results (winning model XP = 1; R: model with performance = 0.00, performance and LP = 0.19, performance and LLP = 0.59, performance and hierarchy = 0.36, performance, LLP, and hierarchy = 0.18), we kept the version described earlier, which was not affected by abrupt discontinuities -thus facilitating interpretation.To update the hierarchy utility of the chosen goal, we used a term H equal to 1 for hierarchical (G3, G5), -1 for non-hierarchical (G4, G6), and 0 for simple goals (G1, G2). However, hierarchy effects were more prominent for complex than compound goals (Figure 2 ). We thus ran the model comparison pipeline with an alternative coding scheme for H, which was equal to 1 for the hierarchical complex goal (G3), -1 for the non-hierarchical complex goal (G4), and 0 otherwise (G1, G2, G5, G6). Using this scheme did not significantly change the results (winning model XP = 0.96; R: model with performance = 0.02, performance and LP = 0.12, performance and LLP = 0.48, performance and hierarchy = 0.02, performance, LLP, and hierarchy = 0.36).In our model, the term H was used to learn the utility of each goal with respect to hierarchy over time through the shared learning parameter α. However, it is also possible that hierarchy information informs participants' behavior from their first interaction with each goal. We thus ran the model comparison pipeline with a learning rate of 1 for H, equivalent to setting hierarchy values to their coding scheme from the first interaction with a goal. Using this fixed learning rate did not significantly change the results (winning model XP = 1; R: model with performance = 0.03, performance and LP = 0.12, performance and LLP = 0.58, performance and hierarchy = 0.00, performance, LLP, and hierarchy = 0.27).Before analyzing model parameters from the winning model (one which chose goals based on performance and LLP signals), we simulated and fit data with known parameters (best-fit parameters from the Laplacian approximation of HBI) to ensure they could be recovered. We used an options reinforcement learning model to simulate participants' learning process (Appendix F). We observed tight correlations between true and recovered parameters (Figure 6 ), suggesting the experiment was sufficiently powered to assess this model [90] . To ensure we could compare similar models [90] , we tested model recoverability by simulating participants' behavior with each model and the corresponding best-fit parameters (Laplacian approximation step of HBI). Each participant's behavior was simulated 3 times to account for random noise. We again assumed a a reinforcement learning process with options governed the participants' action selection. We then performed model comparison (HBI; [88] ) to determine whether we could correctly identify the simulating parameters. The confusion matrices in Figure 7 illustrate this was the case, although the model with performance-, LLP-, and hierarchy-based goal values was somewhat difficult to distinguish from the model with performance-and LLP-based values alone. In the experiment, participants made two kinds of choices, first picking a goal and then selecting a sequence of actions to try and achieve it. Since our focus was on the former, we only fit the data on the goal selection process, gathering information about action selection and feedback (required, e.g., for the computation of the performance, LP, and LLP features that entered the goal selection component) directly from the participants' data. However, simulating goal selection data -e.g., for parameter recovery (Figure 6 ), model recovery (Figure 7 ), and model validation (ensuring the model captures key aspects of the original data through simulations with the best-fit parameters [90, 91] , Figure 2E ) -requires creating a model that also generates data about the learning component. To this end, we developed models that attempted to learn the correct action sequences in addition to choosing their own goals. The goal selection component of the various models was described in Section 3.2. The learning component detailed below was shared across all models.Each trial begins with a goal selection step (forced, in the training phase, or free, in the learning phase). Next, the agent decides which actions to pursue. The model's fundamental architecture is that of a standard Q-value reinforcement learning model [67] , except that policies were goal-conditioned [70] . The model's Q-value table has shape N G × N A × N S , where N G is the number of possible goals (6), N I is the number of ingredients available actions for selection at the current timestep, and N S is the total number of steps required to make the active goal (either two or four; Appendix B). Any of the four ingredients available for the active goal can be selected at the beginning of each trial and becomes unavailable once the agent chooses them. The model selects actions via a softmax policy [86] over the relevant Q-table until all the slots are filled:P t (a|g * , h) = exp(β learn • F (Q t (g * , a)), h) i exp(β learn • F (Q t (g * , a i ), h)where t is the current trial, a is the chosen action; g * is the active goal; h is the current trial's history (i.e. the actions selected thus far) which, through the filter function F , determines the number of steps left to complete the trial and the actions the agent can consider taking next; β learn is the inverse temperature (the higher the β learn , the more deterministic the decision).Once all slots are full the agent receives a binary feedback r t , equal to 1 if the chosen sequence of actions matched the active goal, and 0 otherwise. The agent then uses this signal to update the value of each chosen action through the delta rule [87] . The filter function F serves to address the correct Q value based on the timestep at which the action to update was chosen:F (Q t (g * , a), h) = F (Q t-1 (g * , a), h) + α learn • (r t -F (Q t-1 (g * , a), h))where α learn is the learning rate. This update function reflects the fact that the agent does not receive partial feedback for correct actions at the right timesteps, and can therefore only identify the appropriateness of chosen actions if the entire chosen sequence is correct.The model's Q table was also subject to forgetting as governed by the variable phi, which gradually brought back all Q values to the initial value Q 0 (0):Q t (g, a) = Q t-1 (g, a) + ϕ • (Q 0 (g, a) -Q t-1 (g, a))∀g ∈ G, a ∈ AExcept for the temporal aspect given by the nature of the environment, the model described until now is not capable of recognizing hierarchical relationships among goals. To enable it to leverage hierarchical structures and speed up learning, we endowed the model with the ability to build "options", i.e. to create and select sub-sequences of actions as a single chunk [92] . Upon receiving positive feedback, the model extends the space of possible actions with the chosen sequence of actions, which can then be considered a viable option whenever the ingredients it requires are available and the slots to fill are of at least the same length as the option itself. Options are selected and then updated in the same way atomic actions are. Note that, in reality, participants could also leverage insights from goals that shared structural similarity without requiring the same atomic actions (e.g., compare G3 and G5, Figure 1 , bottom). While found that the simpler current model was sufficient to capture many aspects of the participants' behavior, future extensions could consider generalization rules that enable the model to share more abstract knowledge across goals.The options model was unfittable through HBI methods due to intractable likelihood [101] . To approximate the learning parameters for each participant (β learn , α learn , and ϕ), we fit a simpler reinforcement learning model (without options), this time through maximum-likelihood methods and only on action selection date, and used the resulting best-fit learning parameters for simulations. To account for random noise, we ran 3 iterations per participant when simulating data for validation and model recovery.Hierarchical features of the environment could interact with the goal selection process in at least two ways. First, hierarchy could have a direct role in goal selection, e.g., people might either prefer or avoid goals that enable them to exploit hierarchical relations. Second, hierarchical structures may impact individuals' learning, hence their LLP -which is then used to select goals. The fact that the winning model for goal selection had an LLP factor, but lacked direct information about the presence of hierarchical relationships among goals, suggests the latter hypothesis is more likely. However, we find that hierarchy plays a major role in the learning process, thereby affecting goal selection indirectly.To illustrate this, we simulated participants' data with two models. The first model, used for all other simulations presented in this article, is a reinforcement learning with options ( [92] ; Appendix F), therefore able to exploit hierarchical relationships between simple and complex goals. The second model is a flat reinforcement learning model, which only has access to atomic actions. Despite the fact that both models lacked hierarchical factors in the goal selection stage, effects of hierarchy similar to those of real participants (Figure 2 ) were found in data simulated with the options reinforcement learning model (Figure 8A-C ). By contrast, removing hierarchical information from the learning component completely removed hierarchy effects in learning and goal selection (Figure 8D-F ). These results suggest at least some participants generated insights for learning by recognizing hierarchical relationships in the learning environment, while not necessarily using this information directly to decide which goals to select. Since the options model was not fittable through HBI methods, quantitative confirmation of these results awaits formal model comparison through more advanced model fitting methods [101] . At the end of the experiment, participants were asked questions about the strategies they followed while playing the online game (Appendix B). Capital letters in the responses transcribed below correspond to participants in the subplots of The first step towards separating participants into clusters was engineering features from the raw data to obtain distinct behavioral signatures. The final subset of features was selected due to its enabling reliable cluster partitions and the low correlation across features (Figure 9A ):1. P(Correct) at learning: proportion of trials of the learning phase in which the chosen goal was achieved. 2. P(G in [G1, G2, G5, G6]): proportion of trials in which a 2-action goal was selected. 3. P(G in [G1, G2, G3, G5]): proportion of trials in which a goal involved in hierarchical relationships was chosen.4. S(Goals): entropy of the chosen goals. 5. S(Actions): entropy of the chosen actions. 6. P(Correct) at testing: proportion of trials of the testing phase in which the proposed goal was achieved (including G7 and G8).To capture temporal dynamics in participants' behavior, all features were computed separately for each of the six blocks of the learning section, except for P(Correct) at testing, which was a single value per subject since there was a single testing block. As a result, the behavioral features were highly dimensional. Thus, before performing clustering, we reduced the dimensionality of the features through principal component analysis (PCA). We identified 11 as the optimal number of principal components (PCs) through the elbow method over the cumulative explained variance. Plotting the PC loadings of the original features suggested PCA maintained the structure inherent in the raw features while enabling a more tractable dimensionality (Figure 9 ). For example, PC1 seemed to track performance indicators, PC2 defined action entropy early in learning, and PC3 represented goal entropy. Participants were then clustered through the K-means method [93] over the PCs. However, similar clusters were obtained through different partitioning methods. The elbow-point heuristic over the within-cluster sum of squares (WCSS) was used to identify 4 as the optimal number of clusters (Figure 10A ). Plotting combinations of the first three PCs confirmed the presence of four distinct groups 10B).We used archetypal analysis -a technique which searches extremal points ("archetypes") in multidimensional data ( [96] , software by [102] ) -to visualize extreme behaviors and exemplify participants' strategies in each cluster. The adjusted mutual information between K-means cluster identity and the closest archetype for each participant was 0.74, confirming much overlap between the two classification methods (Figure 10C ).• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: Error bars and statistics are reported wherever relevant in results and figures.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [NA] Justification: None of the analyses and modeling processes were so computationally expensive to require special computer resources.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: The authors are the creators of the main assets used in the paper and cite other resources used.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: Data collection and analysis methods are sufficiently documented in the main text.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? Answer: [Yes] Justification: Details about data collection with human participants are provided in the main text and the appendices, where the full text of instructions given to participants is reproduced alongside relevant screenshots. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.38th Conference on Neural Information Processing Systems (NeurIPS 2024).Behavioral resultsGiven the novelty of our experimental paradigm, it was crucial to validate the environment and gather intuitions about participants' behavior. Furthermore, both LP and LLP are influenced by how well participants learn to achieve each goal. We thus first analyze group-level learning and goal selection as a function of goals' difficulty levels and hierarchical structures. We then test the specific role of LLP in goal selection through computational modeling (Section5). 
 
 Appendix:  


Decision: While their were some initial concerns, the authors addressed everything that could be considered significant. Several reviewers consider this a particularly strong contribution in this area and I'm inclined to agree.

Reviews: review: paper_summary: paper_summary: This work presents a hypothesis of a latent learning process that can guide autotelic agents in goal selection. Human experiments provide evidence supporting this hypothesis.
strength_weakness: strength_weakness: 1. All the subjects of the human experiment are students, which may introduce bias.
2. The definition of $V_f$ for each kind of model is unclear: e.g. in "performance", what does "currently active goal" mean? Where does $r^t$ come from? What is the relationship between it and the goal? 
3. The design of LLP, which involves selecting multiple action sequences, does not provide a complete explanation of how LLP can improve human choices or actions better than LP.
questions: questions: 1. Hierarchy influences goal selection through latent learning processes (LLP). Will separating the evaluation of hierarchy and LLP affect the results?
2. Section 6 discusses individual differences. Has there been an analysis of the causes behind these differences?
limitations: limitations: Yes.
paper_summary: paper_summary: This paper looks at human goal selection during learning. A known useful signal for goal selection is learning progress (LP). LP measures performance from past observations, and is therefore only sensitive to measurable change in performance. The paper hypothesizes that goal selection is additionally driven by a “latent LP” (LLP) measure, which an agent infers from its knowledge of and interaction with the environment, and which therefore does not depend solely on observable performance improvements. Computationally, for each potential goal that can be set, LP is modeled as a value that is updated based on changes in the performance prediction error, while LLP is modeled as tracking how much of the solution space is currently unexplored (thus, it is a proxy for how close the agent is to solving the goal). A simple goal selection model is proposed under which goals are selected with probability given by a softmax function of goal-specific values. These values are a weighted sum of - among other factors - the aforementioned LP and LLP values. An experimental setup is then introduced in which participants were asked to iteratively select one of many goals and to attempt to solve that goal. The fact that participants had to actively select goals made participants’ goal selection behavior directly observable. Data from human participants reveals that the LLP factor is an important element in goal setting behavior.
strength_weakness: strength_weakness: The definition of LLP introduced here is quite specific to a small and discrete space of action sequences. A more general definition or some preliminary discussion on how one could extend the current work to continuous or countably infinite spaces of action sequences would strengthen the work.
questions: questions: The goal selection model considers a weighted sum for combining the factors that contribute to the goal values. Did you consider or test more complex functions of the factors?
limitations: limitations: Limitations are discussed in the conclusion. The authors do mention that potential future ethical concerns should be addressed, but do not expand on what those concerns may be.
paper_summary: paper_summary: This paper examines how humans select between different possible goals. The authors designed a hierarchical reinforcement learning task where the main dependent variable was which goal participants chose to work on for each trial. Then, they built descriptive computational models to assess what parameters explained participants' goal choices throughout the duration of experiment. The parameters that were most predictive of the participants' goal selections were performance and latent learning progress. 

The experimental task introduced in this paper bears a resemblance to Little Alchemy (https://littlealchemy.com/) insofar as participants combine together different elements to create new elements. Where the task differs is that, rather than freely combining elements without any specific goal in mind, the participants in this task had 6 (not including the testing phase) goal potions to create and they received feedback about whether or not they successful concocted their target potion for the trial.

Their findings contribute to understanding human learning and building autotelic machines. In terms of human learning, the paper provides new evidence that latent learning progress (rather than standard learning progress) drives people to chose some goals over others. In terms of building autotelic machines, new artificial agent models could incorporate latent learning progress when choosing goals to better mimic the adaptive and rapid learning of humans and animals.
strength_weakness: strength_weakness: While this paper has a number of important strengths, the main weakness is that it is not clear how well these findings would generalize to different tasks (especially more real-world tasks):
- First, the experimental paradigm is very specific. Users have to pick from a limited list of goals with the explicit over-arching goal of being able to create all of the potions successfully. Conversely, people usually generate their own goals (e.g., "I'd like to read this book", "I'd like to go on cross-country trip to visit Aunt Ida") rather than picking from a specific list, and they often choose these goals without any supervised over-arching goal.
- Second, the participants were university students (mostly female) participating for course credit. Would the results generalize to different populations?
- Third, the measure of latent learning progress (LLP) is entirely specific to this experimental task. The authors operationally define latent learning progress as 1 - (N actions sequences tested / N possible action sequences). This measure only applies in situations where there are a finite number of options to try. How can future researchers extend this notion of LLP to open-ended tasks that use high-dimensional sensory and motor spaces (where this operational definition would not work anymore)?

To be clear, I don't expect the authors to design a new experimental paradigm or test a new demographic range of participants during the rebuttal period. But it would be helpful for them to address these as limitations.

On a separate note, the statistical analyses need effect sizes. (For example, a simple Cohen's d for each Wilcoxon / t-test would be suffice.) Some of these differences in task performance are statistically significant because p < .05, but are not necessarily behaviorally significant because the difference (effect size) is so small. Also, if I'm reading the stats correctly, the results in text do not match Fig 2A: Complex hierarchical G3 mean performance is reported as .37, but the bar in Fig 2A looks like it couldn't be any higher than .3.


Minor:
- The in-text references are sometimes formatted a little strangely in terms of the number (for example citing 12-17, 11 instead of citing 11-17 or citing 1, 33, 34, 3, 35, 5, 6, 36, 37, 8 instead of citing 1, 3, 5, 6, 8, 33-37).
- The explanation of the experimental game (section 3.1) is not that clear. The figure (Fig 1) is definitely helpful, and the description in the appendix clarifies the game really well. You might consider reformatting the paragraph describing all of the goals into a list format.
- Figure 3 is difficult to understand as-is, but I think it might actually be a really interesting figure. Part of what makes it unclear is the y-axis: What is the "action sequence index"? I suspect there might be an alternative to a dot plot that would be easier to read (maybe even forgoing the y-axis and showing a single strip of color-coded trials for each subject perhaps?). It would also help the readers to add some descriptors to each graph so the reader knows which elements to look for (e.g., setting a few objectives rather than trying all goals, unprincipled switching between goals).
questions: questions: My questions are covered in the Weaknesses portion.
limitations: limitations: See Weaknesses section.
paper_summary: paper_summary: This paper develops a new analysis of autonomous goal selection based on "latent learning progress". The key idea is that agents seek to maximize progress on a latent variable rather than an observed variable like performance. The paper reports an interesting experiment with humans that seeks to test whether people use latent learning progress to guide their goal selection.
strength_weakness: strength_weakness: - Latent learning progress wasn't formally defined until well into the middle of the paper, and there it was only operationalized in terms of the specific experimental setup rather than something more general.

- While motivated by work in AI, the paper isn't really written in a way that will have a broad appeal to an AI audience. I'm not sure how much impact this work will have at least within AI.

Minor:

p. 1: "Along factors" -> "Along side factors"

p. 4: "in which factor" -> "in which factors"; "combination of factors" -> "combinations of factors"
questions: questions: - Can the authors provide a more general formal definition of latent learning progress?

- How can latent learning progress be defined in such a way that it's likely to be both general and useful? My concern is that there are probably many versions of this which are not useful, depending on how one defines the latent variable whose progress is being monitored.
limitations: limitations: The authors have adequately described the paper's limitations. I don't see any potential negative societal impacts.

Comments: title: Discussion period almost over
response
comment: Thank you for your careful feedback, enthusiasm, and support!
Thanks for your response. I'm especially looking forward to seeing how the authors will computationally expand on LLP in the future. (And I appreciate that the authors likely do not want to go into too much detail on how they plan to do so quite yet.) In the meantime, I hope that we'll be seeing this paper at NeurIPS, and I'm raising my score accordingly to reflect that.
We thank the reviewer for taking into consideration all our responses. 

We are grateful for their suggestion on effect sizes, which we will consider for future studies, although we decided to take a relatively standard approach here. 

As for entry points into more naturalistic studies, we view both paths the reviewer mentioned as interesting! We think our study provides an entry point into the study of goals in more naturalistic settings in at least three ways, including the two proposed by the reviewer:
- [Feasibility] This study serves as an entry point in that it establishes the validity of LLP in a simple environment, preparing the ground for more complex situations.
- [Task] The task we developed in this study could serve as a baseline for future work. One possible extension of the task would involve participants coming up with their own “goal potions” rather than selecting them from a predefined menu. In this case, LLP could still be defined in a similar way, provided that the number of possible combinations to achieve the participant-defined goal is known. Another possibility would be to use data from online videogames to study LLP, as was previously done in studies on motivation and (the standard version of) LP ([Brändle et al., 2024](https://osf.io/vg8dz)). 
- [Theory] The theoretical idea of LLP, could be computationally expanded to more complex internal models than simply the number of possible combinations.

We hope this further clarifies the points raised by the reviewer.
Thank you for your thorough response. I've reviewed your general response and your responses to me and the other reviewers. 

I'm enthusiastic that your team plans "to adapt our approach to increasingly more naturalistic, complex settings to pursue a more nuanced understanding." Can you provide any more clarity about how the present study provides an entry point towards this goal? Once the settings are more naturalistic and complex, it seems like you won't be able to use the same computational definition of LLP, so you'll (computationally) need to start from "square one." Do you see the entry point as being the task, which could be adapted to be more open-ended? Or do you see the entry point as the overall theoretical idea of LLP?

In terms of effect sizes, thank you for including these for the revision. For future studies, I encourage the authors to consider effect sizes that aren't normalized by the number of participants. When effect sizes are measurements that are normalized by sample size, it obscures the actual magnitude of the effect. (The raw magnitude/size of the effect is the same whether you test 50 or 500 people; the difference is that 500 will give you more statistical power.) This comes back to my original comment: "Some of these differences in task performance are statistically significant because p < .05, but are not necessarily behaviorally significant because the difference (effect size) is so small."
The discussion period is almost over, so both authors and reviewers please respond to any unaddressed questions. Reviewers, be sure that you have all of the information you need from the authors, since after the 13th, they won't be able to respond.
We are happy to hear that all points were clarified. Thank you for your constructive feedback and for supporting our contribution to NeurIPS!
Thank you for the clarification, and for addressing the potential ethical concerns. I agree entirely with your justification for using a weighted sum of factors. 

I think this is a really strong paper, and agree with the authors' view that it is suitable for NeurIPS. Based on the authors' responses to my review and the other reviews, I have decided to raise my score.
We are glad to hear all questions have been addressed and see the score has been increased. Thanks again for your constructive feedback!
We are glad to hear we were able to address all comments! Thank you again for the constructive feedback.
Thank you for these responses, which address my comments. My score was already high (8), and I don't think the work justifies a higher score (9, which would imply it is "groundbreaking"), but I will advocate for its acceptance.
The new response addressed my question. I would like to increase the score.
We are glad that the reviewer found our response exhaustive, and thank them for bringing up an additional question. Due to space limitations, we did not fully expand on Hierarchical Bayesian Inference (HBI) in the manuscript and instead redirected the reader to the original article that introduces this model fitting method ([Piray et al., 2019](https://doi.org/10.1371/journal.pcbi.1007043)). The cited paper contains full details and information, as well as a [GitHub repository](https://payampiray.github.io/cbm) with the code to implement it. 

Briefly, HBI performs concurrent model fitting and model comparison. It characterizes a population-level distribution of parameters from which individual estimates are drawn, in a way that is proportional to the probability of each subject’s data being generated by each model i.e., the model’s responsibility with respect to each subject. Large values of responsibility (close to 1) for a subject and model indicate the model is likely to be the best underlying model for the subject. The HBI algorithm comprises four steps, which are iterated on until stopping criteria are met: 1) calculate the summary statistics, 2) update posterior estimates over group parameters, 3) update the posterior estimate over each individual parameter, 4) update estimates of each model’s responsibility with respect to each individual subject. For individual parameters, we used the default priors of 0 for the mean and 6.25 for the variance. We refer the reviewer to the original article ([Piray et al., 2019](https://doi.org/10.1371/journal.pcbi.1007043)) for all mathematical details. 

While relatively new, this method has been used successfully in several applications (at the time of writing, Piray et al., 2019 counts over 100 citations). HBI outperforms more traditional statistical tools, such as maximum likelihood model fitting, as it is less prone to overfitting and less likely to favor overly simplistic models (see [Piray et al., 2019](https://doi.org/10.1371/journal.pcbi.1007043)). 

We hope this answer satisfies the reviewer. Given that all other points were addressed in our previous response, we hope the reviewer will consider increasing their score, or help us address any remaining questions by stating them in a comment.
Thank you for the response, which addressed most of my questions. 
The only unclear thing left is about the model fitting described in lines 208-215. Could you clarify what the fitted parameters and loss function are in this fitting process? How did you determine the 'responsibilities' of each model in explaining participants' behavior?