Latent Learning Progress Drives Autonomous Goal Selection in Human Reinforcement Learning
Authors: Gaia Molinaro
Cédric Colas
Pierre-Yves Oudeyer
Anne Collins
Venue: NeurIPS 2024 poster (2024)
Keywords: goals
reinforcement learning
cognitive science
computational modeling
autotelic agents
curriculum development
Primary area: neuroscience_and_cognitive_science
TL;DR: A newly defined, "latent" form of  learning progress provides a valuable signal for goal selection in human reinforcement learning

Abstract: Humans are autotelic agents who learn by setting and pursuing their own goals. However, the precise mechanisms guiding human goal selection remain unclear. Learning progress, typically measured as the observed change in performance, can provide a valuable signal for goal selection in both humans and artificial agents. We hypothesize that human choices of goals may also be driven by _latent learning progress_, which humans can estimate through knowledge of their actions and the environment – even without experiencing immediate changes in performance. To test this hypothesis, we designed a hierarchical reinforcement learning task in which human participants (N = 175) repeatedly chose their own goals and learned goal-conditioned policies. Our behavioral and computational modeling results confirm the influence of latent learning progress on goal selection and uncover inter-individual differences, partially mediated by recognition of the task's hierarchical structure. By investigating the role of latent learning progress in human goal selection, we pave the way for more effective and personalized learning experiences as well as the advancement of more human-like autotelic machines.

Latent Learning Progress Drives Autonomous Goal
Selection in Human Reinforcement Learning
Gaia Molinaro
Cédric Colas Massachusetts Institute of Technology ccolas@mit.ede edu
University of California, Berkeley gaiamoling maaobberreke edu
Pierre-Yves Oudeyer
Anne G. E. Collins
Inria Centre at the University of Bordeaux pierre-yte yverre-yeee... oudeyer@inria.fr
University of California, Berkeley amnecollins@berkeleey edu
Abstract
Humans are autotelic agents who learn by setting and pursuing their own goals. However, the precise mechanisms guiding human goal selection remain unclear. Learning progress. typically measured as the observed change performance, can provide valuable signal for goal selection in both humans and artificial agents. We hypothesize that human choices of goals may also be driven by latent learning progress, which humans can estimate through knowledge of their actions and the environment even without experiencing immediate changes in performance. To test this hypothesis, designed a hierarchical reinforcement learning task in which human participants (N = 175) repeatedly chose their own goals and learned coal-conditioned policies. Our behavioral and computational modeling results confirm the influence of latent learning progress on goal selection and uncover inter-individual differences, partially mediated by recognition of the environment's hicrarchical structure. By investigating the role of latent learning progress human goal selection, we pave the way for more effective and personalized learning experiences as well as the advancement of more human-like autotelic machines.
Introduction
Animals often spontaneously explore their environment the absence of immediate extrinsie payoffs [1H]]. While seemingly wasteful, such intrinsically motivated behavior is argued adaptively guide learning and exploration, paying off across developmental and evolutionary timescales [9] [T] As result, researchers and engineers started endowing artificial systems with intrinsic reward signals that aim to reproduce aspects of human curiosity-driven learning [1] A striking aspect of human intrinsic motivation is autotelic exploration, wherein people self-generate goals, and reward themselves for achieving them 18 20 How people decide which goals to pursue is thus an
outstanding issue in the cognitive sciences 21.. Identifying the key variables affecting goal selection
may in turn enable the development of artificial agents that intelligently set their own objectives, rather than being optimized for predefined ones 222
Alongside factors such as performance and novelty, learning progress (LP) the derivative of performance with respect time has proven a useful signal for goal selection, steering both humans 23 24 and artificial agents (25)28) away from tasks that are either too simple or too hard. Most existing approaches approximate LP using past observations (e.g., differences in recent and earlier performance (33), such that LP is greatest when performance changes rapidly ([29), but see (0 However, there often situations where externl change visible, yet some other form of progress toward the desired outcome is made. Imagine being tasked with identifying the correct
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

sequence of numbers to open a combination lock, which you might attempt through trial and error Throughout most of this scenario, repeated failures would yield no difference in performance, hence no empirical learning progress (as typically defined). Nonetheless, provided that the lock has a limited number of slots and numbers and that you can avoid repeating incorrect combinations, every attempt is a step toward the solution. Based on this observation, we propose extending the definition of L.P to include a latent variant, distinct from its manifest counterpart in that it relies on an internal model of the environment and one's 's behavior in addition to performance. Following this example, we formulate latent learning progress (LLP) as a measure of progress an agent infers by leveraging knowledge about the environment and its interactions with it, e.g., through reasoning. For example, LLP could be tracking the proportion of hypotheses tested over the entire solution space, thus changing with every new set of actions taken and dropping upon identifying the correct one. Unlike LP, LLP does not require observing performance improvements to provide informative signals about progress. At least in certain settings, LLP could be a more valuable signal for goal selection than the classically defined LP, as it proactively estimates future progress without waiting for performance improvements.
Here, we hypothesize that LLP is signal humans use for goal selection, and that it can be considered separately from LP. Specifically, we test the role of LLP in human goal selection by introducing a novel learning paradigm with goal choices as the key dependent variable. Several features of this environment (detailed in Section make it possible to distinguish between LP and LLP and test the latter's contributions to autotelic exploration in sample of human participants. Among these, a hierarchical component is introduced in the learning environment to further target LLP, which might be sensitive to seeraarchy-bssed inferences leading to sharp changes in the agents' knowledge of the environment. These features make our environment suited for probing LLP and testing its contributions to autotelic exploration in sample of human participants. By combining behavioral findings and computational modeling of human decisions in the experiment, we find support for our hypothesis that LLP captures human goal choices better than standard LP in this hierarchical reinforcement learning task. We also find inter-individual differences in goal selection strategies, warranting further studies on the factors underlying such idiosyncrasies. In our discussion of the results, we argue that adopting the notion of LLP may foster the development of autotelic machines.
2 Related work
Intrinsic motivation The various forms of intrinsic motivation can broadly be classified as either knowledge-baadd (KB) or competence- based (CB) [ 28 29 31 32]. KB intrinsic motivations de- rive from comparisons between pre- pre-existing and newly acquired information and are well-documented
in the animal world (e.g.. I
3
5
6
8 (7)) In artificial agents. KB signals of novelty, diversity,
or prediction error broaden exploration when provided alongside extrinsic rewards [3 15
16
38
39
or even on their own [II see 40 for review). However, human curiosity is often driven towards
stimuli of intermediate complexity 7 42). rather than extremes. Accordingly, some artificial
models aim to maximize an "intermediate" difference between predictions and observations [30 43 Intrinsic motivations based on learning progress (LP) can avoid arbitrary cutoffs for what signifies intermediate complexity. LP is particularly useful as a CB motivation, prompting agents to improve Indeed, humans often find pleasure in CB intrinsic motivations highlight the autotelic property of human learning, which is often driven by self-defined objectives. Two concurrent challenges in cognitive science and artificial intelligence involve identifying the signals people use to develop their learning curricula 19 21 and adapting those principles to machines [22
their performance on seff-determined goals (29, 32
44
activitics whose difficulty matches their abilitics
455
46
Goal selection in humans Goals, defined as "a cognitive representation of a future object that the organism is committed to approach" [47], have long been a key concept in psychology (c.g.. (48
with rich literature on how various types of goals interact with performance and motivation [52 53
In cognitive neuroscience, goals are typically imparted by the experimenter and are not the main focus of the analyses [54]- though recent work illustrated the impact of goals on core aspects of neural, cognitive, and behavioral responses ( 20 21 55] see [9] for review). Nevertheless, among the principles thought to guide human goal selection is the tendency to choose activities that balance the expected outcome's desirability and the probability of attaining it (60) When free to allocate their time to various tasks or choosing activities for pure enjoyment, people naturally tend to match the activity's level of challenge to their current abilities 45 46 61), which might be optimal for learning
62 63] However, most studies fail to provide precise computational details on the mechanisms of

---

goal selection. Although not explicitly addressing goals, two recent experiments attempted to fill this gap. In particular, [23 tasked participants with repeatedly choosing which of four learning activities (each with a different difficulty level) to engage with, and found that performance and LP jointly contribute to activity selection. Similarly, [24] found that people use novelty and I.P when exploring various activities based on their own curiosity. Here, we propose that in deciding which goals to set, humans may rely not only on manifest signals of performance but also on an internally tracked, latent form of learning progress.
Goal selection in artificial agents Enabling agents to learn autonomously has been longstanding challenge in artificial intelligence and central tenet of many successes in the field [43] 64)68). However, most existing models are optimized for a predetermined objective. Overcoming this limitation, autotelic agents are designed to autonomously select goals and learn to achieve them begging the question of how to optimally set goals for efficient learning. Even random goal sampling (70-72 72)
22 25
marrop-ytmiiiiiitiionn objectives [73 74] present some advantages [75), but agents can be programmed to adjust their goal selection to currently owned skills and further benefit from autotelic properties. One such approach involves two-part adversarial system, wherein a "teacher" proposes challenging tasks and a "student" learns to maximize performance over them 44 78. Algorithmic implemennations of LP also help agents avoid wasting time and resources on objectives that are too far-reaching or even impossible (e.g., due to inherent noise in the system; 28 ).). These approaches calculate LP relative performance, not considering changes in the agent's 's goal-conditioned competence that might occur latently. Thus, to our knowledge, LLP-mediated goal selection in artificial agents is an unexplored avenue.
3 Experiment design and modeling approach
3.1 Experimental setup
Most reinforcement learning problems consider single task to master. Similarly, experimental psychologists usually instruct participants to pursue specific objective. By contrast, we developed an environment in which goal selection was the main dependent variable, enabling us to study key determinants of goal setting in humans. The experiment was implemented in the Unity game engine and presented as an interactive online game, in which healthy human participants (N 175; see Appendix A for details) played the role of alchemists and could learn to make different potions from sets of ingredients. Below we refer to potions as "goals" and ingredients as "actions".
Goal selection
Action selection
?
Feedback
vade
AAAAA A&AA
8m 664A
Goel
Simple
Simple
Complex
Complex
Compound
Herarchy
NA
NA
Yes
No
Yes
No
Actions
U
Figure Structure of learning trials and summary of the goal space. (Top) example trial. (Bottom) Schematic representation of goals and their characteristics.
The paradigm included training, learning, and testing phases. Learning phase trials (144) were composed of a goal selection stage, an action selection stage, and a feedback stage (Figure I top; Appendix B). In the goal selection stage, participants viewed colored potions available for selection, with icons indicating the number of ingredients required and the four different ingredients available to make them. the action selection stage, participants chose which ingredients to add to the cauldron. In the feedback stage, participants received binary feedback as a result of their actions (the empty flask in the foreground either filled up or remained empty based on whether participants chose the correct ingredients in order). In the 12 training trials preceding the learning phase, goal selection
3

---

was forced to ensure participants experienced each goal at least twice. In the final testing trials, participants were prompted to try and make each potion 4 times without feedback.
Potions could either be made of basic ingredients ("simple" and "complex" goals GI-G4) or potions of different colors ("compound" goals G5-G6) and required either two (simple and compound goals GI-G2, G5-G6) or four actions to complete (complex goals G3-G4; Figure [ bottom). Two potions could be made from either basic ingredients or other potions (potions for G3-G6). One complex goal ("hierarchical". G3) was made by combining the ingredients of the two simple potions (GI, G2), whereas the other complex goal was not mooonhiiarrchica"" G4). One complex goal ("hierarchical", G5) was made by flasks of the two simple potions (G1, G2). whereas the other complex goal was not "hoonhierarcchicl" G6). Thus, G3 and G4, and G5 and G6 were matched in terms of actions required to create the corresponding potions but differed in whether knowledge from other potions could be used to find the corresponding solutions faster. Two additional, simple goals made of the first two and last two ingredients of G4 were presented to participants in the testing phase (G7 and G8; Appendix B). Unless otherwise specified, we focus on goals G1-G6, which participants engaged with in the training and learning phases. Further details about the goal space are provided in Appendix B was never shown to participants. The correspondence between goal identity, color, position on the screen, and solution was pseudo-randomized across participants.
The information displayed in Figure [
Various features make the experimental design well-suited to study the contributions of LLP to goal selection by helping decorrelate LP from LLP. First, the correct response was static and signaled through deterministic feedback. While the standard LP measure only flattens upon repeated subse- quent attempts, deterministic feedback enables L.P to reach its limit after a single correct response. Second, the action space is limited and solutions are unique, so negative feedback can be used to exclude the attempted action sequence from the set of possible solutions. This feature was especially meaningful for studying LLP, which can be immediately updated following failed attempts with novel action sequence. Third, the choice space is vast enough that rewards are sparse early on in learning, which further decorrelates LP and LLP since LP is until the first successful attempt, while LLP is updated at the first trial no matter the outcome. Fourth, hierarchical relationships exist among some, but not all goals. Upon becoming aware of hierarchical relationships, participants could take shortcuts to identify the correct recipe for the goals involved, leading to sharp changes in LLP even with small differences in performance. This feature additionally enables us to test for hicrarchy effects on goal selection beyond the indirect effects of hierarchy on learning through LLP..
3.2 Computational modeling
On each trial, participants first selected goal and then sequence of actions. Unless otherwise specified, we only modeled goal selection, conditioning on true participants actions within each trial after goal selection. However, action selection influences performance, LP, and LLP. As in existing studies [23], we model human goal selection as a multi-arm bandit problem, where the probability of selecting a goal depends on its subjective value relative other goals. Multiple factors f may jointly contribute to the overall subjective value of goal [23 24]. which is thus calculated as a weighted sum of the goal's value relative to each factor Vi times its weight 3,. The probability P'(g") ) of choosing goal g" among possible goals G on trial t is obtained through a softmax function over the goal values (as is standard in the decision-makin literature [6)):
E, 3,- (g()))
pt P'(g*) = Eyec expl (,, By V7(9))
Subjective goal values are updated as a function of experience through the delta rule [87] (²+
Vi('") + -++'""">v;'">>>>> 6)((")
where a is a learning rate for value updates and 5, is factor-dependent Models differed in which factor, or combination of factors, composed the overall goal value. Below we detail the individual
factors we considered.
Performance On each trial, the utility of the selected goal with respect to performance is updated based on the goal-contingent feedback received that trial pe (1 for positive feedback, 0 for negative feedback):
performance (g) - Vif Vperformance (g")
Performance estimates were initialized as V'O perffommance (g*) = 1 where 6 is the number of goals.

---

Learning progress Learning progress (LP) was updated through the change in reward prediction error for the current goal after initializing (g") ) = (see [44 for similar approach). "Reward", here refers to the feedback r participants received on each trial. However, note that no external rewards were delivered for successfully obtaining the desired potion (Appendix (C. On each trial, the reward prediction error was calculated as the difference between the current performance value estimate and the feedback obtained. The reward prediction error on trial t was compared to the reward prediction error on the previous trial 1:
('((g") [(r*
V performance (g")) (rr-1
V performance (g"))) - VVLr(g")
Latent learning progress Goal utilities with respect to latent learning progress (LLP) were updated based on the difference between the estimate at trial and the current trial's LLP:
"'L((")) LLP g* (LL
with (g)) = 1. Until the goal was learned (when LLP immediately became 0) subjective (unobservable) estimates of LLP were approximated as LLP, = 1 Naciio anqueeen loand
The number
Nyvedhte oxparsons
of possible action sequences is 12 for 2-action goals and 24 for 4-action goals. The number of action sequences tested is the sum of unique action sequences the agent has tried up until trial I. A goal considered learned when the subject has obtained positive feedback for the current goal least once (but see Appendix D for alternative heuristics and approximations). Therefore, this formulation of LLP captures the size of the space left to explore find the correct sequence of actions for the current goal, i.e. the "distance" from the solution. As result, negative 3LLP values indicate a high willingness to pursue activities one is close to solving. This choice of formulation rather than its opposite, i.e. one where high values indicate high levels of progress toward perfect performance was made to maintain continuity in the LLP function, which is after solution is found.
Hierarchy Hierarchical relationships between goals could impact goal selection in at least two ways. First, indirectly, by enabling inferences about the solution for goals involved in the hierarchy, thus speeding up exploration and changes in LLP. Second, directly, by acting on biases people might have for selecting goals that share common structures. To account for the latter, we also included hierarchy factor, whose estimates were updated via:
Mmerrrchy (g)) Hg- Vineeurchy(()) (g")
with Vr nicrarchy (g") 0 and H equal to 1 for hierarchical (G3, G5), 11 for non-hierarchical (G4, G6), and 0 for simple goals (G1, G2: but see Appendix D for an alternative scheme and update rule).
Model space Because performance is powerful motivator in people's decisions about time allocation 23
45 we include it in all our models. In addition to a performance-onl model, we fit and compared models with additional factors for LP, LLP, hierarchy, and both LLP and hierarchy. Thus, fit parameters included shared a across factors and weighting parameters 3 for each factor in the model. Adding control variable for choice perseveration improved the likelihood of goal choices but did not affect modeling results (Appendix D) so we removed to facilitate interpretation.
Model fitting and comparison Models were fit and compared through hierarchical Bayesian inference (HBI; (88)) based on the likelihood of participants' goal selection choices [18 (8). We compare models through protected exceedance probability (PXP), i.e., the probability that each model the most frequently expressed in the studied population after accounting for chance [89]. and average responsibility (R), which captures how well each model explains participants behavior. Models and parameters for the winning model were recoverable [90] (Appendix [E). Simulating participants' behavior [9I] also required modeling the learning process, which we assumed followed a reinforcement learning architecture with options 92 (Appendix F).
Behavioral results
Given the novelty of our experimental paradigm, it was crucial to validate the environment and gather intuitions about participants' behavior. Furthermore, both LP and LLP are influenced by how well participants learn to achieve each goal. We thus first analyze group-level learning and goal selection as a function of goals' difficulty levels and hierarchical structures. We then test the specific role of LLP in goal selection through computational modeling (Section 5)
5

---

Participants learn successful goal-conditioned policies Learning phase performance was above chance for each goal (Wilcoxon tests were used place of parametric t-tests since the data was not normally distributed according to Shapiro-Will tests of normality; chance levels were 0.083 for 2-action and 0.042 for 4-action goals; all average performances Σ 0.29, all W(174) 2 160, all p S 0.004; Figure A). As expected, performance was higher for 2-action (M 0.51 + 0.02) than 4-action goals (M 0.33 + 0.02, W(174)= = 1207, p < 0.001, r = 0.51). Similar patterns for goals GI-G6 were observed in the testing phase, where performance was not contingent goal selection (Figure B.
Hierarchical inference supports learning If individuals recognized hierarchical relationships during learning, they could use them to draw inferences about the correct recipes for the potions involved. For instance, upon discovering that G5 could be made from the potions created in G1 and G2, participants may correctly infer that G3 is achieved by chaining the recipes for G1 and G2 (Figure I bottom). Consistent with the hypothesis that least some participants exploited the environment's hierarchical structure. learning performance was higher for hierarchical goals compared to their non-hierarchical counterparts (complex: hierarchical (G3) M 0.37 + 0.02, non-hierarchical (G4) = + 0.02, W(174) = 4168.5, p 0.004, r = 0. compound: hierarchical (G5) M 0.53 + 0.02, non-hierarchical (G6) M 0.49 + 0.02, W(174) = 6036.5, p = 0.032, r = 0.11; Figure ( Similarly, participants took fewer attempts to learn solutions to hierarchical than non-hierarchical
goals (complex: hierarchical (G3) M 16.66 = 1.19, non-hierarchical (G4) M 22.1 = 1.23, W(120)
1884.5, p= 0,009, r= 0.17; compound: hierarchical (G5): M = 11.66 + 0.58, non-hierarchical (G6) M 13.21 = 0.53, W(155) = 3889.5, p 0.03, r = 0.1 Figure Fig However, hierarchy effects were not present in the testing phase, likely due ceiling effects (complex: hierarchical (G3) M = 0.56 - 0.03, non-hieranee non-hicrarchical (G4) M = 0.52 = 0.03. W(174) = 1592.5, p = 0.223, r= 0.07; compound:
hierarchical (G5) M 0.73 = 0.03, non-hierarchical (G6) M = 0.73 + 0.03, W(174) 1158.5. p
0.765, r = 0.02; Figure 2 3).
Goal selection is sensitive to performance and hierarchy In assessing whether difficulty levels and hierarchy affected participants' goal selection, we found it displayed opposite patterns to learning performance (Figure The average probability of selecting goal was lower for 2-action (M- 0.15 + 0.00) than 4-action goals (M = 0.19 + 0.01; W(174)= = 4214, p < 0.001, r = 0.27). Overall, participants chose the hierarchical (G3; M 0.18 + 0.01) less than the non-hierarchical complex goal (G4; M = 0.21 + 0.01, W(174) = 5231.5, p 0.002, r= 0.17). Differences between hierarchical (G5; M 0.15 + 0.01) and non non-hierarchical compound goal selection were not significant (G6; M = 0.16 + 0.00, W(174) = 6444, p = 0.128, r = 0.08). At the individual level, the greater the
10-
10-
10-
thh beating shhaee
0.
0.5
N.S.
06
Proence
0.6
PIDW ROOoee
04
0.4
02
2.02
0.
D
Goal
#kmaxihe adected
0.5
0.0
Paramatee estmee
03
N.S.
08
Ovecal
Heeady
-2.5
0.75
04
08
0:50
0.4
-7.5
0.25
0.2
0.0
Paerenaco Rgy
0.00
Partccpont
Figure 2: Behavioral and modeling results. (A) Learning performance above chance and showed hierarchy effects. (B) Test performance was better than chance, but no significant hierarchy effects were detected. (C) On average, fewer attempts were needed to learn hierarchical (G3) compared to non-hierarchical (G4) 4-action goals. (D) Partial hierarchy effects are present in goal selection. The winning model (triangle markers) reproduces this pattern. (E) Model responsibilities for individual participants and the overall studied population. model. Bars and shading indicate the SEM, dots individual participants. *** p < 0.001, ** p < 0.01
(F) Best-fitting parameters (HBI) for the winning

---

difference in performance between 2-action and 4-action goals, the lower the probability of selecting a 2-action goal (Spearman's p= -0.35. < 0.001), and the greater the performance difference between hierarchical and non-hierarchical goals, the lower the probability of selecting hierarchical goal (complex: p = -0.68, p 0.001; compound: == -0.51, < 0.001). These results indicate that goal selection depends performance and hierarchical relationships between goals. We next turn to computational modeling to detail the contributions of performance, hierarchy, LP, and LLP to goal selection.
5 Modeling results
We that predicted L.P supports human goal selection, thus expecting models relying on this signal to fit participants' goal selection data better than models that do not.
Latent learning progress guides goal selection The model with performance and LLP utility factors was the most strongly represented across participants (PXP = 1; R = 0.63) and replicated the main goal selection patterns seen in participants' data (Figure 2 Some participants were also well-captured by model that additionally integrated hierarchy into goal values (R = 0.22). We note that hierarchy likely plays a more direct role learning (thereby impacting LLP) than in goal selection, given that hierarchy effects in goal selection and performance could be recovered after eliminating hierarchical components from goal selection, but not learning (Appendix GG. The model selecting goals based on performance and LP did not account well for participants behavior (R- 0.12). Models with just performance (R 0.00) or performance and hierarchy factors (R = 0.04) were marginally represented. The winning model's 's best-fitting Ppertoemance and ŠLLP were on average negative, suggesting most individuals focused on activities where performance was low - indicating the solution had yet to be found and where LLP was low (Figure 2F) indicating learning potential was high (Section 5). However, more nuanced patterns and temporal dynamics may exist at the individual level.
6 Inter-individual differences in learning and goal selection
Several interesting strategies emerged upon inspecting individual behaviors (Figure 3). We first exemplify them before using data-driven approach to classify participants according to their approach to the game, revealing differences in how hierarchy impacts learning and thus goal selection. Finally, we assess whether LLP supports goal selection across groups.
People express variety of goal and action selection strategies Many participants repeatedly selected the same goal often testing possible solutions in systematic order (Figure 3 only switching to a new goal after learning the current one (Figure A). After learning to make all potions, some participants began alternating goals, if testing or training their ability retain multiple solutions (Figure BA. Although most participants showed interest in learning all goals (Figure JABB) small subset of individuals preferred setting few objectives, even a single goal, in which they excelled (Figure (P)), while disregarding others. Lastly, some participants moved across goals and ingredients in seemingly umprincipled manner, presumably because they were confused or unmotivated (Figure 3D)) A subset of the participants seemed to be aware of adopted strategies (see Appendix for example verbal reports). We next clustered participants explore how idiosyncrasies in learning related to goal selection. Note that participants in Figure were chosen as examples and do not directly relate to the clustering analyses presented below.
01
Figure 3: Example behaviors from four participants. (A-D) Chosen action sequence as function of trial number (training and learning phases), color-coded by goal and feedback.
7

---

Clustering of individual strategies reveals differences in hierarchical learning We used K- means clustering [93] to identify 4 groups of participants, where K was determined through the elbow method (Appendix [] The behavioral metrics used to identify the clusters (e.g.. performance and goal selection entropy and distributions) can be found in Appendix ! To avoid statistical double- dipping [94], we limit ourselves to qualitative observations on cluster-based behavior. To better illustrate salient characteristics of each cluster, we also identified "archetypal" participants through archetypal analysis a method, often used to profile video game players [95]. which detects data points that can reproduce the original population when linearly recombined [96]. We then matched each archetypal feature with the best-aligned cluster (Appendix [I] and identified the participant closest to the corresponding archetype as an extreme example of the cluster it belonged to (Figure MM M-P).
Participants in cluster 1 (N 39) had below-average performance at learning (M= = 0.18 = 0.03) and testing (M = 0.27 + 0.06) and favored 2-action goal. They often failed to learn or retain solutions and switched goals frequently (Figure[MM In the largest cluster 2 (N = 75: Figure 33) performance was excellent at learning (M 0.59 + 0.0) and testing (M 0.86 = 0.03). This group leveraged hierarchical structures to speed up learning (Figure 4B.F) which also impacted goal selection. Non-hierarchical goals were favored (Figure ) and often learned later than hierarchical ones (Figure 4N). Cluster participants (N 53) showed relatively high performance at learning (M = 0.26 = 0.04) and testing 0 = 0.06; (Figure 4C), but seemed to struggle with all 4-action goals performing worse in them but choosing them more- and took less advantage of hierarchical relationships across goals (Figure&G..K.O). Given its small size (N = 8), we invite caution in interpreting the results for cluster 4. However, this group showed an interesting, strong tendency repeatedly select the same, small set of goals at which they succeeded (often preferring 2-action goals; Figure .,P). As a result, their performance was excellent in the learning phase, where goals were freely chosen (M- = 0.67 + 0.13, F.,L but dropped in the test phase, where goals were imposed (M = 0.46 + 0.16).
Despite such distinct patterns of behaviors, LLP was a central component of goal selection in all clusters (Figure Q-TT) suggesting LLP is a flexible signal that adapts to idiosyncrasies in people's learning while similarly guiding goal selection (but see Appendix G for further discussion of how learning interacts with goal selection). At the same time, the best-fit parameters of each cluster captured specific properties of the different groups (Figure 4Q-T insets). The efficient learners of cluster 2 relied more strongly on LLP (cluster 2 M = -3.17 = 0.18, other clusters M -2.46 0.24) and updated both performance- and LLP-based goal values faster than other groups (cluster 2 = 0.73 + 0..03, cluster 2 = 0.6 + 0.03, U(173) = 4979, p 0.001, r 0.28). Cluster 3, which seemed less likely rely on hierarchy, was more attuned to performance when selecting goals (cluster 3 MMSgsstteeeee = -4.63 + 0.22, other clusters M M Aafoomunc = -3.67 + 0.16, U(173) = 2274, p = 0.002, = 0.24)). Cluster which lacked precise strategies, was less sensitive to both performance (cluster M pattomess = -2.93 + 0.26, other clusters and LLP (cluster 1 M Myestomanc = -1.7 + 0.16, other clusters M Sgattmmence -3.06 + 0.14, U(173) = 3980, p < 0.001, r 0.36).
+ 0.16, Mann- Mann-Whitney U(1 U(173)= 2696, p 0.001,
M @yestmmancs -4.25 + 0.14, U(173) = 3778. p <
7 Discussion
Human behavior is often motivated by self-defined challenges. Despite the centrality of goals in human learning 19. how people select their own objectives without external guidance or incentives remains to be specified. Following advances in artificial open-ended skill learning (26428), learning progress (LP) has emerged as useful signal for human goal selection 23 24 Here, we introduced a "latent" form of L. (LLP), which is informed by an agent's model of the environment and memory of the actions pursued in it, in addition to the manifest changes in performance tracked by standard LP. In purposefully developed hicrarchical reinforcement learning task, we find that people's 's goal selection is driven by LLP, calling for an extension of the LP notion to include its latent variants.
Inter-goal relationships in our test environment enabled us to check whether hierarchy additionally drives goal selection, as intuition suggests (e.g., one might decide to become competent in baking by first relying on meady-made dough and only later setting the lower-level goal of making dough from scratch). We find that hierarchy impacts goal selection indirectly by enabling inferences and thus affecting I.P.. That is, participants' learning is sensitive to hierarchical structures, leading to changes in L.P which, in turn, affect goal selection. We also gathered partial evidence for a standalone role of hierarchy in curriculum development, which further studies may address.
8

---

Clusters, S. MHO
Clustr NOTS
Cuester N169
Cluster Nn
H
K
Figure 4: Behavioral and modeling results by cluster (1-4, left to right). (A-D) Learning performance. (E-H) Probability that each goal was learned over the number of times was selected. (I-L) Goal selection probabilities. (M-P) Behavior of each cluster's closest-matching archetypal participant. (Q-T) Model responsibilities and best-fitting parameters.
We found marked inter-individual differences in strategies for goal setting and pursuit. However, our sample was restricted relatively homogeneous group of undergraduate university students. Future studies and computational models may extend participation to broader subject pool and specifically address individual differences goal selection and achievement, as well as their relationship with demographics and cultural background, cognitive abilities, and psychopathology (cf. (97). which might inform personalized teaching strategies and productivity tools. As our knowledge of human goal setting becomes more precise, however, ethical concerns regarding the use of behavioral sciences marketing and management should be addressed, particularly in cases where highly personalized methods of influence could transform advertising into manipulation.
Elucidating the computational mechanisms driving human goal selection may propel advances in open-ended machine learning. Our suggestion, wherever relevant, is to incorporate LLP in artificial autotelic agents' goal selection. Compared to classic LP, LLP is sensitive to latent changes in an agent's ' abilities which LP would only be able to capture after repeatedly observing positive feedback. Using LLP could result in faster adaptation of the agent's goal selection, ultimately speeding up its learning by enabling it to detect progress before receiving the first reward and letting proceed to more challenging tasks sooner.
Some limitations of the present study may need to be addressed before other fields can fully benefit from the results we presented. In our task, people chose goals from predefined menu of options. Although this facilitates the study of goal setting, people often invent their own goals by combining observations and imagination (98). For simplicity, our modeling focused on goal selection. Future research, however, should jointly model goal choices and goal- condiiinit action selection, since as our data suggests learning participates in complex interactions with the goal selection process. While we provide a simple and task-specific formalization of LLP, generalized definition is necessary to understand the differences between LLP and other intrinsic motivation signals and ease the implementation of L.LP-baee goal selection in autotelic machines. By providing initial evidence for I.I.,, we hope to inspire the establishment of even more precise signals for goal selection in both humans and artificial agents.

---

Acknowledgments
In producing this work, G.M. was supported by the Chateaubriand Fellowship of the Office for Science & Technology of the Embassy of France in the United States; P-Y.O. was supported by ANR Al Chair DeepCuriosity, A.G.E.C. was supported by NIH Grant 100M-119988300)
References
[1] Berlyne, D.E. (1960) Conflict, arousal, and curiosity. McGraw-Hill Book Company
[2] Blain, B. and Sharot, T. (2021) Intrinsic reward: potential cognitive and neural mechanisms. Curr. Opin. Behav. Sci 39, 113-118
[3] Chew, B. et al. (2021) A neurocomputational model for intrinsic reward. J. Neurosci, 41, 8963-8971
[4] Gottlicb, J. and Oudeyer. P.Y. (2018) Towards a neuroscience of active sampling and curiosity. Nat. Rev. Neurosci. 19, 758-770
[5] Kidd, and Hayden, B. Y. (2015) The psychology and neuroscience curiosity. Neuron 88, 449-464 460
[6] Harlow, H.F. (1950) Learning and satiation of response in intrinsically motivated complex puzzle performance by monkeys. J. Comp. Physiol. Psychol. 43, 289-294
[7] Ryan, and Deci, (2000) Intrinsic and extrinsic motivations: Classic definitions and new directions. Contemp. Educ. Psychol. 25, 57
[8] White, R.W. (1959) Motivation reconsidered: the concept of competence. Psychological Review 66, 297
[9] Barto, A.G. et al. (2004) Intrinsically motivated learning of hierarchical collections of skills. In ICDL
[10] Oudeyer, P.Y. and Smith, L.B. (2016) How evolution may work through curiosity-driven developmental process. Top. Cogn. Sei. 8, 192-502
[11] Singh, et al. (2010) Intrinsically motivated reinforcement learning: An evolutionary per- spective. IEEE Trans. Auton. Mental Develop. 2, 70-82
[12] Baldassarre, G. and Mirolli, M., eds. (2013) Intrinsically Motivated Learning in Natural and Artificial Systems. Springer Berlin Heidelberg
[13] Bellemare, M. al. (2016) Unifying count-based exploration and intrinsic motivation. In NeurlPS, vol. 29, Curran Associates, Inc.
[14] Burda, Y. et al. (2019) Large-scale study of curiosity-driven learning. In ICLR
[15] Eysenbach, B. al. (2018). Diversity is all you need: Learning skills without reward function. arXiv.
[16] Pathak, D. al. (2017) Curiosity-driven exploration by self-supervised prediction. In ICML, pp. 2778-2787. PMLR
[17] Schmidhuber. J. (2010) Formal theory of creativity, fun, and intrinsic motivation (1990-2010). IEEE T. Auton. Ment. De. 2, 230-247
[18] McDougle, S.D. et al. (2022) Executive function assigns value to novel goal-congruent out- comes. Cereb. Cortex 32, 231-247
[19] Molinaro, G. and Collins, A.G. (2023) A goal-centric outlook on learning. Trends. Cogn. Sci.
[20] O'Reilly, R.C. (2020) Unraveling the mysteries of motivation. Trends Cogn. Sci. 24, 425 425-434
10

---

[21] Juechems, K. and Summerfield, C. (2019) Where does value come from? Trends Cogn. Sci. 23, 236-850
[22] Colas, C. et al. (2022) Autotelic agents with Intrinsically motivated goal-conditioned reinforce- ment learning: A short survey. J. Artif. Intell. Res. 74, 1159-1199
[23] Ten, A. et al. (2021) Humans monitor learning progress in curiosity-driven exploration. Nat. Commun. 12, 5972
[24] Poli, F. al. (2022) Contributions of expected learning progress and perceptual novelty to curiosity-driven exploration. Cognition 225, 195119
[25] Forestier, S. et al. (2022). Intrinsicallyy motivated goal exploration processes with automatic http://......r........66/2/..... 17t.5///2XXiv..080021555
curriculum learning. arXiv. b 4855:/ atr//.
[26] Akakzia, A. et al. (2020). Grounding language to autonomously-acquired skills via goal generation, arXiv. https::/arxivvorgg. wtt.://gg/bbbboo..//o// 07185
[27] Blaes, S. et al. (2019) Control what you can: Intrinsically motivated task sal--ilannin agent. NeurIPS
[28] Colas, C. et al. (2019). CURIOUS: Intrinsically motivated modular multi-goal reinforcement learning. arXiv. /ttp://arxiv org/ abs/ 08110066280
[29] Oudeyer, P.Y. and Kaplan, F. (2007) What is intrinsic motivation? typology of computational approaches. Front. Neurorob. 1, 108
[30] Lopes, M. and Oudeyer, P.Y. (2012) The strategic student approach for life-long exploration and learning. In ICDL, pp. 1-8.
1ttp:///.0..10///errrrr 1w110109/DeeL...mm 2012.6400828 6400807
[31] Rayyes, R. (2023) Intrinsic motivation learning for real robot applications. Front. Robot. Al 10, 1102438
[32] Santucci, V.G. et al. (2013) Which is the best intrinsic motivation signal for learning multiple skills? Front. Neurorobot. 7,22
[33] Bromberg-Martin, E.S. and Hikosaka, O. (2009) Midbrain dopamine neurons signal preference for advance information about upcoming rewards. Neuron 63, 119-126
[34] Charpentier, C.J. et (2018) Valuation knowledge and ignorance in mesolimbic reward circuitry. PNAS 115, E7255-E7264
[35] Golman, R. and Loewenstein, (2018) Information gaps: A theory of preferences regarding the presence and absence of information. Decision 5, 143
[36] Molinaro, G. et al. (2023) Multifaceted information-sseking motives in children. Nat. Commun. 14, 5505
[37] Sharot, T. and Sunstein, C.R. (2020) How people decide what they want to know. Nature Human Behaviour 4, 14-19
[38] Mazzaglia, P. et al. (2022) Curiosity-driven exploration via latent bayesian surprise. In AAAI, vol. 36, pp. 7752-7760
[39] Poupart, P. et al. (2006) An analytic solution to discrete bayesian reinforcement learning. In ICML, pp. 697-704
[40] Linke, et (2020) Adapting behavior via Intrinsic reward: survey and empirical study.
J. Artif. Intell. Res. 69, 1287-1332
[41] Kidd, C. et al. (2012) The Goldilocks effect: Human infants allocate attention to visual sequences that are neither too simple nor too complex. PloS One 7, e36399
[42] Geana, A. et al. (2016) Boredom, information-seckiin and exploration. In CogSei, pp. 1751-6
11

---

[43] Schmidhuber, J. (1991) A possibility for implementing curiosity and boredom in model- building neural controllers. In From Animals Animats (Meyer. J.A. and Wilson, W.W, eds.). MIT Press
[44] Schmidhuber, J. (2013) Powerplay: Training Increasingly general problem solver by continually searching for the simplest still unsolvable problem. Front. Psychol. 4
[45] Baranes, A.F. et al. (2014) The effects of task difficulty, novelty and the size of the search space on intrinsically motivated exploration. Front. Neurosci. 8
[46]
Csikszentmihalli, M. (2009) Flow: The psychology of optimal experience. Harper and Row
[47] Elliot, A.J. and Fryer, J.W. (2008) The goal construct in psychology. In Handbook of motivation science (Shah, Y.J. and Gardner, W.L., eds.), pp. 235-250. The Guilford Press
[48] Carver, C.S. and Scheier, M.F. (1998) On the Belf-Regulation of Behavior. Cambridge University Press
[49] Hull, C.L. (1934) The rat's speed-of- of-locomotion gradient in the approach to food. J. Comp.
Psychol. 17, 422
[50] Miller, G.A. et al. (1960) The integration of plans. Henry Holt and Co
[51] Powers, W.T. (1973) Feedback: Beyond behaviorism: Stimulus-response laws are wholly predictable within control-system model of behavioral organization. Science 179, 356-356
[52] Gollwitzer, P.M. (2018) The goal concept: helpful tool for theory development and testing
in motivation science. Moriv. Sci. 4, 185-205
[53) Locke, E.A. and Latham, G.P. (2002) Building practically useful theory of goal setting and task motivation: A 35-year odyssey. Am. Psychol. 57, 705-717
[54] Karayanni, M. and Nelken, I. (2022) Extrinsic rewards, intrinsic rewards, and non-optimal behavior. J. Comput. Neurosci. 50, 139-143
[55] Amir, N. et al. (2023). States as goal-directed concepts: an epistemic approach to state- representation learning. arXiv. htttps://arxiv.org// abs//312.222222. 02367
[56] Castegnetti, G. et al. (2021) How usefulness shapes neural representations during goal-directed behavior. Sci. Adv. 7, eabd5363
[57] Frömer, R. al. (2019) Goal congruency dominates reward value in accounting for behavioral and neural correlates of value-based decision-making. Nat. Commun. 10, 4926
[58] Molinaro, G. and Collins, A.G. (2023) Intrinsic rewards explain context-sensitive valuation in reinforcement learning. PLoS Biology 21, e3002201
[59] Sepulveda, P. et al. (2020) Visual attention modulates the integration of goal-relevant evidence and not value. eLife 9, c60705
[60] Huang, S.c. (2023) The Temporal Dynamics of Goal Systems: How Goal Progress Changes the Drivers of Motivation. In Goal Systems Theory: Psychological Processes and Applications. Oxford University Press
[61] Metcalfe, J. (2002) Is study time allocated selectively to region of proximal learning? J. Exp. Psychol. Gen. 131,349
[62] Chaiklin, S. (2003) The zone of proximal development in vygotsky's analysis of learning and Instruction. In Vygotsky's Educational Theory in Cultural Context (Kozulin, A. et al., eds.), Learning in Doing: Social, Cognitive and Computational Perspectives, pp. 39-64. Cambridge University Press
[63] Guadagnoli, M.A. and Lee, T.D. (2004) Challenge point: A framework for conceptualizing the effects of various practice conditions in motor learning. J. Mot. Behav. 36, 212-224
12

---

[64] Botvinick, M. et al. (2017) Building machines that learn and think for themselves. Behav. Brain Sci. 40, e255
[65] Oudeyer, P.Y. (2017) Autonomous development and learning in artificial intelligence and robotics: Scaling up deep learning to human-like learning. BBS 40, e275
[66] Silver, D. al. (2016) Mastering the game of go with deep neural networks and tree search. Nature 529, 484-489
[67]
Sutton, R.S. and Barto, A.G. (2018) Reinforcement Learning: An Introduction. MIT Press
[68] Turing, A. (1950) Computing machinery and intelligence. Mind 59, 433-460
[69]
Nair, A. et al. (2020) Contextual imagined goals for self-supervised robotic learning, In CoRL, pp. 530-539
[70] Schaul, T. et al. (2015) Universal value function approximators. In ICML, pp- 1312-1320. PMLR
[71] Andrychowicz, M. et al. (2017) Hindsight experience replay. In Advances in Neural Informa- tion Processing Systems, vol. 30. Curran Associates, Inc.
[72] Laversanne-Finot, A. al. (2021) Intrinsically motivated exploration of learned goal spaces. Front. Neurorob. 14
[73] Pong, V.H. et al. (2019). Skew-fit: State-covering self-supervised reinforcement learning. arXiv. https://arxis www.ariiir...o.oeooooe.oe ort/ abs//90//191..// 03698
[74] Pitis, S. et al. (2020) Maximum entropy gain exploration for long horizon multi-goal reinforce- ment learning. In ICML, pp. 7750-7761
[75] Portelas, R. et al. (2020) Automatic curriculum learning for deep rl: short survey. In IJCAI, pp. 4819-4825 4825
[76] Campero, A. et al. Learning with amigo: Adversarially motivated intrinsic goals. arXiv. /a//a/ba//000 12122
/ttps://arxiv..
[77] Portelas, R. et al. (2020) Teacher algorithms for curriculum learning of deep rl in continuously parameterized environments. In CoRL, pp. 835-853
[78] Sukhbaatar, S. et al. (2017). Intrinsic motivation and automatic curricula via asymmetric self-play. arXiv. htttps///axxv..ooo
[79] Bagaria, A. et al. (2023). Scaling goal-based exploration via pruning proto-goals. arXiv. 200.////202..493333
[80] Baranes, A. and Oudeyer, P.Y. (2013) Active learning of inverse models with intrinsically motivated goal exploration robots. Rob. Auton. Syst. 61, 49-73
[81] Forestier, S. and Oudeyer, P.Y. (2016) Modular active -urioe discovery of tool use. In IROS, pp. 3965-3972
[82] Fournier, al. (2018). Accuracy-based curriculum learning in deep reinforcement learning. arXiv. https://axxiv.rrrg
09614
[83] Florensa, et. (2018). Automatic goal generation for reinforcement learning agents. arXiv. 06366
https://arzivvorr/abbes
[84] Moulin-Frier, C. et al. (2014) Self-organization of early vocal development in infants and machines: the role of intrinsic motivation. Frontiers in psychology 4, 1006
[85] Santucci, V.G. al. (2016) GRAIL: goal-discovering robotic architecture for Intrinsically- motivated learning. IEEE Trans. Cogn. Develop. Syst. 8, 214-231
[86] Luce, R.D. (1959) On the possible psychophysical laws. Psychol. Rev. 66. 81
13

---

[87] Wagner, A.R. and Rescorla, R.A. (1972) Inhibition in pavlovian conditioning: Application of theory. Inhibition and learning, pp. 301-336. Erlbaum
[88] Piray, P. et al. (2019) Hierarchical bayesian inference for concurrent model fitting and compar- ison for group studies. PLoS Comput. Biol. 15, e1007043
[89] Rigoux, L. et al. (2014) Bayesian model selection for group studies rtvdies-rrvisited. Neuroimage 84, 971-985
[90] Wilson, R.C. and Collins, A.G. (2019) Ten simple rules for the computational modeling of behavioral data, eLife 8, c49547
[91] Palminteri, S. et al. (2017) The importance falsification in computational cognitive modeling. Trends. Cogn. Sci. 21, 425-433
[92] Sutton, R.S. et al. (1999) Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence 112, 181-211
[93] Kaufman, L. and Rousseeuw, P.J. (2009) Finding groups in data: an introduction to cluster analysis. John Wiley & Sons
[94] Ball, T.M. al. (2020) Double dipping in machine learning: problems and solutions. Biol. Psychiatry Cogn. Neurosci. Neuroinaging 5, 261
[95] Drachen, A. et al. (2016) Guns and guardians: Comparative cluster analysis and behavioral profiling in destiny. CIG, pp. 1-8
[96] Cutler, A. and Breiman, L. (1994) Archetypal analysis. Technometrics 36, 338-347
[97] Blain, B. et al. (2023) Sensitivity to intrinsic rewards is domain general and related to mental health. Nat. Mental Health. 1, 679-691
[98] Colas, C. al. (2020). Language cognitive tool to imagine goals in curiosity-driven exploration. arXiv. https://arxiv http:///rxvvvor/g//o////// ///aabb/220000 09253
[99] Sugawara, M. and Katahira, K. (2021) Dissociation between asymmetric value updating and perseverance in human reinforcement learning. Sci. Rep. 11, 3574
[100] Smith, A.C. et al. (2004) Dynamic analysis of learning behavioral experiments. J. Neurosci 24, 447-461
[101] Rmus, M. et al. (2023). Artificial neural networks for model identification and parameter esti- mation in computational cognitive models. Biorxiv. http:://wwwwb..xx.oooooooo// 10. 111111121.0.../ 09. 14. 557793v2
[102] Motevalli Soumehsaraei, B. and Barnard, A. (2019). Archetypal analysis package. CSIRO. http:////ob..orr//.002//////////
14

---

Appendix overview
To help the reader, we provide an overview of the contents covered by the ensuing appendices:
A. Participants
Information about the human participants who took part in the experiment. B. Experimental paradigm details; Detailed information about the protocol and the learning paradigm's 's environment.
C. Experiment instructions Instructions provided to participants the start of the experiment.
D. Modeling robustness checks, Supplementary analyses verify the robustness of our model- ing results in the face of small modifications to the model designs.
E. Parameter and model recoverability Results of the parameter and model recoverability
studies.
F. Options reinforcement learning model for simulations Description of the design and use of models with a learning component (used for simulations only).
G. The role of hierarchy in learning and goal selection Results and discussion of some evidence for role of hierarchy learning in the context of the present experimental paradigm, and how it relates to goal selection.
H. Verbal reports of example participants strategies Example verbal reports from participants about their adopted learning strategies.
I. Clustering methods]
Details about the clustering methods.
A Participants
One hundred and seventy-nine participants were recruited through the University of California, Berke- ley, Research Participation Program pool of participants and received course credit compensation for their time. Participants completed the experiment online from their own devices. We excluded four participants who either responded affirmatively when directly asked whether they took written notes about the correct recipe for each potion, or mentioned writing down solutions on their phone or piece of paper when reporting their strategies in follow-up questionnaire. Thus, 175 participants (76% female, age: range = 18-47, M = 21.21 = 3.75, 94% right- were included in the final data set. All were native or fluent English speakers with no medical history of brain injury, mental/psychiatrii illness, or drug/alcohol abuse. The Institutional Review Board at the University of California, Berkeley, approved the study.
B
Experimental paradigm details
B.1
Gameplay
The paradigm was divided into training, learning, and testing phases. Each learning phase trial encompassed goal selection, action selection, and feedback stage. In the goal selection stage, participants viewed a blackboard depicting six "goal" potions, characterized by their color. Next to each potion were either two or four empty slots indicating the number of ingredients required to make the potion - and illustrations of the four different ingredients available for that potion. Potions could either be made of basic ingredients (a frog, bunch of mushrooms, horseshoe, and a butterfly), or potions of different colors (e.g., teal, orange, red, and blue-colored potions). Two potions appeared twice on the left side of the board, as they could be made from either basic ingredients or other potions. Participants were prompted to select one of the available potions on the left side of the blackboard with mouse click (Figure D At the beginning of the action selection stage, the blackboard contents changed to depict the goal potion and either two or four empty slots, which indicated the number of ingredients required. Four different ingredients appeared on separate shelves on the right side of the screen. When a participant clicked on an ingredient, it moved from the shelves to cauldron depicted the center the screen and appeared in the corresponding slot on the blackboard. An empty flask appeared on the table next to the cauldron for the duration of the action selection stage. Participants were prompted to select ingredients, one at time, until all slots were filled (Figure top). During the feedback stage, the flask either remained empty and shook left and right, signaling failure to make
15

---

the goal potion, or became filled with the same color as the goal potion and wiggled up and down, signaling that the chosen ingredients were added to the cauldron in the correct order. No points or otherwise explicit feedback messages were provided.
The learning phase (144 trials) was divided into 6 blocks, at the heginning of which participants were allowed to take a break if needed. The training phase (12 trials, 2 per goal) served to help participants get familiar with the game flow and was similar the learning phase, except that goal selection was forced. This section comprised 12 trials (two per goal), and was intended to help participants get familiar with the game flow. The testing phase was similar to the learning phase, except that goal selection and feedback stages were skipped. Here, participants were instead prompted to make a given potion without receiving information about whether the selected order and ingredients were correct. A set of instructions preceded the gameplay (Appendix (). Importantly, participants were not told to try and learn all the correct mappings between actions and goal achievement. We did not disburse performance bonuses, nor were there any external incentives for successful goal completion. Instead, participants were free to explore the goal space as they pleased for fixed number of trials.
At the end of the experiments, participants completed short demographics questionnaire and optionally answered the questions "Did you try use any strategy while playing the game? If so, what was it?", and "Did you notice any structure in the game? If so, what did you notice?". A subset of participants were also asked, "Did you have specific aims or objectives while playing? If so, what were they?". Participants were also invited to leave written feedback.
B.2
Goal space
The six different goals (potions) available for selection were constructed illustrated in Figure bottom. While we present them in an orderly fashion to ease readability, participants were not told about the underlying structure of the set of goals. The relative position on the screen of goals with different features was pseudorandomized across participants. Below is a description of the six available goals from the training and learning phases:
D
Gl: simple goal 1. This potion was made of two basic ingredients.
G2: simple goal 2. This potion was made of two basic ingredients, none of which were used to make G1.
G3: hierarchical complex goal. This potion was made of four basic ingredients, in an order that reflected the recipe for G1 followed by that for G2. Thus, participants who became aware of the structure could leverage knowledge of G1 and G2 to learn G3, or vice-versa.
G4: non-hierarchical complex goal. Like G3, this potion was made of four basic ingredients. However, the order in which they had be added the cauldron did not reflect G1 or G2 in any way. Thus, learning G1 or G2 did not offer any learning advantage for G4 or vice-versa.
G5: hierarchical compound goal. This potion had the same color as G3. However, potions of different colors were available for making G5, as opposed to the basic ingredients available for G4. The potions required to make G5 correctly were GI followed by G2. Learning this potion could, in theory, be used gain insight into how G3 is made, or vice-versa.
G6: non-hierarchical compound goal. This potion had the same color G4 but, similar to G5, was made of two other potions rather than basic ingredients. These were potions that were not available as goals, such that learning G6 could not leverage provide insight into other potions' recipes. However, participants could infer that the recipes for the first and second potions used to make G6 were the first two and last two ingredients used in G4.
Two additional goals were presented to participants in the testing phase:
G7: simple goal 3. This potion was made of the first two basic ingredients of G4.
* G8: simple goal 4. This potion was made of the last two basic ingredients of G4.
C Experiment instructions
We report the instructions provided to participants verbatim below. Each item corresponds to a screen presented to participants, who paced their own reading by moving to the next screen upon pressing
16

---

the spacebar key on their keyboard. They were instructed to do so by the message "Press SPACEBAR continue" appearing the bottom of each screen. Wherever applicable, images accompanying each item are references in the following list.
6666
K
6444
AAAA
6446
Figure 5: Images accompanying the experiment instructions.
1. Welcome to our study! Thank you very much for deciding to participate.
2. This experiment requires your full attention. We will only be able your data if you participate fully the experiment, which should only take about 35 minutes when you are fully engaged.
3. Before we begin, please maximize your browser window and turn off all notifications (Facebook, Instagram, texts, etc.) on both your phone and computer.
4. Please do not disengage (for example, by switching tabs in your browser navigating away from the task) while you take part in the experiment, including during breaks.
5. If you're not engaged, your data are not valid and cannot used for the experiment, and therefore you may not get credit. Thank you for understanding.
6. In this game, you will play the role of an alchemist and try to make potions from a set of ingredients.
7. On each trial, you will see a blackboard that shows all the potions you can try to make. [Figure 3
8. Note that this is just example setup, and things may look slightly different in the actual game.
9. Each potion is made of either or ingredients. The empty slots next to each potion indicate how many ingredients that potion requires. [Figure 5B]
10. The images shown on the far right of the blackboard illustrate which ingredients you can try and make each potion. [Figure [C]
11. In the game, you will first select a potion by clicking on it. [Figure 5D]
12. Once you have chosen a potion to make, you will select one ingredient at a time from the shelves on the right to try and make it. [Figure SJ
13. Once you click on an ingredient, it gets added to the cauldron. [Figure 5F]
14. The blackboard will help you keep track of which ingredients you have selected and which potion you trying to make. [Figure
17

---

15. Each ingredient can only be selected once, and the order in which they are added matters.
16. If you select the correct ingredients in the correct order, you will see the empty flask on the table fill up with the potion you selected. [Figure 5
17. If the ingredients were incorrect or not chosen in the right order, the flask will remain empty. [Figure [
18. After you have clicked on the required number of ingredients, you will get choose potion again. You are welcome to try and make the same potion again or choose a different one on each trial. [Figure 5J]
19. Some potions can be made two ways. [Figure 5k]
20. Either by combining basic ingredients: [Figure5_...
21. Or by combining other potions: [Figure 5M]
22. Things may seem confusing at first, but the game will become clearer as you keep trying!
23. To help you familiarize yourself with the game, for the first few trials the computer will choose potions for you make.
24. Are you ready? Press SPACEBAR begin.
D Modeling robustness checks
We performed several checks to verify our modeling results were robust to small variations in the adopted methods. To save time and resources in these additional analyses, compare models through exceedance probability (XP) rather than the more computationally expensive PXP [89].
D.1 Adding a perseveration term to all models
Since adding perseveration term can henefit models of human choices 99 we also ran a version of the model comparison pipeline in which each model included such a feature in the goal valuation process. The utility Vp Pesseverattinn grew each time goal was selected the extent determined by a. Adding perseveration to each model did not change the results (winning model XP= 1; R: model with perseveration only 0.00, perseveration and performance 0.00, perseveration, performance, and LP =0.12, perseveration, performance, and LLP= 0.70, perseveration, performance, and hierarchy = 0.06. perseveration, performance, LLP, and hierarchy 0.17). For simplicity and ease of interpretation, we thus opted for models without perseveration.
D.2 Using vs. 3 successful attempts to define when a goal is learned
Our implementation of LLP drops to 0 after participant has learned the solution to a goal, as (assuming perfect memory) no further learning progress can be made once the solution found. Since we did not have access to the true cutoff for when a goal was "learned"" we placed an arbitrary cutoff after the first time the solution was identified. However, participants may require more than single trial to retain goal's correct action sequence. Therefore, we also developed models where LLP dropped to 0 after 3 successful attempts. Using this heuristic instead of the 1 successful attempt heuristic did not change the results (winning model XP I; R: model with performance 0.03, performance and LP = 0.18, performance and LLP = 0.66, performance and hierarchy 0.00, performance, LLP, and hierarchy = 0.13).
D.3 Using Bayesian cutoff to define when goal is learned
To test the reliability of a heuristic method to decide after how many trials goal was "learned" for the purpose of updating LLP, we also used Bayesian cutoff to set more precise threshold. This was calculated by computing confidence intervals over each subject's 's learning curve and determining the trial at which performance surpassed chance levels with 95% confidence (see [100] for details on methods). Using the Bayesian threshold for learning did not change the results (winning model XP =1; 1; R: model with performance = 0.00, performance and L.P 0.14, performance and I.IP= 0.62, performance and hierarchy 0.04, performance, LLP, and hierarchy = 0.21).
18

---

D.4 LLP as the proportion of possible solutions tested
Within the specifics of our task, LLP decreased every time a participant chose new action sequence for the active goal, eventually reaching 0 once the participant found the solution. This ensured there were no discontinuities LLP. However, another intuitive way to model LLP is by incrementing every time participant attempted new action sequence, until they found the solution:
aclin powill cction sequexees
if G is not learned otherwise
LLP
0
Since this formulation did not significantly change the results (winning model XP 1; R: model with performance = 0.00, performance and LP = 0.19, performance and LLP = 0.59, performance and hierarchy 0.36, performance, LLP, and hierarchy 0.18), kept the version described earlier, which was not affected by abrupt discontinuities thus facilitating interpretation.
D.5 Alternative coding scheme for the hierarchy feature
To update the hierarchy utility of the chosen goal, we used a term H equal to 1 for hierarchical (G3, G5), -1 for non non-hierarchical (G4, G6), and 0 for simple goals (G1, G2). However, hierarchy effects more prominent for complex than compound goals (Figure We thus ran the model comparison pipeline with an alternative coding scheme for H, which was equal to 1 for the hierarchical complex goal (G3), -1 for the con-hierarchical complex goal (G4), and 0 otherwise (G1, G2 G5, G6). Using this scheme did not significantly change the results (winning model XP = 0.96; R: model with performance 0.02, performance and LP 0.12, performance and LLP 0.48, performance and hierarchy = 0.02, performance, LLP, and hierarchy 0.36).
D.6 Alternative coding scheme for the hierarchy feature
In model, the term H used to learn the utility of each goal with respect hierarchy time through the shared learning parameter a. However, is also possible that hierarchy information informs participants' behavior from their first interaction with each goal. We thus ran the model comparison pipeline with a learning rate of for H. equivalent to setting hierarchy values to their coding scheme from the first interaction with a goal. Using this fixed learning rate did significantly change the results (winning model XP = I; R: model with performance = 0.03, performance and LP = 0.12, performance and LLP 0.58, performance and hierarchy = 0.00, performance, LLP, and hierarchy = 0.27).
E
Parameter and model recoverability
Before analyzing model parameters from the winning model (one which chose goals based on performance and LLP signals), simulated and fit data with known parameters (best-fi fit parameters from the Laplacian approximation HBI) to ensure they could be recovered. We used an options reinforcement learning model simulate participants' learning process (Appendix We observed tight correlations between true and recovered parameters (Figure 6). suggesting the experiment was sufficiently powered assess this model (90)
prstemonce
Bur
5
1.0
0252,
088
0.8
Racovered
Recovered b.
Recovered
0.2
-10
0.0
-10
-7.5
-5.0
-2.
0.0
0.00
0.25
0.50
0.75
1.00
Generated
Generated
Generated
Figure 6: Parameter recovery. Spearman's correlations between true and recovered parameters for the winning model.
19

---

To ensure we could compare similar models [90], we tested model recoverability by simulating partic- ipants' behavior with each model and the corresponding best-fit parameters (Laplacian approximation step of HBI). Each participant's s behavior was simulated 3 times to account for random noise. We again assumed a reinforcement learning process with options governed the participants' action selection. We then performed model comparison (HBI; (88) D to determine whether we could correctly identify the simulating parameters. The confusion matrices in Figure 7 iillustrate this was the case, although the model with performance- LLP., and dierarchy-basse goal values was somewhat difficult to distinguish from the model with performance- and LLP-based values alone.
Exceedance probabilly
B
Average model responsibillly
Perf
Per
0:51
0:22
0:04
True model
Pert UP
True model
Pert LP
Pert LLP
Peat LLP
005
0.02
0:60
0
Purt Heer
Purt Heet
Per L.P Hew
Pert LLP Hes
0.00
0:44
003
Peet UP
Pert LpP
Paaf Her
Part LUP Her
Pist LLP
Perf Hear
Pert LLP Her
Part
Recovered model
Recovered model
Figure Confusion matrices for model recovery. (A) Exceedance probabilities for true and recovered models. (B) Average responsibilities for true and recovered models. Model name abbreviations: Perf performance, LP learning progress, LLP latent learning progress, Hier hierarchy.
F Options reinforcement learning model for simulations
In the experiment, participants made two kinds of choices, first picking a goal and then selecting a sequence of actions to try and achieve it. Since our focus was on the former, we only fit the data on the goal selection process, gathering information about action selection and feedback (required, e.g., for the computation of the performance, LP, and LLP features that entered the goal selection component) directly from the participants' data. However, simulating goal selection data - c.g., ... for parameter recovery (Figure 6). model recovery (Figure 7 and model validation (ensuring the model captures key aspects the original data through simulations with the best-fit parameters 990 91]], Figure - creating model that also generates data about the learning component. To this end, we developed models that attempted to learn the correct action sequences in addition to choosing their own goals. The goal selection component of the various models described in Section 3.2 The learning component detailed below was shared across all models.
Each trial begins with goal selection step (forced, in the training phase, or free, in the learning phase). Next, the agent decides which actions to pursue. The model's 's fundamental architecture is that of standard Q-value reinforcement learning model (67), except that policies were coal-conditioned 77 The model's Q-value table has shape NG x NA x Ns. where Ng is the number of possible goals (6), N, is the number of ingredients available actions for selection at the current timestep, and Ns is the total number of steps required to make the active goal (either two or four; Appendix [B) Any of the four ingredients available for the active goal can be selected the beginning of each trial and becomes unavailable once the agent chooses them. The model selects actions via a softmax policy [66] over the relevant Q-table until all the slots filled:
P'(agg, (h) eap(3rcam F(Q' ,)),, C.p((Breem F(Q'(g) (g" aa),,,,
where is the current trial, a is the chosen action; g" is the active goal; h is the current trial's history (i.e. the actions selected thus far) which, through the filter function F, determines the number of steps left to complete the trial and the actions the agent can consider taking next; Sream is the inverse temperature (the higher the Sream, the more deterministic the decision).
Once all slots are full the agent receives a binary feedback r', equal to the chosen sequence of actions matched the active goal, and 0 otherwise. The agent then uses this signal to update the value
20

---

of each chosen action through the delta rule |87 The filter function F serves to address the correct Q value based on the timestep at which the action update was chosen:
(gQ"(0" ()) - Fg"" (g),,,))) + Oleam (f'-F(Q"- ())
where Oteam is the learning rate. This update function reflects the fact that the agent does not receive partial feedback for correct actions at the right timesteps, and can therefore only identify the appropriateness of chosen actions if the entire chosen sequence is correct.
The model's Q table was also subject forgetting as governed by the variable phi, which gradually brought back all Q values to the initial value Qo (0):
= Q-- + a. (Qulg. - '' E G,
Except for the temporal aspect given by the nature of the environment, the model described until now is not capable of recognizing hierarchical relationships among goals. To enable it to leverage hierarchical structures and speed up learning, we endowed the model with the ability to build "options", i.e. to create and select sub-sequences of actions as a single chunk [92]. Upon receiving positive feedback, the model extends the space of possible actions with the chosen sequence of actions, which can then be considered a viable option whenever the ingredients it requires are available and the slots to fill are of at least the same length as the option itself. Options are selected and then updated in the same way atomic actions are. Note that, in reality, participants could also leverage insights from goals that shared structural similarity without requiring the same atomic actions (e.g.. compare G3 and G5, Figure bottom). While found that the simpler current model was sufficient to capture many aspects of the participants' behavior, future extensions could consider generalization rules that enable the model to share more abstract knowledge across goals.
The options model was unfittable through HBI methods due intractable likelihood [IOI. To approximate the learning parameters for each participant (Stearn. Olearn and Φ), we fit a simpler reinforcement learning model (without options). this time through maximum- likelihood methods and only on action selection date, and used the resulting best-fit learning parameters for simulations. To account for random noise, we ran iterations per participant when simulating data for validation and model recovery.
maximum-likelihood
G
The role of hierarchy in learning and goal selection
Hierarchical features of the environment could interact with the goal selection process in at least two ways. First, hierarchy could have direct role goal selection, e.g., people might either prefer or avoid goals that enable them to cxploit hierarchical relations. Second, hierarchical structures may impact individuals' learning, hence their LLP which is then used select goals. The fact that the winning model for goal selection had an LLP factor, but lacked direct information about the presence of hierarchical relationships among goals, suggests the latter hypothesis is more likely. However, we find that hierarchy plays a major role in the learning process, thereby affecting goal selection indirectly.
To illustrate this, we simulated participants' data with two models. The first model, used for all other simulations presented in this article, is reinforcement learning with options (122; Appendix FP). therefore able to exploit hierarchical relationships between simple and complex goals. The second model a flat reinforcement learning model, which only has access to atomic actions. Despite the fact that both models lacked hicrarchical factors in the goal selection stage, effects of hierarchy similar those of real participants (Figure 2) were found in data simulated with the options reinforcement learning model (Figure -C. By contrast, removing hierarchical information from the learning component completely removed hierarchy effects in learning and goal selection (Figure 8DFF). These results suggest at least some participants generated insights for learning by recognizing hierarchical relationships in the learning environment, while not necessarily using this information directly to decide which goals to select. Since the options model was not fittable through HBI methods, quantitative confirmation of these results awaits formal model comparison through more advanced model fitting methods [10I).
21

---

Data semoleled with opkoma
05
Brth than
01
02
Q1
100
150
10
Dara simulatedd withost options
04
05
br Kerre
07
02
00
00
20
100
150
3os
Figure 8: Simulations with two different learning models. (A-C) Simulated participants' behavior from generating model with options shows hierarchy effeets in learning and goal selection. (D-F) Simulated participants' behavior from a generating model without options fails to replicate hierarchy effects.
H
Verbal reports of example participants' strategies
At the end of the experiment, participants were asked questions about the strategies they followed while playing the online game (Appendix B). Capital letters in the responses transcribed below correspond to participants in the subplots of Figure
A. "Yes, I tried to find the easiest potion recipes firsts (two ingredients) and only focus on the recipes one at a time. When found one, would move to the next casiest. Eventually, I knew all of them but needed engrain them into my memory. I then repeatedly moved from one recipe to the next practicing. Finally, began to randomly select potions as a means of testing and improving my memory."
B. "I tried using permutation skills had learned in class I had taken on discrete math. This was especially helpful for the potions that required ingredients, where permuted the possible solutions into different sets depending example, I would choose a first and second ingredient, which left me with possible combinations for the remaining ingredients. If neither of those sequences worked, I would choose a different second ingredient and try the 2 possible sequences that could be made with this new combination of first and second ingredients. If that again didn't 't work, I would choose the last possible second ingredient. If was still not successful in finding the correct sequence, would try different first ingredient and attempt this entire process all over again."
what the first 2 ingredients were. For
C. "I used the one that let me win each time."
D. "No didn't 't use strategy because was confused on what to do in the beginning of the game."
I Clustering methods
The first step towards separating participants into clusters was engineering features from the raw data to obtain distinct behavioral signatures. The final subset of features was selected due to its enabling reliable cluster partitions and the low correlation across features (Figure
1. P(Correct) at learning: proportion of trials of the learning phase in which the chosen goal was achieved.
2. P(G in [G1, G2, G5, G6]): proportion of trials in which a 2-action goal was selected.
3. P(G [GI, G2 G3, G5]): proportion of trials in which goal involved in hierarchical relationships was chosen.
22

---

4. S(Goals): entropy of the chosen goals.
5. S(Actions): entropy of the chosen actions.
6. P(Correct) at testing: proportion of trials of the testing phase in which the proposed goal was achieved (including G7 and G8).
To capture temporal dynamics in participants' behavior, all features were computed separately for each of the blocks of the learning section, except for P(Correct) testing, which was a single value subject since there was single testing block. As result, the behavioral features were highly dimensional. Thus, before performing clustering, we reduced the dimensionality of the features through principal component analysis (PCA). We identified 11 the optimal number of principal components (PCs) through the elbow method over the cumulative explained variance. Plotting the PC loadings of the original features suggested PCA maintained the structure inherent in the raw features while enabling a more tractable dimensionality (Figure 99) For example, PC1 seemed to track performance indicators, PC2 defined action entropy early in learning, and PC3 represented goal entropy.
A
0.
wilearing
075
0
0.-
0.29
PIG (010 Ga asp-0002 0042
0.20
2000
-017
060
025
0
08
Goal entrapy
0.20
25
PrComect 2t tosting
0.47
0.078
0.000
0.13
0069
-076
08
PICOFodd attomrig
P10 (01 o2. ca 6m
Adton enreey
PiCanadd al Banning
-.6
PO Po az 03 08)
Spotims)
Prcomeei buston
P 0z
Figure 9: PCA feature correlations and loadings. (A) Pearson's correlations among features used for
PCA. (B) Loadings of the chosen features on the first PCs.
Participants were then clustered through the K-means method 9 over the PCs. However, similar clusters were obtained through different partitioning methods. The elhow-point heuristic over the within-cluster sum of squares (WCSS) was used identify as the optimal number of clusters (Figure TO.). Plotting combinations of the first three PCs confirmed the presence of four distinct groups I03).
We used archetypal analysis - a technique which searches extremal points ("archetypes") in multidi- mensional data (196), software by [122] to visualize extreme behaviors and exemplify participants' strategies in each cluster. The adjusted mutual information between K-means cluster identity and the closest archetype for each participant was 0.74, confirming much overlap between the two classification methods (Figure[OOC),
23

---

A
C
Cluster Archeopp Archolppo
porti-4
2500
2000
Number clusters
PC
PC1
PC2
Figure 10: Clustering results. (A) Scree plot showing changes in WCSS function of the number of components used for K-means clustering. (B) Positioning of the identified clusters relative to the first three PCs. (C) Simplex plot showing archetype contributions each participant, color-coded by the participant's K-means cluster identity and marker-coded by the archetype closest to each participant.
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Abstract and introductions clearly state the claims made, including contribu- tions, and reflect the scope of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The paper discusses the limitations of the work in the Discussion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
24

---

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
No theoretical results are presented.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: All necessary information to reproduce the main experimental results is provided in the main text and/or the appendices.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often
25

---

one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification: The authors reserve the right to perform further analyses the present data before publicly releasing them.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
cablic/guides/ httgg//dd//ooeeuuiiiaae CootsSubmissionool111
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.
ssinnoliccy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: The authors specify model fitting and comparison methods in the Computa- tional modeling section.
26

---

Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: Error bars and statistics are reported wherever relevant in results and figures. Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of eтrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [NA]
Justification: None of the analyses and modeling processes were so computationally expen- sive to require special computer resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https http::/://eeuueeeeeeeeeeeoe htt.a/awwww....ppppppp.ccccccaaa
27

---

Answer: [Yes]
Justification:
The authors have reviewed the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The authors discuss the potential positive and negative societal impacts of the work performed in the Discussion section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML...
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification:
No data models are released.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
28

---

We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The authors are the creators of the main assets used in the paper and cite other resources used.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paw.:ssi.iaasaiiiia...... thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors are encouraged to reach out the asset's 's creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
Data collection and analysis methods sufficiently documented in the main
text.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: Details about data collection with human participants provided in the main text and the appendices, where the full text of instructions given to participants is reproduced alongside relevant screenshots.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
29

---

Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation,
or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification:
The experiment did not involve any known risks and was approved by the local
IRB.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
30


Reviews:
[{"review_id": "QLwaWLBler", "reviewer": "Reviewer_kdQC", "summary": "This work presents a hypothesis of a latent learning process that can guide autotelic agents in goal selection. Human experiments provide evidence supporting this hypothesis.", "questions": "1. Hierarchy influences goal selection through latent learning processes (LLP). Will separating the evaluation of hierarchy and LLP affect the results?\n2. Section 6 discusses individual differences. Has there been an analysis of the causes behind these differences?", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Autotelic agents represent an important research direction. This work provides evidence from human experiments on latent learning processes, which could later be used to develop effective and personalized learning progressions for human-like autotelic machines.", "weaknesses": "1. All the subjects of the human experiment are students, which may introduce bias.\n2. The definition of $V_f$ for each kind of model is unclear: e.g. in \"performance\", what does \"currently active goal\" mean? Where does $r^t$ come from? What is the relationship between it and the goal? \n3. The design of LLP, which involves selecting multiple action sequences, does not provide a complete explanation of how LLP can improve human choices or actions better than LP.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer kdQC for examining our submission and recognizing its importance and potential impact on machine learning. In the rebuttal period, we hope the reviewer could also help clarify what would improve soundness and presentation beyond the point raised so far. Regarding contribution, we would like to stress the fact that NeurIPS welcomes submissions from “Neuroscience and cognitive science” – therefore, we invite the reviewer to consider our contribution to such fields. We address all of the reviewer’s stated concerns below. \n\n---\n\n**Weaknesses**\n\n*All the subjects of the human experiment are students, which may introduce bias.*\n\nThis point was addressed in the global response.\n\n--- \n*The definition of  V_f for each kind of model is unclear: e.g. in \"performance\", what does \"currently active goal\" mean? Where does r^t  come from? What is the relationship between it and the goal?*\n\nThe currently active goal is the goal a participant has chosen on a given trial. To make our statement clearer, we now say “selected goal” instead of “currently active goal”. $r^{t}$ comes from the feedback provided to participants (the target potion either fills up or remains empty), as specified in the same sentence:  “1 for positive feedback, 0 for negative feedback”. Feedback is goal-contingent, such that participants only receive feedback relative to the selected goal. We clarify this further in the text: “On each trial, the utility of the selected goal with respect to performance is updated based on the goal-contingent feedback received on that trial $r^{t}$”.\n\n---\n*The design of LLP, which involves selecting multiple action sequences, does not provide a complete explanation of how LLP can improve human choices or actions better than LP.*\n\nAs mentioned in the paper, “unlike LP, LLP does not require observing performance improvements to provide informative signals about progress.” This would make LLP particularly useful in situations (mentioned in the paper) “where no external change is visible, yet some other form of progress toward the desired outcome is made. Imagine being tasked with identifying the correct sequence of numbers to open a combination lock, which you might attempt through trial and error. Throughout most of this scenario, repeated failures would yield no difference in performance, hence no empirical learning progress (as typically defined). Nonetheless, provided that the lock has a limited number of slots and numbers and that you can avoid repeating incorrect combinations, every attempt is a step toward the solution.”   \n\n---\n**Questions**\n\n*Hierarchy influences goal selection through latent learning processes (LLP). Will separating the evaluation of hierarchy and LLP affect the results?*\n\nHierarchical components are a design choice in our experimental paradigm. Removing the hierarchical feature would likely make it more difficult to tease LLP and LP apart. We find, however, that exactly how hierarchy and LLP relate to and influence each other is an interesting question. Our current findings suggest that “hierarchy impacts goal selection indirectly by enabling inferences and thus affecting LLP”.\n\n---\n\n*Section 6 discusses individual differences. Has there been an analysis of the causes behind these differences?*\n\nThe goal of the current paper was not to identify the sources of individual differences in goal selection. Noting that such differences exist is in fact a novelty of our submission. That said, we agree with the reviewer that the question of individual differences is worthy of further exploration, and we mention it would be interesting if future studies crossed findings similar to ours with participants’ data related to “demographics and cultural background, cognitive abilities, and psychopathology”.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hz0MgIMgAz", "reviewer": "Reviewer_y4se", "summary": "This paper looks at human goal selection during learning. A known useful signal for goal selection is learning progress (LP). LP measures performance from past observations, and is therefore only sensitive to measurable change in performance. The paper hypothesizes that goal selection is additionally driven by a “latent LP” (LLP) measure, which an agent infers from its knowledge of and interaction with the environment, and which therefore does not depend solely on observable performance improvements. Computationally, for each potential goal that can be set, LP is modeled as a value that is updated based on changes in the performance prediction error, while LLP is modeled as tracking how much of the solution space is currently unexplored (thus, it is a proxy for how close the agent is to solving the goal). A simple goal selection model is proposed under which goals are selected with probability given by a softmax function of goal-specific values. These values are a weighted sum of - among other factors - the aforementioned LP and LLP values. An experimental setup is then introduced in which participants were asked to iteratively select one of many goals and to attempt to solve that goal. The fact that participants had to actively select goals made participants’ goal selection behavior directly observable. Data from human participants reveals that the LLP factor is an important element in goal setting behavior.", "questions": "The goal selection model considers a weighted sum for combining the factors that contribute to the goal values. Did you consider or test more complex functions of the factors?", "limitations": "Limitations are discussed in the conclusion. The authors do mention that potential future ethical concerns should be addressed, but do not expand on what those concerns may be.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This is a very good paper. The writing is clear and the motivation and discussion of prior work in the introduction and section 2 is excellent. Understanding human goal selection is important. Not just in its own right, but also for human-AI interaction settings where better models of human behaviour can help to train better AI assistants or companions. Furthermore, a better understanding of how humans direct their learning may inspire future research on things like curriculum design for RL. The experiments presented in this paper are novel, rigorous, and clearly support the claim that LLP informs goal selection. Moreover, there is a wealth of additional information and additional results in the appendices.", "weaknesses": "The definition of LLP introduced here is quite specific to a small and discrete space of action sequences. A more general definition or some preliminary discussion on how one could extend the current work to continuous or countably infinite spaces of action sequences would strengthen the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their excellent summary of our article and their thorough feedback. We address their suggestions for improvement below. \n\n---\n\n**Weaknesses**\n\n*The definition of LLP introduced here is quite specific to a small and discrete space of action sequences. A more general definition or some preliminary discussion on how one could extend the current work to continuous or countably infinite spaces of action sequences would strengthen the work.*\n\nThis point was addressed in the global response. \n\n---\n\n**Questions**\n\n*The goal selection model considers a weighted sum for combining the factors that contribute to the goal values. Did you consider or test more complex functions of the factors?*\n\nIn our initial analyses, we indeed considered more complex functions of the factors. . Specifically, we tried modeling more complex interactions in our current setup (e.g., interactions between performance and other goal selection motives), but did not find this procedure to improve fit. Moreover, we note that we already observe interesting dynamics with a simple model, which is preferable for interpretability. It is possible that more complex functions of the factors might be relevant in more complex settings, and this will be an important question for future research. Here, we follow the standard “Occam’s razor” approach of the lowest sufficient complexity of the model given our data (Wilson and Collins, 2019). \n\n---\n\n**Limitations**\n\n*Limitations are discussed in the conclusion. The authors do mention that potential future ethical concerns should be addressed, but do not expand on what those concerns may be.*\n\nWe thank the reviewer for the opportunity to expand on such concerns, and we now state that : “As our knowledge of human goal setting becomes more precise, however, ethical concerns regarding the use of behavioral sciences in marketing and management should be addressed, particularly in cases where highly personalized methods of influence could transform advertising into manipulation.”", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qVJ4OVuWg5", "reviewer": "Reviewer_Grub", "summary": "This paper examines how humans select between different possible goals. The authors designed a hierarchical reinforcement learning task where the main dependent variable was which goal participants chose to work on for each trial. Then, they built descriptive computational models to assess what parameters explained participants' goal choices throughout the duration of experiment. The parameters that were most predictive of the participants' goal selections were performance and latent learning progress. \n\nThe experimental task introduced in this paper bears a resemblance to Little Alchemy (https://littlealchemy.com/) insofar as participants combine together different elements to create new elements. Where the task differs is that, rather than freely combining elements without any specific goal in mind, the participants in this task had 6 (not including the testing phase) goal potions to create and they received feedback about whether or not they successful concocted their target potion for the trial.\n\nTheir findings contribute to understanding human learning and building autotelic machines. In terms of human learning, the paper provides new evidence that latent learning progress (rather than standard learning progress) drives people to chose some goals over others. In terms of building autotelic machines, new artificial agent models could incorporate latent learning progress when choosing goals to better mimic the adaptive and rapid learning of humans and animals.", "questions": "My questions are covered in the Weaknesses portion.", "limitations": "See Weaknesses section.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Goal selection (and intrinsic motivation more broadly) is a fundamental question in the study of both biological and artificial intelligence. This paper tackles a significant research question by using a novel experimental paradigm. The original paradigm introduced in this paper could also be modified for future work to address other questions like open-ended learning or could be made more broadly available online to test more diverse pools of participants in the future. \n\nThe writing is very clear through out (and any lingering questions I had about the experimental task were clarified effectively in the appendix). \n\nAnother wonderful strength of the paper is their ability to report on inter-individual differences. They are able to fit descriptive models to each individual subject.", "weaknesses": "While this paper has a number of important strengths, the main weakness is that it is not clear how well these findings would generalize to different tasks (especially more real-world tasks):\n- First, the experimental paradigm is very specific. Users have to pick from a limited list of goals with the explicit over-arching goal of being able to create all of the potions successfully. Conversely, people usually generate their own goals (e.g., \"I'd like to read this book\", \"I'd like to go on cross-country trip to visit Aunt Ida\") rather than picking from a specific list, and they often choose these goals without any supervised over-arching goal.\n- Second, the participants were university students (mostly female) participating for course credit. Would the results generalize to different populations?\n- Third, the measure of latent learning progress (LLP) is entirely specific to this experimental task. The authors operationally define latent learning progress as 1 - (N actions sequences tested / N possible action sequences). This measure only applies in situations where there are a finite number of options to try. How can future researchers extend this notion of LLP to open-ended tasks that use high-dimensional sensory and motor spaces (where this operational definition would not work anymore)?\n\nTo be clear, I don't expect the authors to design a new experimental paradigm or test a new demographic range of participants during the rebuttal period. But it would be helpful for them to address these as limitations.\n\nOn a separate note, the statistical analyses need effect sizes. (For example, a simple Cohen's d for each Wilcoxon / t-test would be suffice.) Some of these differences in task performance are statistically significant because p < .05, but are not necessarily behaviorally significant because the difference (effect size) is so small. Also, if I'm reading the stats correctly, the results in text do not match Fig 2A: Complex hierarchical G3 mean performance is reported as .37, but the bar in Fig 2A looks like it couldn't be any higher than .3.\n\n\nMinor:\n- The in-text references are sometimes formatted a little strangely in terms of the number (for example citing 12-17, 11 instead of citing 11-17 or citing 1, 33, 34, 3, 35, 5, 6, 36, 37, 8 instead of citing 1, 3, 5, 6, 8, 33-37).\n- The explanation of the experimental game (section 3.1) is not that clear. The figure (Fig 1) is definitely helpful, and the description in the appendix clarifies the game really well. You might consider reformatting the paragraph describing all of the goals into a list format.\n- Figure 3 is difficult to understand as-is, but I think it might actually be a really interesting figure. Part of what makes it unclear is the y-axis: What is the \"action sequence index\"? I suspect there might be an alternative to a dot plot that would be easier to read (maybe even forgoing the y-axis and showing a single strip of color-coded trials for each subject perhaps?). It would also help the readers to add some descriptors to each graph so the reader knows which elements to look for (e.g., setting a few objectives rather than trying all goals, unprincipled switching between goals).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive comments and their extremely detailed, helpful feedback. We address their suggestions for improvement below. \n\n---\n\n**Weaknesses**\n\n*First, the experimental paradigm is very specific. Users have to pick from a limited list of goals with the explicit over-arching goal of being able to create all of the potions successfully. Conversely, people usually generate their own goals (e.g., \"I'd like to read this book\", \"I'd like to go on cross-country trip to visit Aunt Ida\") rather than picking from a specific list, and they often choose these goals without any supervised over-arching goal.*\n\nWe completely agree with the reviewer! In fact, we noted “In our task, people chose goals from a predefined menu of options. Although this facilitates the study of goal setting, people often invent their own goals by combining observations and imagination”. Having a highly controlled setup in which to study goal selection was essential for a first understanding of people’s approach, and our clustering analyses and plots of individual behaviors highlight vast variability in goal selection strategies even in such a limited space. Building on our initial findings, it may be possible to start exploring how people create and pursue more free-form goals (if not as rich as the examples mentioned by the reviewer, at least more self-designed than the ones we studied here). We note that our is a standard approach to studying a novel, complex question: we start from a simplified experimental design to establish findings in well controlled setting, and plan to adapt our approach to increasingly more naturalistic, complex settings to pursue a more nuanced understanding. \n\n---\n*Second, the participants were university students (mostly female) participating for course credit. Would the results generalize to different populations?*\n\nThis point was addressed in the global response.\n\n---\n*Third, the measure of latent learning progress (LLP) is entirely specific to this experimental task. The authors operationally define latent learning progress as 1 - (N actions sequences tested / N possible action sequences). This measure only applies in situations where there are a finite number of options to try. How can future researchers extend this notion of LLP to open-ended tasks that use high-dimensional sensory and motor spaces (where this operational definition would not work anymore)?*\n\nThis point was addressed in the global response.\n\n---\n\n*On a separate note, the statistical analyses need effect sizes. (For example, a simple Cohen's d for each Wilcoxon / t-test would be suffice). [...].*\n\nWe now provide the effect size for all Wilcoxon tests as the standardized effect size r (Z/√N; Rosenthal et al., 1994; see attached PDF, “Updated statistics with effect sizes”). Note that we have replaced the symbol “Z” with “W” for non-standardized test statistics in Wilcoxon tests.\n\n---\n*Also, if I'm reading the stats correctly, the results in text do not match Fig 2A: Complex hierarchical G3 mean performance is reported as .37, but the bar in Fig 2A looks like it couldn't be any higher than .3.*\n\nWe are grateful to the reviewer for noticing this mismatch! The stats are correct, and we have updated the figure to match them (see attached PDF, “Updated Figure 2”). The previous figure was also correct, but aggregated the data slightly differently. The two are now consistent. \n\n---\n\n**Minor**\n\n*The in-text references are sometimes formatted a little strangely in terms of the number [...].*\n\nWe thank the reviewer for noticing the citation formatting error, which we have fixed for the camera-ready version.\n\n---\n*The explanation of the experimental game (section 3.1) is not that clear. The figure (Fig 1) is definitely helpful, and the description in the appendix clarifies the game really well. You might consider reformatting the paragraph describing all of the goals into a list format.*\n\nWe thank the reviewer for their suggestion and for consulting the Appendix thoroughly. We had initially placed a full description of the task in the main body of the paper but had to move it to the Appendix due to space limitations. We intended to provide the reader with enough information about the task to understand the main takeaways and refer interested readers, with a more specific interest in cognitive science experiments, to details in the Appendix.\n\n---\n*Figure 3 is difficult to understand as-is, but I think it might actually be a really interesting figure. Part of what makes it unclear is the y-axis: What is the \"action sequence index\"? I suspect there might be an alternative to a dot plot that would be easier to read (maybe even forgoing the y-axis and showing a single strip of color-coded trials for each subject perhaps?). It would also help the readers to add some descriptors to each graph so the reader knows which elements to look for (e.g., setting a few objectives rather than trying all goals, unprincipled switching between goals).*\n\nWe thank the reviewer for their insightful analysis of Figure 3. “Action sequence index” refers to the unique combination of ingredients selected by the participants and their order (e.g., the first point would be [0, 1, 2, 3] – where 0 is the top-most ingredient on the screen, 1 the second, etc.). To improve clarity, we have relabeled the y-axis “Action sequence”, since the index is not defined in the text. We omitted the axis labels to avoid cluttering the figure, making it unnecessarily large, and overloading the reader with information. We used the preceding paragraph to guide the reader through specific aspects of the figure they should be focusing on, and would therefore find it redundant to repeat the information in the figure caption.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "m07xpj1PYT", "reviewer": "Reviewer_HMNe", "summary": "This paper develops a new analysis of autonomous goal selection based on \"latent learning progress\". The key idea is that agents seek to maximize progress on a latent variable rather than an observed variable like performance. The paper reports an interesting experiment with humans that seeks to test whether people use latent learning progress to guide their goal selection.", "questions": "- Can the authors provide a more general formal definition of latent learning progress?\n\n- How can latent learning progress be defined in such a way that it's likely to be both general and useful? My concern is that there are probably many versions of this which are not useful, depending on how one defines the latent variable whose progress is being monitored.", "limitations": "The authors have adequately described the paper's limitations. I don't see any potential negative societal impacts.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The core theoretcal idea (latent learning progress) is well-motivated and novel.\n\n- The experiment is well-designed, interesting, and rigorously analyzed.\n\n- The modeling approach is very thorough.\n\n- The paper is clearly written.\n\n- The literature review is comprehensive.\n\n- Potentially impactful within cognitive science.", "weaknesses": "- Latent learning progress wasn't formally defined until well into the middle of the paper, and there it was only operationalized in terms of the specific experimental setup rather than something more general.\n\n- While motivated by work in AI, the paper isn't really written in a way that will have a broad appeal to an AI audience. I'm not sure how much impact this work will have at least within AI.\n\nMinor:\n\np. 1: \"Along factors\" -> \"Along side factors\"\n\np. 4: \"in which factor\" -> \"in which factors\"; \"combination of factors\" -> \"combinations of factors\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive assessment of our submission, and address their comments individually below. \n\n---\n\n**Weaknesses**\n\n*Latent learning progress wasn't formally defined until well into the middle of the paper, and there it was only operationalized in terms of the specific experimental setup rather than something more general.*\n\nWe provide an intuitive explanation of LLP in the introduction, and save a formal definition for later in the paper, where all other related concepts have been introduced, to facilitate the reader’s understanding of such definition. The point about a generalized version of the LLP definition was addressed in the global response.\n\n---\n*While motivated by work in AI, the paper isn't really written in a way that will have a broad appeal to an AI audience. I'm not sure how much impact this work will have at least within AI.*\n\nAs noted above, NeurIPS welcomes submissions from “Neuroscience and cognitive science” – therefore, we invite the reviewer to consider our contribution to such fields. That said, we provide several connections to the artificial intelligence and machine learning literature in the Introduction, Related Work, and Discussion sections of the article, and would welcome the reviewer’s suggestions to further bridge across different fields. \n\n---\n\n**Minor**\n\n*p. 1: \"Along factors\" -> \"Along side factors\"*\n*p. 4: \"in which factor\" -> \"in which factors\"; \"combination of factors\" -> \"combinations of factors\"*\n\nMinor typos have been fixed in the camera-ready version.\n\n---\n\n**Questions**\n\n*Can the authors provide a more general formal definition of latent learning progress?*\n\nThis point was addressed in the global response.\n\n---\n\n*How can latent learning progress be defined in such a way that it's likely to be both general and useful? My concern is that there are probably many versions of this which are not useful, depending on how one defines the latent variable whose progress is being monitored.* \n\nThis point was addressed in the global response.\n\n---\n\n**Limitations**\n\n*The authors have adequately described the paper's limitations. I don't see any potential negative societal impacts.*\n\nWe included potential societal impacts in our initial submission, but we expanded it in response to reviewer y4se: “As our knowledge of human goal setting becomes more precise, however, ethical concerns regarding the use of behavioral sciences in marketing and management should be addressed, particularly in cases where highly personalized methods of influence could transform advertising into manipulation.”", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to express our gratitude for the reviewers’ positive and constructive feedback.

The reviewers highlighted several strengths of the paper, including the important research direction it falls in line with and its tackling a fundamental question. They noted the potential impact of our findings on research in cognitive science, but also the design of autotelic machines and the development of AI assistants or companions that better align with human goals. The reviewers also appreciated the novelty of both the theoretical idea and the empirical approach, which could be extended to further address questions on open-ended learning. According to the reviewers, the writing was clear, the supported evidence was comprehensive, and the analysis thorough.   

The reviewers also noted some limitations. We address some common questions below, and invite reviewers and ACs to refer to individual responses for other points raised in the review process. We hope the provided answers and clarifications help improve our submission’s score wherever possible, and otherwise kindly ask the reviewers to express remaining concerns so we can best address them. 

Reviewers kdQC selected “fair” as a contribution score, and HMNe mentioned that the paper’s appeal to an AI audience could be improved. First, we would like to point out that NeurIPS welcomes (and has welcomed in the past) submissions from “Neuroscience and cognitive science” (as mentioned in the [call for submissions](https://neurips.cc/Conferences/2024/CallForPapers)), where we find our paper makes the most immediate contributions. That said, we made sure to provide several connections to the artificial intelligence and machine learning literature in the Introduction, Related Work, and Discussion sections of the article. Reviewer Grub perfectly captured our positioning in terms of impact, stating that “In terms of human learning, the paper provides new evidence that latent learning progress (rather than standard learning progress) drives people to choose some goals over others. In terms of building autotelic machines, new artificial agent models could incorporate latent learning progress when choosing goals to better mimic the adaptive and rapid learning of humans and animals”.

Reviewers kdQC and Grub pointed out that our participants were students from a university cohort. We would like to note that this is standard practice in the field. We, as well as other researchers studying higher cognition in various domains, have observed that qualitative effects found in university student populations typically generalize well to a broader, more diverse population of healthy adults. The reviewers are, however, correct that we cannot guarantee that our findings will indeed generalize to the general population and that this will be more important as we consider individual differences alongside whole-group qualitative effects. We recognized this limitation in our initial submission: “However, our sample was restricted to a relatively homogeneous group of undergraduate university students.” We now also mention that “Future studies and computational models may extend participation to a broader subject pool and specifically address individual differences in goal selection and achievement [...]”. Nonetheless, LLP was a key factor across participant clusters. While the distribution of clusters might change across populations (i.e., different strategies might be more or less common in certain populations), we predict LLP to be widely used. 

All reviewers noted the lack of a generalized definition of LLP beyond the scope of our experimental paradigm, especially with respect to situations where possible solutions are not finite. First, we note that while simplistic, our novel experimental paradigm is already quite complex relative to the existing setups. However, we agree with the reviewers this is a limitation of the work as it currently stands. As noted in the article, “While we provide a simple and task-specific formalization of LLP, a generalized definition is necessary to understand the differences between LLP and other intrinsic motivation signals and ease the implementation of LLP-based goal selection in autotelic machines”. However, our focus in this initial work was to provide evidence for LLP use in humans, and it was essential to establish this aspect to “inspire the establishment of even more precise signals for goal selection in both humans and artificial agents”. We also note that even in situations where infinite spaces are theoretically possible, humans often restrict possibilities to a countable amount, and can therefore keep learning by exclusion like participants in our study did. For example, in a search problem such as looking for one’s glasses, one could consider virtually infinite locations, but rarely does so – making our current definition of LLP easily applicable. In more complex, real-world situations, we speculate that people may rely on a different internal model of their environment and their non-observable progress as they attempt to solve it – however, future work will be needed to extend LLP in these situations and describe such internal model in full detail.

Overall, we find the reviewer’s feedback encouraging and constructive, and we look forward to continuing the discussion.

Meta-review: While their were some initial concerns, the authors addressed everything that could be considered significant. Several reviewers consider this a particularly strong contribution in this area and I'm inclined to agree.
Decision: Accept (poster)