MAGELLAN: Metacognitive predictions of learning progress guide autotelic LLM agents in large goal spaces
Authors: Loris Gaven
Thomas Carta
Clément ROMAC
Cédric Colas
sylvain lamprier
Olivier Sigaud
Pierre-Yves Oudeyer
Venue: ICML 2025 poster (2025)
Keywords: LLM agents
Open-Ended Learning
Learning Progress
Goal-conditionned RL
Automatic Curriculum Learning
Primary area: deep_learning->large_language_models
TL;DR: We introduce MAGELLAN, a metacognitive framework that lets LLM agents learn to predict their competence and learning progress online to guide their curriculum in large goal spaces.

Abstract: Open-ended learning agents must efficiently prioritize goals in vast possibility spaces, focusing on those that maximize learning progress (LP). When such autotelic exploration is achieved by LLM agents trained with online RL in high-dimensional and evolving goal spaces, a key challenge for LP prediction is modeling one’s own competence, a form of metacognitive monitoring. Traditional approaches either require extensive sampling or rely on brittle expert-defined goal groupings. We introduce MAGELLAN, a metacognitive framework that lets LLM agents learn to predict their competence and learning progress online. By capturing semantic relationships between goals, MAGELLAN enables sample-efficient LP estimation and dynamic adaptation to evolving goal spaces through generalization. In an interactive learning environment, we show that MAGELLAN improves LP prediction efficiency and goal prioritization, being the only method allowing the agent to fully master a large and evolving goal space. These results demonstrate how augmenting LLM agents with a metacognitive ability for LP predictions can effectively scale curriculum learning to open-ended goal spaces.

MAGELLAN: Metacognitive predictions of learning progress guide autotelic LLM agents in large goal spaces
Loris Gaven
Thomas Carta
Clément Romac 12 Cédric Colas 13 Sylvain Lamprier' Olivier Sigaud
Verre-Yves, Oudeyer
Abstract
Evaluste goals
MAGELLAN
Open-ended learning agents must efficiently prior- itize goals in vast possibility spaces, focusing on those that maximize learning progress (LP). When such autotelic exploration is achieved by LLM agents trained with online RL in high-dimensional and evolving goal spaces, a key challenge for LP prediction is modeling one's ' own competence, a form of metacognitive monitoring. Traditional approaches either require extensive sampling or rely on brittle expert-defined goal groupings. We introduce MAGELLAN, metacognitive frame- work that lets LLM agents learn to predict their competence and LP online. By capturing seman- tic relationships between goals, MAGELLAN en- ables sumple-efficient LP estimation and dynamic adaptation to evolving goal spaces through gener- alization. In an interactive learning environment, we show that MAGELLAN improves LP predic- tion efficiency and goal prioritization, being the only method allowing the agent to fully master a large and evolving goal space. These results demonstrate how augmenting L.LM agents with metacognitive ability for LP predictions can ef- fectively scale curriculum learning to open-ended goal spaces.
(G)
(P
LLM
goal outcome
14a5-4rred Bansh
selected goal
Very large evolving goal space
LORK #ORowe
Online policy
LLM
Figure I. Navigating large goal spaces with MAGELLAN: Dur- ing training, our LLM agent uses MAGELLAN to estimate its past and current competence to compute absolute learning progress (ALP) on each goal. Given the per-goal ALP, the L.LM agent's s goal selector chooses the next goal to practice proportionally their ALP. The LLM agent then performs a trajectory to achieve this goal and the outcome is used update both the LLM agent with online RL and MAGELLAN'S competence estimation.
the sake of learning and discovery (Berlyne, 1954; Kidd & Hayden, 2015). Crucially, humans are autotelic learners intrinsically motivated to represent, invent, select and solve their own goals (Colas et al., 2022a). To navigate in a possi- bly infinite space goals, without time to explore exhaus- tively, they are equipped with intrinsic motivation signals (Baldassarre & Mirolli, 2012; Gottlieb & Oudeyer, 201 8). Research on human curiosity has shown the key role of one such intrinsic signal: Learning Progress (LP), i.e. improve- ment of one's 's own ability to solve goals (Kaplan & Oudeyer, 2007). Computational modeling work showed both how it enables efficient automatic curriculum learning (Lopes Oudeyer, 2012a; Poli al., 2024b) and how it generates developmental trajectories that simulate key properties in the development of human infants (Oudeyer & Smith, 2016). Recently, several experimental paradigms where humans were free to explore various learning activities confirmed that humans use metacognitive LP monitoring to explore and prioritize goals (Ten et al., 2021; Leonard et al., 2023; Sayalı et al., 2023; Poli et al., 2024a).
1. Introduction
Humans are open-ended learners, continuously exploring and developing new skills through their lifetime. A key mechanism to enable this remarkable capacity is curiosity- driven learning the intrinsic motivation to explore for
IIriaa (Flowers), University of Bordeaux, France 'luggin Face "MIT, Computational Cognitive Science Lab. Cambridge, MA, USA "Univ Angers, LERIA, SFR MATHSTIC. F-49000 Angers, France 'Sorbonne Université, ISIR, Paris, France. Corre- spondence to: Loris Gaven <loris <loris.gaven
gaven @ in
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

MAGELLAN
Inspired by open-ended learning in humans and other natu- ral systems, research in Al and artificial life has studied how to build machines with similar capabilities (Schmidhuber, 2013; Jiang et al., 2023; Sigaud et al., 2023). A promising di- rection focuses on developing autotelic artificial agents that, like humans, can self-generate their learning curriculum by progressively exploring goals with maximum LP (Baranes & Oudeyer, 2013; Colas et al., 2022b). This approach ef- ficiently allocates the agent's learning time by avoiding goals that are either too easy or too difficult (Portelas et al., 2020b; Romac et al., 2021), enabling even physical robots to acquire complex skills like tool use in just a few dozen hours (Forestier et al., 2022). However, while these meth- ods show promise in constrained settings, scaling them to open-ended learning remains challenging. The key difficulty lies in efficiently estimating an agent's current competence and expected LP across potentially infinite, evolving, and nigh-dimensional goal spaces- fundamental challenge we address in this paper.
based generalization of agents in a textual environment. In partieular, we study the following scientific questions:
Q1. Given initial set of language goals, how does MAGELLAN's estimation of learner's competence com- pare to more classic methods? How does this estimation scale with the size of the goal space?
Q2. Can MAGELLAN be used by an online RL LLM agent to self-organize an efficient learning curriculum over these goals?
Q3. How well can MAGELLAN's estimation generalize to predict the agent's 's competence on unseen goals?
Q4. When these new unseen goals are introduced through- out training, can MAGELLAN leverage its generalization abilities integrate new goals into the curriculum seam- lessly?
We show that MAGELLAN 1) accurately and efficiently approximates LP, 2) allows an LLM agent to master all goals from Little-Zoo while prior methods fail when not provided extensive expert knowledge, and 3) generalizes its LP estimation to never-seen goals, enabling faster adaptation to evolving goal spaces. Moreover, we show MAGELLAN learns to cluster goals and achieves results comparable to an LP estimator with expert-defined groups.
Parallel to this, a recent breakthrough has emerged in train- ing large language model (LLM) agents to learn goal- directed behaviors through online interaction with their en- vironment (Carta et al., 2023; Wen et al., 2024a;b). These LLM agents possess critical capability for open-ended learning: they can leverage the structure of language to generalize effectively, transferring skills learned from prac- ticed goals to semantically similar goals. Using language- instructed agents also offers extensive expressiveness for specifying goals, as required for the open-ended learning set- ting. This usually induces huge goal spaces, for which prior- itization is crucial. However, current approaches leveraging LP for this prioritization fall short at handling such discrete, high-dimennn nigh-dimensional and structured goal space. They either work only on small low-dimensional goal spaces (Baranes & Oudeyer, 2013; Portelas et al., 2020a; Kanitscheider et al., 2021; Zhang et al., 2024) or rely on expert-defined goal groupings to reduce the number of goals (Colas al., 2019; Akakzia et al., 2021; Kumar et al., 2024). In particular, none of them are able to capture the semantic relationships between goals to efficiently estimate an LLM agent's 's gener- alization abilities.
2. Related Work
2.1. Goal selection in autotelic agents
Autotelic agents exploring vast goal spaces face a critical challenge: they must prioritize which goals to pursue to efficiently develop general competence (Colas et al., 2022b). The automatic curriculum learning (ACL) community has developed various approaches to address this challenge (Portelas et al., 2020b), leveraging different forms of in- trinsic motivation: pursuing goals of intermediate difficulty (Held et al., 2017: Racaniere et al., 2020; Castanet et al., 2023). seeking novelty or uncertainty (Warde-Fatte Farley et al.,
2019; Pong et al., 2019; Pitis et al., 2020), maximizing regret w.r.t. an optimal policy's ' performance Unsupervised Environment Design (UED) methods (Dennis et al., 2020; Jiang et al., 2021) or maximizing learning progress (LP) (Stout & Barto, 201 O; Matiisnn et al., 2020; Fournier et al., 2018; Portelas et al., 2020a; Colas et al., 2019; Kanitscheider et al., 2021; Kovač al., 2023; Zhang et al., 2024). ACL methods also differ in their use of the goal space: while some methods compute metrics over the goal space and sample from it (e.g. (Matiisen et al., 2020; Kanitscheider et al., 2021; Portelas et al., 2020a)), others generate goals (e.g. (Held et al., 2017; Castanet et al., 2023; Dennis et al., 2020)). LP-based methods have proven partic- ularly robust, especially upon a limited training budget in
mostly used in
In this paper, we study how to estimate LP over natural language goals such that an LLM agent learning with online RL in an interactive environment could increase its overall competence efficiently possible. For this, intro- duce MAGELLAN, for MetAcognitive GEneralization of Learning progress in LANguage model agents. MAGEL- LAN leverages the LLM inside the agent to learn an LP estimator that automatically learns semantic relationships and tracks competence transfer between goals in a sample efficient manner (see Figure 1). We evaluate MAGELLAN in the Little-Zoo environment specifically designed as carefully controlled experimental setup for commonsense-
2

---

MAGELLAN
which it is not possible learn all goals (Lopes & Oudeyer, 2012b). They adapt to the agent's capabilities without re- quiring environment knowledge (e.g. which is necessary when using regret) and avoid common pitfalls like getting stuck on goals where progress plateaus or chasing uncon- trollable novelty. The key challenge with LP approaches lies in efficiently estimating progress over large goal spaces, which is the focus of our work.
2022). However, these methods did not update the LLM's knowledge through environmental interactions. A new direc- tion emerged with GLAM (Carta et al., 2023), followed by (Wen et al., 2024b;a; Zhou et al., 2024) that ground LLMs in interactive environments using online RI. The resulting LLM agents demonstrated remarkable generalization across language tasks; however, they lack the autotelic mechanisms necessary for navigating expansive goal spaces. In this pa- per, enhance SAC-GLAM (Gaven al., 2024) LLM agent with metacognitive abilities, enabling it to estimate its I.P and prioritize goals within large language spaces
2.2. Computing LP over goals
Learning Progress (LP) measures the expected future im- provement in achieving a goal through practice (Oudeyer et al., 2007). Since future progress cannot be directly mea- sured, most approaches use past progress as proxy, with the recent exception of (Kumar et al., 2024)'s Bayesian pre- diction model. The most direct approach to estimate LP is to regularly reevaluate the agent's ' competence for each goal (Kanitscheider et al., 2021; Zhang et al., 2024), which accurately captures competence transfer - the phenomenon where practicing one goal affects performance other goals. However, this becomes computationally prohibitive for large discrete goal spaces and is just impossible for con- tinuous ones. One way to address this is to only rely on online estimations, where a goal's 's estimated competence is only updated when this goal is practiced. Online estimations nonetheless fail to capture competence transfer and existing methods addressed this by grouping goals with similar com- petence together. For continuous spaces, approaches either learn to partition the space directly when dimensionality is low (Oudeyer et al., 2007; Baranes & Oudeyer, 2013; Porte- las et al., 2020a), or first embed high-dimensional goals into a lower-dimensional space before partitioning (Laversanne- Finot et al., 2018; Kovač et al., 2023). For discrete spaces, methods typically rely on expert-defined groupings (Stout & Barto, 2010; Matiisen al., 2020). However, these grouping approaches are inherently brittle: they assume transfer between groups while potentially masking competence vari- ations within groups. This limitation is particularly acute for high- nigh-dimensional structured spaces like natural language, where competence transfer naturally occurs between se- mantically similar goals regardless of predefined groupings. Instead, MAGELLAN leverages an L.LM's semantic under- standing to dynamically model competence transfer between goals, enabling efficient and adaptive LP estimation without requiring predefined groupings or exhaustive evaluation.
3. Methods
In this section, we detail how MAGELLAN learns a metacognitive module that estimates and generalizes an agent's 's LP over language goal spaces. We then explain clas- sic IP baselines against which MAGELLAN is compared. Finally, we introduce the Little-Zoo environment, specifi- cally designed to study commonsense-based generalization abilities of LLM agents when facing large language goal space.
3.1. Problem statement
Let M = (S, T, be an MDP, with S a set of states, T the transition function, A the action space and R the reward function. Let G be a goal space and 11 the policy space. We define a competence function CM, G - R that indicates the competence of policy final aim is to find the optimal policy x² that maximizes
1I for goal in M. The
JM(TT Ey-u(cy [Eg.u(cy(CM..(()))
(g),
where U(X) is the uniform distribution over a set X.
In this paper, we focus on episodic online goal-conditioned RL with sparse and binary rewards, defined on a goal- augmented MDP A, G. R), with R: SS G + (0; 1} a binary success function indicating whether state sat- isfies goal g. Here, we define G (So x 7|So C S). with I an instruction space and So the set of initial states. We consider a textual environment where a prompting func- tion Φ : 5 x 1 + DK is given to transform any pair (state, instruction) into a textual prompt of K tokens in a given vocabulary V. Thus, from the agent side, the policy π se- lects any action On E by sampling from a categorical distribution x (.lō(sh, i)) at any step h of the episode.
2.3. Autonomous LLM agents
For our competence function we use the success probability P (so. a) defined as the probability for x, starting from sp. to fulfill within H steps: Pa (S0, i) = Ex (7) [r...b with r, = 1(5s) E T, R(sn, 2) = 1) the goal outcome
Recent work has explored using Large Language Models (LLMs) to solve complex tasks in interactive environments. Early approaches focused on direct action prediction using LLMs, either incorporating environmental feedback (Huang et al., 2022; Yao et al., 2022: Hao et al., 2023; Shinn et al., 2023; Wang et al., 2024) or operating without (Ahn al.,
IIn all generality a competence function does not assume the goal be inside the MDP. For instance, g could ask for a maximum number of steps.
3

---

MAGELLAN
of episode 7 for instruction i, the indicator function and (0,
focuses on success probability (i.e., we want Co. R Prr (g)), we train Co, using stochastic gradient de- scent to minimize the binary cross-entropy: [(O+) B(g.)>D, [BCE(r, (9)(.
ot (r, i)) the distribution of episodes of H steps in-
duced by x to fulfilling i from so. In this setting our objec- tive becomes:
We maintain another buffer B, storing the last N weights of our competence estimator: B, = [Or-N, 0:41-N,..., Or]. Weights are added the buffer every time the competence estimator is updated, enabling access to estimations of the policy's competence from time to t N. Using this in- formation, we estimate the absolute LP (ALP) Baranes & Oudeyer (2013); Kanitscheider et al. (2021), tracking both progress and forgetting, as follows:
(Tx) E ((0)))())) [Pxx (so. i)].
However, given the possibly huge number of goals (su, i), the direct maximization of the problem becomes particularly inefficient. Our aim is to leverage transfer of competence between goals and focus during training on the ones max- imizing LP. We denote as x² the policy obtained after episodes using an RI. algorithm, with I the set of all tra- jectories collected during training using {* [[I, d|]]. The goal of each episode is sampled using task selec- tor 'iG that selects a goal based on collected trajectories Ma(TT) g. Given a budget of T training episodes, we thus consider the problem of approaching the optimal se- lector n2= = arg maxna J(="). In particular, we build on prior work to construct GGG on a proxy of the LP at each training episode f. We define the LP for any goal g = (so; i) as the improvement of the policy at episode after k episodes on goal g: LP (g) = Fgtii(() (g) - Pr (g). As highlighted in Section 2.2, since computing the future competence on all goals is intractable, prior approaches ap- proximate future progress with past progress (I.P.. (g) R (g) - p=,- * (g)). Nonetheless, accurately estimating past and current competence remains a challenge in large discrete goal spaces.
AÎ.P n. (9) ( (g) -O (g)l-
(1)
This ALP estimation can subsequently be used to structure the agent's curriculum. We apply the multi-armed bandit goal selection scheme introduced by (Lopes & Oudeyer, 2012b) where each arm a goal, and its utility is MAG- ELLAN's estimate of the ALP of this goal. Goals are then sampled proportionally to their estimated ALP with an- nealing e-greedy scheme (e decreasing from I to 0.2). In practice, we train two separate versions of the same initial L.M (using LoRA adapters (Hu et al., 2022)): one for the policy and one for MAGELLAN's current competence esti- mator. We show Appendix D.1 ablations on architectural choices indicating that 1) keeping the LLM frozen leads to poor results, highlighting the need for dynamic repre- sentation space (see also Figure 6). and 2) training separate LoRA adapters for the policy and MAGELLAN leads to more stability.
3.2. Metacognitive generalization of learning progress in language model agents
With MAGELLAN, we propose learn estimators of the current and past policy's 's competence for any goal. As op- posed to prior works, which either consider all goals in- dependently or use goal groupings, we argue that learning goal-conditioned estimators would allow generalization be- tween similar goals without defining any clear group. We propose to leverage the LLM used by our agent to learn the parameters O, of competence estimator Ce. (g) for policy ne on goal g- We compute Co, (g) by giving g in the LLM's prompt, which produces a latent representation on top of its final decoder block for the last token. We use a Multi-Layer Perceptron (MLP) to output the estimated competence based on this representation. We train both the LLM and the MIP, leveraging the LLM'' ability to project goals into a latent space where semantically similar goals close. By updating the estimated competence of one goal, this allows MAGELLAN to also update close goals.
3.3. Classic ALP baselines
Following the literature on ALP in Section 2.2, we imple- ment classic approaches, focusing on two dimensions. First, we consider Online (Baranes & Oudeyer, 2013; Matüsen et al., 2020) vs Evaluation- ALP (Kanitscheider et al., 2021; Zhang et al., 2024) estimation. Then, we consider directly using the goal space (Portelas al., 2020a; Kan- itscheider et al., 2021) or using expert-defined groups of goals with assumed competence transfer (Stout & Barto, 2010; Colas et al., 2019). The latter requires extensive expert knowledge (EK) given the absence of automatic ap- proach for discrete goal spaces. As expert-defined groups are created beforehand, no competence transfer is assumed across groups, which is likely happen in spaces like nat- ural language, where transfer occurs between semantically close goals regardless of groups
In practice, we maintain a buffer De which contains, for the M most recent training episodes at t (i.e., T T"), their corresponding pair of goal and out- come (i.e. (g = =80, i), r+a) for each r). As this work
We thus implement four baselines (see all details in Ap- pendix C.3): Eval-ALP: Every N episodes, training stops and the
4

---

MAGELLAN
agent is separately evaluated on each goal obtain competence estimate. The per-goal ALP is the absolute difference between estimates at t and & N. The same goal selection scheme as in MAGELLAN is used according to the per-goal ALP estimations.
Eff.
Transf.
No EK
EK-Eval-ALP
low
+
Eval-ALP
low
++
EK-Online-ALP
high +
X
Online-ALP
high
-L-E Every N episodes, training stops and the agent is evaluated on multiple goals randomly sam- pled in each expert-defined group to obtain a per-group averaged competence. The per-group ALP is computed using the absolute difference between the competence at l. and / - N. The goal selection process first selects group using the same selection scheme as MAGEL- LAN for goals. Given a selected group, a goal from this group is randomly selected.
MAGELLAN
high
Table I. Comparison of ALP estimation methods. We use the fol- lowing dimensions: computational Efficiency, competence Trans- fer tracking. and required Expert Knowledge.
guage. It should feature a diverse set of goals with varying difficulty levels, enabling the assessment of the agent's 's abil- ity to learn and generalize complex skills. Additionally, these goals should be organized into hidden families based on commonsense knowledge, allowing for targeted evalua- tion of generalization capabilities.
Online-ALP: At each goal practiced, the observed policy's competence is added to a buffer of 2M past experiences for this goal. The ALP is computed using the absolute difference between the average compe- tence over the last and first M experiences in the buffer. The same goal selection scheme as MAGELLAN and Eval-ALP is used.
Existing environments for LLM agents do not have such re- quirements. Creative environments like Minecraft (Johnson et al., 2016) Crafter (Hafner, 2022) rely on image-based observations and require an image captioner to use LLM agents. Textual environments like BabyAl-text (Carta et al., 2023) focus on navigation skills without any commonsense- based generalization. Although WordCraft (Jiang et al., 2020) incorporates commonsense-basse goals, no relation- ship between goals exists, limiting the analysis of the agent's s generalization abilities. To address these gaps, we introduce Little-Zoo, novel environment explicitly designed to meet these criteria.
EK- EK-Online-AL.. At each goal practiced, the observed policy's competence is added to a buffer of 2M past experiences for the goal's 's expert-defined group. The per-group ALP is computed using the absolute differ- ence between the average competence over the last and first M experiences in the group's buffer. The same goal selection scheme as EK-Eval-ALP is used.
We summarize the four methods above and MAGELLAN in Table based on their Efficiency (i.e. computational cost introduced by additional evaluations), Competence Transfer tracking and no Expert Knowledge requirement. We con- sider method's efficiency as "high" if it does not require any additional evaluation (i.e. it only uses the performance observed on goals sampled), and as "low" otherwise. We evaluate the competence transfer tracking using the follow- ing criteria:
Built upon the Playground enviromment (Colas et al., 2020), Little-Zoo is fully text-based, with observations, goals, and actions expressed in natural language. It features objects that can be combined together and are grouped into the following hidden categories: furniture (which cannot be combined), plants, herbivores, and carnivores. Given the set of all objects and a set of instructions, Little-Zoo's goal space is the combination of all possible instructions and scene initializations. The feasibility of a goal thus depends on the objects available, making most combinations infeasi- ble and not trivial to detect (see Figure 2 and Appendix A.3). For instance, these are respectively feasible and infeasible goals: "Goal: Grow deer. You see: baby deer, bookshelf. water, tomato seed."; "Goal: Grow deer. You see: baby deer, bookshelf, baby lion, tomato seed." (water is missing in the last one). Instructions are hierarchically structured, ranging from simple grasping tasks to more complex sequences in- volving object interactions (e.g. ""growing" animals). The complete goal space contains approximately 20 million com- binations. In our experiments, we subsample goals with the following proportions: 80% of the goals are impossible 16% involve grasping, 3.2% involve growing plants, 0.7% involve herbivores, and 0.1% involve carnivores. These proportions correspond to proportions in the complete goal
absence of +: the estimated competence is updated only on sampled goals.
+: the estimated competence is updated on predefined
group the sampled goal belongs to.
++: the estimated competence is updated on a dynami- cally learned group the sampled goal belongs to.
+++ the estimated competence is updated on all goals.
We provide in Appendix B the same table for all prior works covered in Section 2.2.
3.4. The Little-Zoo environment as testbed
Evaluating commonsense-basee generalization of LLM agents in textual environments requires several key prop- erties. The environment must be fully text-based, with all observations, actions, and goals expressed in natural lan-
5

---

MAGELLAN
Goal space
performance will probably be very close the random base- line. We thus provide an additional group containing all impossible goals. In all our experiments, we use success rate (i.e. average outcome over multiple trials for a goal) noted SR as the observed competence.
Insrrcsioon
Sraap Eable"
Hiant
SP
CCrw Ilon
Growr shrap?
We first study how well MAGELLAN's competence esti- mation compares to baselines (Q1). Then, we study how the different methods (except Eval-ALP and EK-Eval-ALP, which are too costly to run while EK-Online-ALLP provides provvdde good estimation of their performance) compare when scaf- folding the LLM agent's curriculum (Q2). We show how these competence estimators also generalize to goals not seen during training (Q3). Finally, we study how all meth- ods adapt as the goal space evolves (Q4) by replacing all goals at different points throughout training.
Difficutty
Grow Akon"
P conned Hel7
swaly
truumcaariing fring
LLM agent
Sweelory/ cannt
Adttrr
On a0M
Ashio pOAg
C)
Figure 2. A) I. Little-Zoo's ' tech tree. B) Liutl-Zoo's goal space is composed of all the possible combinations between instructions and objects that can be in the scene. Most object configurations make an instruction infeasible (e.g. "grow lion" is impossible with the second configuration, as water, needed to obtain plants, is missing). C) Little-Zoo provides a textual description that is given in our LLM agent's 's prompt.
4.1. How well does MAGELLAN estimate competence (Q1)
To assess the ALP methods' ability to efficiently estimate competence, we designed an experimental setup in which our LLM agent was trained for 50k episodes on the Little- Zoo environment with varying goal space sizes (25k, 50k, 100k), while keeping the same repartition between goal types. As computing the expected ALP to train this agent is intractable, one could argue that Eval-AL.P is the best approximation. However, it remains too computationally costly to run, even when performing only 50k training episodes with 25k goals. We thus chose to sample goals according to EK-Eval-ALPPs estimations. To obtain an ac- curate estimate, we perform 2048 per-group evaluations every 1000 episodes. The per-group competence evaluated by EK-Eval-ALP is consequently our competence reference, and we compare the other methods against it. For Eval- ALP, we consider it have zero error, and its computational cost can be estimated without rumning it. For MAGELLAN and Online-ALP, we average the ver-goal competence over groups to compute the error w.r.t. EK- EK-Eval-ALL Figure 3 shows the average error on competence throughout training along and the cost of competence evaluation (i.e. the total number of episodes used only to evaluate competence).
space (see Appendix A.4).
Lottle-Zoo is a deterministic, fully-observable and episodic environment: the agent begins an episode by standing on nothing, with full visibility of the whole scene. The action space consists of 8 actions, including movement to objects, grasping, and releasing objects. Observations include the objects in the scene, the ones in the agent's inventory, as well as the object the agent is standing on. See Appendix A for details on Little-Zoo.
4. Experiments
We provide empirical answers to our scientific questions using experiments with 8 different random seeds in the Little-Zoo environment. For our LLM agent, we use SAC- GLAM (Gaven et al., 2024) to finetune Flan-T5 248M (Raf- fel et al., 2020) as per SAC-GLAM's experiments. We compare MAGELLAN to the classic approaches presented in 3.3. For methods accessing external expert knowledge, we use Little-Zoo's hidden goal families (grasp any ob- ject, grow plant, grow herbivore, grow carnivore) but add only possible goals in these groups. Indeed, these baselines are based on groups predefined in advance by human ex- perts with strong assumption: the goals within a group share the same learning dynamics and therefore the agent's 's competence is the same over all goals in the group. If im- possible goals were included, these groups would lose their relevance. Moreover, because of the large number of im- possible goals, the average competence within each group will always be very close to O. There will be no progress niche that the method can use to generate curriculum, and
As indicated in Table MAGELLAN performs on par with Eval-ALP, showing that it accurately estimates the transfer of competence while using online estimations. We also observe similar competence errors to methods using expert-defined groups, hinting at MAGELLAN's abilities at learning semantical relationships between goals. We pro- vide a more in-depth analysis of such relationships in Sec- tion D.4. Finally, MAGELLAN achieves this performance without an estimation cost.
To demonstrate that our method generalizes beyond the Little-Zoo environment, we conducted an additional exper- iment using goals derived from the OpenR1 1-enRIIMMath Math-2200
6

---

MAGELLAN
4.2. Training an LLM agent with MAGELLAN (Q2)
0.30
NAGELLAN
23k goals
As demonstrated in 4.1, MAGELLAN provides superior competence estimation than Online-ALP. We further in- vestigate whether this improvement translates into better curriculum and improved overall goal mastery. We train our LLM agent on the goal space of Little-Zoo with 25k goals for 500k episodes using four methods: MAGELLAN, Online-ALA EK-OOlineeALL and "Uniform", where goals are sampled uniformly. We do not report EK-Eval-ALP, as report EK-Online-ALP which produces similar compe- tence estimation with no cost. We report the agent's SR every 5000 training episodes by evaluating it on 64 goals uniformly sampled for each category. Figure 5 shows the evolution SR averaged over all categories. Our results show that MAGELLAN is the only method without expert- defined grouping to obtain an SR of least 90% in all categories. It also masters the categories significantly faster than baselines. Despite MAGELLAN's 's similar competence estimation as EK-Online-ALP, the latter learns faster by leveraging the expert-defined groups to better explore. This is because MAGELLAN explores by uniformly sampling goals whereas EK-Online-ALe uniformly samples groups, easily discarding impossible goals.
Competence Estimation
0.25
goals
0.20
Online ALP
100k
0.15
EK-Evai-ALP
0.30
EK- OnOninne-ALP
0.05
0.00
0.2
0.6
0.8
1.0
Competence Estimation (Episodes)
1e10
Figure 3. Scaling of competence estimation error and competence estimation cost (i.e. total number of additional evaluation episodes) when increasing the goal space size.
dataset (Face, 2025). In this setup, rather than training real agent, we simulate the learning of an agent that progressively acquires skills in three categories: Algebra, then Geometry, and finally Number Theory. We compare the competence estimation from MAGELLAN and Online- ALP to the underlying true competence. Figure 4 shows that MAGELLAN accurately tracks the agent's competence and distinguishes between the different mathematical cat- egories, clearly outperforming Online-ALP. Additional re-
1.2
MAGELLAN
Online-Au ALP
Uniform
EK-Onnine-AAL
sults, including experiments in the Baby AI-TAT-Text environ- ment Snevalier-Boisveee et al., 2019; Carta et al., 2023)
Success Rate
0.8
and an ablation study on the impact of LLM size within MAGELLAN, are provided in Appendix D.2.
0.6
0.4
MAGELLAN
OrlrwsU
Toe Succesa Probubiiity
0.2
Algebra
Geometry
Success Probablicy
Seccees Proteeiilkyy
0.8
9.8
100K
200k
300k
400k
500k
Episodes
0.6-
9.8
8.1
Figure 5. Evolution of the observed competence (SR) when eval- uating policies on 64 training goals per category every 5000 episodes. We report the average SR over evaluated goals along with standard deviation (8 seeds). Icons indicate the average time step at which method mastered a goal (i.c. SR >> 90%). We add stars to MAGELLAN, denoting significantly earlier mastery of a category compared the method with the star's color (p-value < 8 x 10 *). The dotted line (EK- Online--nnn ALP) indicates that the method relies expert knowledge.
0.0
10k
15c
2dc
Sc
10k Episodes
15c
28k
Epicodes
Number Theory
Frebabiity....
0.88
0.6
0.0
10x Spicodes
15k
4.3. MAGELLAN's generalization abilities (Q3)
Figure 4. Competence estimation on OpenR -Math---M MAG- ELLAN (blue) accurately tracks competence across Algebra, Ge- ometry, and Number Thcory, closely matching true success proba- bilities and outperforming Online-AI.P (orange).
We move further and study the generalization abilities of both our LLM agent and competence estimators. While Section 4.2 evaluates each policy on 64 goals per category that belong to the training goal space, this section reports
7

---

MAGELLAN
evaluation on held-out composed of unseen goals. As in the previous section, evaluations were performed every 5000 training episodes. However, instead of reporting the policy's s observed competence (SR) during evaluation, we report the difference between the observed competence and the competence estimated by the policy's ALP method. We show in Table 2 the average error over training.
1.0
Irpooable
hattnore
Grapp
Dana platt
Estimated Success Probabiity
06
0
By only tracking competence on goals practiced by the policy (i.e. the ones from the training goal space), Online- ALP cannot provide any estimation for new unseen goals and uses its default competence of 0. This leads to the largest error among methods except for "Grow carnivore" goals as the policies trained with Online-ALP never mastered these goals (see Figure 5). MAGELLAN successfully generalizes its competence estimation and obtains a small error. Finally, EK RK-Online-ALL produces accurate estimations based expert knowledge of which group each test goal belongs to. Appendix D.4 provides detailed results indicating our LLM agents do not perfectly generalize, which explains MAGELLAN and EK-Eval-ALP estimation error.
0.0
1.0
Impoesible
hartivere
Drup
Crow sarswors
0.8
Estimated Success Probablity
0,0
Table We evaluate the policies trained with each ALP method on a held-out test set. We show the difference between the observed and predicted competence. Online-ALL performance on "Grow carnivore" is simply explained by the fact that its policies never mastered this goal category.
0.0
(b)
Categories
MAGELLAN (Mean Std)
Online-ALP (Mean Std)
EK-Online-ALPP (Mean
Figure 6. MAGELLAN's LLM embedding space displayed using t-SNE with goals used in Q2 (Train) and Q3 (Test), along with the estimated success probability and linear interpolation between goals. We show the embedding space for a single seed (a) before training and (b) at the end of the 500k training steps. We see that impossible goals have been left aside, and that the other goals with high estimated success probability are clustered consistently.
Grasp
0.01 0.00
0.98 0.00
0.01 # 0.00
plant
0.78 0.07
Grow herbivore
0.08 = 0.05
0.34 0.18
0.06 + 0.02
Grow carnivore
0.30 = 0.16
0.00 0.00
0.28 + 0.08
Mean
0.11 # 0.06
0.53 + 0.06
0.09 + 0.03
We further investigate MAGELLAN's generalization abil- ities by projecting train (Q2) and test (Q3) goals from single seed into the LLM embedding space MAGELLAN used. The embedding space is plotted both before and af- ter training (Figure 6) with projections obtained via t-SNE (van der Maaten & Hinton, 2008). The initial embedding space lacks any discernible structure for goal classification. Post-training, the space exhibits significant restructuring, with similar goals accurately clustered. A small subset of "Grow carnivore" goals are misclassified as impossible, likely due to incomplete mastery of this category by the LLM agent.
4.4. MAGELLAN's adaptation to evolving goal spaces (Q4)
Finally. we investigate how each ALP method can adapt when the goal space evolves. For this, isolate the training of one seed using MAGELLAN in Section D.3. Every 50k episodes over the 500k training episodes, we stop training, replace the training goals with the ones in our held-out test set, and start four trainings (with 8 seeds each) with this new goal space: one with each ALP method for 50k steps, We expect MAGELLAN to quickly adapt to new goals leveraging semantical relationships between the new and old goals. Maline-ALP starts with competence estimation of O on the new goals as in Section 4.3. For EK- EK--Onine-AAL. to the episode where we change the goal space, we make it track the policy's competence in parallel to MAGELLAN, It thus starts with per-group competence estimation when the goals are replaced along with the information of which
Furthermore, the spatial arrangement of goals correlates with estimated success rates: newly learned goals tend to lie near the boundary with impossible goals. Additionally, goals from the test set are well clustered, demonstrating strong generalization. We show in Appendix D.4.2 that MAGELLAN also makes different clusters for impossible goals based on the infeasibility reason.
8

---

MAGELLAN
LAN's ability to capture semantic relationships between goals allows it assess the agent's competence on unseen goals and rapidly adapt as the goal space evolves. While our study provides an in-depth analysis of MAGELLAN's capabilities in a controlled textual environment, the method itself is highly generalizable. It offers a goal prioritization strategy applicable to any learner operating in a large, high- dimensional, structured, and discrete goal space. Notably, goal spaces involving code appear partieularly promising, given the proficiency of current LLMs in code generation (Wang et al., 2024; Pourcel et al., 2024). Additionally, traditional automatic curriculum learning settings-where goals are not language-defined could also benefit from MAGELLAN's ability to uncover relationships between goals. Beyond artificial learners, MAGELLAN may also prove valuable for human learning, particularly in educa- tional domains where learners must navigate large space of language-defined problems, such as mathematical word problems (Doroudi et al., 2019). Classical LP measures have already been shown to enhance personalized curricula in educational technologies (Clement et al., 2015), suggesting MAGELLAN's potential impact in this area.
Mann
Saaceens Sate
002
390
002
5p uodes
Splaces
(a)
(b)
Figure 7. Adaptation tests: Using a single's seed training of 500k episodes, we stop and replace all goals with unseen ones every 50k episodes. We then resume training and sample goals using each method for 50k training episodes. We show two isolated and representative points of goal replacement: (a) there is no ALP on any goal (after 50k training episodes), and (b) some goals (here, "Grow carnivores" after 150k training episodes) have high ALP. We report the evolution of SR when evaluating the policies on 64 goals per category from the new training set every 5000 episodes. Results show the average competence over evaluated goals along with standard deviation (8 seeds).
expert-defined group each new goal belongs to. We study all adaptation training in Appendix D.5 but isolate and study in Figure 7 two of them chosen as representative of the scenarios encountered:
Impact Statement
Large Language Models (LLMs) are now widely used in real-world applications, and the recent rise of LLM agents empowered with actions such as web search or tool use greatly enhanced their capabilities. In this work, we study open-ended learning mechanisms for such LLM agents, paving the way for agents solving very large and diverse task spaces. While our results indicate significant improvement for LLM agents to learn in such task spaces, our experiments were limited to small-scale LLMs and well-controlled ex- perimental testbeds. Consequently, we do not recommend generalizing our findings to real-world open-ended learn- ing settings. Additionally, our work introduces an efficient method for measuring Learning Progress (LP), an intrinsic motivation signal notably used for prioritizing tasks given to human learners in multiple real-world educational tech- nologies, However, our experiments did not include human learners, and the real impact of our method on accurately measuring human LP remains to be evaluated.
Scenario zero LP (Figure 7a): The agent has mas- tered the "Grasp" and "Grow plants" goals and has 0 ALP across all goals. In this scenario, all ALP es- timations are equivalent. EK-Online-LP manages to discover new ALP niches faster as all impossible goals are in the same group.
Scenario high LP (Figure 7b): The agent is getting high ALP it is learning some "Grow carnivores" goals. Here, MAGELLAN outperforms baselines by generalizing its ALP estimation and continuing train- ing on these goals. MAGELLAN even gets on par performance with EK-Online-LP.
5. Conclusion
In this paper, we introduce MAGELLAN, metacognitive module designed to efficiently estimate an LLM agent's s learning progress (LP) across large language-defined goal spaces. By leveraging the LLM within the agent, MAGEL- LAN accurately and efficiently estimates LP by learning semantic relationships between goals. This approach funda- mentally differs from prior methods, which either struggle to scale to large goal spaces or rely on exfert-defined goal categories. Using MAGELLAN's LP estimations, the L.LM agent can effectively structure its curriculum, enabling it to master all goals within an extensive goal space. In con- trast, previous methods achieve only partial mastery in the absence of expert-defined categories. Moreover, MAGEL-
A Acknowledgment
This work was granted access to the HPC resources of IDRIS under the allocation A017101 1996 made by GENCI. We acknowledge funding from the European Commission's Horizon Europe Framework Programme under grant agree- ment No 101070381 (PILLAR-robots project), the French Defense Innovation Agency, and the ANR AI Chair grant ANR-19-CHIA-ONN We also thank Nicolas Yax for kindly proofreading the paper and providing valuable feedback.

---

MAGELLAN
References
2023, 23-29 July 2023, Honolala, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 3714-3731. PMLR, 2023.
Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes, O.,
David, B., Finn, C., Gopalakrishnan, K., Hausman, K., Herzog, A., Ho. D., Hsu, J. Ibarz, J., Ichter, B., Irpan, A., Jang, E., Ruano, R. M. J., Jeffrey, K., Jesmonth, S., Joshi, N. J., Julian, R. C.,, Kalashnikov, D., Kuang, Y., Lee, Levine,
Chevalier-ooisver,, M., Bahdanau, D., Lahlou, S., Willems, L., Saharia, C., Nguyen, T. H., and Bengio, Y. Babyai: platform study the sample efficiency of grounded lan- guage learning. In International Conference Learning Representations, 2019.
Levine, S., Lu, Y., Luu, L., Parada, C., Pastor,
P. Quiambao, J., Rao, K., Rettinghouse, J., Reyes, D. M., Sermanet, P., Sievers, N., Tan, C.,, Toshev, A., Vanhoucke, V., Xia, F. Xiao, T., Xu, P., Xu, S., and Yan, M. Do as i can, not as i say: Grounding language in robotic affordances. In Conference on Robot Learning, 2022.
Clement, B., Roy, D., Oudeyer, P.-Y,, and Lopes, M. Multi- armed bandits for intelligent tutoring systems. Journal of Educational Data Mining, 7(2), 2015.
Akakzia, A., Colas, C., Oudeyer, P.-Y., Chetouani, M., and Sigaud, O. Grounding language autonomously- acquired skills via goal generation. In International Con- ference Learning Representations, 2021.
Colas, C., Fournier, P., Chetouani, M., Sigaud, O., and Oudeyer, P.- CURIOUS: intrinsically motivated mod- ular multi-goal reinforcement learning. In International conference machine learning, pp. 1331-1340. PMLR, 2019.
Baldassarre, G. and Mirolli, M. Intrinsically motivated learning systems: an overview. Intrinsically motivated learning natural and artificial systems, pp. 1-14, 2012.
Colas, C., Karch, T., Lair, N., Dussoux, J.-M., Moulin- Frier, C., Dominey, F., and Oudeyer, P.-Y. Language cognitive tool imagine goals in curiosity-driven exploration. arXiv:2002.09255: [cs], 2020.
Baranes, A. and Oudeyer, P.-Y. R-IAC: Robust intrinsi- cally motivated exploration and active learning. IEEE Transactions on Autonomous Mental Development, 1(3): 155-169, 2009. ISSN 1943-0612. doi: 10.1109TTAMD. 2009.2037513. Conference Name: IEEE Transactions on Autonomous Mental Development.
Colas, C., Karch, T., Moulin C., and Oudeyer, P.- Y. Language and culture internalization for human -ike autotelic ai. Nature Machine Intelligence, 4(12): 1068- 1076. 2022a.
Colas, C., Karch, T., Sigaud, O., and Oudeyer, P.- Y. Autotelic agents with intrinsically motivated goal- conditioned reinforcement learning: short survey. Jour- nal of Artificial Intelligence Research, 74:1 1199, 2022b.
Baranes, A. and Oudeyer, P.-Y. Active learning of inverse models with intrinsically motivated goal exploration in robots. Robotics and Autonomous Systems, 61())49-77, 2013. ISSN 0921-8890. doi: r00022.5.. 008.
Dennis, M., Jaques, N., Vinitsky, E., Bayen, A., Russell, S., Critch, A., and Levine, S. Emergent complexity and zero-shot transfer via unsupervised environment design. Advances in Neural Information Processing Systems, 33, 2020.
Berlyne, D. E. A theory of human curiosity. British Journal of Psychology, 1954.
Blaes, S., Vlastelica Pogančić, M., Zhu, J., and Martius, G. Control what you can: Intrinsically motivated task- planning agent. In Advances in Neural Information Pro- cessing Systems, volume 32. Curran Associates, Inc., 2019.
Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. QLoRA: Efficient finetuning of quantized L.MSs. In seirth- Conference Neural Information Pro- cessing Systems, 2023.
Carta, T., Romac, C., Wolf, T., Lamprier, S., Sigaud, O., and Oudeyer, P.-Y. Grounding large language models in inter- active environments with online reinforcement learning. In Proceedings of the 40th International Conference on Machine Learning, pp. 3676-3713, 3713. PMLR, 2023, ISSN: 2640-3498.
Doroudi, S., Aleven, V., and Brunskill, Where's the reward? review of reinforcement learning for instruc- tional sequencing. International Journal of Artificial Intelligence in Education, 29:568-6221 2019.
Face, H. Open r1: fully open reproduction of deepseek- rl, January 2025. URL https: //t:::///tthbt com/ huugiingface//pen--r
Castanet, N., Sigaud, O.,, and Lamprier, S. Stein variational goal generation for adaptive exploration in multi-goal reinforcement learning. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), International Conference on Machine Learning, ICML.
Forestier, S. and Oudeyer, P.-.. Modular active curiosity- driven discovery of tool use. In Proceedings of the 2016
10

---

MAGELLAN
IEEE/RSJ International Conference on Intelligent Robots and Systems, Proceedings of the 2016 IEEE/RSJ Inter- national Conference Intelligent Robots and Systems, 2016.
Jiang, M., Dennis, M., Parker-Holder, J., Foerster, J., Grefen- stette, E.,, and Rocktäschel, T. Replay-guided adversarial environment design. In Proceedings of the 35th Inter- national Conference on Neural Information Processing Systems, NIPS '21, pp. 1884-1897, Red Hook, NY, USA, 2021. Curran Associates Inc. ISBN 978-1-71384-539-3
Forestier, S., Portelas, R., Mollard, Y., and Oudeyer, P.- Y. Intrinsically motivated goal exploration processes with automatic curriculum learning. Journal of Machine Learning Research, 23(1), January 2022. ISSN 1532- 4435.
Jiang, M., Rocktäschel, T., and Grefenstette, E. General in- telligence requires rethinking exploration. Royal Society Open Science, 10(6):230539, 2023.
Johnson, M., Hofmann, K., Hutton, T., and Bignell, D. The malmo platform for artificial intelligence experimentation. In Ijcai, volume 16, pp. 2246-4247 2016.
Fournier, P., Sigaud, O., Chetouani, M., and Oudeyer, P.-Y. Accuracy-basse based curriculum learning in deep reinforce- ment learning, 2018.
Kanitscheider, I., Huizinga, J., Farhi, D., Guss, W. H., Houghton, B., Sampedro, R., Zhokhov, P., Baker, B., Ecoffet, A., Tang, J., Klimov, O., and Clune, J. Multi- task curriculum learning in complex, visual, hard- exploration domain: Minecraft, 2021.
Gaven, L., Romac, C., Carta, T., Lamprier, S., Sigaud, O., and Oudeyer, P.-Y. Sac-glam: Improving online rl for Ilm agents with soft actor-critic and hindsight relabeling, 2024.
Gottlieb, J. and Oudeyer, P.-Y. Towards a neuroscience of active sampling and curiosity. Nature Reviews Neuro- science, 19(12):758-777) -70000 2018.
Kaplan, and Oudeyer, P.-Y. In search of the neural circuits of intrinsic motivation. Frontiers in neuroscience, 1:9, 2007.
Hafner, D. Benchmarking the spectrum of agent capabilities. In International Conference on Learning Representations, 2022.
Kidd, C. and Hayden, B. Y. The psychology and neuro- science of curiosity. Neuron, 88(3):449-460) 2015.
Kovač, G., Leversanne-Finot A., and Oudeyer, P.-Y.
Hao, S., Gu, Y., Ma, H., Hong, J., Wang, Z., Wang. D., and Hu, Z. Reasoning with language model is planning with world model. In Bouamor, H., Pino, J., and Bali, K. Empiri-
Grimgep: Learning progress for robust goal sampling in visual deep reinforcement learning. IEEE Transac- tions on Cognitive and Developmental Systems. 15(3): 1396-1407, 2023. doi: 10.1 2.1110//CDDS
(eds.), Proceedings of the 2023 Conference cal Methods in Natural Language Processing, pp. 8154 8173, Singapore, 2023. Association for Computational Linguistics. doi: 10.18653/11/200 1ttp33//1/2222.omm wwt.:/022..mmmmmmmmmmmm mmm.emmmmmlm5mm000.777 507.
Kumar, N., Silver, T., McClinton, W., Zhao L.., Proulx, S., Lozano-Pérez, T., Kaelbling, L. P., and Barry, J. Practice makes perfect: Planning to learn skill parameter policies. In Robotics: Science and Systems (RSS), 2024.
Held, D., Geng, X., Florensa, C., and Abbeel, P. Automatic goal generation for reinforcement learning agents. In International Conference on Machine Learning, 2017.
Laversanne-Finot, A., Pere, A., and Oudeyer, P.-Y. Curiosity driven exploration of learned disentangled goal spaces. In Billard, A., Dragan, A., Peters, J., and Morimoto, J. (eds.), Proceedings of The 2nd Conference on Robot Learning. volume 87 of Proceedings of Machine Learning Research, pp. 487-504. PMLR, 29-31 Oct 2018.
Hu, E. J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
Leonard, J. A., Cordrey, S. R., Liu, Z.,, and Mackey, P. Young children calibrate effort based on the trajectory of their performance. Developmental Psychology, 59(3): 609, 2023.
Huang, W., Abbeel, P., Pathak, D., and Mordatch, I. Lan- guage models as zero-shot planners: Extracting action- able knowledge for embodied agents. In International conference machine learning, pp. 9118-9147, PMLR, 2022.
Lopes, M. and Oudeyer, P.-Y. for life-long exploration and learning. In 2012 IEEE international conference on development and learning and epigenetic robotics (ICDL), pp. 1-8. IEEE, 2012a.
The strategic student approach
Jiang, M., Luketina, J., Nardelli, N., Minervini, P., Torr, P., Whiteson, S., and Rocktäschel, T. Wordcraft: An environment for benchmarking commonsense agents. In Language in Reinforcement Learning Workshop at ICML 2020, 2020.
Lopes, M. and Oudeyer, P.-Y. The strategic student approach for life-long exploration and learning. In 2012 IEEE

---

MAGELLAN
International Conference on Development and Learning and Epigenetic Robotics (ICDL), pp. 1-8, 2012b. doi: MAL109/DeeLm.222224440555 ISSN: 2161-9476.
Portelas, R., Colas, C., Weng, L., Hofmann, K., and Oudeyer, P.-Y. Automatic curriculum learning for deep rl: A short survey. In International Joint Conference on Artificial Intelligence, 2020b.
Matiisen, T., Oliver, A., Cohen, T., and Schulman, studeet curriculum learning, 2020.
Pourcel, J., Colas, C., Molinaro, G., Oudeyer, P.-Y., and Teodorescu, L. ACES: Generating diversity of chal- lenging programming puzzles with autotelic generative models. In The Thirly-eighth Annual Conference Neu- ral Information Processing Systems, 2024.
Matthews, M., Beukman, M., Lu, C., and Foerster, J. Kinetix: Investigating the training of general agents through open-ended physies-basee based control tasks, 2024.
Moulin-Frier, C. and Oudeyer, P.-Y. Exploration strate- gies in developmental robotics: A unified probabilistic framework. In 2013 IEEE Third Joint International Con- ference on Development and Learning and Epigenetic Robotics (ICDL), pp. 1-6, 2013. doi: 10.1109/Devl.m. 2013.6652535. ISSN: 2161-9476.
Qwen, :, Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F. Wei, H., Lin, H., Yang. J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J.. Lin, J., Dang, K., Lu, K., Bao, K., Yang, K., Yu, L., Li, M., Xue, M., Zhang, P., Zhu, Q. Men, R., Lin, R., Li, T., Tang, T., Xia, T., Ren, X., Ren, X., Fan, Y., Su, Y., Zhang, Y. Wan, Y., Liu, Y. Cui, Z., Zhang, Z., and Qiu, Z. Qwen2.5 technical report, 2025. URL https: //arxiv. www..org//abooo.. 2212 15115.
Moulin-Frier, C., Nguyen, S. M., and Oudeyer, P.-Y. Self- organization of early vocal development in infants and machines: the role of intrinsic motivation. Frontiers in Psychology, 4, 2014. ISSN 1664-1078, doi: 10.3389/ fpsyg. 20syg2001300100 Publisher: Frontiers.
Racaniere, S., Lampinen, A., Santoro, A., Reichert, D., Firoiu, V., and Lillicrap. T. Automated curriculum gener- ation through setter-solver interactions. In International Conference on Learning Representations, 2020.
Oudeyer, P.-Y. and Smith, L. B. How evolution may work through curiosity-driven developmental process. Topics in Cognitive Science, 5002 2016.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21 (140):1-67, 2020. ISSN 1533-7928.
Oudeyer, P.-Y., Kaplan, F., and Hafner, V. V. Intrinsic motivation systems for autonomous mental development. IEEE Transactions on Evolutionary Computation, 11(2): 265-286, 2007. ISSN 1941-0026, doi: 10.1109/ 2006.890271. Conference Name: IEEE Transactions Evolutionary Computation.
Romac, C., Portelas, R., Hofmann, K., and Oudeyer, P.-Y. TeachMyAgent: a benchmark for automatic curriculum learning in deep RI. In International Conference on Machine Learning, pp. 9052-9063, PMLR, 2021. ISSN: 2640-3498.
Pitis, S., Chan, H., Zhao, S., Stadie, B., and Ba, J. Maximum Entropy Gain Exploration for Long Horizon Multi-goal Reinforcement Learning. In Proceedings of the 37th International Conference Machine Learning, pp. 7750- 7761. PMLR, November 2020. ISSN: 2640-3498.
Sayah, C., Heling, E., and Cools, R. Learning progress mediates the link between cognitive effort and task en- gagement. Cognition, 236: 105418, 2023.
Poli, F., Meyer, M., Mars, R. B., and Hunnius, S. Ex- ploration in 4-year-old children is guided by learning progress and novelty. Child Development, 2024a.
Schmidhuber, J. PowerPlay:
Training an increasingly gen- eral problem solver by continually searching for the sim- plest still unsolvable problem. Frontiers Psychology, 4, 2013. ISSN 1664-1078. doi: 10.3389/fpsy.22013.00115 Publisher: Frontiers.
Poli, F. O'Reilly, J. X., Mars, R. B., and Hunnius, S. Cu- riosity and the dynamics of optimal exploration. Trends in Cognitive Sciences, 28(5):441 1-113,4533100 2024b.
Pong, V. H., Dalal, M., Lin, S., Nair, A., Bahl, S., and Levine, S. Skew-fit. fit: State-covering self-supervised re- inforcement learning. In International Conference on Machine Learning. 2019.
Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K., and Yao, Reflexion: language agents with verbal reinforce- ment learning. In Oh, A., Naumann, T., Globerson, A.. Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 8634 8652. Curran Associates, Inc., 2023.
Portelas, R., Colas, C., Hofmann, K., and Oudeyer, P.-Y. Teacher algorithms for curriculum learning of deep rl in continuously parameterized environments. In Conference on Robot Learning, pp. 833-853, PMLR, 2020a.
Sigaud, O., Baldassarre, G., Colas, C., Doncieux, S., Duro, J., Terrin-Gilbee N., and Santucci, V. G. definition
12

---

MAGELLAN
of open-ended learning problems for goal-conditioned agents. ArXiv, abs/231 1.0044,, 2023.
Stout, A. and Barto, A. G. Competence progress intrinsic motivation. In 2010 IEEE 9th International Conference on Development and Learning, pp. 257-262, 2010. doi: 10.1 100////........0..555...
ISSN: 2161-9476.
Ten, A., Kaushik, P., Oudeyer, P.-Y., and Gottlieb, J. Hu- mans monitor learning progress in curiosity-driven ex- ploration. Nature Commurications, 12(1):5972, 2021.
ISSN 2041-1723. doi: 10. 1460003//1144--01-226////
Publisher: Nature Publishing Group.
van der Maaten, L. and Hinton, G. Visualizing data using t-sne. Journal of Machine Learning Research, 9(86);
2579-2605, 2008.
Wang. G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., Fan, L., and Anandkumar, A. Voyager: An open- ended embodied agent with large language models, 2024. ISSN 2835-8856.
Farde-Farley, D., de Wiele, T. V., Kulkarni, T., Jonescu, C.,, Hansen, S., and Mnih, V. Unsupervised control through non-parametric discriminative rewards. In International Conference on Learning Representations, 2019.
Wen, M., Deng, C., Wang. J., Zhang, W., and Wen, Y. Entropy-regularize token-level policy optimization for large language models, 2024a.
Wen, M., Wan, Z., Wang. J., Zhang, W., and Wen, Y. Rein- forcing LLM agents via policy optimization with action decomposition, 2024b.
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. React: Synergizing reasoning and act- ing in language models. In International Conference on Learning Representations (ICLR), 2022.
Zhang, J. Lehman, J., Stanley, K., and Clune, J. OMNI: Open-endedness via models of human notions of inter- estingness. In The Twelfth International Conference on Learning Representations, 2024.
Zhou, Y., Zanette, A., Pan, J., Levine, S., and Kumar, A. ArCHer: Training language model agents via hierarchical multi-turn RL. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 62178-62209 PMLR, 21-27 Jul 2024.
13

---

MAGELLAN
Appendices
This supplementary material provides additional results, discussion, and implementation details.
Section details our Little-Zoo environment.
Section A.1 explains the environment mechanics and how the different actions affect the environment. It also gives more details about optimal trajectories.
Section A.2 details the goal space generation.
Section A.3 explains the different types of impossible goals. This section also gives examples for each type of impossible goal.
SSection A.4 contains the goal space's distribution per goal type.
Section extends the table from Section 3.3, comparing the previous LP estimation methods.
Section C contains the implementation details of MAGELLAN and the baselines.
Section C.1 explains the LLM-based RI. agent used in our experiments, the associated hyperparameters and the prompt used.
Section C.2 details MAGELLAN architecture, hyperparameters and the prompt used.
Section C.3 contains the additional details for implementing the baselines.
Section C.4 is about the compute budget.
Section extends the results given in the Section 4 on the experiments of the paper.
Section D.1 an ablation on MAGELLAN's architecture.
Section D.2 develops the results given in Section 4.1 with plots for goal spaces of different size.
Section D.3 contains several additional results for Section 4.2.
Section D.3.1 gives the evolution of the success rate per method and per goal type.
Section D.3.2 an analysis of the goal sampling strategy of each method.
Section D.4 develops several aspects of MAGELLAN's generalization abilities from Section 4.3.
Section 1 specifically examines the ability generalize success probability estimations goals different goal types.
Section D.4.2 describes the embedding evolution during training.
Section D.4.3 conducts detailed analysis of the embeddings of impossible goals.
Section D.5 expends on the results found in Section 4.4.
Section D contains all the curves from the experiment about the adaptation to an evolving goal space.
Section D.5.2 provides a quantitative analysis of this experiment.
14

---

MAGELLAN
A. Little-Zoo environment
A.1. Environment mechanics
There are 5 categories of clements the agent can interact with: the objects (i.c. "bookshelf"), the water, the plants (i.c. tomato) which can bee seed and grown, the herbivores (i.e cow) which can be baby or grown, and the carnivores (i.e. lion) which can be baby or grown.
To interact with any element of the environment, multiple actions are accessible. First, the agent can "Go to { j" to move to any element. You can only move to an element (not random point in the plane) and after a "Go to" action you are placed on the element. When performing the "Grasp" action, the element the agent is standing on is added to its inventory (up to 2 items). Some elements can interact with each other. To make two elements interact, the agent must release one element in its inventory by using the "Release {element] j action while standing on the other element. Interactions between three elements are also possible: the agent has hold two elements in its inventory and perform "Release all" " while standing on the third element. The "Release" action only accessible if an interaction be made between the element to release and the one the agent is standing on. Therefore, the agent must carefully manage its inventory to avoid filling it with useless elements without being able to release them. At each turn you get full description of the environment with "You see {element 1}, felement 2}, felement 3}, felement 4}".
To grow plant, an herbivore or a carnivore, the agent needs to follow a certain sequence of actions:
To "Grow a plant", ", the agent has to "Release water" while "Standing" on the plant seed.
To "Grow an herbivore", the agent has to "Release grown plant" while standing on baby herbi- vore.
To "Grow carnivore", the agent has to "Release grown herbivore" while standing on baby carnivore.
Table 3 summarizes the optimal action strategy for each category. In order to maintain a coherent level of difficulty between the different categories, we fix the maximal number of actions allowed to the agent to solve a goal as 150% of the minimal number of actions required to solve the goal. The required exploration to discover strategies has, therefore, the same level of difficulty for all goals.
Table 3. The optimal trajectories per categories.
Minimal number of actions
Maximal number of actions
Categories
Optimal action strategy
Grasp X
Go X. Grasp
Go to water, Grasp, Go plant, Release water
Grow plant
Go water, Grasp. Go plant,
Grow herbivore
7
11
Release water, Grasp, Go herbivore, release plant Go to water, Grasp, Go plant, Release water, Grasp, Go to herbivore, release plant, Grasp, Go to carnivore, release
Grow carnivore
10
15
A.2. Goal space generation
To define our goal space, we start from the sets of objects O, plants P, herbivores H. carnivores C and the element water (each set being of size 6). One goal g in the goal space is defined using 5 elements (Eg, E1, Ez, Es, Ea) (O, P. H. C, water) and by using the objective function F, from the set (Grasp(x), Grow(x)) on Eg. Thus, we get the goal
Fr, F.z F.,, FA). As each set contains 6 elements, we get total of 19, 531, 250 goals. The vast majority of these goals are impossible for various reasons (see Appendix A.3).
In our experiments, we subsample two goal spaces: a train and held-out test space. To generate them, we sample the total number of impossible goals nimpo we want have, then sample among the possible goals: Phmme goals where Fg O,
15

---

MAGELLAN
Rempe goals where Eg E P, Mempo goals where Eg H, and 5"
Mampo 5
goals where Eg C. Appendix A.4 gives more details
on the goal repartition.
A.3. Example of impossible goals
In our environment, goal is defined combination of elements present in the environment and an instruction, such as "grasp the rabbit". Some combinations are incompatible such as:
Grasp rabbit
You see: bookshelf, baby lion, desk, baby Cow, where there is no rabbit. The generation of our environment leads to having 80% of impossible goals in the goal space, which is similar to any complex environment (Zhang et al., 2024; Matthews et al., 2024). However, conversely to (Zhang al., 2024), identifying impossible goals and avoiding to select them is not trivial as there are multiple reasons why goal impossible.
The easiest reason for impossibility is the absence of the element the agent has to act
Absence of the element to grasp
Goal: Grasp bed
You see: bookshelf, baby lion, desk, baby cow.
Absence of the element grow
Goal: Grow deer
You see: baby giraffe, bookshelf, water, tomato seed.
Another reason is that the element cannot follow the dynamic required in the objective:
Growing an element that is neither an animal nor plant Goal: Grow desk
You see: bed, baby bobcat, baby elephant, door.
A more difficult reason that requires a better understanding of the underlying mechanisms of the environment is the lack of at least one element necessary to reach the objective grow:
The lack of water:
Goal: Grow cucumber
You see:
cucumber seed, carrot seed, baby coyote, baby wolf,
Goal: Grow coyote
You see:
cucumber seed, berry seed, baby coyote, baby cow.
The lack of a plant preventing the agent to grow an herbivore resulting the impossibility to grow carnivore:
Goal: Grow coyote
You see: water, baby elephant, baby coyote, baby cow.
Consequently, to efficiently build a curriculum, being able to understand why goals impossible and quickly generalize to infer the other impossible goals is crucial.
A.4. Goal repartition
At the end the goal space generation, we have 19531250 goals distributed as presented in Figure 8a. However, using all the goals was computationally impossible considering all the comparisons we made. Moreover, the repartition with 92% of impossible goals and 7.53% of "Grasp" goals made the epsilon-greed exploration of our goal selector too slow to discover LP niches in acceptable compute budget. Subsequently, following the distribution given in Figure 8b. Both goal spaces are composed at 80% of impossible goals, the remaining goal categories becoming less and less represented as the difficulty increases. This repartition synthetically simulates how natural language goals mostly lead to sequences of words without any meaning. This strueture makes the strategy of uniformly sampling goals inefficient and underlines the necessity of a curriculum to be able to reach the hardest goals in the allocated training budget (500k episodes in our experiments).
sample training goal space and testing goal space
16

---

MAGELLAN
12.44%
22.0%
2.
02
0.2
50
Irpse
Grapp
CRS
Drose
Cape
00-
Inso0sble
Grosp
CNEW
ShoP
Catogerics
Cotegories
(a) Goal distribution per categories on the goal space.
(b) Goal distribution per categories in our experiments.
Figure 8. We first generate the full goal space whom the distribution is given Figure 8a then for computational smaller goal space with the following distribution Figure 8b.
sample
B. Comparison of LP methods
We compare prior work computing LP for automatic curriculum learning under the dimensions from Section D.2, We show the comparison in Table 4 evaluating methods:
Efficiency: Computational cost of additional evaluation episodes not used to train the policy.
Competence transfer tracking: How well does the method track all the possible competence transfer.
No expert knowledge required: they require any external expert knowledge such as pre-defined goal groupings.
We consider method's efficiency as "high" if it does not require any additional evaluation (i.e. it only uses the performance observed on goals sampled), and as "low" otherwise. We evaluate the competence transfer tracking using the following criteria:
absence of +: the estimated competence is updated only on sampled goals.
+: the estimated competence is updated predefined goal subset the sampled goal belongs to.
++: the estimated competence is updated on dynamically learned goal subset the sampled goal belongs to.
++: the estimated competence is updated on all goals.
17

---

MAGELLAN
Methods
Efficiency
low low Competence transfer tracking
expert knowledge required
(Oudeyer et al., 2007)
high
(Baranes & Oudeyer, 2009)
high
(Baranes & Oudeyer, 2013)
high
(Moulin-Frier & Oudeyer, 2013)
high
(Moulin-Frier ct al., 2014)
high
(Portelas et al., 2020a)
high
(Forestier & Oudeyer, 2016)
high
(Kovač et al., 2023)
high
(Stout & Barto, 2010)
high
(Lopes & Oudeyer, 2012b)
high
(Matiisen al., 2020)
high
(Fournier al., 2018)
high
(Colas ct al., 2019)
high
(Blacs et al., 2019)
high
(Akakzia al., 2021)
high
(Kumar al., 2024)
high
(Kanitscheider ct al., 2021)
+++
(Zhang et al., 2024)
+++
Table Comparison of prior work w.rt. their estimation approach. We use following dimensions: computational efficiency, dynamical competence transfer assumptions and no required expert knowledge.
C. Implementation details
To facilitate reproduction and future work, provide code at https://ggthubb.com/Eoorersssss MAGELLAN.
C.1. LLMeb RL agent
We the SAC-GLAM method from (Gaven al., 2024) with the hyperparameters listed in Table 5. Following SAC- GLAM, the Q-value head is a two-layer MLP with 1024 ReLU-activated units, applied to the last hidden state of the decoder. The policy and the critic share the same LoRA adapters. Additionally, we apply a warm-up phase of 10 updates, during which only the Q-function is trained.
Table 5. SAC hyperparameters
Variable
Value
Update frequency
64
Number of updates
1
Batch size
256
Discount factor
0.99
Optimizer
Adam
Critic learning rate
1 x 10-4
Actor learning rate
1 x 10
Entropy coefficient
auto
Entropy coefficient initialization
0.05
Target entropy
-0.125
Entropy coefficient learning rate
1 x 10-3 3
n-step
Replay buffer capacity
500000
The prompt used as the LLM agent's observation shown in Prompt C.I.
18

---

MAGELLAN
Prompt C.I: RI Agent observation
Goal: Grow lion
You water, carrot seed, baby lion, baby cow You are standing nothing
Inventory (0/2): empty
Action:
C.2. MAGELLAN
Our competence estimator uses single-layer MLP with 128 units and Tanh activations, applied the last hidden state of the LLM. It is updated every 32 policy updates, with batch sampling that prioritizes recent data. The sampling distribution is determined by Σッ Σー= where represents the position of a data point in the buffer D (with 1 being the oldest and M the buffer size). This strategy ensures that the competence predictor remains responsive while preserving batch diversity, incorporating data from different versions of the learning agent to help mitigate noise introduced by the fluctuations in the RL agent's 's learning process.
To compute the LP, we store the weights of our competence estimator in buffer This is done by saving the LoRA adapters and the MLP weights whenever the competence estimator updated. The window for LP computation is defined by [B] update frequency. The LP is calculated by comparing the competence estimation obtained with the oldest and most recent weights in the buffer.
The MAGELLAN hyperparameters are provided in Table 6.
Table 6. MAGELLAN hyperparameters
Variable
Value
start
end
0.2
decay period
320
B size
100
D size
5000
Batch size
256
Optimizer
Adam
Learning rate
10-4
Update frequency
32
The prompt given to the competence estimator is the same as the prompt given to the RL agent Prompt C.2.
Prompt C.2: MAGELLAN prompt
Goal: Grow lion
You see: water, carrot seed, baby lion, baby cow You are standing on: nothing
Inventory (0/2): empty
Action:
C.3. Baselines
We compared our method against four baselines: Online-ALP, Eval-ALP, EK-Online-ALPP, and EK-Eval-ALP. Below, we outline the implementation details for the different families of methods:
Non-EK methods For methods that do not rely on expert knowledge, competence and LP estimations are computed individually for each goal.
19

---

MAGELLAN
EK methods For methods incorporating expert knowledge, goals grouped into five buckets, with competence and LP computed at the bucket level.
Online methods In online methods, a buffer of size M stores goal-outcome pairs. The agent's 's current competence is estimated from the most recent half of the buffer, while past competence derived from the oldest half. Non-EK methods maintain separate buffer for each goal, whereas EK methods use buffer for each bucket.
Eval methods In evaluation-based methods, the agent's competence is assessed every N episodes. For non-EK methods, the agent is evaluated l times on each individual goal, while for EK methods, evaluations are performed times per bucket.
The hyperparameters for Online methods presented in Table while those for Eval methods are shown in Table 8.
Table 7. Online methods hyperparameters
Variable
Value
start end
0.2
6 decay period
320
Buffer size
100
Table 8. Eval methods hyperparameters
Variable
Value
start end
0.2
decay period
320
Eval frequency
1000
Number eval
2048
C.4. Compute budget
To optimize GPU VRAM usage during training, we employ 4-bit quantization techniques as described in (Dettmers et al., 2023). We use vectorized version of Little-Zoo with 32 instances of the environment running (synchronously) parallel. In order to accelerate training our LLM agent (both its policy with online RL and MAGELLAN), we leverage Lamorel? deploy 2 instances the LLM in parallel. Thus, we distribute both the forward passes compute actions' probability or LP and training in a Data Parallelism setting. When using Flan-T5 250M, each LLM instance is distributed (Vertical Model Parallelism)
Nvidia H100 80GB GPUs requiring thus total of 2 Nvidia H100 80GB GPUs to experiment (1 GPU 2 LLM instances). For one seed of one ALP method trained in Section 4.2, performing 500k training episodes requires 80 GPU hours on the Nvidia H100 80GB.
20

---

MAGELLAN
D. Additional results
D.I. Ablations on the MAGELLAN architecture
In order to implement MAGELLAN, we try various architectures presented in Figure 9. In all of them, to reduce the computational cost, we froze the LLM and used LoRA adapters (Hu et al., 2022) for training the policy (along with its Q-value) and the SR estimator. The architectures tested are:
A: Different adapters for the policy and MAGELLAN (Figure 9a). This is the architecture adopted in the paper. The environment representation for the policy and the goal space representation for the competence estimator are learned separately on different adapters.
B: Shared adapters for the policy and MAGELLAN(Figurg 9b). We test whether a shared useful representation between the policy and competence estimator can emerge through training.
C: Shared adapters but only trained with the policy (Figure 9c). We single set of LoRA adapters train both the policy and the competence estimation module, but apply the latter's gradient only to the MLP outputting the competence. This ablation tests if the policy loss on its own creates semantical relationships between goals. We argue that training of the policy should push towards a clustering of the goal space.
D: Adapter only for the policy, frozen LLM representation for MAGELLANCC MAGELLAN can learn to predict competence using the LLM's ' original latent space. Learning to estimate competence over a fixed representation is closer to prior works such as ALP-GMM (Portelas et al., 2020a).
9d). In this ablation, test if
We compare the learning dynamics of the four architectures in Figure 10. We observe no difference in their ability learning "grasp" goals. However, the agents with Architecture A continue progress other goal types whereas the ones using Architectures B, C, and D stagnate between 50k and 100k episodes before progressing again. At the end of training (500k episodes), agents using architecture A have learned all goal types. The ones based on Architectures B and C learned up to "grow herbivore" goal types. The agents that utilize Architecture D succeed plateaued on "grow plant".
To gain a finer comprehension of the learning dynamics implied by the different architectures, we look in Figure 11 at the embeddings at the end of training for each architecture.
Architecture A (Figure 1 11a) and architecture B (Figure 11b) shared very similar representations with in both cases goals clustered between "ggrap", "grow" and impossible goals. Inside the cluster "grow" the different element types (i.e. plants, herbivores, carnivores) are also clustered. Nonetheless, as seen in Figure 10, Architecture B does not master "grow carnivore" goals and classified them as impossible. This indicates that obtaining a shared latent space for the policy and MAGELLAN is possible but slows down skill acquisition.
Architecture C (Figure 11c) where MAGELLAN uses the representation solely changed by the policy still manages to accurately estimate competence. However, predicting competence is much more difficult using this architecture as the goal space is not modified ease competence estimation. Yet, using only the policy gradient still improves the initial latent space of the LLM as shown by Architecture D (Figure 1 l1d).
21

---

MAGELLAN
LLM Agent
LLM Agent
Pollcy
LP: 1
Policy
LP
T
C
C
C
MAGELLAN's gradient
MAGELLAN'S gradlent
Policy's 's gradient
Policy's gradient
G
CLn
Tr. & i
Tu CpM
LoRA adapters
LoRA adapters
LLM
LLM
Prompt with:
Prompt with:
Prompt with: observation
Prompt with: initial observation o instruction
observation o
Initial observation
instruction
instruction
instruction i
(a)
(b)
LLM Agent
LLM Agent
Palicy
LP:
Policy
LP:
C
C
C
C
MAGELLAN's gradient
MAGELLAN's gradient
Palicy's is gradient
Palicy's 's gradient
Thp
LoRA adapters
LoRA adapters
LLM
LLM
Prompt with: observation o
Prompt with:
Prompt with:
Prompt with:
initial observation
observation
initial observation
Instruction i
Instruction i
Instruction I
instruction i
(c)
(d)
Figure 9. Different architectural choices in MAGELLAN: (a) we learn separate LoRA adapters between the policy and MAGELLAN (used in the paper); (b) share adapters and update them using both the policy and MAGELLAN gradient; (c) we share adapters but they are only updated by the policy gradient; (d) MAGELLAN directly uses the latent representation produced by the pretrained LLM.
22

---

MAGELLAN
1.2
Architecture A
Architecture C Architecture
Architecture
Success Rate
0.8
0.4
0.0
200k
300k
500k
Episodes
Figure 10. Training curves of the four different possible architecture for MAGELLAN. We 8 seeds plot the mean and the standard deviation (shadow area around the solid line).
1.0
1.0
Impossiile
owe tettiore
Tran
Risosbble
Grew hortkore
Tak
Grapp
carriore
Trxt
Graap
cambrore
0.8
0.8
Estraatdd Success Prbabiliyy
Pobabilry
0
0.8
0.0
0.0
(a) Architecture A.
(b) Architecture B.
1.0
1.0
Impostible
Ferbiicr
Trn
moosoble
Grew herthore
Crow stant
Growe plant
0.8
Estimated Success Probabiiiyy
Frbabiiity
0.6
0.4
Edlimated
0.2
0.0
0.0
(c) Architecture C.
(d) Architecture D.
Figure The LLM embedding space of MAGELLAN displayed using t-SNE with goals used Q2 (Train) and Q3 (Test), along with MAGELLAN's estimated success probability and linear interpolation between goals. We show the embedding space for single seed for the four architectures described Appendix end of 500k training episodes.
23

---

MAGELLAN
D.2. Q1. Competence estimation properties
D.2.1. PER-GOAL COMPETENCE ESTIMATION
In this section, we report the evolution of the per-goal competence estimation of experiments from Section 4.1. We show separately for the three goal space size: 25k (Figure 12), 50k (Figure 13) and 100k (Figure 14).
Grasp
Grow Plants
3.0
1.0
Eddmasdd Success Protaa
0.0
Edhmand
08
Eacodds
Spluodes
Grow Herbivores
Crow Carnivores
2.0
Estmatud Supcess Prbbsoiir
0.00
0.-
0.0
0.6
Eacoodes
Eptoodes
Figure 12. Evolution of competence estimation for each ALP method each goal category for 25k goals. We show the average competence and standard deviation across 8 seeds use EK-Eval-AL3 sample goals.
MAGELLAN
E6-OnineAALP PISPuRHA1P
Estmane Success Protott ProtEAAER
02
0.0
86
Soltardee
2.2
1001 Success Proba-ktyy
0.6
06
0.4
0.4
022
02-
10c
206
254
136
23c
33c
Figure Evolution of competence estimation for cach ALP method each goal category for 50k goals. We show the average competence and its standard deviation across 8 seeds that use "K-Eval-ALPP to sample goals.
24

---

MAGELLAN
MAGELLAN Onlne-ALP
OOOnine ALP ECEvaHALP
Grasp
Grow
Gptraed Suc00s Pebbsaats
0.4
0.4
02-
02-
216
Grow Herbivores
Grow Cariivrre
12
3.8
Esdmatedd Success Procaait
1.0
0.0
0.
londes
Fplondee
Figure 14. Evolution of competence estimation for each ALP method on each goal category for 100k goals. We show the average competence and its standard deviation across 8 seeds that KE-EEvl--ALA sample goals.
D.2.2. COMPETENCE ESTIMATION ON THE BABYAI-TEXT GOAL-SPACE
We replicated the experiment from Section 1. simulating a learning agent and estimating its competence online using MAGELLAN and Online-ALP. In this setting, the goal space is drawn from the BabyAl-Texx environment, consisting of five goal categories of increasing difficulty: Go to "object", Pick up "object", Open "door", Put "object A " next to "object B", and Pick up "object then go "object B". As shown in Figure 15, MAGELLAN outperforms Online-ALL successfully tracking the agent's competence progression across the different task categories.
MAGELLAN
Online ALP
Tue Success Probsbility
Go to
Pickup
Open
Success Probabiitty
Success Probability
Success Probability
0.6
0.6-
0.6
0.4
0.0
0.0
0.0
Episodes
Episodes
Episodes
Put next to
Pickup then go to
1.0
1.0
Success Probabiityy
Success Probabiiity
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.0
Sk
10k Episades
15k
20k
0.0
Sk
10k Episodes
15k
20k
Figure 15, Competence estimation on BTby MAGELLAN accurately tracks competence across five goal types of increasing difficulty and consistently outperforms Online-AL.P.
25

---

MAGELLAN
D.2.3. IMPACT THE LLM USED IN MAGELLAN
In this section, we compare different LLMs -lan-TS-smaal, Flan-T5-base, and Qwen2.5-0.5B (Qwen et al., 2025)) evaluate their impact the competence estimation performance of MAGELLAN. The experiments simulate agent learning using goal spaces: OpenR Maath-220k and BabyAl-Tex, (Figures 16 and 17). Results show that all tested LLMs achieve similar performance, suggesting that MAGELLAN's competence estimation remains robust across range of lightweight language models.
MAGELLAN (Qwen2.5-0.50)
MAGELLAN (Flan-TS-base)
MAGELLAN (Flan-TS-small)
Online ALP
True Success Probability
Algebra
Geometry
Number Theory
1.0
1.0
1.0
Success Probablitty
Success Probabilty
Success Probability
0.8
0.8
0.2
0.0
20k
0.0
10k
15k
20k
Episodes
Episodes
Episodes
Figure Effect LLM choice on MAGELLAN's competence estimation for OpenR1 -Math-200k. All models yield similar performance.
MAGELLAN (Qwen2.5-0.33
MAGELLAN (lan-TS-basee)
MAGELLAN (Faalll--mallll
Online ALP
Tue Success Probability
Go to
Pickup
Open
Success Probablity
Success Probabiltty
Success Probability
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.0
0.0
5k
10k
15k
20k
5k
10k
15k
20k
0
Sk
10k Episodes
15k
20k
Episodes
Episodes
Put next to
Pickup then go
1.0
1.0
Success Probability
Success Probability
0.4
0.2
0.2
10k Episodes
Sk
20k
Episodes
Figure Effect of LLM choice on MAGELLAN's 's competence estimation for Al-Txt- Performance remains consistent across models.
26

---

MAGELLAN
D.3. Q2. Training an LLM-based RL agent with MAGELLAN
D.3.1. PER-GOAL SUCCESS RATE
We detail in Figure 18 the evolution of the success rate per method and per goal type. All methods perform similarly on the type "Grasp". However, MAGELLAN outperforms all the baselines that do not rely expert knowledge (Uniform and Online-ALP) on all other types. Learning dynamics fostered by MAGELLAN are close to the ones generated with F.K-Online-AL.P which relies expert knowledge. Such results hint towards clustering of the goal space by MAGELLAN to efficiently and reliably estimate the SR of the LLM agent. This hypothesis is strengthened by the plot of the embedding space Appendix D.4.
MAGELLAN
Uniform
Online ALP
BK Online ALP
Grasp
Grow Plants
Rato
0.0
Success Rate
0.0
0.6
0.0
0.0
100k
230k
400k
500k
0.0
200k Ealsodas
300k
4C0K
500k
Epicodes
Grow Herbivores
Grow Carnivores
1.0%
1.0
0.8
Success Rate
0.9
Succesa
W
0.2
6.2-
0.0
0.0
200k
Episodes
Figure 18. Evolution of average SR for each ALP method for each goal category. To get the success rate within category. evaluate the policy on 64 goals of this category. We use 8 seeds plot mean and standard deviation.
27

---

MAGELLAN
D.3.2. GOAL SAMPLING STRATEGIES
We analyze the sampling strategies of the different methods. Figure 19, it appears that MAGELLAN has a sampling strategy close to the one of EK-Online-ALP. The main difference is that EK-Online-ALi. does not select impossible goals thanks to the use of expert knowledge. MAGELLAN's sampling strategy quickly begins to sample goals of type "Grow herbivore" after sampling goals of type "Grow plant", which contrasts with the strategy of Online-ALP. Indeed, Online-ALP mostly selects goals already encountered, getting stuck on one goal type before moving to another. The similarity between MAGELLAN and EK- Le--Oniine-AL indicates that MAGELLAN is able to cluster the goal space dynamically, using this information to generalize its competence estimation to sample interesting goals not yet discovered. More results on the dynamic clustering of the goal space by MAGELLAN are given in Appendix D.4.2.
100
100
80-
Goals propertion
Goals proportion
40
0
100K
200k
300k
400K
500k
0
100K
200k
300k
400k
500k
Episodes
Episodes
(a) MAGELLAN's sampling strategy.
(b) EK-Online-ALP's sampling strategy.
80
Goals proportion
40
100k
200k
300k
400k
500k
Episodes
(c) Online-ALL ALP's sampling strategy.
Figure 19. Goal sampling strategies of MAGELLAN, EK-Online-All ALP,, Maline-ALP. We do take into account the 20% uniformly sampled goals from the e-greedy exploration.
28

---

MAGELLAN
D.4. Q3. MAGELLAN's generalization abilities
D. PER-GOAL SUCCESS PROBABILITY ESTIMATION TEST SET
In this section, we provide detailed analysis of the results presented in Section 4.3, specifically examining the ability to generalize success probability estimations test goals across different goal types (see Figure 20). As expected, Online-ALP exhibits the largest errors, as it can only assign success probability of 0 to unseen goals. In contrast, both MAGELLAN and EK-Online LK--Online-ALP achieve highly accurate estimations for "Grasp", "Grow plant", and "Grow herbivore". However, both methods tend to overestimate the generalization abilities of the policy on "Grow carnivore" goals.
Casimared
Oooerved Rucoes probettiyy
35A
Nerthors
Herbhones
Carricorne
(a) MAGELLAN.
(b) AL-OOnine-ALL.
Estestte provubally
Plants
4.7
60
30.
(c) OnLine-ALP.
Figure 20. Per-goal estimation of the success probability for each method. The average result (over 8 seed) is the solid line and the shaded zone represents the standard deviation.
29

---

MAGELLAN
D.4.2. EVOLUTION THE EMBEDDINGS
In Section 4.3, we present the embedding at the beginning and the end of the training for the seed 0.3 In this section, we explore how the goal space is dynamically modified throughout training with Figure 21. It is a chronogram of the embedding space used by MAGELLAN projected using t-SNE (van der Maaten & Hinton, 2008) with the estimated success probability over the whole space calculated using linear interpolation with Gaussian filter. Each embedding (except the first one that is the initial state) is plotted after the agent masters goal category.
At the beginning, Figure 21a, no structure is discernible in the goal space and the estimated success probability is uniform around 0.5.
After mastering the goals of type "Grasp", Figure 21b shows several clustered appeared. All "Grasp" goals are in the same cluster with a high estimated success probability zone. The goals of type "Grow plant" are also clustered together and close to the zone of high estimated success probability. That is hint that MAGELLAN has already picked them as the next candidate type for the curriculum. It also correctly places the "Grow plant" goals from the test set in the same cluster as the ones from the train set. However, it still mixes them with impossible goals, underlying that it does not fully master this type of goal. The "Grow herbivore" and "Grow carnivore" mixed with other impossible goals.
After achieving mastery of "Grow plant", in Figure 21c, we both "Grasp" and "Grow plant" are correctly clustered in the zone of high success probability, with the goals from the test set correctly placed into the two clusters. The goals from the type "Grow herbivore" and "Grow carnivore" are clustered together and almost separated from the impossible goals. Their cluster is close to the zone of high estimated probability.
Once "Grow herbivore" has been solved, we see in Figure 21d that the three achieved goal types are clustered in three groups, all in the zone of high estimated success probability. The "Grow carnivore" are far from this zone but still clustered together, apart from the impossible ones. also appears that the model places "Grow carnivore" goals from the test close to the "Grow herbivore".
Finally, when the type "Grow carnivore" is mastered, in Figure 21e, all the goals from the possible goal types (from both train and test) are placed in the zone of high success probability. The goals of type "Grow X" form three clusters tightly packed. The "Grasp" type separated from the other types.
What is clear from the chronogram that MAGELLAN's learning modifies the embedding space in way that facilitates prediction of the probability of success.
"The dynamic the same for the other seeds.
30

---

MAGELLAN
03
Extraand Succes Rewao
(a) Beginning of training, no goal mastered.
(b) "Grasp" type of goals mastered.
Cot
6a
(c) "Grow plant" type of goals mastered.
Gota With 200 Meroere
Crop
Crew
Eormand facred Frtusday
(d) "Grow herbivore" type of goals mastered.
Groal
TAN
Eatrond Rucene Sobariigg
O
"Grow carnivore" type of goals mastered.
Figure 21. Chronogram of the embedding space of the seed 0, at the beginning and after mastering each type of goal.
31

---

MAGELLAN
D.4.3. EMBEDDING OF IMPOSSIBLE GOALS
As detailed in Section A.3, goal is considered impossible either due the absence of required element when attempting to grow furniture. Accurately identifying impossible goals and generalizing this knowledge is crucial for strong performance in the Little-Zoo environment. In this section, we analyze how MAGELLAN handles impossible goals.
Figure 22a illustrates the clustering of different categories of impossible goals. The impossible "Grasp" goals form a compact cluster, while the "Grow" cotes-catego as "Grow plant", "Grow herbivore", "Grow carnivore", and "Grow furmiture" exhibit four less-defined clusters. Additionally, a large, mixed cluster contains various impossible "Grow" goals. However, when examining the same embeddings through the lens of missing elements, shown in Figure 22, distinet structure emerges: MAGELLAN also organizes goals based the absent element in the scene.
For instance, the previously observed clusters of impossible "Grow" goals largely align with the "missing water" category, as water is prerequisite for all "Grow" goals. Moreover, the central mixed cluster from Figure 22a is further clustered into sub-clusters based on the missing element. In particular, the red cluster at the bottom of Figure 22 represents the absence of a plant, encompassing both "Grow herbivore" and "Grow carnivore" goals. This suggests that MAGELLAN effectively captures underlying environmental dependencies to determine goal feasibility. This ability to infer structural properties of the environment likely contributes the strong generalization performance observed in Section 4.3 and Section 4.4.
1.0
groep
grow cemiieee
Imgrreltie
monreltt
0.8
Estimated Success Probablitt
grow hatbivore
0.6
0.
0.2
0.0
1.0
Maaing mann Ibem
Masing plart
Tra
Missing tivora
Test
wattrr Itam
0.8
Estimated Success Probobility
0.4
0.2
(b)
Figure 22. Plot the t-SNeeprojected embeddings of impossible goals, where each goal is color-coded based (a) the type impossible goal (b) on the specific missing element required to make possible.
32

---

MAGELLAN
D.5. Q4. Leveraging the generalization abilities when facing new goals
D.5.1. 10 ADAPTATION CASES THROUGHOUT TRAINING
In Section 4.4, we present adaptation tests where the goal space evolves by replacing goals with new unseen goals from similar categories. We conduct the test by training agent using MAGELLAN for 500k steps. Then, every 50000 episodes (with {1; 3; 4; 5; 6; 7; 8; 9; 10J), we stop training and replace MAGELLAN by one of the four ALP methods of goal sampling (MAGELLAN EK- OK--OnnineAAPPP Online-AL.P, and Uniform). Finally, we resume the training for 50k steps and measure the SR. For EK-Online-ALA we make it track the agent's competence based on goals sampled by MAGELLAN during the first phase, allowing it to start with an estimation on the new unseen goals. Our test measures the ability of a method to estimate and quickly update competence on unseen goals. Figure 23 details all the results obtained at the ten points we use in the experiments.
We can divide the different experiments in typical scenarios:
Scenario zero LP (Figure 23a): the agent has mastered the "Grasp" and "Grow plant" goals and has 0 ALP across all goals. In this scenario, all ALP estimations are equivalent. EK-Online-L.P manages to discover new ALP niches faster as all impossible goals are in the same group.
Scenario with LP:
High LP (figures 23b, 23c, 23d): the agent is getting a high ALP as it is learning some "Grow carnivore" goals. Here, MAGELLAN outperforms baselines by generalizing its ALP estimation and continuing training on these goals.
Medium LP at the end of training (figures 23e to 23j): the agent has almost learned all the goals, and it only remains few goals in the "Grow carnivore" category on which its performance is not stable. When changing the goal space, Online-ALP and Uniform take a lot of steps to find the remaining goals with LP in the new goal space, destabilizing the agent. As a result, the agent's SR decreases just after the transition and does not recover after 50k training epispdes. MAGELLAN and Online-ALB maintain their high SR.
We notice that the learning curve of the agents using MAGELLAN and EK--Onine-ALPP almost identical, but method does not rely expert knowledge and naturally clustered the goal space as seen in Figure 6b.
D.5.2. GLOBAL SAMPLE EFFICIENCY ASSESSMENT
To obtain a quantitative analysis of the results from Section 4.4, we plot in Figure 24 the average sample efficiency (after x episodes) of each method, averaged over the 10 tests, with k E (10000; 20000; 30000; 40000; 500003. The sample efficiency after episodes is calculated as 1 10 I S * + 0) - (500000 * n.) dt.
We observe that the Uniform method is the only approach exhibiting a negative average sample efficiency, which expected since it predominantly samples impossible goals, thereby destabilizing the agent. In contrast, the other methods demonstrate increasing sample efficiency as progresses, reflecting improved success probability estimation and, consequently, more effective goal sampling. However, Online-ALI exhibits only marginal improvement, it must continually re-estimate success probabilities across the entire goal space.
Both MAGELLAN and EK-Online-ALP, which leverage generalization to estimate success probabilities for novel goals, achieve higher sample efficiency. Notably, EK-Online-ALP benefits significantly from expert knowledge in clustering the goal space and identifying impossible goals. However, its strong performance is contingent on the assumption that new goals belong to the same categories as those in the training set, which may limit its generalization capacity.
33

---

MAGELLAN
1.20
03
0.5
M
Sytowee Ree
MIGHLLAN
Ciill Weeki
250
XXX
230
(a)
(b)
(c)
(d)
1/ 11e
of
Sucren 500
unatim
12A
Colascks
1.30
736
Eolsscea
124
Colockas
7306
Imm MAN
Suthee Date
11
NIGELANS 90-A
NAGELANS W-L
PaPatee
(i)
Figure Adaptation tests: Using single's seed training of 500k episodes, we stop and replace goals by new unseen goals every episodes. We then resume training by sampling goals using each of our four methods' ALP estimation (MAGELLAN, EK-Online-LLP. Online-ALP Uniform) and perform training episodes. We report the evolution observed competence (SR) when evaluating the policies 64 goals per category from the new training every 5000 episodes. We report the average competence over evaluated goals along with standard deviation.
Uniform
0.10
MAGELLAN
EX-Online-ALP
Sample Efficlency
0.00
0.04
0.02
-0.02
10k
20k
30k
50k
Length the test (Episodes)
Figure 24, Average sample efficiency (after K. the length of the test) of each method average over the 10 tests.
34


Reviews:
[{"review_id": "PVl0k2nWCF", "reviewer": "Reviewer_9oss", "summary": "The paper introduces MAGELLAN—a metacognitive module that enables autotelic LLM agents to estimate their own learning progress (LP) over large, discrete, and evolving goal spaces. The approach leverages the inherent semantic understanding of an LLM to learn a goal‐conditioned competence estimator that generalizes across similar natural language goals without relying on expert-defined groupings. Experiments in a custom textual environment (Little‑Zoo) demonstrate that MAGELLAN can accurately track learning progress, generalize to unseen goals, and adapt rapidly when the goal space evolves. Overall, the paper claims that this method enables the agent to build a self-organized curriculum, leading to faster and more complete mastery compared to several baseline LP estimation techniques.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer 9oss for their thorough feedback, finding our approach using an LLM to estimate LP innovative, highlighting the comprehensiveness of our experimental evaluation and acknowledging the effectiveness of MAGELLAN. In the next paragraphs, we answer reviewer 9oss’ concerns.\n\n## Single and limited environment\nWe appreciate the reviewer’s feedback and understand their concerns on our limited evaluation domain. As a similar question was raised by most reviewers, we provide a single response, included in reviewer’s 3cqQ rebuttal. **This response notably shows new results on additional experiments launched according to reviewers’ comments.**\n\n## LLM choice\nWe agree with the reviewer that studying MAGELLAN’s sensitivity to various LLMs would make our evidence more robust. We thus **ran an additional experiment** for our Section 4.1 where MAGELLAN’s error on 25k goals is measured with **Flan-T5 80M and Qwen2.5-0.5B** in addition to Flan-T5 248M  (which is already in the paper). We show the results in the following plot: https://github.com/ghjnkmjl745678/MAGELLAN_ICML/blob/main/errors_littlezoo_goals.png.\n\nOur results indicate that larger models lead to slightly more precise competence estimations in Little-Zoo. However, this difference is not significative as the LLM is finetuned by MAGELLAN to adapt its embedding space for competence estimation in the given environment. We will add these results to Section 4.1.\n\nLastly, the reviewer asks about scalability. In our experiments, we used LoRA (Hu et al., 2022), limiting the number of weights to finetune. Nevertheless, larger LLMs remain slower and more computationally consuming both at inference (i.e. to estimate the LP over the goal space and choose the next goal) and during finetuning. We will add this discussion to our conclusion.\n\n## Other methods and baselines\nWe thank the reviewer for their suggestion of additional references to discuss. Section 2.1 discusses goal selection approaches in general (including methods not using LP) and we would be happy to add these references (and the ones proposed by reviewer pnae on UED methods) in this section. Could the reviewer be more precise on which work we missed? \n\n## Failure modes\nWe thank the reviewer 9oss for their question. In our experiments, we observed no significant failure cases impacting the curriculum. \n\nHowever, we identify two possible failures cases resulting of bad LP estimation:\n- *The LP is overestimated*: In that case, the goal will be quickly sampled by the multi-armed bandit selector and its LP estimation will be reduced.\n- *The LP is underestimated*: In that case, we must wait for the exploration mechanism in the multi-armed bandit goal sampler to eventually sample the goal, enabling MAGELLAN to adjust its LP estimation. This is the worst case among the two.\n\nAdditionally, applying MAGELLAN to a very large discrete goal space with no semantic structure would lead to poor performance (i.e. similar to Online-ALP). Moreover, we do not explicitly study the case where a goal semantic is not aligned with common knowledge internalised by the LLM (e.g. in Little Zoo, a rabbit that would act as a carnivore). However, as shown with the initial embedding space of Flan-T5 248M (Figure 5.a), MAGELLAN will progressively move it from the herbivore cluster to the carnivore one throughout training.", "claims_and_evidence": "Claims: The paper claims that MAGELLAN (i) efficiently estimates LP without expensive evaluations or expert-defined groupings, (ii) generalizes competence predictions to unseen goals, and (iii) adapts to evolving goal spaces, all of which facilitate improved curriculum learning.\n\nEvidence: These claims are supported by empirical results on the Little‑Zoo environment, where MAGELLAN is compared against baselines like Online‑ALP, Eval‑ALP, and variants that use expert-defined groupings. The experiments show lower competence estimation error, higher success rates, and faster mastery across different goal types.", "methods_and_evaluation_criteria": "The approach combines an online RL framework (building on SAC‑GLAM with a finetuned Flan‑T5) with a metacognitive competence estimator that uses the LLM’s latent representations to predict success probabilities. An MLP is used on top of the LLM output, and a buffer of past model weights helps compute an absolute LP (ALP) metric. The method is evaluated using observed competence (success rate) and competence estimation error, as well as computational cost (in terms of additional evaluation episodes). The use of a custom environment designed to reflect the structure of language‑defined goals is well-motivated for the study’s aims. \n\nThe criteria and baselines chosen are appropriate for the stated problem, though reliance on a single environment limits the scope of the evaluation.", "theoretical_claims": "The paper provides a formal problem statement and introduces a competence function and ALP estimation formulation. There is no in‐depth theoretical analysis or proof (e.g., regarding convergence or sample efficiency) beyond the formulation.", "experimental_designs_or_analyses": "Experiments are performed on the Little‑Zoo environment with varying goal space sizes and include tests for generalization (held‑out goals) and adaptation (evolving goal spaces). Multiple random seeds and thorough evaluation every set number of episodes bolster the reliability of the findings.\n\nThe paper provides detailed plots of competence estimation error, success rates, and t‑SNE visualizations of the embedding space, along with ablation studies on architectural choices.\n\nAlthough the experimental design is comprehensive within the chosen setting, the use of a single, synthetic environment raises concerns about external validity. More experiments on diverse environments would help establish broader applicability.", "relation_to_broader_scientific_literature": "The work is well positioned within the literature on intrinsic motivation, curriculum learning, and autotelic agents.\n\nIt builds on prior work in LP estimation and goal selection (e.g., Online‑ALP, Eval‑ALP) while addressing limitations related to expert-defined groupings.\n\nThe integration of metacognitive prediction via an LLM is a notable contribution. However, additional discussion comparing this approach to alternative methods (e.g., uncertainty‑based exploration or meta‑learning techniques) would further contextualize its impact.", "essential_references_not_discussed": "While the paper cites many foundational works, it might benefit from discussing very recent advances in open‑ended learning and meta‑learning that do not strictly rely on LP. For instance, comparisons with recent methods leveraging uncertainty estimation or self‑supervised approaches in high-dimensional goal spaces would provide a more rounded view of the state of the art.", "other_strengths_and_weaknesses": "Strengths:\n\n1. Innovative use of an LLM to dynamically learn semantic relationships among goals, eliminating the need for brittle, expert‑defined groupings.\n2. Comprehensive experimental evaluation and extensive supplementary material that aid in reproducibility.\n\nWeaknesses:\n\n1. Limited evaluation domain: The reliance on a single synthetic environment (Little‑Zoo) raises concerns about generalizability to more complex, real‑world tasks.\n2. The method’s performance with larger or more advanced LLMs is not explored.", "other_comments_or_suggestions": "1. Consider extending experiments to additional environments or real‑world datasets to validate the generality of the approach.\n2. Provide a more detailed discussion on computational overhead and scalability.\n3. Address potential limitations, such as the sensitivity of performance to the choice of the underlying LLM.", "questions_for_authors": "Failure Modes: Were there any observed cases where the metacognitive predictions led to suboptimal goal selection or curriculum choices? If so, how were these instances addressed or mitigated?\n\nGeneralizability: How do you expect MAGELLAN to perform in environments other than Little‑Zoo, particularly in settings with more complex or less structured natural language goals?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "i5iGO1VV3Q", "reviewer": "Reviewer_s63G", "summary": "A key challenge in learning progress prediction is modeling one’s own competence in a computationally feasible and generalizable way. The paper introduces MAGELLAN, a metacognitive framework that enables LLM agents to learn to predict their competence and LP online. MAGELLAN captures semantic relationships between goals, allowing for sample-efficient LP estimation and dynamic adaptation to evolving goal spaces.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer s63G for their in-depth review and comments on our work, finding the claims clearly supported by our experiments, as well as highlighting the novelty of our method. We now provide answers to the comments and questions asked by the reviewer.\n\n## Table 1\nWe acknowledge that Table 1 (and by extension Table 4) currently lacks an explanation on the unit and the criteria used to assign each method’s score. Note that this was also raised by reviewer 3cqQ. We propose to update the manuscript with the following changes. First, we will replace “+++” by “high” and “ ” by “low” in the Efficiency column. Then, we will add the following explanation providing the unit and criteria:\n\nFor the Efficiency property, we consider a method’s efficiency as “high” if it does not require any additional evaluation (i.e. it only uses the performance observed on goals sampled), and as “low” otherwise.\n\nFor Competence transfer tracking, the “+” are given according to this evaluation:\n- absence of +: the estimated competence is updated only on sampled goals\n- +: the estimated competence is updated on a predefined goal subset the sampled goal belongs to \n- ++: the estimated competence is updated on a dynamically learned goal subset the sampled goal belongs to\n- +++: the estimated competence is updated on all goals\n\n## Modification of the embedding space\nAs the reviewer mentions, it is indeed insightful to analyze how the goal embedding space evolves over time. Such an analysis can already be seen in Appendix D.4.2. The chronogram we present there shows how the different categories are identified and how they interact. We put this analysis in appendices for now as it requires a full page.\n\n## Failure of the baseline\nThe reviewer asks for more clarification on why impossible goals are not included in the predefined categories given to baselines that use expert knowledge. These baselines are based on groups predefined in advance by human experts with a strong assumption: the goals within a group share the same learning dynamics and therefore the agent’s competence is the same over all goals in the group. If impossible goals were included, these groups would lose their relevance. Moreover, because of the large number of impossible goals, the average competence within each group will always be very close to 0. There will be no progress niche that the method can use to generate a curriculum, and performance will probably be very close to the random baseline. We will update the sentence at line 266 to make it clearer.\n\n## Generalization to other environments\nThe reviewer wonders how MAGELLAN would perform in other environments, in particular with a sparser reward signal. As a similar question was raised by most reviewers, we provide a single response, included in reviewer’s 3cqQ rebuttal. **This response notably shows new results on additional experiments launched according to reviewers’ comments.**\n\nFocusing on the reward signal sparsity property, we are not sure if the reviewer discusses the reward used by the RL policy or the signal used to train MAGELLAN and would like to remind that MAGELLAN assumes a goal-conditioned environment with a binary outcome on an episode (i.e. whether the goal has been reached or not). In Little-Zoo, the RL policy also uses a binary reward obtained on the final step only, but this is not mandatory (i.e. MAGELLAN does not make any assumption on the reward signal used by the RL policy).", "claims_and_evidence": "Yes, the claims are supported with clear and convincing evidence, and experiment results that support them.", "methods_and_evaluation_criteria": "Yes, the paper compares MAGELLAN against several suitable baselines and does sufficient analysis to support the claims.", "theoretical_claims": "Yes, they are correct.", "experimental_designs_or_analyses": "Yes. The claims are clearly stated in the introduction, and each claim is soundly supported with empirical evidence.", "relation_to_broader_scientific_literature": "The paper’s key contributions align with and extend prior work on open-ended learning, intrinsic motivation, and curriculum learning. MAGELLAN builds on existing research in LP-based goal prioritization. Unlike prior approaches that rely on expert-defined goal groupings or exhaustive evaluation, MAGELLAN leverages the generalization capabilities of LLMs to dynamically estimate competence and LP, addressing scalability challenges in high-dimensional goal spaces. The paper demonstrates how metacognitive LP estimation can enhance goal selection and learning efficiency in text-based environments.", "essential_references_not_discussed": "I don’t have any additional ones to suggest.", "other_strengths_and_weaknesses": "Strengths:\n- The paper is very well-written, with thorough experiments to support each claim.\n- The proposed methodology is novel and is a smart way to combine training LLM agents with RL while simultaneously estimating learning progress.\n\nWeaknesses:\n- Table 1, what is the unit or measurement of how many “+” are given?\n- line 266, “These baselines would totally fail if given all goals from the same hidden family, regardless of their feasibility”. It would be helpful to provide additional clarification on why this failure occurs.", "other_comments_or_suggestions": "- line 263, “we compare MAGELLAN to the classic approaches presented in 3.3”, missing “Section”\n- Figure 4, it will be helpful to have a legend for the task icons, including an indication of which tasks have to be done before others.\n- It will be interesting to see how the tasks embedding on the t-SNE plot changed across the whole training, instead of just seeing the before and after plots (Figure 5).", "questions_for_authors": "1. How do you think that this approach can be generalized to other text-based environments (e.g., nethack, which has much sparser reward signals)?\n2. Learning progress can also be roughly estimated by telling an LLM the agent’s current capabilities, and directly asking for what the next learnable ones are. What are some ways that these approaches can be compared with MAGELLAN?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Gspyx6reOD", "reviewer": "Reviewer_3cqQ", "summary": "This paper presents a framework for improving competence and learning progress (LP) estimation, used for the goal section of LLM agents in very large (even infinite) evolving goal spaces. The proposed method leverages the semantic relationships between goals and an LLM’s internal semantic knowledge to improve competence prediction. The method is compared to baseline LP estimation methods that rely on empirical evaluations of goals, both online during training or offline and with or without expert knowledge. The agent is tested on a proposed environment called Little-Zoo, which is a text-based learning environment designed to assess generalization and adaptation to evolving goal spaces. Using the Little-Zoo environment, the proposed method is shown to have lower competence prediction error than all baselines that do not use expert knowledge, and goal selection based on the estimated LP leads to faster agent learning. The paper also analyzes the generalization and adaptation of the competence prediction across the goal space.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer 3cqQ for their detailed review of our paper, finding our claims well supported by the experiments, the experimental protocol well-designed, and highlighting the interestingness of our section on automatic curriculum learning and our analysis of the LLM’s embedding space. We now address the concerns raised by the reviewer.\n\n## Single and limited environment\nWe understand the reviewer’s concern about the generalizability of our approach, given that only Little-Zoo is used for evaluation. As explained in the manuscript, Little-Zoo provides a controlled setting to rigorously assess MAGELLAN’s properties. Specifically, accessing the true underlying categories of objects allows us to verify whether MAGELLAN correctly identifies them.\n\nAdditionally, multiple reviewers' comments on Little-Zoo made us realize our explanations may have led to misunderstandings. Despite its name, Little-Zoo is neither trivial nor small—it consists of tens of thousands of goals. Crucially, goals are not just instructions but combinations of instructions and scene descriptions (i.e., accessible objects). **Estimating an agent’s LP in Little-Zoo requires understanding object families and discerning what makes an instruction possible in a given scene (i.e., the tech tree)**. The combination of all possible instructions and scene initializations results in 25,000 goals, 80% of which are impossible. Such a goal space, with mostly infeasible goals, also naturally arises when using freely generated natural language goals with an LLM (e.g., in an autotelic agent), as many goals wouldn’t respect the environment’s dynamics. Finally, Little-Zoo builds on prior environments like WordCraft (Jiang et al., 2020) by **introducing a key missing component: semantic relationships between goals**.\n\nRegarding the generality of our results, MAGELLAN relies on extracting structure from the semantic goal space. The LLM’s embedding space notably enables MAGELLAN to generalize (e.g., knowing which animals are herbivorous or that water is needed to grow seeds). MAGELLAN is designed for language-based goal spaces where LLM embeddings are most effective. In environments with non-language goals or sparse goal structures, the LLM’s utility is reduced, limiting generalization. In the worst case, MAGELLAN is expected to perform similarly to Online ALP.\n\n**To further support the broader applicability of MAGELLAN, especially on more complex natural language goal spaces, we conducted additional experiments**. We evaluated MAGELLAN’s ability at estimating a learner’s competence on math problems from the OpenR1-Math-220k dataset. Here, MAGELLAN must deal with highly non-trivial problems and in particular identify what is the type of each problem to generalize its competence estimation. We focus on *Algebra*, *Number theory*, and *Geometry*, leading to more than 20,000 problems. Given the limited time allowed by the rebuttal period, we did not train an RL learner but rather simulated one that has different learning dynamics on each problem type (i.e. how fast its success probability increases). Similarly to our experiments in Section 4.1, we compare the competence estimation of Online-ALP and MAGELLAN (using Qwen2.5-0.5B) when problems are sampled using an Uniform curriculum. The code can be found on our anonymous repository: https://github.com/ghjnkmjl745678/MAGELLAN_ICML/blob/main/math_experiments/.\n\nFigure *“sr_estimation_math.png”* from this repository shows that MAGELLAN accurately estimates each problem’s success probability. This indicates that it is able to find each problem’s type and generalize its competence estimation within each type. In comparison, Online-ALP leads to poor competence estimations given the large number of problems. \n\nWe would like to perform the same experiment with two other LLMs (Flan-T5 80M and Flan-T5 248M) as performed in our additional experiment for reviewers 9oss’s rebuttal. Unfortunately, our cluster is currently under maintenance but we should be able to provide these results within the coming days. **If the reviewers kindly accept to respond to our rebuttal, we will provide these results**.\n\n## Table 1\nWe agree with reviewer 3cqQ that Table 1 (and Table 4) need more explanations. We proposed changes and explanations to add to the manuscript in our rebuttal to reviewer  s63G.\n\n## Generalization abilities\nWhile Table 2 only shows the average error over several evaluations performed throughout training, Appendix D.4.1 and in particular Figure 16 shows the evolution of the error on the held-out test set. One can see that both the observed and estimated success probability of Online-ALP on “Grow Carnivore” goals is near 0. One can also see that this is the only case where this phenomenon happens (i.e. the observed success probability is greater than 0 on all the other plots). We therefore can affirm that this effect only affects the generalization results of Online-ALP on the “Grow carnivore” category.", "claims_and_evidence": "1. The proposed method predicts agent competence on goals more accurately than existing methods.\n\nThis claim is well supported in the paper, as experiments show estimation errors are much lower for the proposed method than all baselines except those that rely on expert knowledge. \n\n2. The proposed method provides an automatic curriculum over a large evolving goal space that allows an agent to more efficiently learn to master the environment.\n\nThis claim has been shown quite convincingly for the proposed Little Zoo environment, as results show goals sampling based on LP predicted using the proposed method learns faster and to a higher final performance than all baseline methods except for one that uses expert knowledge. \n\nHowever, all experiments were conducted in the Little Zoo environment. This weakens this claim since it is difficult to be confident about the generality of these results with any evidence on more environments (such as Minecraft, although it is not text-only and would require a VLM). \n\n3. The proposed method’s competence estimation can generalize to unseen goals and adapt quickly as the goal space evolves.\n\nThe paper conducted one experiment to study generalization (section 4.3) and indeed show the proposed method has the lowest prediction error on a held-out set of tasks. Again, this is somewhat weakened as it was only conducted on Little-Zoo, which may not generalize to other environments.", "methods_and_evaluation_criteria": "Methods are sound. As mentioned above, the sole focus on Little-Zoo as the evaluation environment limits the confidence of the claims made in the paper. Consider evaluating the proposed method on a more diverse set of environments with large goal spaces.\n\nLittle-Zoo is design to have 80% of goals being impossible. This is quite significant. What was the motivation for this? Does this make the prediction task easier since predicting impossibility already gets to 80% accuracy?", "theoretical_claims": "The paper does not make any theoretical claims.", "experimental_designs_or_analyses": "The paper’s experiments are generally well designed and analysis sound. As mentioned above, the focus on Little-Zoo only is a limitation.", "relation_to_broader_scientific_literature": "The paper is well placed within broader scientific literature, with all relevant prior works discussed in the related works section.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "D.3.2 Line 1382 \"usinf\" -> \"using\"\n\nThis section on the auto-curriculum over goals is very interesting.\n\nThe clustering of goals is also a very interesting illustration.", "questions_for_authors": "Unclear how to interpret the plus and minus symbols in Table 1, what do they mean exactly?\n\nIn Table 2, the authors note that the error in Online-ALP's Grow Herbivore is due to the policy not mastering that goal category. How do you measure this effect and how do you determine the extent other values in the table are or are not affected by this?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u2Qv2R2LtL", "reviewer": "Reviewer_pnae", "summary": "This paper tackles the problem of learning LLM agents in large goal spaces. This paper considers the situation where an LLM maximizes the expected success probability over the huge number of goals. To train the LLM more efficiently, the paper proposes a goal selector that chooses the best goal to achieve based on a neural competence estimator based on previously collected trajectories. The proposed method demonstrates better generalizability over the goal space compared to other baselines.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer pnae for their detailed reading of our manuscript, finding our method reasonable and our experiments well-designed and supporting our conclusions. We now address the reviewer’s concerns.\n\n## Novelty / contribution concerns\nWe acknowledge the reviewer’s concerns regarding the simplicity of our baselines and the contribution of our approach. \n\nFirst, the reviewer rightfully pointed out that Online-ALP cannot estimate competence over unseen goals. However, we would also like to remind the reviewer that our literature review on LP estimators in Section 2.2 and Appendix B highlights that: 1) to this day, the baselines we implemented in our paper (including Online-ALP) are **the only approaches that exist for estimating LP over discrete goal spaces** and 2) the baselines based on periodic evaluation of the competence over the full goal space are intractable in large goal spaces. \n\nSecond, the reviewer mentions that our main contribution is the introduction of a parametric LP estimator able to generalize from seen goals to unseen ones. However, while few prior works studied the use of neural networks to learn latent representations of continuous goals, our work is the first to study parametric LP estimators and their generalization abilities for natural language goals (which have become ubiquitous in the era of LLM agents). Moreover, Appendix D.1 shows that learning such an estimator is not trivial. In particular, we show that learning a neural network on top of a fixed and pretrained embedding model totally fails to build semantic relationships useful for precise LP estimation. Our experiments show that MAGELLAN successfully estimates and generalizes LP by leveraging and adapting the embedding abilities of pretrained LLMs.\n\n## Explanations on Little-Zoo\nWe acknowledge that the section explaining Little-Zoo in the main paper (Section 3.4) is fairly short, with details left to Appendix A. As the reviewer suggested, we propose to improve this section, especially to explain how goals are constructed. **In particular, the reviewer’s comment made us realize our explanation may be confusing as it led to a misunderstanding of two fundamental properties of our environment**:\n**1)  goals are not an instruction alone but rather the combination of an instruction and the description of a scene initialization (i.e. objects accessible)**. For instance, here is a feasible goal: “Goal: Grow deer. You see: baby deer, bookshelf, water, tomato seed.”. And here is an impossible one (water is needed to grow the seed): “Goal: Grow deer. You see: baby deer, bookshelf, baby lion, tomato seed.”.\n**2) Little-Zoo has multiple categories of objects (i.e. plants, herbivores and carnivores) and each category requires a specific sequence of actions to grow one of its objects.**\n\nConsequently, estimating (and generalizing) well an agent’s LP in Little-Zoo not only requires one to understand the inner families of objects but also necessitates one to discern what makes an instruction possible or not in a given scene (i.e. discovering the tech tree). \n\nWe give all these details in Appendix B, including the optimal actions sequence to solve the possible instructions, but we will update Section 3.4 so that Little-Zoo’s natural complexity is easier to grasp. \n\n## Essential References Not Discussed\nWe thank the reviewer for identifying references not discussed. In particular, UED approaches are **Automatic Curriculum Learning (ACL) methods** that do not assume a discrete set of goals to sample from and that, to the best of our knowledge, do not make use of Learning Progress. In Section 2.1, we discuss various ACL methods but do not provide an extensive review of them as our paper does not introduce yet a new ACL approach.\n\nIndeed, we focus on 1) improving the estimation of **Learning Progress on natural language goal spaces** and 2) augmenting LLM agents with a metacognitive monitoring skill. \n\nOne can use this LP estimation (as we have done in Section 4.2) to scaffold an RL learner’s curriculum using an ACL method but we did not propose any contribution to the latter. As explained in Section 3.2, we reused for this prior approaches leveraging a Multi-Armed Bandit framework to sample goals according to their LP estimation.\n\nWe thus propose to add UED references to the ACL methods mentioned in Section 2.1.", "claims_and_evidence": "The main claim of the paper is that a neural-based competence estimator trained on small subset of goals seen during training can be generalized to unseen goals. I believe that this claim is well supported by Table 2 and Figure 5.", "methods_and_evaluation_criteria": "I find the proposed method reasonable. However, I have concerns regarding the significance of the performance improvement and the simplicity of the baselines considered. Specifically, the observed improvement is not entirely surprising, as online-ALP is naturally expected to exploit previously seen goals with high competence scores, given that there is no way to estimate competence for unseen goals. The only difference is the introduction of a neural network to enhance generalizability.\n\nAdditionally, I have concerns about the presentation of the proposed benchmark. Since it is newly introduced, I believe the authors should dedicate more space in the main text to explaining it in detail. At a minimum, they should provide examples of goals or describe their structure, as goal structure plays a crucial role in generalization. From my understanding, all goals follow a simple format, such as \"grasp something\" or \"grow something,\" which makes them relatively easy for an LLM to generalize.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Overall, I find the experimental design and analysis well conducted. However, I have some concerns about the simplicity of the goal format and would like to see an analysis on the failure modes of the proposed method.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "To me, this work is closely related to unsupervised environment design (UED), where a neural network generates novel environment instances to facilitate agent learning [1, 2]. If each goal is treated as a separate MDP, the setting in this paper is identical to UED. I recommend discussing the relationship between the proposed method and UED to provide clearer context. I have introduced only the most well-known paper on UED, but numerous related papers have been published over the past five years.\n\n[1] Dennis et al., Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design, NeurIPS 2020 \\\n[2] Jiang et al., Replay-Guided Adversarial Environment Design, NeurIPS 2021", "other_strengths_and_weaknesses": "The writing and presentation were clear to me, except for the explanation of the proposed benchmark.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Please refer to \"Methods And Evaluation Criteria\" for my specific concerns.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}]

Author rebuttal:

Meta-review: This paper introduces a method that enables LLM agents to estimate their own learning progress (LP) across natural language goal spaces by leveraging the LLM's semantic understanding, which facilitates automatic curriculum learning. Reviewers appreciated the idea of leveraging an LLM's capability to estimate learning progress, eliminating the need for expert-defined goal groupings. They also found that the experiments are well designed and strong enough to demonstrate the advantage of the proposed method. However, there was a concern regarding the lack of discussion on unsupervised environment design (UED). I highly encourage the authors to clarify the connection and distinction from UED in the camera-ready version. Assuming that the authors make the suggested change, I recommend to accept this paper.
Decision: Accept (poster)