Unsupervised Learning of Efficient Exploration: Pre-training Adaptive Policies via Self-Imposed Goals
Authors: Octavio Pappalardo
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement Learning
Unsupervised Reinforcement Learning
Meta-Reinforcement Learning
Pre-training
Curriculum Learning
Primary area: reinforcement learning

Abstract: Unsupervised pre-training can equip reinforcement learning agents with prior knowledge and accelerate learning in downstream tasks. A promising direction, grounded in human development, investigates agents that learn by setting and pursuing their own goals. The core challenge lies in how to effectively generate, select, and learn from such goals. Our focus is on broad distributions of downstream tasks where solving every task zero-shot is infeasible. Such settings naturally arise when the target tasks lie outside of the pre-training distribution or when their identities are unknown to the agent. In this work, we (i) optimize for efficient multi-episode exploration and adaptation within a meta-learning framework, and (ii) guide the training curriculum with evolving estimates of the agent’s post-adaptation performance. We present ULEE, an unsupervised meta-learning method that combines an in-context learner with an adversarial goal-generation strategy that maintains training at the frontier of the agent’s capabilities. On XLand-MiniGrid benchmarks, ULEE pre-training yields improved exploration and adaptation abilities that generalize to novel objectives, environment dynamics, and map structures. The resulting policy attains improved zero-shot and few-shot performance, and provides a strong initialization for longer fine-tuning processes. It outperforms learning from scratch, DIAYN pre-training, and alternative curricula.

Published as a conference paper at ICLR 2026
UNSUPERVISED LEARNING OF EFFICIENT EXPLO- RATION: PRE-TRAINING ADAPTIVE POLICIES VIA SELF-IMPOSED GOALS
Octavio Pappalardo Independent Researcher cctavioppappllarddeemaall com
ABSTRACT
Unsupervised pre-training can equip reinforcement learning agents with prior knowledge and accelerate learning in downstream tasks. promising direction. grounded in human development, investigates agents that learn by setting and pursuing their own goals. The core challenge lies in how effectively gener- ate, select, and learn from such goals. Our focus is on broad distributions of downstream tasks where solving every task zero-shot is infeasible. Such set- tings naturally arise when the target tasks lie outside of the pre-training distri- bution or when their identities are unknown to the agent. In this work, we (i) optimize for efficient multi-episode exploration and adaptation within meta- learning framework, and (ii) guide the training curriculum with evolving esti- mates of the agent's post-adaptation performance. We present ULEE, an unsu- pervised meta-learning method that combines an in-context learner with an ad- versarial goal-generation strategy that maintains training at the frontier of the agent's 's capabilities, On XLand-MiniGrid benchmarks, ULEE pre-training yields improved exploration and adaptation abilities that generalize to novel objec- tives, environment dynamics, and map structures. The resulting policy attains improved zero-shot and few-shot performance, and provides strong initial- ization for longer fine-tuning processes. It outperforms learning from scratch, DIAYN pre-training, and alternative curricula. Code is available at: https: wtw.:huus..coo/ttaavoooaaaaaaaaa
INTRODUCTION
Large-scale pre-training has underpinned many real-world successes of deep learning in computer
vision (Krizhevsky et al. 2012; Chen et al. 2020 Grill et al., 2020 He et al. 2022) and language
modeling (Devlin) (018) Brown 2020). These advancements have motivated analogous efforts in
deep reinforcement learning (RL) (Liu & Abbeel 2021 Schwarzer et al., 2021; Seo et al.) 2022),
where the prevailing paradigm remains train agents from scratch for each new task. The ultimate objective is to develop foundation policies: agents equipped with transferable knowledge that ad- dress RL's fundamental challenges of sample inefficiency and lack of generalization. We investigate unsupervised RL, where agents interact freely with their environments without access to extrinsic rewards, means acquire such policies. This setting raises key questions regarding what data an agent should seek and what it should learn from it. These questions are tightly coupled because the collected data a direct consequence of the agent's 's incorporated behaviors. Ultimately, we seek system that can continuously gather useful data at scale in unsupervised, open-ended fashion and leverage to acquire transferable capabilities.
A promising direction is to design agents that autonomously generate their own goals and learn by attempting to solve them (Oudeyer et al. 2007 [air et al. 22018: Forestier et al. 2022). Our focus on how such goals should be generated, selected, and exploited for pre-training. This connects to automatic curriculum learning, yet existing methods have often assumed either a fixed set of goals goal space that is closely aligned with the evaluation tasks. Progress toward foundation policies
Work conducted independent researcher. Currently University College London (UCL.).

---

Published as a conference paper at ICLR 2026
requires assuming little information about downstream objectives and instead pre-training for a broad range of human-relevant tasks. Moreover, a general base policy must have the capacity to contend not only with novel objectives but also with limited information, whether in the form of partial observability, uncertain dynamics, or unspecified goals. Thus, it must be effective across spectrum of difficulty levels, delivering quick performance on easier tasks and sample-efficient adaptation over long timescales on complex ones.
With these desiderata in mind, we introduce metric that guides goal generation using the agent's performance on goals after an adaptation budget. This contrasts with much prior work that evaluates goal desirability based on immediate performance without any task-specific adaptation (Sukhbaatar et al.] [017]] Florensa et al., (0118). We pair this metric with an adversarially trained goal-generation system that proposes challenging yet achievable goals for the pre-trained policy to learn from. This yields an adaptive curriculum that maintains goals at intermediate difficulty and avoids uninforma- tive extremes of too easy or unsolvable goals. Shifting toward a measure of difficulty that relies on post-adaptation performance aligns better with the intended evaluation setting, where the policy must face novel tasks that require adaptation. Moreover, focuses training on more challenging tasks and leads to coverage of broader distribution, with more tasks becoming easy and fewer remaining effectively unsolvable.
Anticipating the need for test-time adaptation naturally points to meta-learning (Duan et al. 2016 Finn et al. 2017), which explicitly optimizes for efficient learning. Meta-learning has primarily been studied with pre-training carried out over hand-designed task distributions, and its integration with unsupervised settings remains largely underexplored (Gupta et al. (018)). We bridge these directions by introducing an unsupervised meta-learning algorithm that meta-learns a base policy using an automatic curriculum of self-generated goals guided by our post-adaptation difficulty metric.
Our main contributions are as follows:
We introduce post-adaptation task-difficllty metric for guiding automatic goal generation and selection in unsupervised autotelic agents.
We present ULEE, an unsupervised meta-learning method composed of (1) an in-context learner trained with self-imposed goals, (2) a difficulty-predicte network estimating post- adaptation performance, (3) an adversarial agent trained find challenging candidate goals, and (4) a sampling strategy that selects goals within a desired difficulty range.
We evaluate ULEE in complex yet computationally accessible environments across multi- ple timescales and types of generalization. It outperforms baselines and ablations in explo- ration, fast adaptation, fine-tuning to fixed environments, and fine-tuning in meta-learning settings. All code is open-soureed.
2 RELATED WORK
Several strategies have emerged to make use of reward-free objectives in RL. These include learning world model (Sekar et al. 2020 Sco et al. 22022] Rajeswar et al. (023), utilizing auxiliary objec- tives that aid representation learning (Jaderberg et al. 2016; Oord et al. 2018 Zhang et al. 2020aa Stooke al. | Schwarzer al. 202 ) and training agents with intrinsic rewards (Chentanez et al.] 2004 Oudeyer et al. [207]] [Schiidhuber] 2010) that lead to useful behaviors. The latter has been major line of research into both pre-training and online exploration, with methods in both areas holding close relationship. Most work in this line of research can be grouped into methods that consider predictions over some aspect of the environment and are guided by error, uncertainty, or progress in these predictions (Schmidhuber 1991 Pathak et al. 2017, Burda et al.) 2018 Pathak 2019h; those whose rewards come from measures of diversity in collected data (Bellemare 2016 Hazan et al., 2019 Lee et al., 2019 Liu & Abbeel 2021: Yarats et al.. 2021); and those who seek to maximize empowerment (Klyubin et al.) 2005; Mohamed Jimenez Rezende 2015 2 Sharma et al. 2019) or are guided by self-imposed goals
al.
et al.
Gregor et al.] 2016 Eysenbach et al.
2018
(Sukhbaatar et al.) 2017; Nair et al. 2018 Florensa et al. 20188) Pong et al. 2019 Forestier et al.
2022). Our work focuses the latter and incorporates ideas from automatic curriculum learning
(Bengio et al. 2009 Andrychowicz et al.) 2017 Florensa et al. 2017 Matrien et al.) 2019 Jiang
al. 2021 which has proved capable of significantly accelerating learning in RI..
2

---

Published as a conference paper at ICLR 2026
Intermediate difficulty. Previous work on unsupervised RL and automatic curriculum generation has explored dictating the agent's 's curriculum by continually selecting tasks of intermediate diffi- culty. In GoalGAN (Florensa al., 2018), GAN (Goodfellow et al. (0144) variant is trained to output goals that are neither too easy nor too hard for the agent. The goal space is fixed beforehand and remains the same for training and evaluation. The eurriculum's 's aim is accelerate learning across all feasible goals. Racaniere et al. (2019) address the same multitask setting but generate goals uniformly across all difficulty levels. Their goal generator trained with custom objectives that promote goal validity, coverage, and controllable difficulty. They introduce a "judge" net- work trained on past goals estimate success probability. Conditioning the judge and generator on environment observations enables them to handle procedurally generated and partially observable environments. Zhang et al.] (2020b) propose curriculum method that does not rely on adversarial training. They treat previously visited states as candidate goals and estimate goal difficulty from the epistemic uncertainty of the value function, approximated using an ensemble of value networks. They show that goals at both extremes of difficulty correspond to low uncertainty. AMIGo (Campero eet al,, (020) is a method to improve online exploration in which a generative network is trained jointly with the agent provide it with progressively more challenging yet achievable tasks. This interaction leads to the exploration of increasingly complex behaviors that assist the agent in find- ing the environment's s sparse extrinsic reward. Like our work, these methods ground their curricula in difficulty- metrics, though in settings that differ fundamentally from ours. None consider unsupervised pre-training followed by transfer to previously unseen tasks, and they don't 't consider scenarios where goal information isn't available. Instead, they train goal-conditioned policies and design curricula either for multitask learning or to accelerate progress on a single extrinsic task available alongside the unsupervised goals. Moreover, their difficulty estimates are derived from the agent's immediate performance, without allowing for task-specific adaptation. ASP (Sukhbaatar et al. 2017) is closer to our setup in that it also employs an adversarial policy ("Alice") to propose goals, but still optimizes a different objective. ASP is evaluated on a single external task using a goal-conditioned policy ("Bob"), which alternates between self- play and training on the target task. During self-play episodes, Bob is rewarded for either imitating or undoing Alice's 'ss actions. OpenAAl eet al. 202T) extend ASP with imitation learning loss and scale it to evaluate the generalization of the goal-conditioned policy to held-out tasks. Beyond goal-based curricula, other works investigate adjusting task difficulty through the automatic selection or modification of training environments
[Wang et al. (2019; Akkaya et al. 2019 Mchta et al. 2020 Team et al. 202 2023).
Unsupervised meta-learning. The setting of unsupervised meta-learning for RL, where meta- learning is performed on automatically acquired tasks, was first explored in /Gupta et al. (0188) There, each latent skill z discovered by Diversity is All You Need (DIAYN) (Eysenbach is associated with a self-generated goal, and the discriminator probabilities D(2ss) are used as the rewards for that goal. .Jabri et al. (2019) extends this approach with a dynamic curriculum and eval- uates it in partially observable visual environments. Alternatively, Mutti et al. (2022) learns a policy that maximizes the entropy of the induced state-visitation distribution, with particular emphasis on adverse environments.
2018)
Mutti
Ada. Our work also bears similarities to Ada (Team et al. 2023 Ada meta-learns an in-context learning agent (Duan al.) 2016; Wang al. 2016 Mishra et al. 2017) on a large task distribution using a curriculum and evaluates performance on novel tasks in a few-shot regime. Unlike our method, Ada trains solely on tasks produced by its environment generator and does not incorporate intrinsic goals. In addition, its policy is explicitly conditioned on goal and environment-dynamic. information, and the learned policy not evaluated as an initialization for extended fine-tuning.
3 METHOD
We introduce ULEE (Unsupervised Learning of Efficient Exploration), an unsupervised pre-training method in which an intrinsically motivated agent meta-learns an adaptive policy from a curriculum of self-imposed goals. This section outlines its main components. Pseudocode and additional details are provided in Appendices|P A. and A.2
Both pre-training and evaluation are performed over families of partially observable Markov decision processes (POMDPs). We denote by gra and y the distributions of pre-training and evaluation environments, respectively. Agent interactions with each environment
Setting and Notation:
3

---

Published as a conference paper at ICLR 2026
M occur in discrete time steps t E fO..... .T. 1}, where 8t S is the underlying state of the environment at time step t, O₁ ~ ((-[s;) is the observation the agent gets, and d E A is the action it takes. Environments in dynamics (M( 8t, a), and reward function (8t, az). In the pre-training phase, rewards are not provided by the environments but determined by the agent's own intrinsic goals. We write j unmu for the reward-free counterpart of μ 4 and pair each M ~ a u with a goal-conditioned reward function obtain full POMDPs. Although the policy operates on observations, for simplicity we system and goal-conditioned reward function privileged access to full state information during pre-training.
work differ in their initial state distribution PM, transition
give the
3.1 PRE-TRAINED POLICY
The Pre-trained Policy is the component trained with self-imposed goals and the sole component evaluated at test time. Prior work on self--gnnraatd goals typically learns goal-conditioned policies. Although goal conditioning scales to broad goal distributions, is ill-suited for evaluation settings in which (i) the goal to be achieved is unknown and must be discovered, (ii) there is no suitable way to encode the goal, or (iii) the goal representation sufficiently out-of-distribution be unin- terpretable by the policy. These scenarios place goal- condiiondi policies outside the regime where they excel known goals with recognizable representations. Some methods address this by treating the goal-conditioned policy as low-level controller within hierarchy and deploying the high-level controller (Kulkarni et al., 2016, Vezhnevets et al.) 2017; Sukhbaatar et al. 2018 Sharma et al.) 2019). In contrast, we pre-train an unconditioned policy that can be directly deployed.
We investigate the agent's ability to solve goals under various adaptation timescales, noting that longer horizons better match the demands of challenging or unfamiliar tasks. To that end, adopt a meta-learning approach to train π, which allows explicit optimization of exploration and adap- tation over multiple episodes. In implementation,
take black-box approach (Duan al., 2016 Wang et al. 2016). where the agent interacts with environments over a sequence of contiguous episodes and selects actions based its full interaction history. By conditioning on past observa- tions, actions, and rewards, the policy learns to adapt in-context. We refer to the whole multi-episode interaction lifetime and train maximize its expected discounted lifetime return.
H ΣΣ (j
T-1
I [(T)- = EM M grpiglM)
Epac Paro
1y7++
(I)
In Equation j E (1,.... H) indexes episodes within a lifetime, r is the reward for the sampled goal g at time step in episode j. 7 E 1) is the discount factor, and p(g M) is the evolving distribution of self-imposed goals induced by ULEE's goal-generation mechanism.
3.2 GOAL CURRICULUM
In our work, goals are obtained as functions of states via f S - G. An agent accomplishes goal at time step t if f (st+1) 9. meaning the goal is reached with action oe, Our method is flexible to the choice of f- Our procedure finds and selects goals that are of intermediate difficulty according to the current capabilities of the Pre-trained Policy. This prevents it from focusing on goals that too hard to get any feedback on or too easy to require new capabilities, and thus provides a strong learning signal (Florensa al. 2018 Zhang al. 2020b). We define the difficulty of a goal g for policy π in environment MM as the complement of ' 's expected success rate over the last K episodes of sequential interaction (Eq. 2. Thus, with H being the total number of episodes the lifetime, the performance over the first H K episodes, which the agent leverages to explore and adapt, is ignored.
H
1ate (0. (o.....T 1): Abl211 = =
d(g: T. M) = -1- 1-Epse PM
Σ
(2)
K
j=H-K+1
4

---

Published as a conference paper at ICLR 2026
For each sampled environment M f and goal p(g M), running for one lifetime yields single-sample empirical estimate of that task's difficulty d(g: n; M). We implement the intermediate-difficltyy curriculum with three subsystems that dynamically adjust p(g M).
3.2.1 GOAL PROPOSAL
Goal-search Policy aas is trained adversarially to reach hard goals. More specifically, it trained to maximize Jgs(Pe gT)))) M. - r'rg") where y a discount factor and the goal-search rewards y88 the difficulties of the candidate goals encountered (Eq. 3
r - (s M) = <, M)
each training environment M, the Goal-search Policy runs for episodes prior to the Pre-trained Policy. Let So, $1,... denote the sequence of states visited across these episodes. Then, the multiset of candidate goals collected by Tga for M is defined as GCM = {f(st) : E {0, n, 2n, }}, where E N controls the temporal spacing between considered goals.
3.2.2 GOAL SELECTION
To decide with which goal to train the Pre-trained Policy among the candidate goals GCM. we perform a random sample among goals at a desired level of difficulty. More specifically, we define a lower bound LB and upper bound UB and sample uniformly among goals whose difficulty lies within these bounds (Eq. 4.
9M US),
S (g E GCM LB S d(g, M) S CBB
(4)
Here, 9M is the goal that is sampled for environment M, and is the current Pre-trained Policy, If = 0, we instead sample uniformly from GCM.
3.2.3 DIFFICULTY PREDICTION
In practice, there is no access to the goal difficulties needed to compute the goal-search rewards or to uncover which goals are in S. Even computing single-sampl sample empirical estimate requires running ı for multiple episodes, which becomes prohibitively expensive and cample-inefficient. Conse- quently, we introduce Difficulty Predictor network that can estimate the difficulty of goals the agent has faced. By replacing the true goal difficulties with the Difficulty Predictor estimates d, we can approximate the behavior of Equations 3 and 4 without additional environment interactions.
As the difficulty of goals depends on the agent's evolving capabilities, we keep buffer By of triplets (g. EM, d(g)) that holds only the most recent goals on which pre-training was performed. In each iteration, train the Difficulty Predictor with supervised L2 regression loss computed from goals in this buffer (Eq.5)
(or(o) = |Byl
Σ (dolg.E) - d(g))²
(5)
(g.E,d)EB,
The target d(g) a one-sample empirical estimate of goal difficulty obtained from the Pre-trained Policy's 's training lifetime on g. EM denotes a generic object (e.g., Su Pas) that carries information regarding the environment M on which the empirical estimate was obtained.
EXPERIMENTS
Unsupervised reinforcement learning methods can differ in the form of knowledge they aim quire, leading to various criteria for their evaluation. In this work, we assess the downstream utility of ULEE's Pre-trained Policy on novel tasks. Specifically, we ask:
Q1 What exploration capabilities does the Pre-trained Policy exhibit? (Sec. 4.3.1
5

---

Published as a conference paper at ICLR 2026
Q2 How well does the policy adapt with limited experience? (Sec. 4.3.2)
Q3 Does the Pre-trained Policy provide strong initialization for fine-tuning under larger adap- tation budgets? (Sec. 4.3.3
Q4 Is the Pre-trained Policy an effective initialization for downstream meta-learning on curated task distributions? (Sec. 4.3.4)
Finally, Section 4.3.5 revisits QI and Q2 to test along an additional axis of generalization.
4.1 BENCHMARKS AND TASKS
We evaluate distributions of pre-sampled, procedurally generated, partially observable grid vironments from the JAX-based Xland-MiniGrid (Nikulin et al.) (024), which retains the diversity of XLand (Team et al., 2021) while enabling orders of magnitude faster experimentation. Envi- ronments contain a goal and a set of rules governing object-object and agent-object interactions (Fig. la Various types of goals, rules, objects exist. We use two distributions of million pre-sampled combinations of {goal, rules, initial objects} adhering to the following specifications:
trivial: prerequisite rules must be triggered before achieving the goal: three distractor objects not involved in achieving the goal are present.
small: 0-2 rules that the agent must trigger before the goal becomes achievable; distractor objects and (-2 distractor rules that lead dead ends present.
We allocate 100000 combinations μ and the remaining 90% to ان uns All environments use 13x13 grid, symbolic 5x5 observations, and an episode horizon of 256 steps with early termi- nation upon success. Each distribution is instantiated with either four- six ex-rroom layouts, yielding our three benchmarks: ARooms-Triviee 4Rooms -mall,, and 6Rooms-Small (Fig- D.
The goal mapping / -> G used during unsupervised pre-training encodes an inductive bias into which behaviors are worth mastering and thus affects the difficulty of generalizing to XLand- MiniGrid extrinsic goals. We consider mapping feounss to the grid's per-object counts, where the agent succeeds when its interactions lead to the same count vector, and mapping fgria that defines goals by the agent's 's position and the grid's exact configuration, excluding the agent's 'ss orientation and pocket; success requires reproducing that configuration. These mappings serve reasonable task- agnostic proxies. We utilize feouns. as the default mapping and evaluate fgria exclusively on 4Rooms- Small, making the distinction explicit. See Appendix A.2. for further details the environments.
(b)
(c)
(d)
Figure 1: Panel (a), reproduced from Nikulin et al. 2024 (CC BY 4.0), shows goal and the rules
that must be triggered to achieve it, represented as a tree of depth 2. Analogously, tasks from the trivial and small benchmarks correspond to depth-0 and depth- I trees. Panels (b)-(d) show example environments from the three benchmarks: 4Rooms- Trivial, 4Rooms-Small, and 6Rooms-Small.
4.2 BASELINES AND ABLATIONS
We study ablations of ULEE's goal-generation mechanism. A first set of ab- + sampling). The search component is adversarial when the Goal-search Policy is trained to reach difficult goals, and random when placed by a random policy. The sampling component is bounded when goals are drawn uniformly from those of intermediate difficulty and uniform when drawn from all candidates. Our main method
Ablation Variants:
lations denotes each variant as ULEE

---

Published as a conference paper at ICLR 2026
(Sec. 3 corresponds to ULEE (adversarial bounded), which refer to simply ULEE. In addi- tion, we introduce ULEE (SED), which estimates difficulty from the Pre-trained Policy's 's immediate rather than post-adaptation performance. After meta-leaminn with tasks multi-episode interac- tions, we compute their single-episode difficulty using K additional episodes run as they were first episodes (resetting memory before each). These extra episodes excluded from the ablation step counts reported in figures and text.
Baselines: We compare our method against DIAYN (Eysenbach et al. 2018), an empowerment- based unsupervised RL method used as a pre-training baseline (Q1-Q3); PPO (Schulman et al, 2017 trained from scratch on fixed tasks standard model-free baseline (Q3); RND (Burda al, 2018 trained from scratch on fixed tasks as popular baseline that uses intrinsic rewards to improve online exploration (Q3); and RI.2 (Duan et al., 2016, Wang et al. 2016) trained from scratch on a
curated task distribution as standard meta- meta-learning baseline (Q4).
Implementations: All policies are optimized with PPO and use the same Transformer-X XL, backbone Dai et al.) 2019 Parisotto et al. 2020), with separate MLP heads for the Actor and Critic. Variants with bounded sampling use LB 0.1 and UB = 0.9. For DIAYN, the skill discriminator has access to the full state and conditions on (s)). Further information is provided in Appendix A.2
4.3
EXPERIMENTAL RESULTS
4.3.1 EXPLORATION
To assess exploration capabilities, measure the percentage of goals from μ reached under budgets of 1-20 episodes. ULEE's Pre-trained Policy substantially outperforms random behavior (the common default when learning from scratch) and DIAYN pre-training, reaching more than twice as many goals at the 20-episode mark across all benchmarks (Fig. ( Ablations highlight the need for mechanisms to avoid pre-training on trivial goals and show that guiding the curriculum by post-adaptation performance, rather than immediate success, becomes increasingly beneficial as benchmark difficulty grows. Variants using an adversarial Goal-search Policy achieve the best results, and bounded goal sampling proves effective when goal search is random.
4Rooms Trivial
4Rooms Small
6Rooms Small
4Rooms-Smal (far)
Goals Reached (%)
18
Episode Number
Eptsode Number
Ep Episade Number
Episode Number
DUVN
Random
ULEE brandom-buunsed)
VLEE (nandom--nittmm
ULEE tndversatt--unitmmo
ULEE (SED)
LEFE
Figure 2: Percentage of ~ goals reached under different exploration budgets. A goal is considered reached at episode if it was achieved in any episode S j. Results are averaged across seeds, with individual seeds overlaid faint thin lines.
4.3.2 FAST ADAPTATION
A policy must not only discover goal solutions but also know how to reliably achieve them once does. Figure 3a evaluates gradient-free, few-shot adaptation over 30-episode lifetimes. ULEE's Pre-trained Policy leverages its interaction history to improve steadily, reaching up to 3x increase in mean return by the 30th episode. Variants trained with random Goal-search Policy manage some adaptation early on but soon stagnate. For DIAYN, adaptation occurs at single, discrete point by selecting the best-performing skill after all have been evaluated. Figure 3b) reports the agent's post-adaptation return (measured the mean over the last 10 episodes) by task percentile. ULEE outperforms all baselines and ablations. However, the hardest cutt-o--distrbuttion tasks remain challenging. In two of the three benchmarks, return shows that as the pre-training budget is increased (up to 5 billion steps), ULEE's post-adaptation performance continues to improve. DIAYN exhibits an initial gain followed by stagnation or
achieved on 60% of tasks. Figure 3e

---

Published as a conference paper at ICLR 2026
Moooms--Tivia
4Rooms-Small
6Rooms Small
4Rooms-5mall (fpe)
0.0
023
Fplcade
028
6100
035
Epteede Number
Episode
Episede Number
Episcd Number
4Rooms-Irrvial
-Rooms--mall
GRooms-Smail
4Rooms-Smail (Cyod)
Task Percentiie
Takk Parcentile
120 Percentle
Task Perentile
ARooms-Trivia
4Rooms-Smail
6Rooms TRooms-Small
4Rooms-Smaa (fari)
Steps
Steps
Steps
Steps
DO0N
Rencom
LLEE mandon-bbunded
ULEE trandom-unffemr)
LLEE Calleeroontl--unieeeen
ULEE (SED)
ULCC
Figure 3: Evaluations vv tasks: (a) performance across episodes during task- adaptation, (b) few-shot performance by task percentile, (c) few-shot performance as pre-training on unu progresses. ULEE pre-training improves over baselines and ablations across all views. The legend in (c) applies to all panels. Reported steps for ULEE variants (c) omit those from the Geal-search Policy, which adds 25%. Results are averaged over 4 seeds, with shaded regions indicating standard deviation.
4Rooms-Tn
4Rooms-
6Rooms Small
4Rooms-Small (Fama)
Episade naturn
00
04
Steps
Steps
Steps
Stops
scnatch))
finetuning
finetuning
Figure 4: Mean, 40th, and 20th percentile returns on a fixed set of evl tasks as learning on them progresses. Results are averaged over seeds, with shaded regions indicating standard deviation.
decline as training continues. Lastly, across all views, using foomts over fgnd during pre-training yields better results for ULEE, highlighting the value of f as a source of inductive bias.
4.3.3 FINE-TUNING ON FIXED TASKS
Beyond few-shot performance, we evaluate the utility of pre-training when longer adaptation bud- gets available. We sample 2 (148 environments from vil and, this fixed set, compare training from scratch to fine-tuning policies pre-trained with ULEE or DIAYN. For budgets up to billion steps, ULEE consistently outperforms the other methods in mean, 40th percentile, and 20th per-
8

---

Published as a conference paper at ICLR 2026
centile returns (Fig. While DIAYN provides slight initial advantage training from scratch in some benchmarks, this benefit is short-lived. The results in Figure underscore the difficulty of the 4Rooms- Small and 6Rooms-Sman benchmarks. Even when targeting fixed, small subset of tasks with extrinsic rewards, no method manages to solve 80% of tasks, and without ULEE, at least 40% remain unsolved.
4.3.4 FINE- TUNING WITH SUPERVISED META-LEARNIE
We next investigate whether ULEE provides strong initialization for supervised meta- reinforcement learning. Figure 5 compares meta-learning from scratch to starting from ULEE's Pre-trained Policy. For budgets of up to 5 billion steps of meta-learning on j ULEE initialization yields higher mean, 40th percentile, and 20th percentile returns f yr tasks. In the 4Rooms-Smon benchmark, using feomts during the unsupervised phase leads to better results early in fine-tuning, but by 5 billion steps, the difference from /gr is negligible.
4Rooms Trivial
4Rooms Small
6Rooms Small
4Rooms-Small (fyma)
Returns
(Frmm scratch)
ULEE finetuning
20th percentle
40th percerti
Figure 5: Mean, 40th, and 20th percentile returns val tasks meta-learning μ ش progresses. Results are averaged over 4 seeds, with shaded regions indicating standard deviation.
4.3.5 GENERALIZATION NEW ENVIRONMENT STRUCTURES
Previous sections evaluated generalization to novel goals and dynamics, but the environments' lay- (grid size and number of rooms) remained unchanged between pre-training and evaluation. To assess generalization along this additional axis, we test the performance of methods on variety of classical MiniGrid tasks after pre-training on 4Rooms-Small (Table [T]. In few-shot evaluations, ULEE (fcoums) attains the best results, achieving a non-zero return on all environments. ULEE (fgna) and DIAYN (fgria) also improve substantially over a random policy.
Table 1:
Mean return on MiniGrid tasks. ULEE and DIAYN methods were pre-trained on 4Rooms- Small and their performance evaluated after 20 adaptation episodes. Results are averaged over seeds, with standard deviations reported. The best-performing method is highlighted in bold.
task
Random
DIAYN
DIAYN (fgria)
ULEE
ULEE (fgria
BlockedUnloockichCTp
0.02 + 0.00
0.02 0.01
0.03 + 0.00
0.43 + 0.19
0.03 = 0.00
Do0r
0.05 0.00
0.00
0.08
0.27 0.35
0.08 00000.00 0.02
DoorKey-8x8
0.01
0.00
0.01
0.24
0.02 = 0.01
DoorKey-16x16
0.00
0.00
0.00
0.11
0.00 = 0.01
Empty-8x8
0.00
0.18
0.72
0.24
0.70 0.25
E6xty-16x16
0.00
0.49
0.7310.42
0.13
0.53 0.36
EmptyRandom--e 8x8
0.24
0.18
0.77
0.39
0.61 = 0.33
EmptyRandom- 16
0.00
0.34
0.39
0.17
0.47 = 0.31
FourRooms
0.00
0.07
0.13
0.04
0.16 = 0.04
LockedRoom
0.00
0.00
0.00
0.01
0.00 0.00
MemoryS8
0.000
0.09
0.01
0.11
0.01
MemoryS16
0.22
0.31
0.42
0.51
0.51 151-0.00 0.06
Unlock
0.03 1..00
0.01
0.10
0.75
0.02
UnlockPiekUp
10.00
0.00
0.00
0.68 L 0.15
0.00

---

Published as a conference paper at ICLR 2026
5 CONCLUSION
We present ULEE, an unsupervised meta-lcarning approach to induce a pre-trained policy with transferable capabilities. ULEE leverages an adversarial curriculum of self-imposed goals to train on tasks that are challenging but solvable for the policy under given adaptation budget. After pre- training on reward-fres grid worlds, we evaluate on novel, partially observable, procedurally gen- erated environments. ULEE outperforms baselines and ablations in zero-shot and few-shot settings and provides strong initialization for fine- fine-tuning on both fixed tasks and curated meta-learning dis- tributions. It demonstrates generalization to new goals, transition dynamics, and grid structures, and scales favorably with the size of the pre- pre-training and adaptation budgets. Future work could refine or extend ULEE's components, for instance by introducing hierarchical structure into the meta-learned policy to address longer-horizon tasks. complementary direction is to integrate vision-language models (VLMs) into the goal-proposal and -prrrdsppecificationn mechanisms align pre-training with human-relevant tasks and potentially enhance applicability to real-world settings.
REFERENCES
1lge Akkaya, Marcin Andrychowicz, Maciek Chociej. Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al. Solving rubik's 's cube with a robot hand. arXiv preprint arX1v:1910.07113. 2019.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience play. Advances in neural information processing systems, 30, 2017.
Adrien Baranes and Pierre- Yves Oudeyer. Active learning of inverse models with intrinsically mo- tivated goal exploration in robots. Robotics and Autonomous Systems, 61(1):49-73, 2013.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances neural information pro- cessing systems, 29, 2016.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference machine learning, pp- 41 41-48, 2009.
Tom B Brown, Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv: 1810.1881...284 2018
Andres Campero, Roberta Raileanu, Heinrich Küttler, Joshua B Tenenbaum, Tim Rocktäschel, and Edward Grefenstette. Learning with amigo: Adversarially motivated intrinsic goals. arXiv preprint arXiv:2006. 1211220006.112,,,. 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Nuttapong Chentanez, Andrew Barto, and Satinder Singh. Intrinsically motivated reinforcement learning. Advances neural information processing systems, 17, 2004.
Cédric Colas, Pierre Fournier, Mohamed Chetouani, Olivier Sigaud, and Pierre- Yves Oudeyer. Cu- rious: intrinsically motivated modular multi-goal reinforcement learning. In International con- ference on machine learning, pp. 1331-1340. PMLR, 2019.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdi- nov. Transformer-xl: Attentive language models beyond fixed-length context. arXiv preprint 2019.
Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1810.88100400 2018
10

---

Published as a conference paper at ICLR 2026
Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Pieter Abbeel. RI2: Fast reinforcement learning via slow reinforcement learning, arXiv preprint wrXiv:1611.027700 2016.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint urXiv: 0ativ:8802. /802.06070 2018.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pp. 1 1126-1 135. PMLR, 2017.
Carlos Florensa, David Held, Markus Wulfmeeer, Michael Zhang, and Picter Abbeel. Reverse cur- riculum generation for reinforcement learning. In Conference robot learning, pp. 482-495. PMLR, 2017.
Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. In International conference on machine learning, pp. 1515-1528. PMLR, 2018.
Sébastien Forestier, Rémy Portelas, Yoan Mollard, and Pierre-Y Oudeyer. Intrinsically motivated goal exploration processes with automatic curriculum learning. Journal of Machine Learning Research, 23(152):1 -11 2022.
-aan-Baptiste Gaya, Laure Soulier, and Ludovic Denoyer. Learning a subspace of policies for online adaptation in reinforcement learning. arXiv preprint arXiv:21 2021.
Ian J Goodfellow, Jean Pouget-Abadic Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv preprint 16 1.07507, 2016.
Btan-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural 2020.
information processing systems,
Abhishek Gupta, Benjamin Eysenbach, Chelsea Finn, and Sergey Levine. Unsupervised meta- learning for reinforcement learning. arXiv preprint arXiv: 180.004000 2018.
Elad Hazan, Sham Kakade, Karan Singh, and Abby Van Soest. Provably efficient maximum entropy exploration. In International Conference Machine Learning, pp. 2681-2691. PMLR, 2019.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Allan Jabri, Kyle Hsu, Abhishek Gupta, Ben Eysenbach, Sergey Levine, and Chelsea Finn. Un- supervised curricula for visual meta-reinforcemmntt learning. Advances in Neural Information Processing Systems, 32, 2019.
Max Jaderberg, Volodymyr Mnih, Wojeiech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement lcarning with unsupervised auxiliary tasks. arXiv preprint arxiv: 1.05397, 2016.
Minqi Jiang, Edward Grefenstette, and Tim Rocktäschel. Prioritized level replay. In International Conference on Machine Learning, pp. 4950. PMLR, 2021.
Aya Kayal, Eduardo Pignatelli, and Laura Toni. The impact of intrinsic rewards exploration in reinforcement learning. Neural Computing and Applications, pp. 1-35, 2025.
A.S. Klyubin, D. Polani, and C.L. Nehaniv. Empowerment: universal agent-centric measure of control. In 2005 IEEE Congress on Evolutionary Computation, volume 1, pp. 128-135 Vol.1, 2005. doi: 10. 109//1C..200.115555555 1554676.

---

Published as a conference paper at ICLR 2026
Grgur Kovač, Adrien Laversanne-Fines and Pierre- Yves Oudeyer. Grimgep: learning progress for robust goal sampling in visual deep reinforcement learning. IEEE Transactions on Cognitive and Developmental Systems, 15(3):1396-1477, 2022.
Alex Krizhevsky, Iya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. Advances in neural information processing systems, 25, 2012.
Tejas D Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. Advances in neural information processing systems, 29, 2016.
Lisa Lec, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine, and Ruslan Salakhutdi- nov. Efficient exploration via state marginal matching. arXiv preprint arXiv:1906.00. 2019.
Hao Liu and Pieter Abbeel. Behavior from the void: Unsupervised active pre-training. Advances in Neural Information Processing Systems, 34: 1 8849-18477. 2021.
Tambet Matiisen, Avital Oliver, Taco Cohen, and John Schulman. Teacher-student curriculum learn- ing. IEEE transactions on neural networks and learning systems, (19))3732-37444 2019.
Bhairav Mehta, Manfred Diaz, Florian Golemo, Christopher J Pal, and Liam Paull. Active domain randomization. Conference on Robot Learning, pp. 1162-1176. PMLR, 2020.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Picter Abbeel. A simple neural attentive meta- learner. arXiv preprint arXiv:1707.0314), 2017.
Shakir Mohamed and Danilo Jimenez Rezende. Variational information maximisation for intrinsi- cally motivated reinforcement learning. Advances in neural information processing systems, 28, 2015.
Mirco Mutti, Mattia Mancassola, and Marcello Restelli. Unsupervised reinforcement learning in multiple environments. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 36, pp. 7850-7858, 2022.
Ashvin Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. Advances in neural information processing systems, 31, 2018
Alexander Nikulin, Vladislav Kurenkov, Ilya Zisman, Artem Agarkov, Viacheslav Sinii, and Sergey Kolesnikov. Xland-minigrid: Scalable meta-reinforceement learning environments in jax. Ad- vances in Neural Information Processing Systems, 37:43809-43835. 2024.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:1807.037448 2018
OpenAl OpenAI, Matthias Plappert, Raul Sampedro, Tao Xu, 1lge Akkaya, Vineet Kosaraju, Peter Welinder, Ruben D'Sa, Arthur Petron, Henrique P d O Pinto, et al. Asymmetric self-play for automatic goal discovery in robotic manipulation. arXiv preprint arXiv:2101.04882. 2021.
Pierre-Yve Oudeyer, Frdric Kaplan, and Verena V Hafner. Intrinsic motivation systems for au- tonomous mental development. IEEE transactions evolutionary computation, 11(2):26--866, 2007.
Emilio Parisotto, Francis Song, Jack Rae, Razvan Pascanu, Caglar Gulcehre, Siddhant Jayakumar, Max Jaderberg, Raphael Lopez Kaufman, Aidan Clark, Seb Noury, al. Stabilizing transformers for reinforcement learning. In International conference on machine learning, pp. 7487-7498.
PMLR, 2020.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference machine learning, pp. 2778-2787. PMLR, 2017.
Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supervised exploration via disagreement. In International conference machine learning, pp. 5062-5071, PMLR, 2019.
12

---

Published as a conference paper at ICLR 2026
Vitchyr H Pong. Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine. Skew- fit: State-covering self--upervised reinforcement learning. arXiv preprint 1rXiv:1903.036%: 2019.
Sebastien Racaniere, Andrew K Lampinen, Adam Santoro, David P Reichert, Vlad Firoiu, and Timothy P Lillicrap. Automated curricula through setter-solver interactions. arXiv preprint urXiv:/900) 12892, 2019.
Sai Rajeswar, Pietro Mazzaglia, Tim Verbelen, Alexandre Piché, Bart Dhoedt, Aaron Courville, and Alexandre Lacoste. Mastering the unsupervised reinforcement learning benchmark from pixels. In International Conference on Machine Learning, pp. 28598 28617. PMLR, 2023.
Jürgen Schmidhuber: Curious model-building control systems. Proc. international joint confer- ence on neural networks, pp. 1468-1463, 1991.
Jürgen Schmidhuber. Formal theory of creativity, fun, and intrinsic motivation (1990-2010). IEEE fransactions on autonomous mental development, 2(3):300-277, 2010.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprini urXiv:1707.063444 2017.
Max Schwarzer, Nitarshan Rajkumar, Michael Noukhovitch, Ankesh Anand, Laurent Charlin, R De- von Hjelm, Philip Bachman, and Aaron C Courville. Pretraining representations for data-efficient reinforcement learning. Advances in Neural Information Processing Systems, 11111266616999 2021.
Ramanan Sekar, Oleh Rybkin, Kostas Danilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning explore via self-supervised world models. In International conference on machine learning, pp. 8583-8592. PMLR, 2020.
Younggyo Seo, Kimin Lee, Stephen L James, and Pieter Abbeel. Reinforcement learning with action-free pre-training from videos. In International Conference on Machine Learning, pp- 19561-19579. PMLR, 2022.
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-aware unsupervised discovery of skills. arXiv preprint arXiv:/907, 0e00.00077..777 2019.
Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from reinforcement learning. In International conference on machine learning, pp. 9870-9879. PMLR, 2021.
Sainbayar Sukhbaatar, Zeming Lin, Ilya Kostrikov, Gabriel Synnaeve, Arthur Szlam, and Rob Fergus. Intrinsic motivation and automatic curricula via asymmetric self-play. arXiv preprint arXiv: 2017.
Sainbayar Sukhbaatar, Emily Denton, Arthur Szlam, and Rob Fergus. Learning goal embeddings via self-play for hierarchical reinforcement learning. arXiv preprint 18X.v0881..0008. 2018.
Adaptive Agent Team, Jakob Bauer, Kate Baumli, Satinder Baveja, Feryal Behbahani, Avishkar Bhoopchand, Nathalic Bradley- Bradiey-Schmieg, Michael Chang, Natalie Clay, Adrian Collister, et al. Human-timescale adaptation in an open-ended task space. arXiv preprint arXiv:2301.07608, 2023.
Open Ended Learning Team, Adam Stooke, Anuj Mahajan, Catarina Barros, Charlie Deck, Jakob Bauer, Jakub Sygnowski, Maja Trebaez, Max Jaderberg, Michael Mathieu, et al. Open-ended learning leads to generally capable agents. arXiv preprint arXiv:2107. 2021.
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hicrarchical reinforcement learning. In International conference machine learning, pp. 3540-3549, PMLR, 2017.
Jane X Wang. Zeb Kurth-Nelson, Dhruva Tirumala. Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. arXiv preprint arXiv: L05663, 2016.
13

---

Published as a conference paper at ICLR 2026
Rui Wang. Joel Lehman, Jeff Clune, and Kenneth O Stanley. Paired open-ended trailblazer (poet): Endlessly generating increasingly complex and diverse learning environments and their solutions. arXiv preprint arXhv:1901.0155. 2019.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Reinforcement learning with pro- totypical representations. In International Conference on Machine Learning, pp. 11920-11931. PMLR, 2021.
Amy Zhang, Rowan McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. arXiv preprint arXiv:2006. 10742, 2020a.
Yunzhi Zhang, Pieter Abbeel, and Lerrel Pinto. Automatic curriculum learning through value dis- agreement. Advances in Neural Information Processing Systems, 15:768876657 2020b.
14

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 PSEUDOCODE
Algorithm 1: ULEE. Unsupervised Learning of Efficient Exploration
Initialize: Pre-trained Policy Goal-search Policy Tga, Difficulty Predictor dg. and goal buffer Bg
while training do Sample batch of environments {Mi)er ~ pmmup
>> Search for candidate goals
foreach M, do Run Risi obtain candidate goals GCM, (Sec. 3.2.1 and environment information εM; end
b. Sample goals of intermodlate difficulty foreach M, do
Compute predicted difficulties de (g. EM. for all E GCM,
Filter to target difficulty range: S + { E GCM, LB s EM, UB (Eg 4 if S * then
10
Sample gaz, Unif(S)
13
Sample gM, Uni((CCM,)
end 14
15
end 16
D Train Pre-trained Policy
17
goaats 0 while lifetimes not finished do
19
Depiate t foreach M, do
20
21
advance aT' lifetime on M, under goal gM; append transitions to Dupdate
22
check for success {3t (0....,T-- f+++1) = =g in completed episodes and update
guressss
23
end
24
update Dupdne maximize meta-RL. objective J ((r) (Eq. end
25
26
Compute empirical difficulty estimates d(gM.) from Dools according to Eq.
27
Push ((MM, EM,, ((gg)))) into buffer Bg (replace the N oldest tuples if full) b Train fi culty Predf ctor
28
Update de on data from By minimize (or(#) (Eq.
Train Goal-search Policy
for num gs updates
30
Dustme t foreach M, do run Tgai collect trajectories into Dupsttes setting 15th + dg (f(st), (Eq.
31
32
end
update Tg2 on Dupante to maximize (Tgx)) = BMS.. Ex-0 y/r2""
34
end
36
end
A.2 EXPERIMENTAL DETAILS
A.2.1 ENVIRONMENTS AND GOAL MAPPING
Our benchmarks are derived from Minidrri This section extends the information given in Section 4.1 for full details, see Nikulin et al. (2024). Each environment offers six discrete prim- itive actions. There are 70 unique objects that can appear in the environments. Objects are iden- tified by type (c.g., key, ball, pyramid) and color. The (extrinsic) task space spans 13 goal types (e.g., Agent HoldGoal (a) whether agent holds a, TileOnPosition iononoaal (a, x, y) whether (x, y) position; Ti LeNeerrreeeee
(a, b) whether is tile to the left
15

---

Published as a conference paper at ICLR 2026
of a). The rule space spans 10 rule types (e.g., AgentHoldRule(aa + If agent holds replaces it with c; NiieeeaRRiightuul
(a, b) - c - If is one tile to the right of a, re- places one with and removes the other). Each grid cell is associated with corresponding ID that identifies its kind (e.g., wall, red pyramid, empty space). fcounts maps states to a vector that records the number of occurrences of each ID. Thus, an intrinsic goal under fcounss specifies how many instances of each unique object should appear on the grid. For both intrinsic and extrinsic tasks, reward is given only upon success. If the agent succeeds in time step receives a reward of 1.0 - 0.9 ff/maxx.uu h//mmmxxtstepp steps), where max_steps is the maximum number of steps the agent can take before the episode is considered a failure. We use early termination upon success. The agent's x observations are not occluded by walls.
Our configuration for benchmarks differs from -ini--isss default behavior in two ways: (i) for 13x13 13 grids, we set the episode horizon (max_steps) to 256 rather than the default 509; (ii) we sample the initial state 80 ~ Pas once at the beginning of each lifetime and keep it fixed across all episodes within that lifetime. The default configuration resamples so at the start of every episode.
Our evaluations on MiniGrid tasks Section 4.3.5 use the default configurations of XLand- MiniGrid's 's pre-built MiniGrid environments.
A.2.2 HARDWARE AND SEEDS
Each algorithm is trained with 4 random seeds. For every training seed, we draw two additional independent seeds: one to construct the benchmark split between fu J. / dumurr and fa and one to run the fine-tuning experiments (Sections 4.3.3 and 4.3.4). Hyperparameters for each method are held fixed across benchmarks and seeds. All experiments are executed on a single NVIDIA RTX 4090 GPU.
A.2.3 ALGORITHM IMPLEMENTATIONS AND HYPERPARAMETERS
Architectures
Policies: Policies use a Gated Transformer-XL. architecture with blocks, hidden size 192, and 4 attention heads (head dimension 48). Gates use a bias of 2. The network attends to cached activations of up to 128 steps into the past, yielding an effective memory horizon of 256 (memory Jen x num. blocks). During training, gradients are propagated for up to 64 steps. Two separate MLP heads, each with a size of (256, 256), are used as the Actor and Critie. Each full policy network has x 1.7M parameters. ReLU activations are used throughout. For meta-learned policies (ULEE's Pre-trained Policy and the RL baseline), the per-step input is {or, de, (t-1, where de e (0. 1} indicates the beginning of a new episode (dummy d:-1 and rt-1 are used at the start of each lifetime). The meta-learner state is reset only upon encountering a new environment (at the start of a lifetime).
Difficulty Predictor and discriminator: ULEE's Difficulty Predictor and DIAYN's discriminator MIPs of size (256, 256) that operate on learned encodings of their inputs (see Input encodings below). Both have access to full state information during pre-training. DIAYN's discriminator predicts latent skills from encodings of visited states (s)) where f coincides with ULEE's goal-mapping function and serves as source of inductive bias. The Difficulty Predictor's inputs are (gg) (with f(yy = g) and εM- = [gna(So).. where su ~ PM (see 3.2.3 fgna removes the agent's 'ss i orientation and pocket information from the state data.
RND's target and predictor are MLPs with two hidden layers of size 256. They operate on encodings of observations. For benchmarks using Scomss: each observation is mapped to a vector that records the number of occurrences of each cell type; the MI.P is applied directly to this vector and produces 256-dimensional output. For benchmarks using fgnid obser- vations are first encoded with a small CNN (see Input encodings below); the MLP is then applied to this encoding with output dimensionality of 64. RND is used as baseline in Section 4.3.3 where agents are trained to solve 2 248 extrinsic tasks. Since a single policy is trained across all tasks, also use single shared sarget-predictts pair.
RND target and predictor:
Input encodings: To process the symbolic grid observations and states, each cell's symbol is em- bedded via two learned embedding tables (shape and color). Then, a convolutional neural network
16

---

Published as a conference paper at ICLR 2026
is applied, with the final output flattened. Additional inputs, when present, concatenated to the flattened vector. Actions and skills have their own embedding tables. All learned embedding tables produce 16-dimensional vectors. For the Difficulty Predictor, the two input states concatenated across the channel dimension before applying the CNN.
DIAYN skills: DIAYN is trained with 10 skills (following Gaya al. (202)) and Kayal et al. 2025 ) We add a fixed per-skill bias vector to the policy logits; these vectors are orthogonally
initialized and scaled by factor of We find this addition important for learning distinguishable skills.
Training
The pre-training phase for ULEE and DIAYN spans billion environment steps. We collect perience in parallel from batches of 048 environments and, after 5 120 steps per environment, resample a new batch from 14 uuuuu ULEE updates its policy every 256 steps per environment, while DIAYN updates every 512 steps. Policy updates for all algorithms use PPO with the Adam optimizer (hyperparameters in Table
For ULEE, an additional 25% of steps (on top of the counts reported above) are allocated to the Goal- search Policy Tgs. For each newly sampled environment M, aya first executes two 256-step episodes to collect candidate goals. It adds one candidate to GCM for every 15 states visited (Section 3.2.1 After the Pre-trained Policy and Difficulty Predictor are updated with data from M, ags executes three additional 256-step episodes during which it is trained to reach difficult goals (Section 3.2.
Task difficulty (Equation [ is estimated empirically as the Pre-trained Policy's success rate over its last five episodes that task. For bounded sampling, we use a difficulty lower bound LB of 0.1 and upper bound UB of 0.9. The goal buffer B, used to train the Difficulty Predictor stores goals (together with their empirical difficulty estimates) from the last five batches of 2 048 intrinsic goals (Byyl = 5 x 2048). After the Pre-trained Policy æ finishes training on a new batch and the corresponding goals added to By, we train the Difficulty Predictor for 2 epochs over the entire buffer using minibatch size of 256 and a learning rate of x 10-1 (Sec. 3.2.3).
DIAYN's s discriminator updated every 512 steps per environment and is trained for a single epoch over all collected transitions with a learning rate of 3 x 10-4
For RND, we compute advantages for extrinsic and intrinsic rewards using the same 7 and values as in Table These are combined using extrinsic-reward coefficient of 1 and an intrinsic-reward coefficient of 0.1 before policy updates. The predictor network updated every 256 steps per environment and trained for single epoch over all collected transitions. Its learning rate is 1 x 10 for benchmarks that use foomns and x 10 for benchmarks that use fgrid-
A.2.4 EXPERIMENTAL PROCEDURES
Exploration: For each seed in Figure 2 results are obtained from evaluation on set of 16:88 vironments sampled from y _ Methods interact for 20 sequential episodes with each environment. goal is considered reached episode j if it was achieved in any episode < j. For DIAYN, we run two episodes with each skill, randomizing their order. Each skill is used at least once before any repeat.
Fast adaptation: For each seed in Figures 3a and 36 results are obtained from evaluation on set of 16 384 environments sampled from p Methods interact for 30 sequential episodes with each environment. For DIAYN, we run two episodes conditioning with the best-performing skill. Figure 3a shows results across all 30 episodes, whereas Figure (3b) reports aggregate performance over the last 10. Figure 3e tracks performance pre-training on f " at each evaluation point, we sample a fresh set of 1 024 environments from the last 5 episodes of 25-episode interactions.
each skill, followed by ten episodes
u throughout
rval and measure the mean performance
Fine-tuning on fixed tasks: For Section 4.3.3 each seed is fine-tuned on a fixed set of 2 248 environments sampled from -ا Figure 4 tracks performance this fixed set throughout training, with performance measured as the mean over the last 10 episodes of 30-episode interactions. For DIAYN, before fine-tuning we run 10 episodes of each skill per environment. In each environment,
17

---

Published as a conference paper at ICLR 2026
Table 2: Hyperparameters for policy updates. All policies are updated using PPO with the Adam
optimizer. Entropy coefficients are listed separately as they vary across different methods.
Hyperparameter
Value
Learning rate
2 10
Adam
1 x 10-5
Discount factor 7
0.99
Update epochs
Number of minibatches
Clipping
0.2
GAE X
0.95
Value loss coefficient
0.5
Max gradient norm
0.5
Entropy coefficient:
Goal-search Policy
0.01
Pre-trained Policy
0,005
DIAYN
0.03
DIAYN when fine-tuning
0,01
PPO (from scratch)
0.005
RND
0.005
RL2 (from scratch)
0.005
we select the best- pest-performing skill and keep it fixed during fine-tuning. These preparatory steps are excluded from the figure.
Fine-tuning with supervised meta-learning: Figure reports performance on ا throughout meta-learning y train At each evaluation point, sample fresh set of 1 024 environments from ov and measure the mean performance over the last 5 episodes of 25-episode interactions.
results for each seed each MiniGrid environment averaged over 2 048 runs; each run consists of 30 sequential episodes, and we report the performance over the last 10 episodes.
MiniGrid: In Section 4.3.5
A.2.5 HYPERPARAMETER SELECTION AND STABILITY
As is common in meta-learning and unsupervised pre-training, conducting an exhaustive hyperpa- rameter search was infeasible due the length and computational cost of each run. Our hyper- parameter studies therefore aimed to: (1) assess whether ULEE requires careful tuning in order to learn, and (2) identify a reasonable, stable configuration rather than optimize for the best possible values. Given this constrained scope, we do not present quantitative comparisons or prescriptive rec- ommendations on the choice of hyperparameters. Instead, this section documents the explorations that were performed and the observations that informed our final choices.
Our investigations found that ULEE does not require careful tuning to learn effectively. Across all configurations tried, observed no signs of collapse, and the results reported in the main paper were obtained after limited tuning. All tuning and atability-analliis experiments were run on seeds that were different from those used the final experimental results reported in Section
When selecting hyperparameters for each method, we focused mainly on the following criteria:
We monitored performance on tasks sampled from tran throughout unsupervised pre-training reward-free environments from unsup We additionally inspected skill dis- tinguishability (via per-skill heatmaps on empty environments and via the discriminator loss) and the entropy of each skill. When searching for fine-tuning hyperparameters (Sec- tion 4.3.3), 3), we evaluated performance on fixed extrinsic-reward tasks as fine-tuning pro- gressed.
DIAYN:
RL2 (from scratch): We monitored performance on tasks sampled from traiin during super- vised meta-learning tasks from n trin
18

---

Published as a conference paper at ICLR 2026
PPO (from scratch): We monitored performance set of 048 tasks sampled from n while training on them.
RND: We monitored performance on a set of 2 048 tasks sampled from yuri while training them.
We monitored performance on tasks sampled from μ tnain throughout unsupervised pre-training on reward-free environments from μ pu Importantly, during the selection process, we did not inspect any of the evaluation metries presented in Figures 2 3 and In particular, we did not test or select hyperparameters based on downstream funentuning performance. We also did not tune any hyperparameters specific to fine-tuning policies pre-trained with ULEE; for those experiments we reused the values employed during pre- training.
ULEE:
Below we summarize which hyperparameters were varied and which were kept fixed. For those that were varied, our tuning process was limited in several ways: at most three hyperparameters were varied at a time, each combination was tested with only 1-2 seeds, experiments were run exclusively on the 4Rooms-Sman benchmark, and runs were shorter than those used in the final results.
Network architectures: We did not perform any hyperparameter tuning on network architectures.
Number of steps and parallel environments: We did not tune the number of parallel environments, the split of (goal, rules, initial objects} } tuples between μ ir and ا the maximum number of steps per episode, the number of steps per environment before resampling, or the total number of training steps.
Policy updates: We tested variations of the entropy coefficient, discount factor 7, and learning rate. All other hyperparameters were fixed advance. The values explored were:
DIAYN: entropy coefficient (0.003, 0.005, 0.01, 0.02, 0.03, 0.13, learning rate {3 x 10-5, 10-",3 .2 10-4, 3 x 10 4,5 10 10-4).
DIAYN (fine-tuning): entropy coefficient (0.005, 0.01, 0.03). learning rate (2 x 10-1).
RL2: entropy coefficient {0.001, 0.003, 0.005, 0.007, 0.01, 0.03, 0.05). learning rate {1 x 10 10-4,2 x 10 4.3 10
1004,0 4.5 x 10 10-4, x 10-33, discount factor {0.99, 0.995, 0.999).
PPO: entropy coefficient (0.003, 0.005, 0.01), learning rate {3 x 10-5, 10-4, 10-1,3 x 10 1.0 5 x 10-1).
RND: entropy coefficient {0.003, 0.005, 0.01}, learning rate {3 x 10-5, 5, x 10-4, 5,5 x 10-4).
ULEE Pre-trained Policy: entropy coefficient f0.003, 0.005, 0.01, 0.03), learning rate (3x 10-5, x 10-4, 4.2 x 10-4, x 10-4, 5 x 10-4}.
.
ULEE Goal-search Policy: entropy coefficient {0.003, 0.005, 0.01, 0.033, learning rate {2 x 10-4).
We found that learning rates between x 10-1 and 3 x 10 worked well across all methods. Using discount factors of 0.995 and 0.999 in RL strongly hindered performance.
-ppyN-specific typerparameters: We varied the dimensionality of skill embeddings {16, 128), the discriminator learning rate {3 10-3, 1 10-4, 4,3 x 10-*), and the number of steps per en- vironment per update (256, 512, 1024]. We also experimented with adding a fixed per-skill bias vector to the policy logits (Appendix A.2.3]. This proved important for obtaining distinguishable skills in setups where the agent's and objects' initial positions are randomized. We explored two ini- tializations for the bias vectors (orthogonal, random directions in the unit sphere) and several scale factors {1,5, 8, 10, 15, 20}. Both initialization schemes were effective and scale factors between 5 and 10 performed best, though smaller and larger values still helped. Additionally, we tested dis- abling the policy updates for the first 10 batches of environments, which did not help, and varying the initialization of the discriminator final layer, which improved stability.
RND-specifie hy byperparameters: We varied the dimensionality of the embedding produced by the target network (32, 64, 256). the intrinsic-reward coefficient (0.02, 0.05, 0.1, 0.3, 0.5, 1, 2}. and the predictor-network learning rate (1 10-5 5 x 10-5, x 10-1,3 10-4).
19

---

Published as a conference paper at ICLR 2026
ULEE-specific hyperparameters:
Difficulty Predictor.
We varied its learning rate (3 x 10 10 10
4,5x 10-4} and the 10~+.3x 10-4,5
number of batches of intrinsic goals stored in the training buffer {1,3,5). ULEE learned under all configurations. Differences were generally modest; a learning rate of x 10 produced the largest performance drop.
Goal-search Policy and entropies. As noted earlier, we varied the entropy coefficient of the Goal- search Policy and Pre-trained Policy updates. For the Goal-search Policy, we observed little effect on performance. For the Pre-trained Policy, the different values influenced the shape of pre-training curves but did not reveal clear best choice. We also varied the number of Goal-search Policy updates per batch of environments (num-gs.updatee) in Algorithm D {3, 6} and observed similar results.
Goal sampling strategy. We tested three sampling schemes: (1) uniform sampling from goals within difficulty bounds (Section 3.2.2). with bounds {(0.1, 0.9), (0.3, 0.7) (2) sampling a target diffi- culty for each environment M from a Gaussian with mean in {0.4, 0.6, 0.8} and standard deviation in {0.2, 0.4] and picking the goal from GCm whose predicted difficulty is closer to that value; (3) assigning weights to candidate goals in GCMr (based on their estimated difficulty) using a Gaussian density function with mean (0.5, 0.6) and standard deviation in {0.2, 0.43 and sampling in ac- cordance with these weights. ULEE learned under all schemes and hyperparameters tried. The best performance was obtained with scheme and 3 using a Gaussian mean of 0.6 and standard deviation of 0.2, and with scheme I using a lower bound LB of 0.1 and upper bound U B of 0.9.
Steps per update. We varied the number of steps per environment per update {256, 512} and ob- served faster learning with the smaller value.
No search was performed over hyperparameters not mentioned this section. As noted earlier, our hyperparameter search was limited in scope: the goal was identify reasonable values and evaluate whether ULEE exhibited failure modes or strong sensitivity to particular hyperparameters. Thus, the final set of hyperparameters should not be interpreted as either optimal or deeply tuned. Ablations used the same hyperparameters selected for ULEE.
A.3 LEARNING PROGRESS
A complementary line of work in automatic curriculum generation employs learning progress (LP), rather than difficulty, the guiding metric for goal selection (Baranes & Oudeyer 2013 Co- las et al. 2019; Forestier et al. 2022, Kovač et al. (022). Learning progress is estimated by tracking agent's 's performance exhibiting the largest improvements (or deteriorations) are prioritized. Typically, LP is com- puted from changes in the policy's immediate performance training proceeds, without allow- ing for any task-specific adaptation. Analogous to difficulty-based methods, I.P can be adapted so as measure changes in post-adaptation performance. Let 7 denote training time and क., the policy after training for т. If d(g; πI, .M) denotes post-adaptation performance metric on task (g, M),, e.g., an empirical estimate of Eq. _ then possible LP-style goal desirability score is P. (g, M; = |d(g: ,,, .) M)-d(g; -r-Ar, M)], where AT > 0 is the measurement window. In- vestigating post- cost-adaptation learning progress and its interplay with post-adaptation difficulty-basee curricula and unsupervised goal generation is promising direction for future work.
tasks over time and quantifying its rate of change. Tasks
A.4 USE OF LARGE LANGUAGE MODELS
Large language models (LLMs) were used solely as coding assistants and for polishing the paper's writing. They were not used generate, develop, or analyze the underlying scientific content or technical contributions.
20


Reviews:
[{"review_id": "6fyy4hDqtx", "reviewer": "Reviewer_SaiF", "summary": "The paper proposes ULEE (Unsupervised Learning of Efficient Exploration), an unsupervised meta-RL pretraining framework that trains an in-context learner via an adversarial goal-generation strategy. Goal difficulty is defined by the post-adaptation success rate, yielding an intermediate-difficulty curriculum.", "questions": "* Does the learned difficulty correlate with intuitive task hardness? A qualitative or heuristic-based comparison between high and low-difficulty goals would be helpful.\n* Does the goal-search policy reliably propose high-difficulty goals? What are the difficulty distributions of goals sampled by the goal-search policy (and a random policy)?\n* What other environment information $\\xi_M$ could be used, especially for environment domains other than grid-worlds?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-motivated. The curriculum is based on performance after in-context adaptation, not immediate performance, which aligns with the intended meta-RL setting. Empirically, ULEE improves exploration, shows faster few-shot adaptation, and provides stronger initializations for finetuning.", "weaknesses": "* The empirical impact of defining difficulty via post-adaptation performance, rather than immediate performance, remains unclear without an ablation. A direct ablation (e.g., a sensitivity study over $K$) would strengthen the paper.\n* The baselines do not include recent meta RL and unsupervised RL methods.\n* Experimental scope is limited to grid-world domains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "awxbvOqjJI", "reviewer": "Reviewer_HPvB", "summary": "The paper introduces Unsupervised Learning of Efficient Exploration (ULEE) as an unsupervised meta-learning approach that pre-trains a policy capable of rapid adaptation to new tasks. It achieves this by generating challenging-but-solvable goals based on the policy's estimated post-adaptation performance. The resulting policy is an in-context learner that adapts to new goals, dynamics and map structures using only its interaction history (observations, actions, rewards), requiring no explicit goal input. On the XLand-Minigrid benchmark, ULEE outperformed baselines in terms of fast adaptation, and provided a better initialization for both extended fine-tuning and supervised meta-learning. It also demonstrated generalization to novel environment structures. \n\n**Recommendation:**\\\nThis paper falls outside my area of expertise, but appears to have a well motivated and interesting problem setting and strong empirical results. However, I have some questions about the methodology regarding seeding and validation/test sets. Therefore, in its current state, I will recommend to reject. However, I will be open to change my score if my questions are answered satisfactorily.", "questions": "- Are there better ways to isolate and analyse the exploration capabilities of the Pre-trained Policy? For example, subtract all evaluation goals reached in a single episode and only include the ones reached with more episodes? Or visualize the exploratory behaviour of the policies in some way? \n- There appears to be no explicit mention of a test versus validation set. Are the final results evaluated on an independent testing set of environments (that has not been used for validation, hyperparameter tuning, or generally for algorithm design)? Similarly, were the seeds used for the final evaluation (testing) different from the ones used for validation, tuning and design? \n- I could not find mention of the hyperparameter tuning approach for your method and the baselines. How is it ensured that your approach did not accidentally benefit from an advantageous hyperparameter combination or tuning budget? Did you use separate seeds for tuning and final evaluation? \n\n\n**Things to improve that did not impact decision:**\n- Some of the related work mentioned in Section 2 is missing an explicit comparison with the paper's approach.\n- Line 227: The variable $n$ is introduced there but not defined or mentioned in the text.\n- Line 294: I don't quite understand how $f_{counts}$ works. \n- Figure 3c: It is unclear to me what exactly Figure 3c is showing. Is it showing post-adaptation return on the evaluation set, evaluated at different points during pre-training?\n- Table 1: The bold highlight is very difficult to differentiate from the regular numbers.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Although this is not my area of expertise, the paper's motivation and positioning within existing literature appears strong. \n- The problem of pre-training for adaptation is very interesting.\n- The empirical results are very strong.", "weaknesses": "- Section 4.3.1 does not sufficiently answer Q1. In this section the fraction of evaluation goals reached as a function of the number of evaluation episodes is shown in Figure 2. In my opinion this does not isolate exploration as the cause of evaluation goals reached, nor does it answer _\"what exploration capabilities\"_ the policy exhibits. For example, an increase in evaluation goals reached can also be due to zero-shot generalization, rather than improved exploration/adaptation. \n- Some parts of the experimental methodology is unclear. In particular when it comes to hyperparameter tuning and validation vs test split. \n- Certain details in the main text could be explained better.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NiMEfSdFJs", "reviewer": "Reviewer_ompC", "summary": "This paper introduces ULEE (Unsupervised Learning of Efficient Exploration), an unsupervised meta-learning method designed to pre-train adaptive reinforcement learning policies. The core challenge it addresses is how to effectively generate and select self-imposed goals for an agent to learn from, particularly for broad distributions of downstream tasks. ULEE's main contribution is an automatic curriculum learning strategy guided by a novel post-adaptation task-difficulty metric. This approach optimizes for an agent's performance after a period of adaptation, rather than its immediate performance. The method combines an in-context learner with an adversarial goal-generation system that finds challenging yet achievable goals, effectively maintaining training at the \"frontier of the agent's capabilities\". On XLand-MiniGrid benchmarks, ULEE-pre-trained policies demonstrate improved exploration, adaptation, and generalization to novel tasks , resulting in better zero-shot and few-shot performance and providing a strong initialization for longer fine-tuning processes.", "questions": "1. There are many hyper-parameters in ULEE, learning rates, network architectures, buffer sizes, sampling bounds LB/UB, number of goal-search episodes etc. How critical are the specific hyper-parameter values?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The post-adaptation task-difficulty metric, for me, is novel. It significantly departs from prior works in automatic curricula, which typically evaluate goal difficulty based on the agent's immediate performance. By defining difficulty as the agent's expected success rate after an adaptation budget, the method directly optimizes for the agent's capacity to learn rather than just its current knowledge. This aligns the pre-training objective more closely.\n\n2. The paper originally combines three key components into a single system, ULEE. While concepts like meta-learning, adversarial goal generation, and in-context learners exist, ULEE integrates them synergically. It uses an adversarial \"Goal-search Policy\" to propose hard goals, a \"Difficulty Predictor\" network to estimate their post-adaptation difficulty, and an in-context learner (the \"Pre-trained Policy\") to meta-learn on a curriculum of goals selected for being at the \"frontier of the agent's capabilities\".\n\n3. The authors evaluate the pre-trained policy across a wide spectrum of downstream scenarios.", "weaknesses": "1. The primary weakness of ULEE is its high methodological complexity. The system is not a single algorithm but a complex interplay of four distinct, learning-based components: the Pre-trained Policy ($\\pi$), the Goal-search Policy ($\\pi_{g, s}$), as well as the Difficulty Predictor. Are there practical bottlenecks (e.g., memory, wall-clock time) for this method?\n\n2. The overall system's success depends on these components learning in lockstep. Can this co-adaptive process be brittle? E.g., will a relatively poor Difficulty Predictor lead to a collapse? Adding a robustness analysis to each design ingredient will greatly strengthen the paper.\n\n3. The pre-training and evaluation tasks (4Rooms-Trivial, 4Rooms-Small, 6Rooms-Small) are all drawn from the same \"family\" of XLand-MiniGrid rules, which may not fully cover the claim “generalization to new goals, transition dynamics, and grid structures”.\n\n\nMinors:\n1. I recommend a figure to show the overall framework of the proposed, so that it can be more detailed and accessible.\n\n2. There are some grammatical and typographical errors, for example. In line 65, “more tasks become too…”,", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gBpfHbSZgI", "reviewer": "Reviewer_QuSP", "summary": "The paper tackles meta‑reinforcement learning by combining in‑context RL with a curriculum for self‑generated goals. An ICRL agent is pre‑trained to reach goals proposed by a curriculum pipeline. This pipeline first optimizes a goal‑search policy to estimate the agent’s current capability, then a selector chooses goals within calibrated success bounds. On XLand‑MiniGrid benchmarks, the proposed method, ULEE, improves exploration and adaptation, generalizing to novel objectives, dynamics, and map structures, and achieves better zero‑shot and few‑shot performance.", "questions": "- Why not integrate the selector’s criterion directly into the goal-search reward—for example, use a $|x−0.5|$-shaped reward over success probability or estimated difficulty to penalize goals that are too hard or too easy? What are the trade-offs?\n- Could classical exploration designs like UCB (or Thompson sampling) be used to guide goal sampling—treating difficulty bands or goal families as arms—and does this improve sample efficiency or stability compared to the current selector?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- *Intuitive formulation:* ULEE frames meta-RL as in-context adaptation guided by self-imposed goals, yielding a conceptually clean pipeline (goal search → capability estimation → goal selection). This makes the approach easy to reason about and implement.\n- *Solid empirical evidence:* On XLand–MiniGrid, ULEE consistently improves zero-shot and few-shot performance over strong baselines (learning from scratch, DIAYN, alternative curricula), with generalization to new objectives, dynamics, and map structures.\n-  *High-quality writing and presentation:* The paper is clearly written, with strong contextualization relative to prior work and a well-structured narrative that explains the curriculum components (goal search, capability estimation, selector) and their interplay, which aids reproducibility and understanding.", "weaknesses": "- Despite the intuitive design, the pipeline introduces multiple components and hyperparameters (e.g., difficulty/success bounds), raising concerns about training stability and sensitivity.\n- With adversarial goal-generation, does the search ever propose “degenerate” goals (e.g., trivially satisfiable or reward-hacking)? Although the goal selector may mitigate this issue, randomness in sampling can lead to training collapse.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)