No Regrets: Investigating and Improving Regret Approximations for Curriculum Discovery
Authors: Alexander Rutherford
Michael Beukman
Timon Willi
Bruno Lacerda
Nick Hawes
Jakob Nicolaus Foerster
Venue: NeurIPS 2024 poster (2024)
Keywords: MARL
UED
Robotics
Primary area: reinforcement_learning
TL;DR: An improved score function for unsupervised environment design in binary outcome settings, which we use to train agents for real-world tasks, and an improved adversarial evaluation protocol that assesses policy robustness.

Abstract: What data or environments to use for training to improve downstream performance is a longstanding and very topical question in reinforcement learning. 
In particular, Unsupervised Environment Design (UED) methods have gained recent attention as their adaptive curricula promise to enable agents to be robust to in- and out-of-distribution tasks.
This work investigates how existing UED methods select training environments, focusing on task prioritisation metrics.
Surprisingly, despite methods aiming to maximise regret in theory, the practical approximations do not correlate with regret but with success rate.
As a result, a significant portion of an agent's experience comes from environments it has already mastered, offering little to no contribution toward enhancing its abilities. Put differently, current methods fail to predict intuitive measures of *learnability*. Specifically, they are unable to consistently identify those scenarios that the agent can sometimes solve, but not always.
Based on our analysis, we develop a method that directly trains on scenarios with high learnability. This simple and intuitive approach outperforms existing UED methods in several binary-outcome environments, including the standard domain of Minigrid and a novel setting closely inspired by a real-world robotics problem. 
We further introduce a new adversarial evaluation procedure for directly measuring robustness, closely mirroring the conditional value at risk (CVaR).
We open-source all our code and present visualisations of final policies here: https://github.com/amacrutherford/sampling-for-learnability.

No Regrets: Investigating and Improving Regret Approximations for Curriculum Discovery
Alex Rutherford'
Michael Beukman"
Timon Willi
Bruno Lacerda
Nick Hawes
Jakob Foerster
University of Oxford
Abstract
What data or environments to use for training to improve downstream performance is longstanding and very topical question in reinforcement learning. In particular, Unsupervised Environment Design (UED) methods have gained recent attention their adaptive eurricula promise to enable agents to be robust in- and out- of-distribution tasks. This work investigates how existing UED methods select training environments, focusing task prioritisation metrics. Surprisingly, despite methods aiming to maximise regret in theory, the practical approximations do not correlate with regret but with success rate. As a result, a significant portion of an agent's 's experience comes from environments it has already mastered, offering little no contribution toward enhancing its abilities. Put differently, current methods fail to predict intuitive measures of "learnability." Specifically, they are unable to consistently identify those scenarios that the agent can sometimes solve, but not always. Based on our analysis, we develop method that directly trains on scenarios with high learnability. This simple and intuitive approach outperforms existing UED methods in several binary-outcone environments, including the standard domain of Minigrid and a novel setting closely inspired by real-world robotics problem. We further introduce new adversarial evaluation procedure for directly measuring robustness, closely mirroring the conditional value at risk (CVaR). We open-source all our code and present visualisations of final policies here: attps://gitthe com/ http///aaatttffeeeeeeeee ordd sampling-gngggg for- learnability.
1 Introduction
Curriculum autovaryaatt generating environments for reinforcement learning (RL) agents to train on on-remains longstanding and active area of research [1, 2]. Automated curricu- lum learning (ACL.) methods offer the potential to generate diverse environments, leading to the development of more general and robust agents. Recently, a class of methods under the umbrella of Unsupervised Environment Design (UED) has gained popularity, owing their theoretical guarantees of robustness and empirical improvements in -ittof-distribution generalisation [3-5].
Currently, the most popular and empirically successful subfield of UED develops methods that aim to maximise regret-the difference in performance between an optimal agent and the current agent [3-8]. However, computing regret is intractable in all but the simplest tasks, forcing practical methods to instead approximate it [4, 9]. While the prevailing assumption has been that these approximations are faithful, we investigate this further and find that this not the case: specifically, these scoring functions correlate with success rale rather than regret. This means that these methods tend to prioritise tasks that the agent can already solve, leading to much of the collected experience not contributing to learning an improved policy.
Equua Contribution. Correspondence to arutherford@robots. ac.ac.uk uk.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

While we find that regret performs well in settings where we can compute it confirming that the underlying theory is sound-we show that the common approximations are unreliable. Therefore, we focus on a different scoring mechanism which instead prioritises levels that provide clear learning signal to the agent. More specifically, these levels are those where the agent's 's success rate neither 100% nor 0%, i.e., levels that are neither too difficult nor too easy [1, 10].
Using this scoring function, we develop Sampling For Learnability (SFL), method that estimates learnability by rolling out the current policy on randomly sampled levels and selecting those that the agent solves sometimes, but not always. We find that this simple and intuitive approach outperforms DR [11], Prioritised Level Replay [4, 9] and ACCEL [5] on four challenging environments, including our novel single- and multi- agent robotic navigation domain, -land-Minige [12] and Minigrid [13].
To truly put our method to the test, we develop a new, more rigorous robustness evaluation protocol for ACI., and demonstrate that SFL. significantly outperforms all other methods. Rather than evaluating on a set of arbitrary hand-designed environment configurations, our protocol computes a risk metric on the performance of the method, by evaluating its performance in the worst a% of a newly sampled set of environments,
Our contributions are as follows:
1. We illustrate the inefficacy of current UED methods on several domains, including a novel robot navigation environment.
2. We identify the reason for this observation: current UED regret approximations are flawed.
3. We present Sampling For Learnability (SFL), simple algorithm that trains on environment configurations that have a positive, but not perfect, solve rate, and show that it significantly outperforms current UED approaches on four domains.
4. We introduce a new evaluation protocol: the conditional value at risk (CVaR) of success of trained agent on set of sampled levels. This metric specifically measures the risk of poor generalisation, quantifying robustness in the ACL setting.
2 Background
2.1 Reinforcement Learning & UPOMDPs
We model the reinforcement learning problem as an underspecified partially observable Markov decision process (UPOMDP) [3], denoted by M O,, R. 7). Here, A, S, and O represent the action, state, and observation spaces, respectively. The agent receives an observation (without directly knowing the true state s) and selects an action, which results in transition to a new state, new observation, and an associated reward. θ represents the set of possible parameters, where each 0 E θ defines a specific level. Each 0 corresponds to a particular instantiation of the POMDP with an associated transition function Po S x A + A(S) and an observation function Ig : S - O.
In a multi-agent setting, n agents make decisions simultancously. At each step, agent i chooses an action ag forming joint action a = {aj,.... an) } that transitions the environment according to Pa. Each agent then receives reward based on the reward function S + I, which may be shared among all agents or be agent-specific.
2.2 Unsupervised Environment Design (UED)
UED is an autocurricula paradigm that frames curriculum design as a two-player zero-sum game between level-generating adversary and agent. The agent seeks to maximise its expected return in the standard RL manner, while the adversary can pursue various objectives. Domain Randomisation (DR) fits within this framework by assigning constant utility to each level, reducing level generation to mere random sampling [11]. Worst-case methods, on the other hand, incentivise the adversary to minimise the agent's 's reward, aiming enhance performance the most challenging levels [14]. However, this approach often results in the generation of unsolvable levels [3].
Ragret- UED methods offer an alternative by generating levels that maximise the agent's regret.
Here, the regret of a policy æ on a level / defined as the difference between the policy's discounted return on 0 and the optimal return achievable on that level, expressed U (m²)) U(T), where U

---

denotes the policy's 's discounted return and zq is the optimal policy on 8. This approach deprioritises unsolvable levels and should, in theory, generate levels on which the agent can continue improve. Moreover, using regret the adversary's objective provides additional theoretical benefits. At the convergence of the two-player zero-sum game, the agent's policy enjoys minimax regret robustness, meaning that the maximum regret over the entire level space @ is bounded [3].
However, computing regret in complex settings is intractable because requires access to the optimal policy for each level. As result, current UED methods rely on heuristic score functions that aim to approximate regret. The two most commonly used heuristic approaches are Positive Value Loss (PVL) and Maximum Monte Carlo (MaxMC).
Positive Value Loss (PVL). PVL approximates regret the average of the value loss across all timesteps where is positive. When using GAE [15] (as PPO (16]), PVL. can be written as follows:
PVL Σ max E 1=
where y is the discount factor, is the GAE coefficient and T the length the episode. 6, the TD-error at timestep i, defined as 6, + +V (ot+1) (or),, with V denoting the agent's 's value function, corresponding to the discounted return by following from Or.
Maximum Monte Carlo (MaxMC). Instead of using the bootstrapped value target, MaxMC instead uses the highest return obtained on a particular level:
MaxMC=
Σ T
Ramax V(oi)).
2.2.1 Current UED Methods
Prioritised Level Replay (PLR) [4, 9] involves two key steps: generating random levels and replaying levels from buffer. Initially, random levels are created, and the agent is evaluated on them, with each level being assigned score. High-scoring levels are then added to buffer. Subsequently. levels are sampled from this buffer based on their score and the time clapsed since their last selection, and the agent is trained these sampled levels. PLR has two variants: standard PLR and Robust PLR. In standard PLR, the agent's 's policy is updated using rollouts from the randomly generated environments, whereas in Robust PLR, the policy is not updated during this phase. ACCEL [5] extends the PLR framework by incorporating mechanism that randomly mutates previously high-scoring levels, generating new levels that push the agent to the edge of its capabilities. For additional methods, see Section 8, and for a more detailed introduction to ACL, refer to Appendix E.2.
JaxNav
geal
Start
(a) Minigrid
(b) JaxNav
(c) Real World (Souree: [17])
Figure JaxNAV (b) brings UED, often tested on Minigrid (a), closer to the real world (c)
In this section, we first touch on hardware-acceleratted environments, and robotic navigation. We then go to introduce JaxNav, hardware-accelerne single and multi-agent robotic navigation
environment. While similar on a surface level Minigrid, JaxNav has several additions that make closer to real-world robotics environments.

---

3.1 Hardware Accelerated Environments
Recently, Bradbury et al. [18] released JAX, a Python numpy-likc like library that allows computations to natively on accelerators (such as GPUs and TPUs). This has led to explosion in reinforcement- learning environments being implemented being reduced by hundreds or thousands of times [12, 19-21]. This has enabled researchers to experiments that used to take weeks in few hours [22, 23]. One side effect of this, however, is that current UED libraries are written in JAX, meaning they are primarily compatible with the (relatively small) set of JAX environments.
JAX, leading to the time it takes to train an RI. agent
3.2 Robot Navigation
Motion planning is a fundamental problem for mobile robotics. The general aim is to find collision- free path from a starting location to a goal region in two or three dimensions. We focus specifically on the popular setting of 2D navigation problems for differential drive robots using 2D LiDAR readings as the sensory input for their navigation policies. Given range readings, the robot's current velocity and the direction of its goal, the navigation policy must produce velocity commands to move the robot to its goal location while avoiding static and dynamic obstacles.
3.3 Environment Description
The observation space of JaxNav is highly partially observable and is based on LiDAR readings, depicted by the blue dots in Figure 1b. This is in contrast to Minigrid, which provides a top-down, ego-centric, image-like observation, shown by the highlighted region in Figure la. Additionally, while Minigrid features discrete forward and turn actions, the robots in JaxNav operate in continuous space using differential drive dynamics. Similar Minigrid, agents in JaxNav must navigate from starting location to a goal region, with the goal centre represented by the green cross in Figure lb.
These design choices make JaxNav a close approximation of many real-world robotic navigation tasks [24, 25]. including the ICRA BARN Challenge [26]. which depicted in Figure Ic. This challenge, which has run annually since 2022, aims to benchmark single-robot navigation policies in constrained environments for differential drive robots using 2D LiDAR as sensory input. Even with a cell size of 1.0 m, JaxNav offers similar elearance between robots and obstacles the test maps used in the BARN Challenge, underscoring its relevance not only for evaluating UED methods but also for advancing robotics research. Our environment's 's full design is outlined in Appendix A.
4 Understanding and Improving Level Selection in Goal Directed Domains
In this section, we examine current UED methods, and investigate how they select levels to train on. In particular, we investigate how well currently-used score functions correlate with (a) success rate (i.e., the fraction of times the agent solves the level); and (b) learnability (defined below). We then develop a method which directly samples levels according to their learnability potential, with the following sections detailing our experimental setup and results.
4.1 Defining Learnability
Similarly to the Goals of Intermediate Difficulty objective proposed by Florensa et [1] and the ProCuRI. curriculum strategy proposed by Tzannetos et al. [10], we desire agents to learn on levels that they can solve sometimes but have not yet mastered. Such levels hold the greatest source of possible improvement for an agent's 's policy and so successful autocurricula method must be able to find these. Indeed, given a success rate (i.e., the fraction of times the agent solves the level) of p on a
given level, we follow Tzannetos et al. [10] and define learnability to be p. (1 - p). In a goal-based setting where there is only a nonzero reward for reaching the goal, we justify this definition as follows:
p represents how likely the agent is to obtain positive learning experiences from a level, while 1 - p represents the maximum potential improvement the agent can make on that level. Multiplying these yields (probability of improvement) (improvement potential), i.e., expected improvement.

---

2. Tzannetos et al. [10] derive this definition for two specific, simple, learning settings and show that at each training step, selecting for the highest learnability is equivalent to greedily optimising the agent's s expected improvement in its training objective.
3. p (1-p) p) can also be seen as the variance of Bernoulli distribution with parameter p. i.c. how inconsistent the agent's performance is.
4.2 Analysing Regret Approximations used by UED Methods
(-0.61, p-is
0.25
Paattic Stew by
PeL by Livel
1.8
2.8
Learnailtyy
305
Mein Succmin Rall Levrll
Meenn Stccoess Rati Lewel
Mean SUCORAA Rati by Lovel
(a) MaxMC
(b) Positive Value Loss
(c) Learnabiiity (ours)
Figure Our analysis of UED score functions shows that they are predictive of "learnability".
Having defined learnability, we now turn our attention to the current UED score functions. As demonstrated in Section 7, the latest state-of-the-arr UED methods fail to outperform Domain Randomisation (DR) in the multi-agent JaxNav environment. To highlight the limitations these approaches, examine whether their score functions can reliably identify the frontier of learning, i.., levels that agents can only sometimes solve.
We focus the single-agent version of JaxNav and conduct rollouts using the top-performing seed for PLR-MaxMC on randomly sampled levels over 5000 timesteps. From these rollouts, we compile set of 2500 levels, evenly distributed into 10 bins based on mean success rate values ranging from 0 to 1. We then perform additional rollouts on this collected set, running for 512 environment timesteps (the same number used during training) across 10 parallel workers, and average the results. In Figure 2, we plot the mean MaxMC, PVL and Learnability scores against the mean success rate for each level. We additionally report the Pearson correlation coefficient, r, and p-value for the linear relationship between the success rate and the regret score.
Our analysis reveals no correlation between MaxMC and learnability, and MaxMC instead shows slight correlation with success rate. While PVL has a weak correlation with learnability, the high vari- ance causes already-solved maps to be prioritised alongside those with high learnability. These plots contrast heavily with that of our learnability metric, which directly prioritises levels with the greatest expected improvement. We hypothesise that the root cause of this issue is the agent's 's poor value estimation. In a highly partially observable environment, the agent struggles to accurately estimate the value of state, leading to noisy MaxMC and PVL scores, which in turn hinder UED methods from effectively identifying the learning frontier. Given that reward strongly correlated with success rate, these findings also apply when comparing scores against reward, detailed in Appendix F.
4.3 Sampling For Learnability: Our Simple and Intuitive Fix
Following on from our analysis, now present Sampling For Learnability (SFL), a simple approach that directly chooses levels that optimise learnability. Our approach maintains a buffer of levels with high learnability and trains on a
levels drawn from this buffer alongside randomly generated levels. Algorithm outlines the overall approach for SFL and illustrates the relative simplicity of our method compared to SoTA UED approaches. The policy's weights & are updated using any RL algorithm; we use PPO [16] for all of our experiments. Meanwhile, our method for collecting learnable levels is detailed in Algorithm 2. We find that the default values of T 50, e = 0.5.
Nr 256. L 2000, N 5000 and K
1000 work well across domains. However, the
5

---

per-environment hyperparameters we use are listed in Appendix C. and Appendix I contains plots showing the effect of changing each of these hyperparameters.
Algorithm 1 Sampling For Learnability
Algorithm Collect learnable levels
Initialize: policy Tg. level buffer D while convergee do
Input: policy x B t N random levels
D collect__eeee www.naalleeeeeeeeii/aiae for 1 = .......7 do
Using Alg. 2
Rollout я for L steps for all E B
D. p Ng levels sampled uniformly from D
f- success rate for each rollout
D: - (1 Nt randomly generated levels
Learnability <- p- P)
Collect r's trajectory on Dr and update e
return Top levels in B ranked by Learnability
end for end while
While key limitation of this approach is the additional timesteps required to form the learnability buffer, we find that due to the speed of forward rollouts on JAX-based environments, this does not dramatically increase our overall compute time, see Appendix H for more detailed discussion.
5 Experimental Setup
We now outline the domains used along with our adversarial evaluation protocol. Rather than taking the common approach of reporting average performance on a set of hand designed levels (which by their very nature are arbitrary). we sample a large set of levels and examine each method's performance on their worst-case levels. This directly targets the tails of the level distribution and as such is a superior measure of robustness. We further report the comparative performance of methods on the sampled set to determine the degree to which one method dominates another.
We use four domains for our experiments, JaxNav in single-agent mode, JaxNav in multi-agent mode, the common UED domain Minigrid [13] and XLand-Minigrid [12]. See Appendix for more details about the environments. We 10 seeds for Minigrid and single-agent JaxNav, and 5 seeds for multi-agent JaxNav and -Land-Minigre In all of our plots, we report mean and standard error.
Since SFL performs more environment rollouts, perform fewer PPO updates in single-agens agent JaxNav and XLand- Mann-Miiigrrd to ensure that SFL uses as much compute time as ACCEL. In Minigrid, the additional environment interactions take negligible amount of time, so we run the same number of PPO updates for all methods. For multi-agent JaxNav, we compare each method using the same number of PPO updates. See Appendix H for more information the relative speed of each method, and how many updates we run for each method. Generally, the additional SFL rollouts take much less time than the updates themselves, due the massive parallelisation afforded by hardware-acceleratee environments. Additionally, recent work suggests that world-models could also allow more samples to be taken than the base environment allows [27-30], highlighting the future potential of this approach.
We compare against several state-of-the-ar art UED methods as baselines, implemented with JaxUED [22]. We use ACCEL, with the MaxMC score function, where the agent trains on ran- domly generated, mutated and curated levels. We also include a "robust" version [4], where no gradient updates performed on the former two sets of levels. This uses three times many environment interactions and is roughly twice as slow as SFL for single-agent JaxNav. We use PLR with both the PVL and MaxMC score functions. We also include robust version of PLR which only performs gradient updates on the curated levels; this uses twice as many environment interactions and 80% slower than SFL single-agent JaxNav. We also use Domain Randomisation (DR), which trains only on randomly generated levels, with no curation or prioritisation.
A Bisk-Based Evaluation Protocol
The standard approach to evaluating UED agents is to test them on a set of hand-designed holdout levels [3-5]. Whilst this evaluation approach illustrates the performance of agents on human-relevant tasks, we believe it has several limitations. First, the hand-designed levels are arbitrary, performance on them not representative of general performance; second, does not test a central claim of UED: that it trains agents which are robust to worst-case (yet solvable) environments. To address this,

---

propose a novel evaluation protocol that rectifies both of these problems: measuring the conditional value at risk (CVaR) of the success of the trained agent on set of sampled levels.
To calculate the CVaR we sample N (10, 000 in practice) random, but solvable, levels and rollout the agent policy for 10 episodes on each level. We then find the a% of levels which the agent performs worst. To mitigate bias, we rollout the agent again on these levels, and report the average success rate on this a% subset, i.e., the CVaR at level This metric directly quantifies the performance of training method on its own worst-case levels, which measures its ability to produce robust agents. We perform this computation for each seed, and report the mean and standard error over seeds, at various levels. We further use the N sampled levels to calculate the following metrics.
Mean Success Rate We average the success of each method over all N levels and then report this as the mean success rate and its standard error over multiple independent seeds. Due to space requirements, these results included in Appendix G.
Domination Comparison To identify the degree to which one method dominates another, we obtain the average solve rate of each method (averaging over seeds) per environment. We then plot heatmap, where cell (z, y) contains the number of levels method A solves x% of the time while method B solves them y% of the time. This metric measures how many environments one method strictly solves more often than another.
7 Results
7.1
Single-Agent JaxNav
Figure 3a shows the CVaR results on single-agent JaxNav. We find that optimising for learnability- as our method does-results in superior robustness over a wide range of a values, despite all methods performing similarfy with distribution). In this plot, we also plot the results of an oracle method named Perfect Regret. This uses the same procedure as SFL but with the score function: 1 p(success). Importantly (and different to all other methods), this method only samples solvable levels, so this metric corresponds closely regret. While not shown here, using the same metric with unrestricted level sampling- -hich is more realistic setting -pefforss poorly due to it prioritising unsolvable levels. In Figure 4, we perform pairwise comparisons of each baseline against our approach. We find that there are large number of environments that all methods solve (the bright top-right corner). However, the bottom- right generally brighter than the top-left, indicating that SFL performs better in general. Overall, SFL''s superiority, and Perfect Regret's strong performance, indicates that the flawed approximations lack of performance. We provide further evidence for this claim
100% (which amounts expected success rate over the entire
of regret are responsible for UED's 's in Appendix I.2, where we use learnability score function within PLR and ACCEL.
a5% level
100
Mean Success Rate on ovaluation Set
1.0
Avg Win Rate % on worst-case at%
80
0.8
SFL
Perfect Regret (Oracle)
0.6
ACCEL-MaxMC
40
PLR-PVL
0.4
PLR-MaxMC
20
PLR-LIVL
0.2
RobustPLR
RobustACCEL
0.0
1%
10%
100%
500
1000
1500
2000
PPO Update Step
(a) CVaR of success at a level.
(b) Performance on hand uend-designed test set.
Figure Single-agent JaxNav performance on (a) a-worst levels and (b) challenging hand-designed test set. Only Perfect (Oracle) Regret matches SFL across both metrics.
Solvable means that the goal state be reached in particular level, i.e., it is not impossible complete.
7

---

ACCEI MMNMMC
30
6.6
3.4
5.6
0.8
5.0
0.0
3.4
0.5
5P%
Figure 4: Single-agent JaxNav comparison results. For each figure, cell (ar, y) indicates how many environments have method X solving them x% of the time and method Y solving them y% of the time. The density below the diagonal indicates that SFL is more robust than DR, ACCEL and PLR.
7.2 Multi-Agent JaxNav
Figures and illustrate the performance of all methods on multi-agent JaxNav throughout training. We train with 4 agents and report performance over both hand designed test set and a randomly sampled of 100 maps. The levels used in the hand designed feature cases with 1, 2, 4 and 10 agents. The levels in the sampled set all feature 4 agents and solvability is checked for each agent's individual path. As we train with IPPO, regret scores are calculated on per-agent basis and the score of a level is computed as the mean across individual agent scores. For n agents, learnability computed as Σi=(p; (P; (1 p:)), where p; is the success rate for agent i on on given level. We find that JaxNav significantly outperforms all UED methods.
are given in Appendix and
Moan Success Nata on Evallactoon Set
1.0
80
MCXML
60
PLRMaxMCC
0.6
40
0.4
DR
SFL
ACCEL-MaxMC
0.2
PLR-PVL
PLR-Max MCxMC
0.0
15
10%
100%
5000
10000
15000
20000
PPO Update Step
(a) CVaR success at level.
Success on hand- hand-designed test
Sampled
Moan Retum on Sampped Set
0.8
0.6
SFL
SFL
-8
0.2
ACCEL-M MaXM
ACCEL-MaxMC
PUR-PVL
PLR-PVL
PLR-MaxMC
12
PLR-MaxMC
0.0
5000
10000
15000
20000
10000
15000
20000
(c) Success on Sampled Test Set
(d) Reward on Sampled Test Sct
Figure 5: Performance of Multi-Agent Policies 5 seeds. SFL outperforms all UED baselines in each these. We do not include oracle regret, since it cannot be easily measured in this setting.
7.3 Minigrid
We next move on to the standard domain of Minigrid (see Figure 7). Here we find that most methods perform similarly on the hand-designed test set; however, SFI. significantly outperforms all other methods the adversarial evaluation, indicating it results in more robust policies.

---

0.0
ACDOL ACEELIMAANC MAMMC
PANA
3.0
676
3.4
5.8
0.8
0.0
3.4
0.5
SPL
Figure 6: Multi-agent heatmap results. The bright areas toward the right of the plots indicate that SFL outperforms the baselines we compare against.
Avg Winn Rate worst-case 4% level
100
Set
1.0
80
0.6
SFL
DR
0.4
ACCEL-MaxMC
PLR-MaxMC
20
0.2
RobustPLR
RobustACCEL
0.0
100%
PPO Update Step
CVaR of success at level.
(b) Performance on hand-designed test set.
Figure 7: Minigrid performance on (a) worst-case and (b) holdout levels. SFL is more robust than the baselines on worst-case levels.
7.4 XLand-Minigrid
Our final evaluation domain is Xinnnd-Mnsssid's [12) meta-RI. task using their high-3m benchmark. We report performance using our CVaR evaluation procedure and, in line with [12], as the mean return on an evaluation set during training. Our results are presented in Figure 8, with SFL outperforming both PLR and DR. During evaluation each ruleset was rolled out for 10 episodes. Due to the large number of levels being rolled out to fill SFL's buffer, SFL was slower than DR and PLR. As such, we report results for SFL compute-time matched to PLR.
Avg Win Rate % on worst-case 0% level
SFL
25
PLR
1.75
DR
1.50
1.25
1.00
Mean Return
10
0.75
5
0.50
SFL
PLR
DR
0
1%
10%
100%
0
50
1000
150
200
250
300
Update Step
(a) CVaR of success at level.
(b) Mean return on evaluation set.
Figure 8: XLand-Minigrid performance over five seeds on (a) a worst-case and (b) the evaluation set. SFL outperforms both PLR and DR.

---

8 Related Work
Unsupervised Environment Design (UED) has emerged as prominent method in the ACL field, promising robust agent training through adaptive curricula. Early works focused on learning potential, where the improvement in an agent's 's performance determined the choice of training levels [1, 2, 31, 32]. However, caansttess-orienter methods such as adversarial minimax introduced the notion of training on levels that minimise agent performance, though these often resulted in infeasible scenarios offering no learning benefits [3, 14, 33]. Minimax regret (MMR), more refined robustness approach, alleviates some of these issues by ensuring the chosen levels are learnable [3-5]. However, recent work [8] demonstrated that even true regret does not always correspond to learnability, and this mismatch can lead to stagnation during training. Our work extends this line of research by utilising a scoring mechanism that estimates expected improvement, targeting environments with positive but not perfect solve rate. Unlike existing MMR methods, our approach directly optimises for learnability, instead of using an imperfect proxy for regret, leading to more effective training on our domains.
Relatedly, Tzannetos et al. [10] introduce ProCuRL, which uses a similar learnability score as SFL (and further introduce an approximation to the solve rate p using the agent's ' value function). However, their problem setting is distinct from ours as they assume only a limited fixed pool of tasks are utilised during training, with the goal of improving an agent's performance over a uniform distribution over this pool. We, instead, consider the standard UED setting where we can sample an effectively unbounded number of tasks from some large distribution Q with the goal of achieving an agent that is robust to worst-case settings and can generalise to unseen problems. Following on from this, Tzannetos et al. [34] extend ProCuRL to the setting where the target distribution of tasks is given, and they take into account both how learnable the selected task is, as well as how correlated it is with learnable tasks from the target distribution. While this approach outperforms the original method, it tackles a different problem to SFL and UED in general, i.e., where the target distribution is known.
Finally, robust RL methods have the goal of improving an agent's robustness to environmental disturbances, and worst-case environment dynamics [35-47]. However, these methods generally consider continuous perturbations instead of a mix of discrete and continuous environment settings. Furthermore, these methods tend to overly conservative and prioritise unsolvable levels.
Discussion and Limitations
In this work we only consider deterministic, binary outcome domains and due to the nature of the learn- ability score, SFL is only applicable to such settings. In other domains, we could potentially reuse the intuition that p(l-p) p) is the variance of a Bernoulli distribution; in a continuous domain, analogous metric would be the variance of rewards obtained by playing the same level multiple times. Further- more, our implementation of SFL is in JAX but the method is general. However, one must take the cost of SFL's ' additional environment rollouts into account when considering implementing our algorithm; we chose JAX because its speed and parallelisation significantly alleviates this constraint. Next, while most current SOTA UED methods, including SFL, randomly generate and curate levels, this approach may become infeasible when the environment space is vast, as random generation may have very low likelihood of generating valid levels. Finally, while JaxNav does have deterministic dynamics, Fan et al. [48] successfully transferred RL-based multi-robot navigation policy from simulator of identical fidelity to the real world, suggesting this should be equally possible with JaxNav.
10 Conclusion
In this paper, we investigate the scoring functions used by current regret-based UED methods and analyse whether they can accurately approximate regret. We find that this is not the case and that these prioritisation metries instead correlate with success rate, leading experience not contributing to learning an improved policy. Inspired by this analysis, we develop a method based on an intuitive notion of learnability and find that this improves the robustness of measuring evaluation protocol, reporting
large amount of
the final policies. We also introduce new risk measure on performance over the a% worst-case (but solvable) levels for each method. We hope that our findings inspire future work on more general, donain-agnostic scoring functions, and we open-source all of our code to facilitate this process. Ultimately, we believe this work is a stepping stone towards bridging the gap between popular testbeds for UED and real-world applications.
10

---

Acknowledgements
This work received funding from the EPSRC Programme Grant "From Sensing to Collaboration" (EP/V000748/1). MB is funded by the Rhodes Trust. JF is partially funded by the UKI grant EP/Y028481/1 (originally selected for funding by the ERC). JF is also supported by the JPMC Research Award and the Amazon Research Award.
References
[1] Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforce- ment learning agents. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmüsson, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Rexearch, pages 1514-1523, PMIR, 2018. URI. http://prooeeeeing mlr. www.mmwww...ee.....e..e.. wwwppeeeaaaa florensa18a. html.
[2] Rémy Portelas, Cédric Colas, Katja Hofmann, and Pierre- Yves Oudeyer. Teacher algorithms for curriculum learning deep RL continuously parameterized environments. In 3rd Annual Conference on Robot Learning, CoRL. 2019, Osaka, Japan, October 30 November 1, 2019, Proceedings, volume 100 of Proceedings of Machine Learning Research, pages 835-853. PMLR, 2019. URL http::/pprccediinn// mlr. press/ ppesss/vl////oeoe//o v100/portelas20e http//potttlaa...... html.
[3] Michael Dennis. Natasha Jaques, Eugene Vinitsky. Alexandre M. Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. In Advances Neural Information Processing Systemx, 2020. URL attpp:://rroceednng neurips. cc/ paper/2022 aaper/22200aash/988565 100.335bbb000 (tte3355bbaaa1944449999 1942491 6856 2ww.9996666AAAbra........
[4] Minqi Jiang. Michael Dennis, Jack Parker-Holder, Jakob N. Foerster. Edward Grefenstette, and Tim Rocktäschel. Replay-guided adversarial environment design. In Advances in Neural Information Processing Syxtems, pages 1884-1897, 2021. URL https://proceeddee 1e915566622666116434666666666668
neurips cc/paper/2000 /2221/ hash/
Abstract.html.
[5] Jack Parker-Holder, Minqi Jiang. Michael Dennis, Mikayel Samvelyan, Jakob Foerster, Edward Grefen- stette, and Tim Rocktäschel. Evolving curricula with regret-based environment design. In Proceed- ings of the International Conference on Machine Learning, pages 17473-17498. PMLR, 2022. URL attps://proceeding mlr -ttp://///////////////eaetetttoa holderr2o.aaeear..mma. html.
[6] Mikayel Samvelyan, Akbir Khan, Michael Dennis, Minqi Jiang, Jack Parker-Holder, Jakob Nicolaus Foerster, Roberta Raileanu, and Tim Rocktäschel. MAESTRO: open-ended environment design for multi- agent reinforcement learning. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023, OpenReview.net, 2023. URL http:::///pprreviee..ee//// pdf pdf?iid=aKIRRDaPfat
[7] Ishita Mediratta, Minqi Jiang, Jack Panker-Holder, Michael Dennis, Eugene Vinitsky, and Tim Rocktäschel. Stabilizing unsupervised environment design with a learned adversary. In Conference on Lifelong Learning Agents, pages 270-291, 2023.
[8] Michael Beukman, Samuel Coward, Michael Matthews, Mattie Fellows, Mingi Jiang. Michael Dennis, and Jakob Foerster. Refining minimax regret for unsupervised environment design. /CML., 2024.
[9] Minqi Jiang, Edward Grefenstelte, and Tim Rocktäschel. Prioritized level replay. Proceedings of the 38th International Conference on Machine Learning, volume 139, pages 4940-44 4950. PMLR, 2021. URL http://proceedhi
mlr wpwpps/v19/////peeee..pp. jiang21b.... html.
[10] Georgios Tzannetos, Bárbara Gomes Ribeiro, Parameswaran Kamalaruban, and Adish Singla. Proximal Curriculum for Reinforcement Learning Agents. Transactions of Machine Learning Research (TMLR), 2023.
[11] Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider. Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation the real world. In International Conference on Intelligent Robots and Systems, pages 23-30. JEEE, 2017. doi: N0.1109/RRS.2211..500 URL https: / //dii org/ 10. 1109/IR09, 2017 2617.660 8202133.
2017.8202133.
[12] Alexander Nikulin, Vladislav Kurenkov, Ilya Zisman, Viacheslav Sinii, Artem Agarkov, and Sergey Kolesnikov. XLand minigrid: Scalable meta meta- reinforcemen learning environments in JAX. In Intrinsically- Motivated and Open- Ended Learning Workshop, NeurIPS2023, 2023. URL https: http///ppee////eeeeee pennrrvveeene/// net/ forum?id=xde forum/id-xALDCaallC.

---

[13] Maxime Chevalier-Boisver Bolun Dai. Mark Towers, Rodrigo de Lazcano, Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan Terry Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. CuRR, abs/2306.13833 2023. doi: 10.48550/
ARXIV2230. 13831. URL https: : org/10 10. 48550/arXiv...... 2306. 13831.
[14] Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Rohust adversarial reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 2817-2826. PMLR, 06-11 Aug 2017. URL https: //proceedings. mlr. wttps//eooppitooo/////o html.
[15] John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. In 4th International Conference on Learning Representations, 2016. URI. http:://axriii wrgg 1506. 02438.
[16] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1 2017. URL http::ttt///tteee www..brgg/aaa.a abo/ .06347.
[17] Zhanteng Xie and Philip Dames. Drl-vo: Learning to navigate through crowded dynamic scenes using velocity obstacles. IEEE Transactions on Robotics, 2023.
[18] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris I Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Vilne, and Qiao Zhang. JAX: compos- able transformations of Python+NumPy programs, 2018. URI. http://gittii thu.. com/ goole/oaa//jj.
[19] Chris Lu, Jakub Kuba, Alistair Letcher, Luke Metz, Christian Schroeder de Witt, and Jakob Foerster. Discovered policy optimisation. Advances in Neural Information Processing Systems, 2022.
[20] Alexander Rutherford, Benjamin Ellis, Matteo Gallici, Jonathan Cook, Andrei Lupu, Gardar Ingvarsson, Timon Willi, Akbir Khan, Christian Schroeder de Witt, Alexandra Souly, et al. Jaxmarl: Multi-agent rl environments in jax. arXiv preprint AXXiv:3311.10090 2023.
[21] Michael Matthews, Michael Beukman, Benjamin Ellis, Mikayel Samvelyan, Matthew Jackson, Samuel Coward, and Jakob Foerster. Craftax: A lightning- benchmark for open-ended reinforcement learning. In ICML, 2024.
[22] Samuel Coward, Michael Beukman, and Jakoh Foerster. Jaxued: A simple and uscable ued library in jax. arXiv preprint, 2024.
[23] Minqi Jiang, Michael Dennis, Edward Grefenstette, and Tim Rocktäschel. minimax: Efficient baselines for autocurricula in jax. In Agent Learning in Open-Endedness Workshop at NewrlPS, 2023.
[24] Xuesu Xiao, Bo Liu, Garrett Warnell, and Peter Stone. Motion planning and control for mobile robot navigation using machine learning: a survey. Autonomous Robots, 46(5):569-55) 597, 2022.
[25] Zifan Xu, Bo Liu, Xuesu Xiao, Anirudh Nair, and Peter Stone. Benchmarking reinforcement learning techniques for autonomous navigation. In 2023 IEFE International Conference on Robotics and Automation (ICRA), pages 9224-9230. IEEE, 2023.
[26] Xuesu Xiao, Zifan Xu, Zizhao Wang, Yunlong Song, Garrett Warnell, Peter Stone, Tingnan Zhang, Shravan Ravi, Gary Wang. Haresh Karnan, et al. Autonomous ground navigation in highly constrained spaces: Lessons learned from the benchmark autonomous robot navigation challenge at icra 2022 [competitions]. IEEE Robotics & Automation Magazine, 29(4): 148-156, 2022.
[27] Danijar Hafner, Timothy P. Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In 8th International Conference on Learning Representations. OpenRe- view.net, 2020. URL https: http:///////e//eeeeee het//forumiid--5110000DD
[28] Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. In 9th International Conference on Learning Representations. OpenReview.net, 2021. URL
fet/ ? d O0aabbayZbb...
[29] Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy P. Lillicrap. Mastering diverse domains through world models. CoRR, abs/2301 000/22010041000 .04104, 2023, doi: 10.48550/ARXIV.230...8 URL https: //doi, org/10. 4855048655... 2301. 04104.
[30] Mare Rigter, Minqi Jiang, and Ingmar Posner. Reward free curricula for training robust world models. CoRR, abs/2306.09 09200,099255550... 2023. doi: 104445500AAXXI22230000202 URL https: ://oii. org/10. /8550/ arXiv 2306 092055.
12

---

[31] Pierre-Yves Oudeyer, Frdrie Kaplan, and Verena V Hafner. Intrinsic motivation systems for autonomous mental development. IEEE transactions evolutionary computation, 11(2):265-286, 2007.
[32] Tambet Matüssnn, Avital Oliver, Taco Cohen, and John Schulman. Teacher-student curriculum learning. volume 31, pages 3732-3740, 2020. doi: 10.1 11111//NNSS.011929922. 1109/TNNLS 2019. 2934906.
URL
[33] Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O. Stanley. Paired Open-Ender Trailblazer (POET): Endlessly generating increasingly complex and diverse learning environments and their solutions. CoRR, abs/1901.00 101000111113,.1111111111111 2019. URL http: /ttp:://aaxvvviie org/ abb//901s111111.. http//991.00000 01753.
[34] Georgios Tzannetos, Parameswaran Kamalaruban, Adish Singla. Proximal curriculum with task correlations for deep reinforcement learning. In Kate Larson, editor, Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI-24, pages 5027-5036 International Joint Conferences Artificial Intelligence Organization, 8 2024. doi: 10.24963/ijcai.222445533 URL https: //doi.org//ni 10 24963/1 jaaiiiiijaaaa/// 2024/556.
[35] Garud N Iyengar. Robust dynamic programming, Mathematicss of Operations Research, 30(2):257-280, 2005.
[36] Laurent Ghaoui Arnab Nilim. Robust solutions to markov decision problems with uncertain transition matrices. Operations Research, (3(5):780-799)) 2005.
[37] Huan Xu and Shie Mannor. Distributionally robust markov decision processes. In Advances Neural Information Processing Systems, pages 2505-2513. Curran As- sociates, Inc., 2010. (FF3cd0081446633a000222226066 -a0-re0004966---attraatttt .ttml.
URL. httpe://proceeding www.snneeipp... http:/ccppppee/////.///
[38] Wolfram Wiesemann, Daniel Kuhn, and Berç Rustem. Robust markov decision processes. Mathematics of Operations Research, 38(1):153 183, 2013.
[39] Shiau Hong Lim, Huan Xu, and Shic Mannor. Reinforeement learning in robust markov decision processes. Advances in Neural Information Processing Systems, 26, 2013.
[40] Vineet Goyal and Julien Grand-Clément. Robust markov decision processes: Beyond rectangularityy Math. Oper. Res., 48(1):203-22) 226, 2023. doi: 10.1287/M00R.2022.1222 URL https://doi..oooo org//0.......... 1287/ moor 2022. 1259.
[41] Hideaki Nakao, Ruiwei Jiang, and Siqian Shen. Distributionally robust partially observable markov decision process with moment-basee based ambiguity. SIAM Optim., 31(1):461 2021. doi: 10.1137/19M1 268410. URL https: //doi. org/10 10. 11 1117/1111111 1911//9111110...1/1
[42] Qianli Shen, Yan Li, Haoming Jiang, Zhaoran Wang, and Tuo Zhao. Deep reinforcement learning with robust and smooth policy. Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 8707 8718. PMLR, 2020. URL http:tt/pproceeeiiiiiie
mlr. wwl...pres//ee http/vv11////// shen20b v119//en200bb html.
[43] Rui Yang, Chenjia Bai, Xiaoteng Ma, Zhaoran Wang, Chongjie Zhang, and Lei Han. RORL: robust offline reinforcement learning via conservative smoothing. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NewrlPS 2022, New Orleans, LA, USA, November 28- Decemher 9, 2022, 2022. URL htt.:htt:///eet //tp:://apppeeeeeeee nip.. cc/paper....pp..//eppe files/ files/paper. paper/ httph/hhhw.ebbaaadaaaaaaaaaaeaaa
Saw.:radd--aataatt------oe Conferenceeoeeeeeeeeeeeeoeeeeeee html.
[44] Nicole Bäuerle and Alexander Glauner. Distributionally robust markov decision processes and their connection to risk measures. Math. Oper. Res., 1773) 2022. doi: 10.1287/M0OR.2021.1115
URL httph::t////t / //w.///////oo/// www.oo.o.....ooooooooooooooooooo org/ 10. 1287 /moor. 2021. 1187.
[45] Shengbo Wang, Nian Si, Jose H. Blanchet, and Zhengyuan Zhou, On the foundation distributionally robust reinforvement learning. CoRR, abs/2311.0901s 2023. doi: 10.48550ARRXIV2311..000 URL
https://doi. org/10. 48550/arXiv 2211 090188.
2311.
[46] Mohak Bhardwaj. Tengyang Xie, Byron Boots, Nan Jiang. and Ching-An Cheng Ad- versarial model for offline reinforvement learning. In Advances in Neural Informtion Processing Systems, 2023. URL http://papers: www..iiipssp..eeeeoe cc/paper fwtefffaleepeeeeeeeeeeeee hatpeer///2////aae/ hash/ 022200eet119001119999900900bbbb0 Abstract Coree-Coooneeeee.cee html. html.
13

---

[47] Chenlu Ye, Rui Yang, Quanquan Gu, and Tong Zhang. Coruption-robust offline rein- forcement learning with general function approximation. In Advances Neural Information Processing Systems. 2023. URL http://papers nips.
hashh/
71552455S162283334333111 70011177660660000 160d- Abstract- Conf erence. html.
[48] Tingxiang Fan, Pinxin Long, Wenxi Liu, and Pan. Distributed multi-robot collision avoidance via deep reinforcement learning for navigation in complex scenarios. The International Journal of Robotics Research, 39(7))856-89))) 892, 2020.
[49] Gregor Klancar, Andrej Zdesar, Saso Blazic, and Igor Skrjanc. Wheeled mobile robotics: from fundamentals towards autonomous systems. Butterworth- Heinemann, 2017.
[50] Pinxin Long, Tingxiang Fan, Xinyi Liao, Wenxi Liu, Hao Zhang, and Jia Pan. Towards optimally decentralized multi-robot collision avoidance via deep reinforcement learning. In 2018 IEEE international conference robotics and automation (ICRA), pages 6252 6259,, IEEE, 2018.
[51] Roni Stern, Nathan Sturtevant, Ariel Felner, Sven Koenig, Hang Ma. Thayne Walker, Jiaoyang Li. Dor Atzmon, Liron Cohen, TK Kumar, al. Multi-agent pathfinding: Definitions, variants, and benchmarks. In Proceedings of the International Symposium Combinatorial Search, volume 10, pages 151-158, 2019.
[52] Peter R Wurman, Raffaello D' Andrea, and Mick Mountz. Coordinating hundreds of cooperative, tonomous vehicles warchouses. Al magazine, 29(1): 2008.
[53] Paolo Fiorini and Zvi Shiller. Motion planning in dynamic environments using velocity obstacles. The international journal of robotics research, 17(7):760-775 772, 1998.
[54] Jur Van Den Berg. Stephen J Guy, Ming Lin, and Dinesh Manocha. Reciprocal n-body collision avoidance.
In Robotics Research: The 14th International Symposium ISRR, pages 19, Springer. 2011.
[55] Michael Everett, Yu Fan Chen, and Jonathan P How. Collision avoidance in pedestrian- environments with deep reinforcement learning. IEEE Access, 9:10357 10377,, 2021.
[56] Qingyang Tan, Tingxiang Fan, Jia Pan, and Dinesh Manocha. Deepmnavigate: Deep reinforced multi-robot navigation unifying local & global collision avoidance. In 2020 JEEE/RS. International Conference on Intelligent Robots and Systems (IROS), pages 6952-6959, IEEE, 2020.
[57] Ruihua Han, Shengduo Chen, Shuaijun Wang, Zeqing Zhang, Rui Gao, Qi Hao, and Pan. Reinforcement learned distributed multi-robot navigation with reciprocal velocity obstacle shaped rewards. IEEE Robotics and Automation Letters, 7(3):589655033, 2022.
[58] Jin-Soo Park, Xuesu Xiao, Garrett Warnell, Harel Yedidsion, and Peter Stone. Learning perceptual ballucination for multi-robot navigation in narrow hallways. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages
IEEE, 2023.
[59] Rémy Portelas, Cédric Colas, Lilian Weng, Katja Hofmann, and Pierre-Yves Oudeyer. Automatic curriculum learning for deep RL: A short survey. In Proceedings of the Twenty-inthh International Joint Conference on Artificial Intelligence, IJCAI 2020, pages 4819-4825, ijcai.org, 2020. doi: 10.24963/DCAI. URL https: / /ttii//o..io/ http///ooooooo///////////// wttp:/o.//.///////eee... 24963/ijcai.. 2020/671.
2020/671.
[60] Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew F. Taylor, and Peter Stone. Cur- riculum learning for reinforcement learning domains: framework and survey. Mach. Learn. Res., 21: 181:1-181:50, 2020. URL hatttp://jmll org/ wwnggpaprr//veee Aapers/v21/22 20212202211 212. html.
[61] Lev Vygotsky et al. Interaction between learning and development. Linköpings universitet, 2011.
[62] Samuel Garcin, James Doran, Shangmin Guo, Christopher G. Lucas, and Stefano V. Albrecht. Dred: Zero-shot transfer in reinforcement learning via data-regularised environment design. 2024. URL attps://doi. org/10. 11/110048888 485444855500// arrviv.... 2402. 2402.03479.
14

---

Appendix
We structure the appendix as follows. Appendix A includes more details about JaxNav, and Ap- pendix B describes the other environments we use. The hyperparameters we use and the hand- designed test sets can be seen in Appendices C and D, respectively. Appendix E discusses multi-robot navigation and automated curriculum learning in more detail.
We next provide more results, including more analysis on the UED score functions in Appendix additional general results in Appendix G and compute-time analysis in Appendix H. Finally, we thoroughly ablate SFL. Appendix I.
A
JaxNav Specification
The environment is designed as follows, with full parameters listed in Appendix C.
Observations The robot's observation at given timestep t is o, [1, de, vel containing the current LiDAR range readings (l,), the direction to the robot's goal (d,), and the robot's 's current linear and angular velocities (v,). The LiDAR range readings I, is vector containing the 100 most recent LiDAR range readings from a 360 are centred on the robot's forward axis, the LiDAR's max range Daidr is set to 6 Given the robot's current position p, and the goal position g, the robot's goal direction de is defined as:
d' polar (g Pp) polar g-P. [[-
if if F S otherwise
Duiar
Diidar)
where polar converts Cartesian vector to its polar representation. All observation entries are normalised using their maximum possible values.
Actions The policy selects two-dimensional continuous action a, [v] wt] representing target linear (uf") and angular velocity (me²). The possible linear and angular velocities are limited to a set range, with actions outside these ranges clipped to be within.
The action a' is translated into movement in the x-y plane using differential drive kinematics model [49] which includes limits on linear and angular acceleration.
Dynamics
Rewards Our reward function is inspired by Long al. [50] and aims avoid collisions while minimising the expected arrival time. Due to the difficulty the task we include shaping terms which give small dense reward at each timestep. The reward r received at timestep
defined as the sum: rs r? + ri + Rrime: where r rewards the robot for reaching the goal, rf penalises collisions, and Reime small penalty each timestep equal -0.01. The goal reaching term r is defined as:
R Rgoal (1p''
|pp' - gll < D Dgoal
r
gll
gl))
otherwise
where Rgosl 4.0, D Dooal 0.3 and wy 0.25. This rewards the agent for reaching the goal, and provides a small dense reward if the agent moves closer to the goal. Meanwhile, the collision penalty term rf is defined:
Roollision if collision
Relose
if min(l") < Delose
otherwise
where 1t are the un-normalised LiDAR readings at timestep f, Reollision = -4, Relose = -..1, and Delcee 0.4m. This avoids collisions and provides a small dense penalty when the agent is close to obstacles, this encourages safe behaviour.
Multi-Agent Reward In the multi-agent version of JaxNav, the reward for each agent is defined as Ar; + (1 - Xx) Σj Ty, i.c. it shares its reward, well the team reward. We use À 0.5.
15

---

B Environment Description
Here we describe the other environments we use, with environment parameters listed in Tables to 3.
Table
JaxNav Parameters
Table 2: Minigrid Parameters
Parameter
Value
Parameter
Value
Num Agents
(4 for multi-agent)
Number of walls 60 Agent view size
Square agent width
0.5 m
Grid cell size
1.0
Dynamics
Table 3: XLand Mand-Miiigrid Parameters
Goal Radius, D goal
0.3
Parameter
Value
Min linear velocity
0.0 m/s
Max linear velocity
1.0 m/s
Ruleset benchmark Environment ID
high-3m
Max linear acceleration
1.0 m/s?
R4-13x13
Min angular velocity
-0.6 rad/s
Image observations
False
Max angular velocity
0.6 rad/s
Max angular acceleration
1.0 rad/s?
Timestep length
0.1 200
Number of LiDAR beams
LiDAR range resolution
0.05
LiDAR max range, Didar
6 m
LiDAR min range
0
Map size
11 x 11 m
Maximum wall fill %
60%
Reward Signal
Goal reaching reward, Rgoal
4.0
Distance change weight, Wg
0.25
Collision penalty, Reollision
-4.0
LiDAR threshold, Delose
0.4 m
LiDAR penalty, Relose
-0.1
Timestep penalty, Riime
-0.01
B.1 Minigrid
Minigrid is a goal-oriented grid world where triangle-like agent must navigate 2D maze. As illustrated in Figure la, the agent only observes a small region front of where is facing and must explore the world to move to goal location.
B.2 XLand-Minigrid
This domain combines an XI XLaand-inspirs system of extensible rules and goals with Minigrid- inspired goal-oriented grid world to create a domain with diverse distribution tasks. Each task is specified by a ruleset, which combines rules for environment interactions with goal, and [12] provide database of presampled rulesets for use during training. Following [12], we use 13x13 grid with 4 rooms and sample rulesets from their high diversity benchmark with 3 million unique tasks. As training involves sampling from a database of precomputed rulesets, ACCEL is not applicable. PLR and SFL select rulesets for each meta-Ro RL step to maximise return a held-out set of evaluation rulesets.
C Hyperparameters
Table 4 contains the hyperparameters we use, with their selection process for each domain outlined below. We tuned PPÒ for DR for each domain and then used these same PPO parameters for all methods, tuning only UED-specific parameters.
16

---

Table 4: Learning Hyperparameters.
Parameter
JaxNav Single-Agent
JaxNav Multi-Agent
Minigrid
XLand
PPO
Number of Updates
2250
22850
4500
Number Meta-Steps
298
# of PPO Updates per Meta-Step
128
0.99
AGAE
0.95
PPO number steps
512
256
32
PPO epochs
4
1
PPO minibatches per epoch
4
16
PPO clip range
0.04
0.2
PPO # parallel environments
256
8192
Adam learning rate
2.4e-4
le-3
Anneal LR
yes
Adam e
le-5
1e-8
PPO max gradient norm
0.5
PPO value clipping
yes
return normalisation
no
value loss coefficient
0.5
entropy coefficient
0.0
0.01
Fully-connected dimension size
512
[16, 256]
Hidden dimension size
512
1024
PLR
Replay rate, p
0.5
0.95
Buffer size, K
1000
8000
40000
Scoring function Prioritisation
MaxMC
Top K
Rank 1.0
Rank 1.0
Temperature,
32
staleness coefficient
0.3
Duplicate check
no
ACCEL
Number of Edits
5
20
Buffer size, K
8000
Prioritisation
Rank
Temperature,
1.0
SFL
Batch Size N
5000
25000
40000
Rollout Length L
2000
5070
Update Period T
50
50
100
4
Buffer Size K
100
100
1000
8192
Sample Ratio p
1.0
1.0
0.5
1.0
C.1 JaxNav
For PPO, we conducted extensive sweep the JaxNav environment ensuring robust DR perfor- mance. We only tuned hyperparameters for single-agent JaxNav, and used the best hyperparameters for multi-agent JaxNav. JaxNav hyperparameter searches were done over seeds.
For PLR, we performed grid search, over replay probabilities (0.5, 0.8), buffer capacity (1000, 4000, 8000), prioritisation (rank, topk), temperature (0.3, 1.0). and & (1,32, 32, 128]. For ACCEL, we searched over the same set and additionally included the number of edits, where we considered values of (5, 20, 50}.
17

---

For SFL, we performed line searches over . of {500, 5000, 25000), L {1000, 2000, 4000), K of {100, 1000, 5000). 7 of {10, 50, 500, 1000, 2000) and p (0.25, 0.5, 0.75, 1.0].
Since this a line search and not a grid search, the total number of tuning runs (and total compute) is less than for PLR and ACCEL (60 runs for SFL vs 90 for PLR and 270 for ACCEL).
C.2 Minigrid
For Minigrid, our JaxNav PPO parameters performed similarly to those given in the JaxUED implementation but allowed to 256 environment rollouts in parallel during training compared to JaxUED's 32. For the UED-specific parameters we used the same sweep settings as for JaxNav, again conducting the search over 3 seeds.
C.3
XLand-Minigrid
For PPO, we used the default parameters provided by [12] and the search for UED parameters was conducted over 1 seed. For PLR, we conducted a grid search over replay probabilities {0.5, 0.95). buffer capacity {20000, 40000), prioritisation {rank, topk), temperature {0.3, 1.0), score function (MaxMC, PVL). For SFL, initial experiments illustrated that, due to the number of environment rollouts used to fill the buffer, it was slower than PLR and DR. To use similar compute budget as PLR, conducted the sweep over only 70B timesteps. For SFL, performed a grid search over N of (40000, 30000), {5070, 7650), K (8192), T {1, 2, 3, 4} and of (0.75, 1.0}.
18

---

HHE D Hand Designed Test Sets
The hand-designed levels used for evaluating policy performance throughout training are illustrated in Figures 9 and 10. The set used for multi-agent policies also includes the first maps in Figure 9, Minigrid's levels are shown in Figure and are same as those used by JaxUED.
P
Figure Hand-Designed Test Set for Single Agent JaxNav Policies.
Figure 10: Hand- Designee Test Set for Multi Agent JaxNav Policies.
SixteenRooms
SixteenRooms2
Labyrinth
LabyrinthFlipped
C
OL
Labyrinth2
StandardMaze
StandardMaze2
StandardMaze3
Figure Hand-designed Minigrid Levels [3, 22, 23).
19

---

E
Additional Background Related Work
E.1 Literature Review of Multi-Robot Navigation
Multi-robot path planning presents unique challenges due to the need for coordination among robots to avoid deadlocks in dynamic environments. Traditional methods often discretise the environment, turning the problem into a multi-agent pathfinding task managed by central planner [51, 52]. While effective at scale, these approaches rely heavily on communication infrastructure, making them impractical in scenarios with unreliable connectivity or third-party obstacles. In contrast, our method leverages decentralised learning approaches without relying centralised communication, making it more robust in partially observable and communication centuiiiaatto--imieeed settings.
Decentralised approaches like velocity obstacles [53] and ORCA [54] offer a solution by mapping environmental constraints into the robot's velocity space. However, these methods are susceptible to measurement errors and often exhibit short-sighted behavior, limiting their real- world applicabil- ity [17]. Our method, by comparison, incorporates broader evaluation framework that assesses performance in adversarial and challenging environments, ensuring robustness beyond simple dy- namic obstacle avoidance.
Recent advancements leverage machine learning to overcome these limitations. CADRL [55] uses RL to address short-sightedd sighteests, but its state-based representation limits its adaptability. Our approach, instead, employs lidar-based observations to model more realistic and complex navigation task, which allows for better generalisation to real-world scenarios.
More sophisticated RL-based approaches, such as those by Long et al. [50] and Tan et al. [56], demonstrate improved performance in open spaces but struggle in constrained settings. Our method specifically addresses this by focusing on environments that are solved intermittently, thereby enhanc- ing the agent's ability to learn in varied and complex settings.
Hybrid methods, combining RL with conventional planning [48], show promise but do not fully address these challenges. In contrast, our method integrates adaptive curriculum that dynamically adjusts based on the agent's 's performance, leading to sustained learning improvements even in diverse and adversarial environments.
The design of reward functions is critical in RL-based navigation. Enhancements using velocity obsta- cles [17, 57] improve performance but still face challenges in real-world transferability. Techniques like perceptual hallucination [58] further enhance robustness by reducing multi-robot planning to static obstacle avoidance, though they typically consider simple scenarios and do not account for dynamic mairt-party obstacles. Our method, however, introduces a novel evaluation protocol that rigorously tests the robustness of learned policies in variety of adversarially generated environments, ensuring better real-world applicability.
E.2 Background on Automated Curriculum Learning
Automated Curriculum Learning (ACL) is a subfield of RL where agents are presented with increas- ingly challenging tasks that are adapted to the agent's current progress [59, 60]. One common idea idea is to train the agent on tasks that are neither too easy nor too hard, such that it achieves maximum learning potential [1, 61]. Autocurricula methods have various aims, such as improving learning speed on set of target environments [62] or increasing robustness to unknown environment config- urations [3, 4]. Unsupervised environment design (UED) focuses on the latter. One commonality between autocurricula methods is that the environment generator controls aspects of the environment, such as the transition dynamics, state and observation spaces, goals, and so on [1, 3]. Each of these environment configurations is commonly referred to as level [3].
Methods also differ in how they generate these environment configurations. One class of methods uses generative models, such as Gaussian Mixture Models (GMMs) [2]. While this approach generally makes the problem theoretically tractable, GMMs are limited to continuous-valuu parameter settings. More recently. other generative models, such as Variational Autoencoders, have been used [62]. However, these models often require data to train, which may unavailable or bias the learning process. Other methods use an RL-based level generator, where the generator's objective is based on how the agent performs on the generated level [3]. This approach has been surpassed by the more recent technique of randomly generating and curating levels [4, 5, 9].
20

---

Extended analysis of UED Score Functions
An extension Section 4.2, Figure 12 illustrates the correlation between mean reward and the two most popular regret scores, MaxMC and PVL. These graphs illustrate that the trends seen for success rate also hold for episodic reward, this expected for our environment as the two are strongly correlated. In Figure 13 we conduct the same analysis for the L1 Loss Score [9], defined as:
T
T
LI Loss
Σ 100
Σ
(Ta)
[-0.61, 2:6.2:255
1.30
0.2-
PA. Store
Loaaabiity
-6.- -000 -Is = Mean Eskode Retum Lovel
-125 -ik -la -60 -25
25 20 639
-125
-050 -15 -18
23
Mean Epiede Rotur by Lovel
Mean Epande Retun
(a) MaxMC
(b) Positive Value Loss
(c) Learnabiliiyy (ours)
Figure 12: Analysis of UED and Learnability Score Functions Against Reward
200
r00.07, p=0.0002
4.0
r=0.07, p=0.0066
3.5
3.5
Score by Level
2.0
2.5
10-
1.0
0.5
0.5
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
250
-12. -100 -7.5 -5.0 -2.5 0.0
2.5
5.0
250
Mean Success Rate by Level
Mean Episode Return by Level
(a) L1 Loss against Success Rate
(b) L1 Loss against Reward
Figure 13: Analysis of L1 Loss score function
21

---

G Additional Results
Figure 14 shows each method's overall solve rate on the set of 10000 sampled solvable levels on log scale. We find that while all methods solve the vast majority of levels, SFL slightly outperforms all the baseline methods. Figure 15 reports the pairwise comparisons of each base against SFL for Minigrid. While, again, most methods solve most levels, SFL has slight advantage.
SRL
SRL DR
Perfect Regret (Gracle)
SPL DA
DR
ACCEL MaceCC
34
ACCEL MaxMC
PIR-MaccAC
PLS-PVL
RobustPLR
TL. MaxMC:
RobustACCEL
PLA/Mivee
TLB L3VL
SUAEPLA
04
13
64.00
Group
(a)
(b)
(c)
Figure 14: Overall solve rate 10000 sampled levels (a) Single-agent JaxNav (b) Minigrid and (c) Multi-agent JaxNav.
0.
0.0
0.0
3.50
0.25
0.30
0.75
100
0.50
0.20
2.30 0.75
1.00
5.10
0.25
0,30
3.70
3.50
0.00
2.45
0.30
0.75
1.00
0.05
0.23
0.50
2.75
1.0
Figure 15: Minigrid results. For each figure, cell (æ, y) indicates how many environments have method solving them 5% of the time and method Y solving them y of the time. In each plot compare a different baseline to our learnability measure.
G.1 Episodic Return Plots
Figure 16 shows episodic return and success rate plots for single-agent JaxNav and Minigrid. We find that the episodic return is very strongly correlated with the success rate, which is why we primarily show the latter in the main text.
G.2 Easy Level Analysis
To assess performance on easy levels we have run our evaluation procedure over 10,000 uniformly sampled levels with fewer obstacles than usual. For JaxNav, used maximum fill % of < 30%, half of the standard 60%. Meanwhile, for Minigrid, we use a maximum number of 30 walls instead of 60. These levels, therefore, generally easier than the levels evaluated in the main paper. Results are reported in Figure 17
On JaxNav. SFL still demonstrates significant performance increase while Minigrid all methods are very similar (with the robust methods performing slightly better for low values of a). Due to the challenging dynamics of JaxNav, even levels with small number of obstacles can present difficult control and navigation problems meaning ACL methods (such as SFL. still lead to a performance differential over DR. Meanwhile, in Minigrid, due its deterministic dynamics, difficulty is heavily linked to the obstacle count as this allows for more complex mazes. As such, DR is competitive to ACL methods in settings with fewer obstacles.
G.3 Analysing the Learnability of Levels
Table 5 shows the mean and median of learnability and success rate for a variety of methods. We find that the average learnability of levels in the PLR/ACCEL buffers is very low. While not shown
22

---

Mcan Success Rate Evaal bation Sec
4
2
SFL
Evel
0.4
Regret (Oracle)
DR
0.3
ACCEL-MaxMC
SFL
PLR-PVL
0.2
4
PLR-MaxMC
ACCEL-MaxMC
PLR-LIVL
0.1
PLR-MaxMC
8
RobustPLR
Mean
RobustPLR
RobrustACCEL
RobustACEEL
0.0
500
1000
1500
2000
1000
2000
3000
4000
PPO Update Stop
PPO Update Step
(a) Return JaxNav
(b) Return Minigrid
Evaluation Set
Mean Ratur on Evaluation Set
0.6
0.6
0.1
0.4
0.2
0.0
0.0
5.00
1000
1500
2000
1000
2000
3000
4000
PPO Update Step
PPO Update Step
(c) Success rate for JaxNav
Success rate Minigrid
Figure 16: Episodic return (top) and success rate (bottom) plots for Jaxnav (left) and Minigrid (right).
Avg Win Rate % worst-case a% level
100
100
80
Avg % worst-case
60
80
SFL
Perfect Regret (Oracle)
SFL
DR
DR
ACCEL-MaxMC
ACCEL-MaxMC
PLR-MaxMC
MLXMM
RobustPLR
RobustPLR
RobustACCEL
50
RobustACCEL
1%
10%
100%
6..1%
1%
10%
100%
(a) Evaluation on easy JaxNav levels.
(b) Evaluation on easy Minigrid levels.
Figure 17: CVaR evaluation on easy levels for single-agent JaxNav and Minigrid.
here, this is also true when selecting only the levels with the top 50 PVL/MaxMC scores. We also find no statistically significant correlation between the learnability of these levels and the PVL/MaxMC scores. We further note that most of the levels UED buffers can already be solved by the agent.
By contrast, the levels that SFL selects have high learnability (note that 0.25 is the maximum learnability value), and solve rate around 50%. This shows that SFL indeed selects levels with high lcarnabllity.
23

---

Table The learnability and success rates for levels within the PLR/ACCHL/SFL buffers averaged over training. At each evaluation step. the average and median values for the entire buffer are calculated and then averaged over training. The mean and standard deviation across three different seeds are reported.
Method
Learnability (Mean)
Learnabllity (Median)
Success (Mean)
Success (Median)
PLR(PVL)
0.01 (0.00)
0.00 (0.00)
0.85 (0.04)
0.96 (0.05)
PLR(MaxMC)
0.02 (0.00)
0.00 (0.00)
0.84 (0.03)
0,98 (0.01)
ACCEI (Mx
0.01 (0.00)
0.00 (0.00)
0.97 (0.01)
1.00 (0.01)
ACCEL(PVL)
0.01 (0.00)
0,00 (0.000
0,94 (0.04)
0.95 (0.05)
SFL (All Sampled)
0.01 (0.00)
0.00 (0.00)
(0.01)
0.99 (0.02)
SFL (Selected)
0.22 (0.01)
0.22 (0.01)
0.59 (0.02)
0.61 (0.03)
G.4 Environment Metries
For single-agent JaxNav, we plot the shortest path length, number of walls and the solvability of levels in the PLR/SFL buffers in Figure 18. We find that SFL has marginally longer shortest paths and marginally fewer walls. The SFL levels are also considerably more solvable.
Aside from solvability, there not a large difference these metrics between PLR and SFL, despite SFL significantly outperforming PLR. Qualitatively, find that in JaxNav, levels with high learnability tend to involve lot of turning and intricate obstacle avoidance (as opposed to long paths). As such, the number of walls and shortest path length do not fully capture level's difficulty.
27.5
1.00
25.0
0.95
8
of Walls
Shortest Path
Solvabiliy
0.90
20.0
6
0.85
17.5
SFL
PLR-MaxMC
PLL
12.5
0.75
Enviranment Step
Environment Stop
Environment Step
Figure 18: Environment Metrics for single-agent JaxNav for SFL, PLR and DR.
For XLand-Minigrids we report the mean number of rules for the PLR/SFL buffer rulesets in Figure 19, which illustrates significant difference in the rulesets seen by the different agents. SFL samples well below the mean value throughout training, whereas PLR starts on par with DR before tending easier as training progresses. This result, coupled with the performance difference, illustrates how SFL's learnability score allows it to find the frontier of learning, leading to more robust agents.
6.0
Moan mbmr Ruless
5.6
5.4
SFL
5.2
PLR
5.0
50
100
150
300
Meta-
Update
Figure 19; Environment Metrics for XLand- Minig--d
24

---

G.5 Level Plots
Here we plot some generated levels for single-agent JaxNav in Figure 20, multi-agent JaxNav in Figure 21 and Minigrid in Figure 22. Overall, ACCEL tends to have the most walls, since its mutation operator is able to add more walls over time. Despite this, the levels do not involve a large amount of complicated turning and maneuvering. The levels selected by SFL,, on the other hand, tend to involve going around many corners, which is an important part of JaxNav. In multi-agent JaxNav, SFL occasionally generates levels where not all of the agents can reach their goals; however, these are still useful to Iearn on, as the other agents can complete their tasks.
For completeness, include levels from XLand in Figure 23 but, unlike the other domains, it is difficult to assess a level's difficulty solely from render.
ACCEL
PLR
DR
SFL
Figure 20: Levels in single-agent JaxNav generated by each method.
ACCEL
PLR
DR
SFL
Figure 21: Levels in multi-agent JaxNav generated by each method.
ACCEL
PLR
DR
SFL
Figure 22: Levels in Minigrid generated by each method.
25

---

DR
PLR
SFI.
Figure 23: Levels in Xland- Miniigdd generated by each method.
H
Timing Results and Speed Analysis
Tables 6 and report compute time for all methods on single-agent JaxNav and Minigrid, respectively, Each individual seed was each run on Nvidia L40s using a server which has 8 NVIDIA L40s', two AMD EPYC 9554 processors (128 cores in total) and 768GB of RAM. These times are without logging, and we find that with logging, SFL is around JaxNav. Therefore, for the results in Figure 17b, we use 6% fewer environment steps for SFL to ensure a fair comparison. For Minigrid, SFL is as fast or slightly faster than the other methods. As this is surprising (since SFL performs significantly more environment rollouts), investigate this further. In Table 8, we compare the time it takes for a single iteration (including training and evaluation) in Minigrid
slower than ACCEL on single-agent
L40s GPU.
We note that the SFL rollouts are fast for two reasons:
1. We aggressively parallelise them, running up 25, 000 environments parallel, which takes about the same time as running only hundreds in parallel.
2. We not compute any gradients for these transitions.
The effect of this that these additional rollouts take significantly less time than the actual training step. Furthermore, UED's training step is more complex than SFL's, since it must maintain a buffer of levels, compute the scores during training, and potentially update the buffer.
The multi-agent results were run on variety of machines, including the aforementioned L40s system, similar system featuring NVIDIA A40's and a workstation containing RTX 4090's. On 4090, SFL run takes 1d 1h 13m 54s while ACCEL takes 18h 17m 26s.
All -Land-Minigria experiments mentioned above. Table 9 reports the compute time for all methods; these times are with logging where each method logs the same data. Note that SFL compute-time matched to PLR for this environment.
Nvidia L40s, with the
specification as
Table Mean and standard deviation of time taken for single-agent JaxNav over 3 seeds.
Table 7: Mean and standard deviation of time taken for Minigrid over 3 seeds.
Method
Compute Time
Method
Compute Time
DR
0:41:33 (0:00:27)
DR
0:28:11 (0:00:00)
RobustACCEL
1:37:59 (0:00:30)
RobustPLR
0:39:17 (0:00:04)
RobustPLR
1:23:48 (0:00:26)
RobustACCEL
0:33:39 (0:00:04)
ACCEL
0:42:09 (0:00:18)
PLR
0:29:19 (0:00:00)
PLR
0:41:48 (0:00:25)
ACCEL
0:29:28 (0:00:00)
SEL
0:45:45 (0:00:00)
SFL
0:28:32 (0:00:00)
26

---

Table 8: PLR and SFL timings for a single minigrid iteration
Table 9: Mean and standard deviation of time taken for aand-riiigrid over 5 seeds.
PLR
SFL
Method
Compute Time
Train Step Get Learnable Levels
37.58
358
0
2.28
DR
4:43:06 (0:00:20)
Eval Step
0.7s
0.7s
PLR
4:51:47 (0:00:32)
SFL
4:51:16 (0:00:20)
Total
38.2s
37.9s
I Ablations
This section of our appendix focuses on running ablations for SFL, investigating the effect of hyperparameters in Appendix I.1, using learnability as score function in UED in Appendix 1.2, and other definitions of learnability Appendix I.3
L1 SFL Hyperparameters
In this section investigate the effects of SFL's hyperparameters. For computational reasons, run these ablation experiments over three seeds; furthermore, we consider only JaxNav's 's single agent setting. Our results, for single-agent JaxNav and Minigrid, respectively, are shown in Figures 24 and 25, with the effects of the hyperparameters as follows:
Number of Sampled Levels N Sampling more levels results in improved performance, but in- creases computation time.
Rollout Length L. For JaxNav, there is small reduction in performance by using rollout length of 1000, but no gain from using 4000 compared to 2000. In Minigrid, all values perform roughly the same, possibly due to the shorter episodes.
Buffer Size K In JaxNav, a smaller buffer outperforms larger ones, whereas in Minigrid, there is no significant difference between K = 100 and K 1000. This could relate to how easy the environment is (corresponding to how long it takes learn particular level). As JaxNav is harder than Minigrid (as it also involves low-level continuous control obstacle avoidance in addition to maze solving), it may be that training on each level more times is beneficial.
Buffer Update Period T Inereasing the time between updating the set of training levels reduces performance. However, sampling more often increases the computational load of the additional SFL rollouts.
Sampled Ratio In JaxNav, higher sampling ratio seems preferable, similarly the "robust" version of PLR [4]. In Minigrid, however, using p = 0.5 performs slightly better than 1.
Sampling in decreasing order of learnability For JaxNav, we trialled selecting levels in decreas- ing order of learnability rather than randomly among the top K. This performs better, and has roughly the same effect as reducing the buffer size. This makes sense, as by reducing the buffer size or selecting in decreasing order, we restricting the range of levels that can be chosen to only the nighest-learnabilee ones.
Number louth samp
JuPee
Befter Swmotng2rraag
Aeg ON hatt brak
20
N.. 25365
1000
5000
7000
Decressing
3%
10%
105%
15%
105%
1%
13%
150%
130%
15%
100%
1%
10%
10%
Figure 24: Analysing the effect of hyperparameters on single-agent JaxNav. Hyperparameters not mentioned in each plot use the default configuration's values: N 5000, T 50, p 0.5. K = 1000, L = 2000.
27

---

Rumber
Faltal Langh
Surter Sez
Suteriidev forid
Cemped Envoreato Reds
100
7:12
7:52
60
/e120
2-022
555
1-3000
002000
7-220
2=0.
5000
L=2000
Ke 2002
7w520
23000
1.4000
7w1200
Figure 25: Analysing the effect of hyperparameters on Minigrid. Hyperparameters not mentioned
in each plot use the default configuration's 's values: N = 5000, T = 100, = 0.5, K = =1000, L 2000.
1.2 UED with learnability as a score function
We now apply our learnability metric score function for UED. In particular, when performing our DR rollouts, we compute the learnability as in Section 4.3 and use that as the score function. Figure 26 shows these results; Learnability improves performance compared to MaxMC, supporting our claim that the score function is the primary limitation of current UED methods.
100
15
177%% unall Lifyt
M/L/PA, creitatte
51. (Naak andi outeen
ACCOulan
5/1 Seani snail lefton
KCE-
(a) PVL SFL's score (b) Learnabllity UED's (c) PVL SFL's score (d) Learnability UED's function in JaxNav. score function in JaxNav. function in Minigrid. score function in Minigrid.
Figure 26: Comparing SFL as a score function for UED and vere-versa (for single-agent JaxNav and Minigrid). In Figures (a. c). the large buffer is of size 1000 while the small is of size 100. We find that SFL with learnability outperforms all other combinations.
1.3 Different Definitions of Learnability
In this section, we investigate alternative definitions of learnability. All of the following functions have a learnability value of 0 for a success rate of 0.0 and 1.0, but differ in how scores are assigned to intermediate success rates p. We aim to assess whether our learnability score's 's main contribution is, in fact, just excluding unsolvable and perfected levels; our analysis indicates that this is not the case.
First, on JaxNav, we investigate the case where the peak learnability is not at p 0.3, but at other values. To do this, we represent the learabiliiyy function as a piecewise quadratic, see Figure 27a for an illustration. During this test, the peak learnability value remained at 0.25 but we varied the success rate at which the peak occurs. The results are in Figure 27b, and we find that a peak of 0.6 performs the best, and that performance slightly degrades as the peak moves towards higher success rates. In particular, performance poor when the peak at p 0.99.
In Figures 27c and 27d, we consider other definitions of lcarnability on JaxNav and XLand. We keep the restriction that learnability equals zero when p or p = The first is Uniform, where all levels with success rates O- < < are assigned an equal score. Linear(0) has learnability linearly increasing from 1.0 to 0.0, and Linear(1) has learnability linearly increasing from 0.0 to 1.0.
We find that Linear(O) (which is an approximation of true regret) performs similarly to the default definition of learnability on JaxNav but struggles XLand. Mcanwhile, Uniform sampling performs worse than our approach but still outperforms all UED methods on JaxNav and and (albeit by smaller margin than SFL.) Finally, Linear(1) performs significantly worse on JaxNav but is the closest to SFL's performance on Xland. These results indicate that our chosen definition of the lcarnability score function is more robust than these alternatives across different domains.
28

---

0.25
0.2
0.15
Peakm0.2 2
Leamability/Ptake-0
Leamssility/Pcaks05555
0.10
Peak=0.5
Learnabilit/Peekk00..
Peak=0.6
warmabiltty/Paaa-0..)
Leamaallity(takk-0..5
Peak=0.3 0.9
Leamability/Ptakke0.44
Lesmmabilit(Peekk022
0.0
0.2
0.4
0.6
0.8
1.0
1%
10%
100%
Success Rate (o)
(a) Learnability Curves
JaxNav: Different Peaks
100
SRL
25
Lineart0:
80
Linear(1)
20
Uniform
60
Linear(0)
15
Lineart1)
40
10
Win Rate
Win Rate
20
10%
100%
10%
100%
(c) Jaxlaav Other Definitions.
(d) XLand: Other Definitions.
Figure 27: Other learnability definitions.
29

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The contributions we the introduction are our paper's analysis and experimental section.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations of our work in Section
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: The paper contains no theoretical results.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Our introduced domain is fully detailed in Section 3 while all hyperparmeters used are provided in the Appendix.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: Several of the results use already open-sourced codehases and the remainder of the code will be released on GitHub shortly after this submssion.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
This information where not stated in the experiments section is included the Appendix.
Justification:
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: All results are run over multiple seeds with error bars reported.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time exccution) needed to reproduce the experiments?
Answer: [Yes]
30

---

Justification: We include a comparison of compute time for the single-agent JaxNav, and list the rough resources used for the multi-agent experiments in Appendix G.
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: htt:://nuurrppp cwtpurppsccppuuuiu Sppublic//thhssuuideeee.
Answer: [Yes]
Justification: We have ensured that research conforms the code of ethics.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There are no societal impacts of this work as references foundational research and the real world application is tied (robotic navigation) is not linked to societal harm.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper does not pose such a risk.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite all external sources of assets and their licenses permit our use case.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our released code constitutes a new assets and will be well documented on GitHub to complement the documentation provided by this paper.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper did not invovle crowdsourcing research with human subjects. 15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper did not invovle crowdsourcing or research with human subjects.
31


Reviews:
[{"review_id": "VXY6TyzpXc", "reviewer": "Reviewer_Su34", "summary": "The paper introduces a new metric based on solve rates for evaluating the learning potential of tasks in a multi-task RL environment. It also presents a new evaluation protocol based on worst-case performance to better characterize the robustness of different methods. The work uses this protocol to compare their method with several Unsupervised Environment Design baselines and concludes that it outperforms the baseline. They also find that their UED baselines perform worse than Domain Randomization on this new evaluation metric, and conduct additional experiments to confirm and analyze these results.", "questions": "* Why did you not simply use your heuristic instead of PVL or MaxMC in prioritized level replay?\n* Why does there appear to be a discrepancy between the UED baseline solve rates reported in this paper and the ones reported in [2]?\n* On line 214 you describe sampling 10,000 solvable levels to evaluate on. Are these unseen levels (at least with high probability) or are they sampled from the training set?\n* You mention in the Limitations section that your method is restricted to Jax-based settings. Is the method limited to those settings, or just your particular Jax-based implementation?", "limitations": "The authors adequately discuss the limitations in the paper and throughout the work. Their SFL method appears to be less general that the UED baselines that they compare against, but performs much better on the domains tested in this work. Simple methods on new benchmarks can serve as stepping stones to more general methods, so I do not believe this impacts their contribution.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper includes many contributions, including a new method, a new evaluation protocol, and new benchmarks of existing methods. These contributions are open sourced and well documented in the appendix to support reproducibility. The work attempts to identify failure modes of a large class of regret-based UED algorithms which could help to move further ACL research toward more promising directions. These methods are effective and widely applicable, but their generalization outside of a small set of baselines has not been thoroughly studied. This work will become increasingly important as RL research moves on to more complex environments.", "weaknesses": "The main weakness of the paper is the claim that domain randomization and their new heuristic outperform all UED algorithms, which is not entirely convincing from the results. The PLR hyperparameters used in this work are different from [1] and [2], for example the prioritization, temperature, and number of edits are all different. PLR in particular has several hyperparameters that need to be tuned for new domains, such as the buffer size, sampling temperature, and staleness coefficient. The authors do not explain whether they tuned the hyperparameters for the baselines or how they tuned the hyperparameters for their own method. I think even a small grid search over reasonable parameters that worked in other domains would help to make the comparisons throughout this paper more convincing.\n\nAnother concern is that this work does not include any standard episodic return plots, which is the main metric of comparison in RL. Without these plots, it is difficult to tell whether their method truly outperforms the baselines, or whether their baselines are properly tuned. For instance, if the SFL algorithm performs better according to the CVaR evaluation but not on a standard test return plot, then it's debatable which method is more useful. That being said, [2] does compare the mean evaluation solve rate of UED baselines on Minigrid, and finds that Accel and PLR both significantly outperform DR. In this work, PLR severely underperforms DR using the same metric in Figure 3. This seems to be strong evidence that its hyperparameters or implementation are not correct.\n\nEven if we assume that the baselines are working as intended, I PLR and SFL have many differences which makes it unclear what change is actually leading to better performance. I think this work would benefit from additional experiments using the learnability metric in SFL as the prioritization metric in PLR, and possibly the alternative, using PVL and MaxMC as the selection metrics in the full SFL algorithm.\n\nOverall the writing and presentation was quite good, but there were a few notable issues. The paper somewhat confounds its description of PLR with Robust PLR in section 2.2.1. PLR trains on randomly sampled levels, while Robust PLR evaluates randomly sampled levels and only trains on levels from its replay buffer. Also, \"learnability\" is not clearly defined until 4.2, after it has been referenced many times. This is ok when discussing \"intuitive notions of learnability\" in a vague sense, but not when making statements such as \"no correlation exists between learnability and regret score\".\n\n[1] Jiang, Minqi, et al. \"Replay-guided adversarial environment design.\" Advances in Neural Information Processing Systems 34 (2021): 1884-1897.\n[2] Parker-Holder, Jack, et al. \"Evolving curricula with regret-based environment design.\" International Conference on Machine Learning. PMLR, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, thank you for your thorough review and helpful suggestions! We also appreciate you mentioning that our paper \"includes many contributions\" and \"could help to move further ACL research toward more promising directions.\" Please find our responses and changes below.\n\n# Hyperparameters and UED underperformance\n\nAs rightfully requested by the reviewer, we have performed extensive hyperparameter tuning for all methods (excluding DR). Using these new, optimised hyperparameters (see Figure 1 in the attached 1-page PDF), **UED outperforms DR** (in Minigrid and Single-agent Jaxnav) but **SFL still outperforms UED**. For DR, we use the PPO hyperparameters as discussed in the main response, and did not tune it further, as it has no other relevant hyperparameters. Please see our main response for details regarding how we performed our sweep.\n\nFurthermore, when comparing our results to previous work please note that in the original Robust PLR paper, they use 25 walls for Minigrid. However, they also show that changing the number of walls can drastically alter results; for instance, in appendix C.1.2, they show that using 50 walls causes DR to be competitive to UED. Following Minimax [1] and JaxUED [2], we use 60 walls. Therefore, we cannot directly compare the results of our work and that of the original Robust PLR & ACCEL papers.\n\n[1] Jiang, et al. \"Minimax: Efficient Baselines for Autocurricula in JAX.\"\n\n[2] Coward, et al. \"JaxUED: A simple and useable UED library in Jax.\"\n\n# Episodic Return Plots\n\nWe primarily include success on the set of held-out levels during training rather than episodic return because, in these goal-oriented domains, the success rate is a more intuitive and clearer marker of performance. This is because the actual environment reward may include reward shaping terms that do not directly correspond to our ultimate aim of reaching the goal (but are nonetheless necessary to learn a policy).\n\nWe also find that the average return in all our domains is closely tied to success rate. This is illustrated in Figure 5 of the manuscript where 5\\(c) reports success rate while 5(d) reports return (note, there is a naming error on the y-axis of Figure 5(d)).\n\n**Therefore, the performance trends illustrated in 3(b), 5\\(c), 7(b) are mirrored in the episodic return throughout training.** We have clarified this in the revised manuscript, and have included reward curves in the appendix.\n\n# Learnability as PLR score function and PVL as SFL's metric\n\nThank you for suggesting this, we agree that this is an important ablation to understand the source of improvement and have included these results in the attached PDF (Figure 3). \n\nWe ran this ablation for both Minigrid and single-agent JaxNav, and in both domains, **SFL + Learnability outperforms all other combinations**. Using learnability with PLR/ACCEL has a slight positive effect for PLR in JaxNav and a negative effect in Minigrid. Using PVL as the score function inside our SFL algorithm performs worse, except when we have a large buffer of levels in JaxNav. \n\nSome intuition for why this is:\n- In the large buffer case for PVL on JaxNav, the training levels are more random, since we also include levels with lower scores. Indeed, with a small buffer, PVL performs very poorly, a result in line with our analysis in Figure 2 of the submitted manuscript as PVL fails to predict the frontier of learnability accurately.\n- Conversely, a large buffer results in a set of levels that is slightly biased towards higher success rates, as illustrated in Figure 2 of the submitted manuscript (i.e., PVL correlates slightly with success rate). This bias helps screen out some unsolvable levels using the scoring function, a feature of regret-based UED algorithms [3]. Additionally, our observation shows that random JaxNav levels often provide a good learning signal, as demonstrated by a sampling ratio $\\rho$ of 0.5 performing similarly to 1.0 in the SFL ablations shown in Figure 2 of the 1-page PDF. Therefore, SFL-PVL with a large batch of levels creates a learning curriculum that leads to decent performance, although it is still inferior to SFL with learnability.\n\n\n**The results of this ablation suggest that the improvements due to SFL can be attributed to both the learnability score function *and* our improved sampling approach.**\n\nNote, however, that we did not hyperparameter-tune these results, as we used the optimised hyperparameters found above, and just swapped the score function.\n\n[3] Jiang, et al. \"Replay-guided adversarial environment design.\" \n\n# Poor explanations and presentation\n\nThank you for pointing out these problems, we appreciate your thoroughness. We have rectified them in our updated manuscript. We have also defined learnability earlier on, making the paper clearer.\n\n# Questions\n> Using our heuristic with PLR?\n\nSee above and Figure 3 in the attached 1-page PDF. We find that SFL + Learnability outperforms the other combinations.\n\n> A discrepancy between UED's performance in this paper and [2]?\n\nSee above, in \"Hyperparameters and UED underperformance\".\n\n> Are the 10,000 solvable levels unseen?\n\nThe levels used during evaluation are sampled from our environment generator separately from the training process. As the space of possible levels is very large, generating the same random level twice is unlikely.\n\n> Is the method restricted to Jax-based settings?\n\nOur implementation is in JAX but the method is general. However, as raised in our limitations section, one must take the cost of SFL's additional environment rollouts into account when considering implementing our algorithm; we chose JAX because its speed and parallelisation significantly alleviates this constraint.\n\n# Conclusion\nWe hope that the reviewer feels we have addressed their questions and welcome any further discussion. We also ask that, if all their concerns are met, the reviewer consider revising their score to recommend accepting the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YkfoaTMUxf", "reviewer": "Reviewer_gAKk", "summary": "This work proposes a new Unsupervised Environment Design (UED) method, called Sampling for Learnability (SFL), developed for navigation tasks. SFL upsamples environment configurations with high learnability scores, i.e., p*(1-p), where p is the success rate in a specific configuration. The paper highlights the failure of some of the state-of-the-art (SOTA) UED approaches on such tasks and argues that they cannot approximate learning agents' regret, unlike SFL. They evaluate SFL in JaxNav, a Jax-based single and multi-agent navigation benchmark they introduce, and MiniGrid. Their empirical results evidence that SFL is robust and can outperform Domain Randomisation (DS) and SoTA UED approaches in JaxNav and MiniGrid. The paper demonstrates these outcomes by following two evaluation protocols, respectively: 1) A risk-based protocol that they introduce, computing the conditional value-at-risk (CVaR) of the distribution of success rate based on randomly sampled levels, and 2) a common protocol in the UED literature that evaluates performance/success in hand-designed test sets, as they focus on complex, yet, solvable configurations.", "questions": "Table 4 showcases compute times for each evaluated approach. Most of these approaches have similar components, except maybe SFL's extra rollout phase. So why is SFL one of the fastest?", "limitations": "- The major limitation of the proposed approach is the rollouts needed to compute the learnability score of randomly sampled levels. Section 4.2 indicates that 5000 levels are sampled, and n-step rollouts, where n is not specified, are generated to compute the scores. It would be more informative if the amount of computation and time spent in this phase were reported (in comparison to the usual curriculum learning task sampling phase). In addition, I wonder how SFL is one of the fastest in the single agent nav environment, as reported in Table 4 in Appendix F, despite having this additional phase.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper is well-written, clearly expresses the motivation and the gap in the literature, and illustratively analyses the failure of existing UED methods in navigation tasks. \n\n- Their contribution is a simple yet novel UED approach called SFL that explicitly addresses this failure mode. Their empirical results through two evaluation protocols highlight that SFL is robust (in terms of CVaR-based metrics) and outperforms existing approaches in expected return/success.\n\n- I agree with the authors' claim that existing UED methods claim robustness yet fail to quantify it accurately. Hence, the robustness evaluation protocol utilized in this work may be considered new in the UED literature.", "weaknesses": "- Section 4.1 analyses MaxMC and PVL, popular UED score functions, in terms of their predictiveness of learnability. However, there is no demonstration of a similar analysis for SFL. Although SFL's scoring function is intuitive and it is not as challenging to guess what the scoring would look like, including such an illustrative comparison would support the claims in this paper.\n\n- The empirical results according to the proposed evaluation protocol for robustness (Figures 3a, 5a, and 7a) show that SFL outperforms the rest of the evaluated methods for alpha < 1 (100%). However, except in the multi-agent case (Fig 5a), the expected success rates indicate marginal or no improvement. In MiniGrid, the difference between DR and SFL drops quickly as alpha gets above 0.1 (10%). This is likely due to SFL not doing so well in easier levels compared to other baselines. Although this is expected, as the proposed metric results in upsampling learnable environments, not easy ones, I suggest a comparison of results in easy levels to conclude whether this is the case or not.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments, especially for highlighting our \"simple yet novel UED approach\" and that our paper is \"well-written\".\nBelow, we address the issues raised in the review:\n\n# Correlation analysis for SFL\n\n>Section 4.1 analyses MaxMC and PVL, popular UED score functions, in terms of their predictiveness of learnability. However, there is no demonstration of a similar analysis for SFL. Although SFL's scoring function is intuitive and it is not as challenging to guess what the scoring would look like, including such an illustrative comparison would support the claims in this paper.\n\n\nWe agree such analysis would be helpful, and we have added this to our updated manuscript. Due to space constraints we have not included this in the attached rebuttal PDF.\n\n\n# Easy Levels\n\nTo assess performance on easy levels we have run our evaluation procedure over 10,000 uniformly sampled levels with fewer obstacles than usual. For JaxNav, we used a maximum fill % of $\\leq 30\\%$, half of the standard 60%. Meanwhile, for Minigrid, we use a maximum number of 30 walls instead of 60. These levels, therefore, are generally easier than the levels we evaluated on in the main paper. Results are reported in Figure 4(a,b) of the attached PDF. \n\nOn JaxNav, SFL still demonstrates a significant performance increase while on Minigrid all methods are very similar. Due to the challenging dynamics of JaxNav, even levels with a small number of obstacles can present difficult control and navigation problems meaning Automated Curriculum Learning (ACL) methods (such as SFL) still lead to a performance differential over DR. Meanwhile, in Minigrid, due to its deterministic dynamics, difficulty is heavily linked to the obstacle count as this allows for more complex mazes. As such, DR is competitive to ACL methods in settings with fewer obstacles.\n\n# SFL runtime speed\n\nThank you for raising this important point, as it highlights how SFL takes advantage of parallel computations on the GPU.\n\nLooking at a single iteration (including training + eval) in Minigrid on an L40S GPU, the time breakdowns are as follows:\n|                      | PLR   | SFL   |\n| -------------------- | ----- | ----- |\n| Train Step           | 37.5s | 35s   |\n| Get Learnable Levels | 0     | 2.2s  |\n| Eval Step            | 0.7s  | 0.7s  |\n| Total                | 38.2s | 37.9s |\n\nWe note that the SFL rollouts are fast for two reasons:\n- We aggressively parallelise them, running up to 5000 environments in parallel, which takes about the same time as running only hundreds in parallel.\n- We do not compute any gradients for these transitions.\n\nThe upshot is that they take significantly less time than the actual training step. Furthermore, UED's training step is more complex than SFL's, since it must maintain a buffer of levels, compute the scores during training, and potentially update the buffer.\n\nFor JaxNav, obtaining the learnable levels takes a little longer, but still takes less time than our logging. Due to this, slight differences in logging between PLR and SFL masked this additional computational cost. To better control for this, we measure how long SFL takes vs PLR in the absence of logging. We find that SFL is about 6% slower than PLR, despite running all of the additional rollouts. In Figure 1(a) of the attached PDF (and our updated manuscript), **we have provided compute-time matched results** for single-agent JaxNav (where SFL runs for 6% fewer timesteps). Despite running for fewer timesteps, SFL still significantly outperforms UED methods. SFL on Minigrid runs as fast, or slightly faster than PLR.\n\nWe apologise for the oversight in not specifying how long our rollouts are. We use a consistent rollout length of 2000 steps. We have updated this in our manuscript.\n\n# Conclusion\nWe hope we have addressed the reviewer's comments and are happy to discuss any of these points further. We would further ask if our responses have addressed the reviewer's concerns, and that they consider increasing their support for our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CANIiAPRWM", "reviewer": "Reviewer_Ns9r", "summary": "This experimental paper proposes a UED method (SFL) for JaxNav, a continuous single- and multi-robot navigation task in a partially observable setting.  The authors document the shortcomings of UED methods, (Domain Randomisation, Prioritised Level Replay, and ACCEL) on this partially observed, continuous action, continuous state setting.  The authors provide a demonstration that the scoring mechanism used by the above UED methods is misguided.  Their method, SFL, uses a “learnability score” which focuses learning on levels for which agents achieve success rates closer to 50%.   To better focus on robustness and generalization, the authors develop an empirical CvaR measure of success for evaluation and comparison of methods.", "questions": "Lines 15-17.  Suggest to delete spurious statements such as “We had tried our best to make current UED methods work for our setting before going back to the basics and developing our new scoring function, which took a few days and worked out-of-the-box.  We hope our lessons and learnings will help others spend their time more wisely.”   Don’t complain!\n\nThe authors should make an effort to assist readers familiar with RL but not ACL methods by providing details in the appendix.  For example in line 175, Algorithm 1 does not provide information on how \\phi is updated.  The update algorithm could be provided in the appendix.\n\nFigure 2 includes statements about “learnability”  prior to a definition of learnability.  \n\nLines 154-157.  Learnability has not yet been defined so statements about slight/no correlation with learnability are vacuous.   It is not clear to reader whether learnability in these lines/Figure 2 is the same as the definition of learnability given later (and used by SLR).\n\nLines 133-135 might better go in the later section on weaknesses/limitations.\n\nLine 214.  The authors do not define “solvable.”  The reader needs to know how you are operationalizing this term.  \n\nLine 274-5.  Unclear what “perfect regret” means.", "limitations": "Limitations are nicely discussed.  It would be cool if the authors could comment on how SFL might be generalized to continuous outcomes and stochastic environments.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Scientists who are interested in developing and implementing the RL agents in real life will find this experimental paper important. \n\nThe paper is technically sound and provides simulation support for the claims.   The authors provide a nice discussion of the weaknesses of SFL (proposed version works only for binary outcome, deterministic environment).  I have not gone over the github site;  aside from the points made below, replicability is good. \n\nThe learnability score is original (obvious after reading the paper, but perhaps not before).  This score nicely takes advantage of the JaxNav environment which provides fast training of an RL agent.  Related work is adequately cited and it is very clear how SFL differs from the 3 UED methods discussed.", "weaknesses": "See questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review, particularly highlighting that our paper is \"technically sound\" and that the learnability score is \"original\". Please find our responses to your comments below.\n\n\n> Lines 15-17. Suggest to delete spurious statements such as “We had tried our best to make current UED methods work for our setting before going back to the basics and developing our new scoring function, which took a few days and worked out-of-the-box. We hope our lessons and learnings will help others spend their time more wisely.” Don’t complain!\n\nWe agree and have removed this from our manuscript.\n\n> The authors should make an effort to assist readers familiar with RL but not ACL methods by providing details in the appendix. For example in line 175, Algorithm 1 does not provide information on how \\phi is updated. The update algorithm could be provided in the appendix.\n\nThank you for your suggestion and we agree that we should provide a more detailed background on ACL in our appendix. On line 175, $\\phi$ is updated using any RL policy learning algorithm (we use PPO) and as such is not affected by the ACL process, we will make this distinction clearer in our updated manuscript.\n\n> Figure 2 includes statements about “learnability” prior to a definition of learnability\n\n> Lines 154-157. Learnability has not yet been defined so statements about slight/no correlation with learnability are vacuous. It is not clear to reader whether learnability in these lines/Figure 2 is the same as the definition of learnability given later (and used by SLR).\n\nThank you for pointing this out! We apologise for this oversight and we have rectified this by defining learnability earlier on in our paper.\n\n> Lines 133-135 might better go in the later section on weaknesses/limitations.\n\nWe appreciate this suggestion and have implemented this in the updated manuscript.\n\n> Line 214. The authors do not define “solvable.” The reader needs to know how you are operationalizing this term.\n\nWe added an explanation to the manuscript., thank you for bringing it to our attention. \"Solvable\" in our paper means that the goal state can be reached in a particular level, i.e., it is not impossible to complete.\n\n> Line 274-5. Unclear what “perfect regret” means.\n\nRegret is defined as the difference in return between the optimal policy on a level, and the current policy. However, since it is intractable in practice to compute this, most methods use approximations (such as PVL and MaxMC). We use \"perfect regret\" to refer to the exact implementation of regret, which is sometimes possible (e.g. in gridworlds we can compute the optimal return). We have made this clear in our updated manuscript.\n\n>  It would be cool if the authors could comment on how SFL might be generalized to continuous outcomes and stochastic environments.\n\nHere are just some possible options to explore. We have added this discussion to our updated manuscript, thank you for raising this point as an important step of future work!\n\n- In a non-binary-outcome domain, we could potentially reuse the intuition that $p(1-p)$ can be seen as the variance of a Bernoulli distribution; therefore, in a continuous domain, an analogous metric would be the variance of rewards obtained by playing the same level multiple times.\n- For stochastic environments, we could form an extended level space, where the random seed and the level $\\theta$ are bundled into $\\tilde \\theta$. In this way, we could turn a stochastic environment into a deterministic one, and since we assume simulator access already, this is not a significantly stronger assumption.\n\n# Conclusion\nThank you again for your helpful review, we hope we have addressed your comments satisfactorily, and welcome further discussion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6zHwN92Ppn", "reviewer": "Reviewer_ycoo", "summary": "The paper investigates the limitations of popular UED approaches, such as PLR and ACCEL, demonstrating that they do not improve upon the Domain Randomisation baselines, where levels are randomly sampled. The author's main claim is that learnability does not correlate with the scoring functions MaxMC and PVL used by current UED approaches, leading to sub-optimal performance.\n\nTo address this issue, they propose Sampling For Learnability (SFL), a simple algorithm where levels are chosen to prioritize learnability. In the context of the paper, the learnability of a level is defined as $p \\cdot (1-p)$, where p is the success rate of the agent on that given level.\nThe authors conduct experiments on three different environments: MiniGrid, Single, and Multi-Agent JaxNav. As a metric, they propose the use of the conditional value at risk (CVaR), evaluating on an α% of levels on which the agent performs worst, from a randomly generated buffer of levels.\n\nThe results show that SFL outperforms current UED approaches. Simultaneously, when it is possible to compute it, true regret outperforms SFL, indicating that the flaw of UED algorithms is indeed in the scoring function approximation. To summarize, the main contributions of the authors are:\n\n- Demonstrate sub-optimality of current UED approaches, due to lack of correlation between their scoring function and learnability\n- Propose a new algorithm, SFL, which prioritizes levels with high learnability", "questions": "- In Algorithm 1, $\\rho$ is set to $0.5$. Hence, half of the levels used for updating the policy $\\pi_\\phi$ are ones with high learnability, and half are sampled at random. Why do not set $\\rho$ to a higher value? I would like to see a plot showing the performance of SFL upon varying $\\rho$, so as to have a better idea of how indeed training in 'more learnable' environments correlates with better performances\n\n- In Alg.1, what are the performances if instead of sampling uniformly from the high-learnable levels $\\mathcal{D}$, you select levels in decreasing order of learnability? \n\n- What is the average/median learnability of the randomly created levels in Alg 2? What about the top-K ones? It is important to see and report it so indeed one is sure that the levels you are using are indeed ones with high-learnability. As far as I can infer from the paper, it could still be that most of the randomly created levels have low learnability.\n\n- Can you provide more details on what you did to make current UED methods work better? The poor performance of UED algorithms was already highlighted in the JAXUED paper, but I am not sure if this is just because they were not optimized fully, e.g. via a more extensive hyper-parameter search\n\n- Did you try using different definitions of Learnability? How did they perform?", "limitations": "No ethical limitations", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "- UED has indeed been seen to perform sub-optimally in practice, i.e. as evidenced in the JaxUED paper. It is thus an important problem to address since limited work has been done\n\n- The investigation of the correlation between learnability and the scoring function is a good justification for the paper itself to exists, and gives a plausible reason for the poor performances of UED on some tasks.\n\n- The SFL algorithm proposed is novel (as far as I know), simple, and easy to understand. The results obtained on the 3 environments they considered seem promising.\n\n- CVaR seems a more principled and justified way to investigate the robustness of Auto-curricula algorithms", "weaknesses": "- Learnability definition: why are environments that are solved exactly half of the time the most valuable ( given the $p(1-p)$ function used to define it)? It seems an arbitrary choice, which should need to be justified better through additional experiments. I do not agree that levels that are solved 5% of the time are as valuable as ones solved 95% for example.\n\n- Many parameter choices are just reported with no justification supporting it ( e.g. hyperparameter $\\rho$ for the algorithm)\n\n- Experiments are poor in broadness. Given the purely empirical nature of the paper, I would expect way more experiments to indeed show that SFL outperforms the UED baselines. This includes more seed for the Multi-agent JaxNav, and more environments. As of now, it is not possible to actually confirm that SFL is indeed better than UED algorithms.\n\n- The writing is poor and sometimes confusing. Learnability is referenced way before being formally defined, the syntax of some sentences is poor, etc... By reading the paper, it seems the writing has been rushed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thorough review and useful comments, particularly highlighting that we are focusing on an \"important problem\", and that our algorithm is \"simple, and easy to understand\". Please find our responses below.\n\n# Weaknesses\n## Different Definitions of Learnability\n\nIntuitively, we justify our definition as follows (in a goal-based setting where there is only a nonzero reward for reaching the goal):\n- $p$ represents how likely the agent is to obtain positive learning experiences from a level.\n- $1-p$ represents the maximum potential improvement the agent can make on that level.\n- Multiplying these yields (probability of improvement) * (improvement potential), i.e., expected improvement.\n\n\n$p(1-p)$ can also be seen as the variance of a Bernoulli distribution with parameter $p$, i.e., how inconsistent the agent's performance is.\n\nWe have added this explanation to the updated manuscript to increase clarity.\n\nTo confirm this choice we have run a set of experiments on single-agent JaxNav where we represent the learnability function as a piecewise quadratic. During this test, the peak learnability value remained at 0.25 but we varied the success rate at which the peak occurs. The results are in Figure 4\\(c) of the attached PDF; overall, a peak around $p=0.5$ performs the best, and moving away from this reduces performance slightly. However, as performance is not dramatically worse, this indicates that the most important aspect of learnability is in rejecting levels with perfect or zero success rates.\n\n## Hyperparameter choices / Ablations\n\nThank you for raising this point. We have tuned all of the hyperparameters, and have run ablations on SFL. **Our conclusions remain unchanged (i.e., that SFL outperforms UED)**, but we have more insights now into which parameters are most important for SFL. Furthermore, we find that SFL is less sensitive to hyperparameters than PLR or ACCEL. The new tuned results are in Figure 1, and the ablations are in Figure 2 of the 1-page PDF.\n\n## More seeds and environments\n\nThank you for raising this point. We have run more seeds for multi-agent JaxNav, bringing the total count to 5, and our results have not changed. We have also included results on an additional environment, XLand-MiniGrid, as described in our global response. Here, SFL also outperforms PLR and DR.\n\nWe now **have results in four environments demonstrating SFL's superiority compared to SoTA UED methods**. We further compare against several ablations. We believe these results are sufficiently broad to showcase SFL's performance.\n\n\n## Writing\n\nThank you for raising the writing quality issues, and we apologise for letting them slip through. We have carefully reviewed the submitted manuscript and have fixed these issues for our updated version.\n\n# Questions\n## Different values of the sampling ratio $\\rho$\n\nWe agree that this is an important question and we have tested various values of $\\rho$ on single-agent JaxNav with results shown in Figure 2 of the attached PDF. Increasing $\\rho$ beyond 0.5 increases performance slightly. \nDue to the high maximum fill % used for JaxNav, we hypothesise that the randomly generated levels are able to present challenging control and navigation problems. \nWhile increasing $\\rho$ makes the agent train on more learnable levels, it may also decrease the diversity of the levels seen by the agent.\n\n## Decreasing order of learnability\n\nThank you for this suggestion, we agree it is a useful question to answer. We have trialled selecting levels in decreasing order of learnability rather than randomly and, as illustrated in Figure 1 of the attached PDF, this has roughly the same effect as reducing the buffer size. \nThis makes sense, as by reducing the buffer size or selecting in decreasing order, we are restricting the range of levels that can be chosen to only the highest-learnability ones.\n\n## Average vs top-k learnability\n\nFor 45 iterations for single-agent JaxNav, we logged the average and median learnability of the entire set of sampled levels, and the subset we use to train on, with results below. In short, **the entire set of sampled levels generally has low learnability, whereas the top 100 levels have much higher values**. This shows that SFL is filtering out levels with very low learnability (i.e., levels that the agent either already perfectly solves or that are currently too hard). This result further explains why we outperform existing methods. We appreciate you raising this point and have added this analysis to the manuscript.\n\n|                  | Mean All   | Median All | Mean Subset | Median Subset |\n|------------------|------------|------------|-------------|---------------|\n| Value            | 0.014 | 0.0        | 0.198  | 0.195    |\n\n\n## What we did to make UED better\n\nFor the existing UED methods, we used the open-source implementations provided by JaxUED and conducted a thorough sweep of hyperparameters, as explained in our general response. Additionally, as JaxUED showed, and as we've also found, DR can be surprisingly competitive if the PPO parameters are tuned appropriately. Indeed, in the multi-agent setting of JaxNav, DR is in line with or outperforms all existing UED methods during CVaR evaluation; albeit the UED methods use hyperparameters tuned on the single-agent JaxNav setting.\n\n## Different learnability definitions?\n\nFollowing your suggestion, we have experimented with different peak values, as outlined above. We have not tried an altogether different definition because ours followed from the intuition explained above. However, our code will be open-sourced, allowing the community to easily try alternative definitions.\n\n\n# Conclusion\nThank you once again for your detailed review. We hope our responses have addressed your concerns, and we welcome any further discussion or questions you might have. If we have successfully alleviated your concerns, we kindly ask you to consider updating your score to recommend accepting our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear reviewers, we appreciate your detailed reviews and concrete suggestions for improvement. We are especially grateful for reviewers mentioning that our paper is "well-written", "technically sound", and "includes many contributions", including an algorithm that is "novel, simple, and easy to understand." We are pleased that reviewers recognise the "simulation support" for our claims and that all "contributions are open-sourced and well documented in the appendix to support reproducibility".

We have implemented your suggested improvements, and run additional experiments, which **confirm our findings that SFL outperforms current UED methods on several domains.**

In our global response, we highlight the main changes we made, with updated and additional results presented in the 1-page PDF of figures.

# New Environment - XLand-Minigrid
*Requested by Reviewer ycoo.*

We have added an additional environment to our experiments by running DR, SFL and PLR on Xland-Minigrid's meta-RL task [1]. See below for details.

**Results**
We report performance using our CVaR evaluation procedure and, in line with [1], as the mean return on an evaluation set during training. Our results are presented in Figures 1(d) and 4(d) of the attached PDF. **SFL outperforms both PLR and DR.** Results are averaged over 5 seeds and during evaluation each ruleset was rolled out for 10 episodes. Training runs were conducted on one L40S and due to the large number of levels being rolled out to fill SFL's buffer, SFL was slower than DR and PLR. As such, we report results for SFL compute-time matched to PLR. Not only does SFL outperform PLR for their respective best set of hyperparameters but it is also much more robust to hyperparameter choice, with only one configuration of PLR's hyperparameters being competitive compared to the large majority of SFL's. More details on this experiment are in our updated manuscript.


**Environment Overview**
This domain combines an XLand-inspired system of extensible rules and goals with a Minigrid-inspired goal-oriented grid world to create a domain with a diverse distribution of tasks. Each task is specified by a ruleset, which combines rules for environment interactions with a goal, and [1] provide a database of presampled rulesets for use during training. Following [1], we use a 13x13 grid with 4 rooms and sample rulesets from their high diversity benchmark with 3 million unique tasks. As training involves sampling from a database of precomputed rulesets, ACCEL is not applicable. PLR and SFL select rulesets for each meta-RL step to maximise return on a held-out set of evaluation rulesets.


**Hyperparameter tuning**
All methods used the default PPO hyperparameters given by [1]. For both PLR and SFL, we performed a grid search using similar compute budgets. Due to space, we cannot provide details here but have in our updated manuscript.

[1] Nikulin, et al. "XLand-minigrid: Scalable meta-reinforcement learning environments in JAX."

# Extensive Hyperparameter Tuning
*Suggested by Reviewers Su34 and ycoo.*

To ensure competitive results, we have performed extensive hyperparameter tuning for each baseline. Updated results are shown in Figure 1 of the 1-page PDF. **We find that SFL still significantly outperforms existing UED methods.**

For PPO we conducted an extensive sweep for the JaxNav environment ensuring robust DR performance. For Minigrid, our JaxNav PPO parameters performed similarly to those given in the JaxUED implementation but allowed us to use 256 environment rollouts in parallel during training compared to JaxUED's 32. We kept these base PPO parameters fixed for all methods.

For both PLR and ACCEL, we swept hyperparameters for Minigrid and JaxNav's single-agent variation. For both methods, we performed a grid search over the replay $p$, in $\{0.5, 0.8\}$, level buffer size $K$, in $\{1000, 4000, 8000\}$, choice of scoring function in $\{ \text{PVL}, \text{MaxMC}\}$ and prioritisation function in $\{ \text{rank}, \text{TopK}\}$. For rank prioritisation, we searched for the temperature in $\{0.3, 1.0\}$, while for TopK we searched for k in $\{1, 32, 128\}$. For ACCEL, we additionally searched for the number of edits in $\{5, 20, 50\}$. Each set of hyperparameters tested was run over 3 seeds. For multi-agent JaxNav, we used the best-performing hyperparameters for single-agent JaxNav.

For SFL, we started with the default parameters listed in the main text and conducted an independent line search over the parameters: batch size $N \in \{ 500, 5000, 25000\}$, rollout length $L \in \{1000, 2000, 4000 \}$, number of levels to save $K \in \{100, 1000, 5000 \}$, buffer update period $T \in \{10, 50, 500, 1000, 2000 \}$ and sampled environments ratio $\rho \in \{0.25, 0.5, 0.75, 1.0\}$. 
Since this is a line search and not a grid search, the total number of tuning runs (and total compute) is significantly less than for PLR and ACCEL (20 runs for SFL vs 90 for PLR & 270 for ACCEL).

# Additional Baselines
*Suggested by Reviewer Su34.*

We have added two new baselines (see Figure 3 in the 1-page PDF):
1. Positive Value Loss (PVL) as a score function for SFL.
2. Learnability as a score function for PLR and ACCEL.

**SFL+Learnability still outperforms both of these variations.**

# Ablations and Effects of SFL Hyperparameters
*Suggested by Reviewer ycoo.*

In Figure 2 of the 1-page PDF, we have investigated the effects of changing hyperparameter values on SFL. We also experimented with different variations of learnability, where the highest learnability score corresponds to success probabilities other than 0.5, for instance, where solving a level 20% of the time is defined as "optimal" learnability (Figure 4\(c)).

Overall, SFL's performance improves when we sample more levels, and it is relatively robust to all other hyperparameters. There is a detrimental effect if we sample the set of learnable levels too infrequently (see the "Buffer Update Period" subfigure).

Meta-review: This paper introduces a novel metric for evaluating the learning potential of tasks in a multi-task RL environment. It also proposes a new evaluation protocol to better characterize the robustness of different methods, and compares its proposed method with several Unsupervised Environment Design baselines using this protocol. This paper shows that its proposed method outperforms the baselines.

This paper is very interesting and I believe that many researchers in the field of RL will find this paper important. All the reviewers recommend to accept this paper, and three reviewers have given a rating of 7. After reading the paper, the reviews, and the rebuttals, I agree with the reviewers and recommend to accept this paper.
Decision: Accept (poster)