CLUTR: Curriculum Learning via Unsupervised Task Representation Learning
Authors: Abdus Salam Azad
Izzeddin Gur
Aleksandra Faust
Pieter Abbeel
Ion Stoica
Venue: Submitted to ICLR 2023 (2023)
Keywords: reinforcement learning
curriculum learning

Abstract: Reinforcement Learning (RL) algorithms are often known for sample inefficiency and difficult generalization. Recently, Unsupervised Environment Design (UED) emerged as a new paradigm for zero-shot generalization by simultaneously learning a task distribution and agent policies on the sampled tasks. This is a non-stationary process where the task distribution evolves along with agent policies; creating an instability over time. While past works demonstrated the potential of such approaches, sampling effectively from the task space remains an open challenge, bottlenecking these approaches. To this end, we introduce CLUTR: a novel curriculum learning algorithm that decouples task representation and curriculum learning into a two-stage optimization. It first trains a recurrent variational autoencoder on randomly generated tasks to learn a latent task manifold. Next, a teacher agent creates a curriculum by optimizing a minimax REGRET-based objective on a set of latent tasks sampled from this manifold. By keeping the task manifold fixed, we show that CLUTR successfully overcomes the non-stationarity problem and improves stability. Our experimental results show CLUTR outperforms PAIRED, a principled and popular UED method, in terms of generalization and sample efficiency in the challenging CarRacing and navigation environments: showing an 18x improvement on the F1 CarRacing benchmark. CLUTR also performs comparably to the non-UED state-of-the-art for CarRacing, outperforming it in nine of the 20 tracks. CLUTR also achieves a 33% higher solved rate than PAIRED on a set of 18 out-of-distribution navigation tasks.

Under review as a conference paper at ICLR 2023
CLUTR: CURRICULUM LEARNING VIA UNSUPER- VISED TASK REPRESENTATION LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
Reinforcement Learning (RL) algorithms are often known for sample inefficiency and difficult generalization. Recently, Unsupervised Environment Design (UED) emerged as new paradigm for zero-shot generalization by simultaneously learn- ing a task distribution and agent policies on the sampled tasks. This is a non- stationary process where the task distribution evolves along with agent policies; creating an instability over time. While past works demonstrated the potential of such approaches, sampling effectively from the task space remains an open chal- lenge, bottlenecking these approaches. To this end, we introduce CLUTR: novel curriculum learning algorithm that decouples task representation and curriculum learning into two-stage optimization. It first trains recurrent variational au- toencoder on randomly generated tasks to learn latent task manifold. Next, a teacher agent creates curriculum by maximizing minimax REGRET-based ob- jective on set of latent tasks sampled from this manifold. By keeping the task manifold fixed, we show that CLUTR successfully overcomes the non-stationarity problem and improves stability. Our experimental results show CLUTRR outper- forms PAIRED, a principled and popular UED method, in terms of generalization and sample efficiency in the challenging CarRacing and navigation environment showing an 18x improvement the Fl CarRacing benchmark. CLUTR also per- forms comparably to the non-UED tate-of-the-- art for CarRacing, outperforming it in nine of the 20 tracks. CLUTR also achieves 33% higher solved rate than PAIRED on a set of 18 out-off df-diittibtttinn navigation tasks.
INTRODUCTION
Deep Reinforcement Learning (RL) has shown exciting progress in the past decade solving many challenging domains including Atari (Mnih al. (2033). Dota (Berner et al. (2019). Go (Silver) et al. (0166). However, deep RL is cample-ineffeiiem Moreover, out-of-box deep RL agents often brittle: performing poorly on tasks that they have not encountered during training, or often failing solve them altogether even with the slightest change ( [Cobee et al. (2019), Azad et al. (022).nhang et al. (2018)). Curriculum Learning (CL) algorithms showed promise to improve (Portelas et al. (2020). Narvekar et al. (2020) RL sample efficiency by employing teacher al- gorithm that attempts to train the agents on tasks falling at the boundary of their capabilities, i.e., tasks that are slightly harder than the agents can currently solve. Recently, a class of unsupervised CL algorithms, called Unsupervised Environment Design (UED) (Dennis et al. (2020) Jiang et al. (2021a)]. has shown impressive generalization capabilities which require no training tasks as input. UEDs automatically generate tasks by sampling from the free parameters of the environment (e.g.. the start, goal, and obstacle locations for navigation task) and attempt to improve sample efficiency and generalization by adapting diverse task distribution at the agent's 's frontier of capabilities.
Protagonist Antagonist Induced Regret Environment Design (PAIRED) (Dennis et al. (2020) is one of the most principled UED algorithms. The PAIRED teacher is itself RL agent with actions denoting different task parameters. PAIRED aims at generating tasks that maximize the agent's 's re- gret, defined the performance gap between optimal policy and the student agent. Theoretically, upon convergence, the agent learns to minimize the regret, i.c., will solve every solvable task. Such robustness guarantee makes regret-based teachers well suited for training robust agents.

---

Under review as a conference paper at ICLR 2023
Despite the strong robustness guarantee, PAIRED is still sample inefficient in practice. Primarily be- cause training a regret- based teacher is hard (Parker-Hodder et al. (2022)). First, the teacher receives sparse reward only after specifying the full parameterization of task; leading to a long-horizon credit assignment problem. Additionally, the teacher agent faces a combinatorial explosion problem if the parameter space is permutation invariant- g.g for navigation task, set of obstacles corre- sponds to factorially different permutations of the parameters More importanty, to generate tasks the frontier of agents' capabilities, the teacher needs to simultancously learn task manifold and navigate it to induce curriculum. The teacher learns this task manifold implicitly based on regret. However, as the student continuously co-learning with the teacher, the task manifold is also evoly- ing over time. Hence, the teacher needs to simultaneously learn the evolving task manifold, as well as how to navigate it effectively-whicl which is a difficult learning problem.
To address the above-mentioned challeeng we present Curriculum Learning via Unsupervised Task Representation Learning (CLUTR). At the core of CLUTR, lies hierarchical graphical model that decouples task representation learning from curriculum learning. We develop variational approx- imation to this problem and train a Recurrent Variational AutoEncoder (VAE) to learn a latent task manifold. Unlike PAIRED, which builds the tasks from scratch one parameter at time, the CLUTR teacher generates tasks in a single timestep by sampling points from the latent task manifold and uses the generative model to translate them into complete tasks. The CLUTR teacher learns the curriculum by navigating the pretrained and fixed task manifold via maximizing regret. By utilizing a pretrained latent task mask-manifodd, the CLUTR teacher can train as contextual bandit overcoming the long-horizon credit assignment problem - and create curriculum much more efficiently im- proving stability at no cost to its effectiveness. Finally, by carefully introducing bias the training corpus (such as sorting each parameter vector), CLUTR solves the combinatorial explosion problem of parameter space without using any costly environment interaction.
Our experimental results show that CLUTR outperforms PAIRED, both in terms of generalization and sample efficiency, in the challenging pixel-based continuous CarRacing and partially observ- able discrete navigation tasks. In CarRacing, CLUTR achieves 18x higher zero-shot generalization returns than PAIRED, while being trained on 60% fewer environment interactions on the Fl bench- mark, modeled on real-life Fl racing tracks, Furthermore, CLUTR performs comparably to the non-UED attention-based SOTA (Tang et al.] (2020)), outperforming it in nine of the 20 test tracks while requiring fewer than 1% of its environment interactions. In navigation tasks, CLUTR achieves higher zero-shot generalization in 14 out of the 18 test tasks, achieving a 33% higher solved rate overall. Furthermore, we empirically validate our hypotheses to justify the algorithmic decisions choices behind CLUTR.
In summary, we make the following contributions: i) we introduce CLUTR, novel UED algorithm by augmenting the PAIRED teacher with unsupervised reseerepresnntattoon learning that is derived from hierarchical graphical model for curriculum learning, ii) CLUTR, by decoupling task rep- resentation learning from curriculum learning, solves the long-horizon credit assignment and the combinatorial explosion problems faced by PAIRED. iii) Our experimental results show CLUTR outperforms PAIRED, both in terms of generalization and sample efficiency, in two challenging sets of tasks: CarRacing and navigation.
2 RELATED WORK
Unsupervised Curriculum Design: Dennis et al. (2020) was the first to formalize UED and in- troduced the minimax regret-based UED teacher algorithm, PAIRED with strong theoretical ro- bustmess guarantee. However, gradient based multi Maeti- RL has no convergence guarantees and often fails to converge in Mrazumae et al. (2019). Pre-existing techniques like Domain Randomization (DR) [Jakobi 1997). Saddghi & Levine (2016), Tobin et al. (2017)) and minimax adversarial curriculum learning( Morimoto & Doya (2005). [Pinto al. (2017) also fall under the
category of UEDs. DR teacher follows a uniform random strategy, while the minimax adversar- ial teachers follow the maximin criteria, i.c., generate tasks that minimize the returns of the agent.
'Consider 13x13 grid for navigation task, where the locations are numbered from to 169. Also consider a wall made of four obstacles spanning the locations: {21, 22, 23, 24). This wall can be represented using any permutation of this set, e.g., (22, 24, 23, 21). {23, 21, 24, 22), resulting in combinatorial explosion.
2

---

Under review as a conference paper at ICLR 2023
POET Wang Wong et al. (2019)) and Enhanced POET (ang al. (2020)) also approached UED, before PAIRED, using an evolutionary approach using a co-evolving population of tasks and agents.
Recently, Jiang et al. 202 proposed Dual Curriculum Design (DCD): novel class of UEDs that augments UED generation methods (e.g.. DR and PAIRED) with replay capabilities. DCD involves two teachers: one that actively generates tasks with PAIRED DR, while the other rates the curriculum to replay previously generated tasks with Prioritized Level Replay(PLR) (Tiang et al. (2021b)). Jiang al. (2021a) shows that, even with random generation (i.e., DR), updating the students only on the replayed level (but not while they are first generated, i.e., no exploratory student gradient updates as PLR) and with a regret-based scoring function, PLR can also learn minimax-regret agents at Nash Equilibrium and call this variation Robust PLR. It also introduces REPAIRED, combining PATRED with Robust PL.. Parker-Holder et al. (022)) introduces ACCEI,, which improves on Robust PLR by allowing edit/mutation of the tasks with an evolutionary algo- rithm. While CLUTR and PAIRED-variand actively adapt task generation to the performance of agents, other algorithms such as PLR generates task from fixed task distribution. Different from PAIRED-variants, which are susceptible to instability due to evolving task-manifold, CLUTR intro- duces novel variational formulation with VAE-style pretraining for task mask-maiffold learning.
Representation Learning: Variational Auto Encoders (Kingma & Welling (013)), Rezende et al. (2014), Higgins et al. (2016)) have widely been used for their ability to capture high-level semantic information from low-level data and generative properties in a wide variety of complex domains al. (2016), [Zhang al. (2021). ZZhag itt al. (2022)), natural language (Bowman et al. (2015), Jain et al. (2017)) speech (Chorowski et al. (2019)) ) and music (Jiang et al. (2020)). VAE has been used RL as well for representing image observations (Kendall et al. (2019). Yarats et al. (202T)) and generating goals (Nair et al. (2018) ) While CLUTR also utilizes similar VAEs, different from prior work, combines them in new curriculum learning algorithm to learn latent task manifold. FForensa et al.) (2018) proposed a curriculum learning algorithm with latent-space goal generation using a Generative Adversarial Network.
such computer vision (Razavi al. (2019), Gulrajani
BACKGROUND
3.1
UNSUPERVISED ENVIRONMENT DESIGN (UBD)
As introduced by Dennis al. 2020) UED is the problem of inducing curriculum by designing distribution of concrete, specifsee enviromments, from an underspecified environment with free parameters. The fully specified environments are represented using Partially Observable Markov Decision Process (POMDP) represented by (A, O, TIS, R, 7), where A, O, and S denote the action, observation, and state spaces, respectively. I - O is the observation function, : - TR is the reward function, T S x A + A(S) is the transition function and is the discount factor. The underspecified environments are defined in terms an Underspecified Partially Observable Markov Decision Process (UPOMDP) represented by the tuple M = (A, O, θ, S-M, T.M, ...M RMM 7). O a set representing the free parameters of the environment and is incorporated in the transition function as TM 5 x x θ + (S). Assigning a value to 0 results a regular POMDP, i.e., UPOMDP 0= POMDP. Traditionally (e.g., in Dennis et al. (202)) and Jiang et al. 2021a θ is considered trajectory of environment parameters 0 or just 0 which we call task in this paper. For example, 0 can be a concrete navigation task represented by a sequence of obstacle locations. We denote concrete environment generated with the parameter 0 E θ as MM g simply Ma. The
value of policy π in M3 is defined as v (²(T)) = EN ", where rz is the discounted reward obtained by in Mg.
3.2 PAIRED
PAIRED Dennis et al. 2020)) solves UED with adversarial game involving three players the agent n and antagonist AA trained tasks generated by the teacher O. PAIRED objective
In the original PAIRED paper, the primary student agent named protagonist. Throughout this paper we refer simply as the agent.
3

---

Under review as a conference paper at ICLR 2023
is: marg,ap m (UTP 0) = Eg ( (TP,TA)), Regret is defined by the difference of the discounted rewards obtained by the antagonist and the agent in the generated tasks, i.c., (EGRET"(TP,) A = A) pT)) The PAIRED teacher agent is defined as A : II - A()"), where II is set of possible agent policies and O" is the set of possible tasks. The teacher is trained with an RL algorithm with U as the reward while, the protagonist and antagonist agents are trained using the usual discounted rewards from the environments. Dennis et al. (2020) also introduced the flexible regret objective. local optima. It is defined by the difference between the average score of the agent and antagonist returns and the score of the policy that achieved the highest average return.
alternate regret approximation that is less susceptible to
4 CURRICULUM LEARNING VIA UNSUPERVISED TASK REPRESENTATION LEARNING
In this section, we formally present CLUTR as a latent UED and discuss details.
4.1 FORMULATION OF CLUTR
At the core of CLUTR is the latent generative model representing the latent task manifold. Let's assume that is a random variable that denotes a measure of success over the agent and antagonist agent and be a latent random variable that generates environments/tasks, denoted by the random variable E. We use the graphical model shown in Figure-[[e formulate CLUTR. Both E' and R are observed variables while z is an unobserved latent variable. R can cover a broad range of measures used in different UED methods including PAIRED and DR (Domain Randomization). In PAIRED, R represents the REGRET.
R
Figure 1: Hierarchi- cal Graphical Model for CLUTR
We use variational formulation of UED by using the above graphical model to derive the following BLBO for CLUTR, where VAE(z, E) de- notes the VAE objective:
ELBO & V VAE(2, B) REGRET
We share the details of this derivation in Section B. of the Appendix. The above ELBO (Eq defines the optimization objective for CLUTR, which can be seen as optimizing the VAE objective with regret-based regularization term and vice versa. As previously discussed, it is difficult to train a UED teacher while jointly optimizing for both the curriculum and task representations. Hence we propose two-level optimization for CLUTR. First, we pretrain VAE to learn unsupervised task representations, and then in the curriculum learning phase, we optimize for regret to generate the curriculum while keeping the VAE fixed. In Section 5.4 we empirically show that this two-level optimization performs better than the joint optimization of Eq i.e., finetuning the VAE decoder with the regret loss during the curriculum learning phase.
4.2 UNSUPERVISED LATENT TASK REPRESENTATION LEARNING
As discussed above, we use a Variational AutoEncoder (AEE to model our generative latent task- manifold. Aligning with Dennis et al. 2020) and Jiang al. (2021a). we represent task 0, sequence of integers. For example, in a navigation task, these integers denote obstacle, agent, and goal locations. We use an LSTM-based Recurrent VAE (Bowman all (2155) to learn task repre- sentations from integer sequences. We learn an embedding for each integer and use cross-entropy over the sequences to measure the reconstruction error. This design choice makes CLUTR applicable to task parameterization beyond integer sequences, e.g., to sentences or images. To train our VAEs, we generate random tasks by uniformly sampling from O"", the set of possible tasks. Thus, we do not require any interaction with the environment to learn the task manifold. Such unsupervised train- ing of the task manifold is practically very useful as interactions with the environment/simulato are much more costly than sampling. Furthermore, they are permutation invariant, ie., essentially represent a set. By sorting the training sequences, we thus avoid the combinatorial explosion faced by the PAIRED teacher.
sort the input sequences, fully partially, when
4

---

Under review as a conference paper at ICLR 2023
Task Representation Learning
Teacher Model
UED Method
Replay Method
Algorithm
DR
PLR
DR
PLR
Random
Robust PLR
Robust PLR
ACCEL
DR Evolution
PAIRED
Implicit via RL
REPAIRED
Learned
PAIRED
Robust PLR
Explicit via
CLUTR
Unsupervised Generative Model
Table I: comparative characterization of contemporary UED methods
4.3 CLUTR
We define CLUTR following the objective given in Eq. 1 CLUTR uses the same curriculum ob- jective as PAIRED, REGRET E) = REGRET (TP, TA) where, 0 denotes a task, i.e., a concrete assignment to the free parameters of the enviromment directly, the CLUTR teacher policy is defined as A II + A(Z), where II is set of possible agent policies and is as the latent space. Thus, the CLUTR teacher is latent environment designer, which samples random z and 0 is generated by the VAE decoder function G Z - O We present the outline of the CLUTR in Algorithm[] CLUTR outline is very similar to PAIRED, differing only in the first two lines of the main loop to incorporate the latent space.
Unlike PAIRED teacher, which generates
Algorithm CLUTR
Pretrain VAE with randomly sampled tasks from + Randomly initialize Agent T Antagonist π and Teacher Å;
while not converged do
Generate latent task vector:
Z from the teacher
Create POMDP M, where = ((z) and G is the VAE decoder function
Collect Agent trajectory -P in Mo. Compute: (= Te7
Collect Antagonist trajectory A Mg. Compute: U (T9( = Σ1 P27
Compute: REGRET , ^х = (T (TT)
Train Protagonist policy 7P with RL update and reward R(TP) U (T
Train Antagonist policy A with RI. update and reward R(TA) = U (mA))
Train Teacher policy Á with RL update and reward R(TA) REGRET end while
Now we discuss a couple of additional properties of CLUTR compared to PAIRED-variants
1. CLUTR teacher samples from the latent space Z and thus generates task in single timestep. Note that this is not possible for the PAIRED/REPAIRED teacher, as they gener- ate one task parameter at time, conditioned the state of the partially-generate task so far.
2. PAIRED-variane typically observe the state the partially generated task to generate the next parameter. Hence depending the state space, they require designing different teacher architectures for different environments. CLUTR teacher architecture, however, is agnostic of the problem domain. Hence the same architecture be used different problems.
4.4 COMPARISON OF CLUTR WITH CONTEMPORARY UED METHODS
As discussed contemporary UED methods can be characterized by their i) teacher type (ran- dom/fixed or, learned/adaptive) and ii) the use of replay. To clearly place CLUTR in the context of contemporary UEDs, we discuss another important aspect of curriculum learning algorithms: how the task manifold is learned. The random UED teachers do not learn task manifold. Regret-based
5

---

Under review as a conference paper at ICLR 2023
teachers such PAIRED and REPAIRED learn implicit (e.g., the hidden state of the teacher LSTM) mask-manifold but it is not used explicitly. is trained via RL based on regret estimates of the tasks they generate. Hence, these task manifolsss depend the quality of the estimates, which in turn depends on the overall health of the multi-agent RI. training. Furthermore, they do not take into account the actual task structures. In contrast, CLUTTR introduces an explicit task-manifold modeled with VAE that presents a local neighborhood structure capturing the similarity of the tasks. Hence, similar tasks (in terms of parameterization) would be placed nearby in the latent space. Intuitively this local neighborhood structure should facilitate the teacher to navigate the manifold. The above discussion illustrates that CLUTR along with PAIRED and REPAIRED form a category of UEDs that generates tasks based on a learned task-manifold, orthogonal to the random generation-based methods, while CLUTT being the only one utilizing an unsupervised generative task manifold. Ta- ble summarizes the differences.
EXPERIMENTS
In this section, we first evaluate CLUTR's performance compared to the existing UEDs in Pixel- Based Car Racing with continuous control and dense rewards. As discussed in Section [4.4] our primary comparison is with PAIRED and REPAIRED-the only two existing UED methods that learn task-manifolds to generate tasks. For completeness, we also compare CLUTR with UEDs with random teachers. Furthermore, we compare with PAIRED on partially observable navigation tasks with discrete control and sparse rewards.
We then empirically evaluate the following hypotheses: H1: CLUTR generates more efficient curriculum. (Section 5.3
H2: Learning task representations and curriculum simultaneously degrades the performance (5.4)
H3: Training VAE on sorted data solves the combinatorial explosion problem. (Section 5.5)
At last, we analytically compare the CLUTR and PAIRED curricula. Full details of the environ- ments, network architectures of the teacher and student agents, the VAE, the training hyperparame- ters, and further analysis and evaluation are discussed the Appendix.
5.1 CLUTR PERFORMANCE ON PIXEL-BASED CONTINUOUS CONTROL CARRACING ENVIRONMENT
The CarRacing environment (Jiang et al. 2021a). Brookman et al. (006)) requires the agent to drive a full lap around closed-loop racing track modeled with Bézier Curves (Mortenson (1999) of up to 12 control points. CLUTR was trained with the Flexible Regret Objective for 2M timesteps- around which the agent's training return converges to its maximum. We also experimented with the standard regret objective and obtained better performance than PAIRED, which we discussed in Section D.2 of the appendix. We train the VAE on Million randomly generated tracks for Million gradient updates. Note that only one VAE was trained and used for all the experiments (10 independent runs). We evaluate the agents on the F1 benchmark (Jiang et al. (2021a) that contains 20 test tracks modeled real-life F1 racing tracks. These tracks are significantly out of distribution than any tracks that the UED teachers can generate with just 12 control points. Further details on the environment, network architectures, and VAE training can be found in Section C.1 and C.4 of the appendix, respectively.
Figure 2a shows the mean return obtained by the CLUTR, PAIRED, and REPAIRED on the 20 Fl test tracks. CLUTR outperforms PAIRED and REPAIRED by huge margin: showing an 18x higher mean return than PAIRED and 1.6x than REPAIRED, outperforming both of them all of the 20 test tracks. Note that CLUTR was trained only for 2M timesteps, while both PAIRED and REPAIRED were trained for 5M timesteps. Figure 26 tracks the agents' generalization capabilities by periodically evaluating them on four unseen tracks Vanilla, Singapore, Germany, and Italy. These tracks were selected aligning with fiang et al. (2021a). Based on these selected tracks, CLUTR shows much better generalization and sample efficiency achieving better performance and faster improvement. Further experiment results are shared in Section of the appendix.
We also compare CLUTR to other existing UEDs with random task generation the F1 bench- mark for completeness. CLUTR outperforms Domain Randomization and PLR, falling short only to Robust PLR, which achieves overall 1.13X higher returns. Nonetheless, CLUTR performs

---

Under review as a conference paper at ICLR 2023
FABC)
16R100
REPAROO
CJITE
0.00
AA0A0-LL L3UIH
(a) Mean Performance the Benchmark. CLUTR achieves an 18x higher return than PAIRED and 1.6x than REPAIRED.
(b) Agent performance on selected
(c) Mean Performance the benchmark, compared to other ex- isting UEDs and the non-UED SOTA.
test tracks during training. CLUTR shows significantly better general- ization and sample efficiency.
Figure 2: CLUTR performance on the CarRacing Tasks compared to PAIRED and REPAIRED. The results show mean and standard error of 10 independent runs. Further details can be found in Section D in Appendix
comparably to Robust PLR on seven the 20 tracks and outperforms in one. Furthermore, CLUTR shows comparable performance to the non-UED attention-based state-of-the-arr method for CarRac- ing Tang et al. (2020)) despite not using a self-attention policy and training on significantly fewer environment timesteps (< 1%). Moreover, CLUTR outperforms it nine of the 20 tracks. Detailed results on individual tracks are presented in Table A of the Appendix.
5.2 CLUTR PERFORMANCE ON PARTIALLY OBSERVABLE NAVIGATION TASKS ON MINIGRID
We also compare CLUTR with PAIRED on the popular MiniGrid environment, originally intro- duced by Boovalier-Boisvert
08
al. (2018) and
adopted by Dennis et al. (2020) for UEDs. In these navigation tasks, an agent explores grid world to find the goal while avoiding obstacles and receives sparse reward upon reaching the goal. To train CLUTR VAE, we generate Mil- lion random grids, with the obstacle locations sorted, and the number of obstacles uniformly varying from zero to 50, aligning with Dennis] et al. (2020). We used the standard regret objec- tives. Note that the results reported in the orig- inal PAIRED paper
00
Trathop
Figure 3: Agent solved rate on selected grids during training. CLUTR shows better sample efficiency and generalization than PAIRED. The results show average of independent runs.
obtained after 3 Billion timesteps of training, while we run both PAIRED and CLUTR for 500M timesteps (5 indepen- dent runs) due to the huge computational resource and time needed run a training with 3 Billion timesteps. For the same computational constraints. we compare only with PAIRED in this envi- ronment. Figure 4 shows zero-shot generalization performance of CLUTR and PAIRED 18 unseen navigation tasks from Dennis et al. (2020) based the percent of environments the agent solved, i.e., solved rate. CL.URR achieves superior generalization solving 64% of the unseen grids, while PAIRED achieves 43%, which is 33% lower compared to CLUTR. From figure 4it it can be seen CLUTR outperforms PAIRED achieving a higher mean solve rate on 14 out of the test navi- gation tasks. Figure shows solved rates on four selected grids (Sixteen Rooms, Sixteen Rooms with Fewer Doors, Labyrinth, and Large Corridor) during training. CLUTR shows better sample efficiency, well generalization than PAIRED.
5.3 EFFICIENCY OF THE CURRICULUM: CLUTR PAIRED
tasks for both CarRacing and navigation tasks. CLUTR shows lower regret than PAIRED, meaning the performance gap between the agent and the antagonist is lower in CLUTR. From curriculum learning perspective, want to train
Figure shows the mean regret the

---

Under review as a conference paper at ICLR 2023
BirphCrosingg 11N5
2.75
200
800
soneroo sewetion
LargeCcedor
Lateeein
CLUTS PNTED
GLUM PARCO
CLUTE FATCOO
CAUTA PATSO
CLUMS PAPIDD
CL/MI FARCD
CAUTh FAACOO
CUUTR PAPAED
CLITR PARLE
Figure 4: Zero-shot generalization of CLUTR and PAIRED, in terms of percent of the environments solved. CLUTR achieves higher solved rate than PAIRED in 14 out of the 18 tasks. We evaluate the agents with 100 independent episodes on each task. Error bars denote the standard error.
CLUTT
0.40
200
030
Deggel
475
026
150
Vear
0:15
6.6
PAREE
0.1
0.90
125
260
TirwSlipp
(a) Mean Regret- CarRacing
(b) Mcan Regret Navigation
Figure 5: Mean regret values during training. CLUTR shows a smaller regret value indicating a less performance gap between the agent and the antagonist. CLUTR also converges faster.
the agent on tasks that are slightly harder than it can already solve or, those tasks that can solve already but obtain better returns. In practice, both the agent and the antagonist trained in the same training context e.g., the same hyper-parameters. model architecture, and tasks, differing only by their random initial weights. Hence, lower regret means that the teacher is generating tasks that are either slightly harder than the tasks the agent can solve now (because the other agent is solving them) or, tasks in which the antagonist is performing slightly better. Hence, the tasks are more likely at the agent's frontier of capability. The curves also show that CLUTR and PAIRED show similar convergence patterns, while CLUTR converges sooner to better local optimum. These observations, in addition to the empirical performance, indicate that CLUTR is generating more efficient curriculum than PAIRED.
5.4 LEARNING TASK MANIFOLD AND CURRICULUM: JOINT VS TWO-STAGED OPTIMIZATION
etuned We hypothesize that learning the task representations and the curriculum simultaneously re- sults in a difficult training problem due the non-stationarity of the task manifold. To test this, instead of keeping the task representations fixed, continue finetuning loss during the teacher-student curriculum learning phase. This experiment shows a 58% perfor- mance drop in the F1 benchmark, labeled 'Finetuned VAE' in Figure 6) This empirically validates our hypothesis that pretraining a latent task space and then learning to navigate it to induce curricu- lum indeed is easier and lend better UED.
decoder the regret
5.5 IMPACT OF SORTING VAE DATA ON SOLVING COMBINATORIAL EXPLOSION
We hypothesized that training a VAE on sorted sequences can solve the combinatorial ex- plosion problem. To test this, CLUTR with altemate VAE trained 5X longer
8

---

Under review as a conference paper at ICLR 2023
G
A
X
V
Environment Interactions (2M)
Environment Interactions (500M)
Figure 7:
Example tracks(left) and grids(right) generated by CLUTR(top)) and PAIRED(bottom)
uniformly sampled at different stages of training. The training progresses from left to right.
on a non-sorted and 10X bigger version of the original dataset. This experiment shows 59% performance drop on the Fl benchmark, labeled 'Shuffled VAF in Figure 6 empiri- cally validating our hypothesis. Further details are discussed in Section
D.3 of the Appendix.
5.6 CURRICULUM COMPLEXITY
Mean Relur on F1 Dalaset
400
In this section, compare the curriculum generated by CLUTR and PAIRED, with smapshots of tasks gen- crated by these methods during different stages of the training (Figure 7 We illustrate one common mode of failure/infffectiveness shown by PAIRED: The curricu- lum starts with arbitrarily complex tasks, which none of agents can solve the initial stage of training. After while, PAIRED starts generating rudimentary degener- ate tasks. If enough training budget is given, PAIRED eventually gets out of the degenerative local minima, and the curriculum complexity starts to emerge. On the other hand, CLUTR does show such degeneration and gen- erates seemingly interesting tasks throughout. The exam- ples shown Figure illustrates this.
300
200
CLUTR
Shuffled WAE
Finetuned VAE
PAIRED
Figure 6: Impact of i) Training VAE on non-sorted data (Shuffled VAE) and ii) Finetuning the task-manifold with re- gret (Finetuned VAE) on Fl benchmark.
CONCLUSION AND FUTURE WORK
In this work, propose CLUTR, unsupervised environment design method via unsupervised task representation learning. CLUTR augments PAIRED with a latent task space, decoupling task representation learning from curriculum learning. CLUTR poses several advantages over PAIRED- variants, including solving the long-horizon credit assignment and the combinatorial explosion of the parameter space. Our experiments show CLUTR outperforms PAIRED-variants in terms of sample efficiency and generalization.
Even though CLUTR and other regret- based UEDs empirically show good generalization on human- curated complex transfer tasks, they rarely can generate human-level task structures during training. interesting direction would to enable UED algorithms to generate realistic tasks. Another important direction would be to reduce the gap between the theoretical and practical aspects of segret-based multi-agent UED algorithms, which are subject to the quality of regret estimates and multi-agent RL training. At last, random generator algorithms like Robust PLR or been shown to perform better than learned generator approaches like CLUTR or PAIRED. An in- teresting direction would be to investigate the conditions/environment under which random gen- erator performs better than an adaptive generator and vice versa. At last, we are excited about latent-space curriculum design and hope work will encourage further research in this domain.
DR have

---

Under review as a conference paper at ICLR 2023
REFERENCES
Abdus Salam Azad, Edward Kim, Qiancheng Wu, Kimin Lee, Ion Stoica, Pieter Abbeel, Alberto Sangiovanni- Langiovanni-Vineentllli and Sanjit A Seshia. Programmatic modeling and generation of real- time strategic soccer environments for reinforcement learning. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 36, pp. 6028-6036, 2022.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint 1r1iv:192..6600 2019.
Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Ben- gio. Generating sentences from continuous space. arXiv preprint arXiv:1571.06339, 2015.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint wrXiv:/606.0155. 2016.
Maxime Chevalier-Boisvert, Lucas Willems, and Suman Pal. Minimalistic gridworld environment for gymnasium. https: com Farama wwwndaaiinnniininniniiooo
2018.
Jan Chorowski, Ron J. Weiss, Samy Bengio, and Aäron van den Oord. Unsupervised speech repre- sentation learning using wavenet autoencoders. IEEE/ACM Transactions on Audio. Speech, and Language Processing, 27(12):2041-205 2019. doi: 10. 109/TAASLP2019229888.
Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, and John Schulman. Quantifying generaliza- tion in reinforcement learning. In International Conference on Machine Learning, pp. 1282-1289 PMLR, 2019.
Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. Advances in neural information processing systems, 13:13049-13061, 2020.
Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. In International conference machine learning, pp. 1515-1528.
PMLR, 2018.
Ishaan Gulrajani, Kundan Kumar, Faruk Ahmed, Adrien Ali Taiga, Francesco Visin, David Vazquez, and Aaron Courville. Pixelvae: latent variable model for natural images. arXiv preprint arXiv: 1611:0611..00133. 2016.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. 2016.
Unnat Jain, Ziyu Zhang, and Alexander G Schwing. Creativity: Generating diverse questions using variational autoencoders. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6485 6494, 2017.
Nick Jakobi, Evolutionary robotics and the radical envelope-of-nne noise hypothesis. Adaptive behavior, 6(2):325-36) 368, 1997.
Junyan Jiang, Gus G. Xia, Dave B. Carlton, Chris N. Anderson, and Ryan H. Miyakawa. Trans- former vae: A hierarchical model for structure-aware and interpretable music representation learn- ing. In ICASSP 2020 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 516-520, 2020. doi: 10.1 10..190ICAASPP477222200066
Minqi Jiang, Michael Dennis, Jack Parker-Holle Holder, Jakob Foerster, Edward Grefenstette, and Tim Rocktäschel. Replay-guided adversarial environment design. Advances in Neural Information Processing Systems, 34:1884-1888 2021a.
Minqi Jiang, Edward Grefenstette, and Tim Rocktäschel. Prioritized level replay. In International Conference Machine Learning, pp. 4940 4950. PMLR, 2021b.
10

---

Under review as a conference paper at ICLR 2023
Alex Kendall, Jeffrey Hawke, David Janz, Przemyslaw Mazur, Daniele Reda, John-Mark Allen, Vinh-Dieu Lam, Alex Bewley, and Amar Shah. Learning to drive in a day. In 2019 International Conference on Robotics and Automation (ICRA), pp. 8248 8254. IEEE, 2019.
Diederik P. Kingma and Max Welling, arXiv:1312.6114 2013.
Auto-encoding variational
bayes.
arXiv preprint
Eric Mazumdar, Lillian Ratliff, Michael I. Jordan, and S. Shankar Sastry. Policy-gradient al- gorithms have no guarantees of convergence in linear quadratic games, 2019. URL https:
Volodymyr Mnih, Koray Kavukcuoglu, David Silver. Andrei A Rusu, Joel Veness, Mare Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 51 (877540):529-5555 533, 2015.
Jun Morimoto and Kenji Doya, Robust reinforrmeee learning. Neural computation, 17(2):335-355) 2005.
Michael E Mortenson. Mathematics for computer graphics applications. Industrial Press Inc., 1999.
Ashvin V Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. In S. Bengio, H. Wallach, H. Larochelle, K. Grau- man, N. Cesa-Bianchi, and R. Garnett ). Advances in Neural Information Processing Systems, www.ang.........p...e.. cc/ Papp.,
volume 31. Curran Associates, Inc., 2018. URL
Sanmit Narvekar, Bei Peng. Matteo Leonetti, Jivko Sinapov, Matthew E Taylor, and Peter Stone. Curriculum learning for reinforcement learning domains: framework and survey. arXiv preprint arXiv:2003.0496) 2020.
Jack
Minqi Jiang, Michael Dennis, Mikayel Samvelyan, Jakob Foerster, Edward Grefenstette, and Tim Rocktäschel. Evolving curricula with regret-based environment design. arXiv preprint arXiv:2203.01300 2022.
Lerrel Pinto, James Davidson, and Abhinav Gupta. Supervision via competition: Robot adversaries for learning tasks. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 1601-1608. IEEE, 2017.
Rémy Portelas, Cédric Colas, Lilian Weng, Katja Hofmann, and Pierre- Yves Oudeyer. Automatic curriculum learning for deep rl: A short survey. arXiv preprint 0rXiv:2003.04664. 2020.
Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 32, 2019.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and ap- proximate inference in deep generative models. In International conference on machine learning, pp. 1278-1286. PMLR, 2014.
Fereshtch Sadeghi and Sergey Levine. Cad2rl: Real single- image flight without a single real image, 2016. URL htt.httpss:./ arxiv. http://wwwww..oo.bb/////o/oo 1at.aa////11....... 04201
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL /uL/:ttte////raa..oo/////////..0 06347
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484-489, 2016.
Yujin Tang, Duong Nguyen, and David Ha. Neuroevolution of seff-interpreetable agents. In Proceed- ings of the 2020 Genetic and Evolutionary Computation Conference, pp. 414 2020.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Do- main randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp. 23-30. IEEE, 2017.
11

---

Under review as a conference paper at ICLR 2023
Rui Wang. Joel Lehman, Jeff Clune, and Kenneth O Stanley. Paired open-ended trailblazer (poet): Endlessly generating increasingly complex and diverse learning environments and their solutions. arXiv preprint arXiv:1901.01733 2019.
Rui Wang, Joel Lehman, Aditya Rawal, Jiale Zhi, Yulun Li, Jeffrey Clune, and Kenneth Stanley. Enhanced poet: Open-ended reinforcement learning through unbounded invention of learning challenges and their solutions. In International Conference on Machine Learning, pp. 9940-
9951. PMLR, 2020.
Denis Yarats, Amy Zhang, llya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Improv- ing sample efficiency model-free reinforcement learning from images. Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 10674-10681, 2021.
Amy Zhang, Nicolas Ballas, and Joelle Pineau. A dissection of overfitting and generalization in continuous reinforcement learning. arXiv preprint 18X1::/806.07337, 2018.
Mingtian Zhang, Andi Zhang, and Steven McDonagh. On the out-of-distribution generalization of probabilistic image modelling. Advances in Neural Information Processing Systems, 34:381 1- 3823, 2021.
Mingtian Zhang, Tim Z Xiao, Brooks Paige, and David Barber. Improving vae-based representation learning. arXiv preprint arXiv:2205. 2022.
ETHIC STATEMENT
Unsupervised Environment Design be applied to many real-world applications and shares many similar ethical concerns and considerations with other Artificially Intelligent(AI) systems. For ex- ample, Al systems can cause more unemployment be used for reasons/applications that have negative societal impact, for which responsible usage of such Al systems must be promoted and es- tablished. During our research, all the experiments were done in simulation and human or living subjects were used.
REPRODUCIBILITY
Our code, saved checkpoints, and training data are available at sttttpss://gthub. http::///iitubb.cooooooo clutr/ clutr
12

---

Under review as a conference paper at ICLR 2023
A APPENDIX
B ADDITIONAL DETAILS OF CLUTR
B.I CLUTR OBJECTIVE DERIVATION
We use hierarchical graphical model to formulate the latent environment design problem. Let's assume that is random variable that denotes measure of success defined using the protagonist and antagonist agents and be a latent random variable. We use the graphical model Figure-8 where generates an environment E and R is the success defined over E. Both E and R are observed variables while is an unobserved variable. covers broad range of measures used in different UED methods including PAIRED and DR (Domain Randomization). In PAIRED, R represents the REGRET as the difference of returns between the antagonist and protagonist agents and it depends on the environments that the agents are evaluated on.
Figure 8: Hierarchi- cal Graphical Model for CLUTR
We use variational formulation of UED by using the above graphical model. We first define the variational objective as the KL-divergence be- tween approximate posterior distribution and true posterior distribution over latent variable 2,
Dxr(u(=))v((/A)) E)) = Es [[og((=]] - ES [logp== F)]
E Ez-g(z) [[ogq(2]] - E Ez- loggp E),] + logp(R, E)
where both R and E are given.
Next, we write the ELBO,
ELBO Fx [z)g Es logp R, F,
Ez~g(=) (2)] Ezna(=) llop (R))(())))))))))(((
Fazg(()) [logg(=)]
Fe [[og(22) F P Dosp R|=)] (oogg()
-Emg(=)]log [log ((2)
p(z)
(KL()(=)lp(z)) - -= [ [[ogp(b)=)]]]]]))))) logp
= VAR(2, E) -ooop R)
We also induce objective that includes minimax REGRET. Let R be distributed according an exponential distribution, pl Rt) E") x (Ep(REGRET(TPPTAAT TA)
derive,
ELBO VAE(2, REGRET(R,
where the normalizing factor is ignored.
B.2 ROBUSTNESS GUARANTEES
CLUTR essentially proposes including a pretrained latent space within the teacher/generator. From the teacher's 's perspective, the difference is while the PAIRED teacher starts from randomly initialized weights, CLUTR starts from the pretrained weights. Thus, CLUTR does not impose new assump- tions possible teacher policies. Furthermore, CLUTR does not change any other specifics of the underlying PAIRED algorithm. Hence, CLUTR holds the same theoretical robustness guarantees provided by PAIRED.
In practice, both CLUTR and PAIRED deviate from these theoretical guarantees. For example, both algorithms approximate the regret value, which is the case for other regret-based UEDs such
13

---

Under review as a conference paper at ICLR 2023
as Robust PLR and REPAIRED @Jian et al. (2021a)). Also, the robustness guarantee depends on reaching the Nash equilibrium of the multiagent adversarial game. However, gradient-based multi- agent RL has convergence guarantees and often fails to converge in practice(Mazumdaa et al. (2019). We also note that, by introducing the latent space, CLUTR VAE might not have access to the full task space due practical limitations on training, e.g., the training dataset not having all possible tasks. However, when the decoder is allowed to be finetuned, CLUTR will have access to the full task space, similar to PAIRED. Our empirical results (discussed in Section 5.4) suggest that keeping the pretrained decoder fixed performs better than finetuning it, so we kept it fixed for our main experiments. We also want to mention that we used the flexible regret objective for CarRacing in Section [5.] When the flexible objective is used, CLUTR (and PAIRED) might not hold the robustness guarantee as it changes the dynamies of the underlying game between the teacher and the agents. However, we also experimented with the standard regret objective and obtained better performance than PAIRED as discussed in Section D.2
C TRAINING DETAILS
C.I ENVIRONMENT DETAILS
The CarRacing environment was originally proposed by OpenAl Gym Broekman al. (2016), and later has been reparameterized by Jiang et al. (2021a) with Bézier Curves( (urves(Mooteen+ Morten- son 1999) for UED algorithms. This environment requires the agents to drive a full lap around a closed-loop track. The track defined by a Bézier Curve modeled with a sequence of upto 12 arbitrary control points, each spaced within fixed radius B/2 of the center of the B x B field. This sequence of control points can uniquely identify a track, subject to a set of predefined curvature constraints Jiang et al. (2021a). The control points are encoded in a 10 x 10 grid-a a discrete down- sampled version of the racing track field. Each control point hence is a integer denoting a cell of the grid and the cell coordinates are upscaled to match the original scale of the field afterwards. This ensures no two control points are too close together, preventing areas of excessive track overlapping. The track consists of sequence of L polygons and the agent receives a reward of 1000/ L upon vis- iting each unvisited polygon and a penalty of -0.1 at each time step to incentivize completing the tracks faster. Episodes terminate if the agent drives too far off-track but is not given any additional penalty. The agent controls 3 dimensional continuous action space corresponding to the car's 's steer: torque E [-1.0, 1.0), gas: acceleration E [0.0. 0, 1.0], and brake: deceleration [0.0, 1.0]. Each ac- tion is repeated 8 times. The agent receive a 96 x 96 x 3 RGB pixel observation. The top 84 x 96 portion of the frame contains clipped, egocentric, bird's eye view of the horizontally centered car. The bottom 12 x 96 segment simulates a dashboard visualizing the agent's latest action and return. Snapshots of the test track in the F1 benchmark are shown Figure
Car Racing:
Minigrid: The environment is partially observable and based on Chevalier- Boisverr et al. (2018) and adopted for UED by Dennis et al. (2020). Each navigation task is represented with a sequence of integers denoting the locations of the obstacles, the goal, and the starting position of the agent: on a 15 x 15 grid similar to Dennis et al. (2020). The grids are surrounded by walls on the sides, making it essentially 13* x 13 grid. Dennis et al. (2020) parameterizes the locations using integers. Each task is a sequence of 52 integers, while the first 50 numbers denote the location of obstacles followed by the goal and the agent's initial location. The sequences may contain duplicates to allow the generation of navigation tasks with fewer than 50 obstacles. Snapshots of the test grids used in our paper are shown in Figure 10
C.2 NETWORK ARCHITECTURES
All the student and teacher agents are trained with PPO Schulman et al. (2017).
Student Architecture
For CarRacing, we use the same student architecture as Tiang al. a The architecture consists an image embedding module composed of 2D Convolutions with square kernels of sizes 2,2,2,2,3,3, stride lengths 1 and channel outputs of 8, 16, 64, 128, 256 stacked together. The image embedding is of size 256 and is passed through a Fully Connected (FC) layer of 100 hidden units and then passed through ReL.U activations. This embedding is then passed through two FC with 100
14

---

Under review as a conference paper at ICLR 2023
Belgum
Braz
Mcxico
Auctrolia
Austra
Portugal
Haleysa
Nethoriands
Spath
France
Hungary
Auzsa
Singspore
USA
Bahrain
China
KK
Mo Naco
Cermany
Ialy
Figure 9: Snapshots of the test tracks in Fl benchmark
FourRoorns
Liby
Labyrinth2
LargeCorridor
Maze
MozeZ
HG
T+
Mazze
MediumMaze
MiniMaze
Nome
KineRoonsFewerDDoo
SimpleCrossing$1118
T
SipleerrossppS9N1
SimpleCrosunn559iz
SimpleCrossingS9NN
Soteen
SuttenffoonsFewebbbor
SmaliCorndor
Figure 10: Snapshots of the test grids for MiniGrid
15

---

Under review as a conference paper at ICLR 2023
hidden neurons, and then softplus layer, and finally added to 1 for the beta distribution used for the continuous action space. Further details can be found in Jiang et al. (202la)
For navigation tasks, a tuple with a 5 5 3 grid observation and a direction integer in [0- - 3j. The grid view is fed to a
the same student architecture Dennis al. 20200. The observation
convolutional layer with kernels of size 3 with 16 filters and the direction integer is passed through FC with units. This is followed by an LSTM size 256, and then to two FC layers with 32 units, which connect the policy outputs. The value network uses the same architecture.
Teacher Architecture
For CarRacing, CLUTR teacher takes random noise and generates a continuous vector, i.e., the latent task vector. We pass the random noise through a feed-forward network with one hidden layer of 8 neurons the teacher. The output of this layer is fed through two separate fully-connected layers, each with a hidden size of 100 and an output dimension equal to the latent space dimension, followed by soft plus activations. We then add 1 to each component of these two output vectors, which serve as the and / parameters respectively for the Beta distributions used to sample each latent dimension. In our experiments, we used a 64-dimensional latent task space. For Minigrid experiments, we use a network architecture similar to Dennis et al. (2020) but take only the random noise as input. The adversary network generates discrete actions, but we map them real numbers to feed into the VAE decoder.
VAE architecture
We use the architecture proposed in Bowman et al. (2015). We use word-embedding layer of size 300 with random initialization. The encoder comprises a conditional 'Highway' network followed by an LSTM. The Highway network is two-staged network stacked top of each other. Each stage computes o (x) @ (G(()) + (1-dc (x)) o Q(x). where x is the inputs to each of the highway network stages, G and is affine transformation, (x)) is sigmoid non-linearization, and @ is element-wise multiplication. G and Q are feed-forward networks with a single hidden layer with equal input and output dimensions of 300, equal to the word-embedding output dimension. We ReLU activation as f. The highway network is followed by bidirectional LSTM with a single layer of 600 units. The LSTM outputs are passed through linear layer of dimension 64 to get the VAE mean and log variance. The mean vectors are passed through a hyperbolic tangent activation and for the navigation tasks linearly scaled in [-4, 4]. The decoder takes in latent vectors of dimension 64 and passes through a bidirectional LSTT with two hidden layers of size 800 and follows it by linear layer with size equaling the parameter vector dimension.
C.3 HYPERPARAMETERS
All our agents are trained with PPO (Schulman al. (0017 ). We did not perform any hyperparame-
ter search for our experiments. The CarRacing experiments used the same parameters used in Tiang al. (2021a), and the Minigrid experiments used the parameters from [Dnnis et al.) (2020). VAE
was trained on the parameters from Bowmmn et al. (015). The detailed parameters are listed in Table 2 and T
Parameter
Value
Batch Size
32
Number of Training Steps
1000000
Reconstruction Weight
Latent Variable Size
64
Word Embedding size
300
Maximum Sequence Length
52
Encoder Activation
Hyperbolic Tangent
Learning Rate
0.00005
Dropout
0.3
Table 2: Hyperparameters for training the Task VAE
16

---

Under review as a conference paper at ICLR 2023
Parameter
CarRacing
MiniGrid
Y AGAE
0.99
0.995
0.9
0.95
PPO rollout length
125
256
PPO epochs
8
5
PPO minibatches per epoch
4
1
PPO clip range
0.2
0.2
PPO number of workers
16
32
Adam learning rate
3e-4
le44
Adam
le-5
le-5
PPO max gradient norm
0.5
0.5
PPO value clipping
no
yes
Return normalization
yes
Value loss coefficient
0.5
0.5
Student entropy coefficient
0
0
Action Repeat
8
Table 3: Hyperparameters for PAIRED and CLUTR PPO training.
C.4 VAE TRAINING DATA
For CarRacing, we follow the same parameterization as Jiang et (2021a): each track is defined with sequence of up to 12 integers denoting control points of Bézier Curve. Each control point represented with an integer. We generate 1M random sorted integer sequences of fixed length 12 with piplicates-whitee
enables generating tracks defined with less than 12 control points. For navigation tasks we use the parameterization of Dennis et al. (2020), generating upto 50 obstacles for each task for 15 x 15 grid, surrounded by walls, effectively each location numbered obstacle locations, and the last two for the goal and agent location, respectively. The parameter vector is thus partially permutation invariant. We uniformly generate 1M sequences variable length between and 52 (inclusive). The obstacle locations sorted.
active area of 13 x 13. Hence, to 169. Every number except the last two of the sequence represent
D DETAILED RESULTS ON CARRACING
D.1 DETAILED COMPARISON ON FULL F1 DATASET
We used the flexible regret approximation for the results presented in the main paper. The flexible regret objective is more robust variant of the standard regret estimation (both introduced in [Dnnnis et al. (0200). is defined by the difference between the average score of the agent and antagonist returns and the score of the policy that achieved the highest average return. Thus, the flexible objective blurs the distinction between the agent and the antagonist. Hence we designate the agent achieving the higher average training return during the last 10 steps as the agent.
Figure compares how different UEDs perform during training by periodically evaluating them on Four Selected Tracks: Vanilla, Singapore, Germany, and Italy. These tracks were selected aligning with Jiang et al. (2021a). Based on these selected tracks, CLUTR performance plateaus around 2.5M timesteps. Robust PLR starts slowly but surpasses all the other methods after 5M timesteps.
Table 4 shows the comparison between the final agents trained with CLUTR and other UED algo- rithms. is to noted that, all the UED methods except CLUTR was trained for 5M timesteps where CLUTR was run for 2M timesteps. CLUTR outperforms PAIRED by big margin with 18x bigger mean return on the entire Fl Dataset. CLUTR also outperforms Domain Randomization, PLR, and REPAIRED and only falls short to Robust PLR. Nonetheless, CLUTR shows competitive results compared to Robust PLR. showing comparable results in seven out of the 20 test tracks and outperforming in the Netherlands track. CLUTR also outperforms the non- UED SOTA on the full Fl dataset. CLUTR outperforms the Attention Agent on 9 out of the 20 tracks and shows comparable performance in
17

---

Under review as a conference paper at ICLR 2023
CLUTR
600
PAIRED
REPAIRED
500
DR
Test Environment Returns
PLR
400
Robust PLR
300
200
100
-100
5
TimeStep
1e6
Comparison of mean agent returns on Four Selected Tracks: Vanilla, Singapore, Ger- many, and Italy. Based on these selected tracks, CLUTR improves bit after 2M timesteps later the performance plateaus. Robust PLR starts slowly but surpasses all the other methods after 5M timesteps.
Figure 11:
D.2 CLUTR WITH STANDARD REGRET LOSS
We also train CLUTR with the standard regret loss for 5M timesteps. Figure compares the impact of standard/fiexible regret loss on the regret and agent returns during training. With standard regret loss, CLUTR shows lower regret value, but shows similar pattern. The CLUTR agent achieves better returns with flexible loss throughout the training.
Figure 13 compares the mean regret and agent training returns with PAIRED, CLUTR with standard loss shows much lower regret than PAIRED (Figure 13a). Figure 13b shows that the CLUTR agents compete closely, while PAIRED antagonist achieves much higher returns than the PAIRED agent which leads to higher regret returns for the teacher agent but results in weak student agent. To test the Zero-shot generalization, we evaluate CLUTR with the standard loss on the full F1 benchmark. Figure [4] shows CLUTR with standard regret loss outperforms PAIRED in all the 20 test tracks. This implies that CLUTR outperforms PAIRED irrespective of the choice of the loss function (stan- dard/flexible). Figure 15 compares the sample efficiency of CLUTR with the standard regret loss with PAIRED by evaluating the agents on four selected tracks (Vanilla, Singapore, Germany, Italy) during training. It can be seen that CLUTR, even without the regret loss, outperforms PAIRED significantly. We note that these test environments were not used in any way, neither during training CLUTR (and PAIRED) nor while designing it.
As mentioned in Jiang et al. a221a)) PAIRED overexploits the relative strengths of the antagonist over the protagonist and generates a curriculum that gradually reduces the task complexity. How- ever, CLUTR overcomes this and generates curriculum where the agent and the antagonist closely compete (Figure 13b) and shows a robust generalization on the unseen benchmark.
D.3 EXTENDED ANALYSIS ON IMPACT OF SORTING TRAINING DATA FOR VAE TRAINING
The non-sorted dataset was generated by shuffling each track of the original VAE training dataset 10 different times, resulting in 10X bigger dataset (10M tracks). It trained for 5X longer for 5M training steps. We planned on training for 10M gradient steps (10X than the original VAE) but stopped at 5M as it converged much sooner. We both CLUTR and CLuTR-shuffed, i.e.,
18

---

Under review as a conference paper at ICLR 2023
Track
DR
PLR
Robust PLR
PATRED
REPAIRED
CLUTR (2M)
Attention Agent
Australia
484
545 23
692 + 15
100
47
683
826
Austria
409 21
4
615 13
92
3
597
511
Bahrain
27
41
15
19
23
40
372
Belgium
328 12811
327
42
72
19
429
668
Brazil
309
17
43
16
256 19
3
145
China
24
8:
24
-101
7
28
344
France
29 3
47
13
240
498
153
Germany
23
388 + 20
48
16
22
404
2/4
Hungary
3
533 26
17
98
414
63
769
Italy
461 27
588 20
625
13
25
16
798
Malaysia
25
283 20
400
126
200
426
300
Mexico
33
561 1 21
12
67
415 30
627
580
Monaco
268
486 19
-28 18
256
46
835
Netherlands
418 26
25
70
307
41
131
Portugal
27
407 :
43
13
265 21
4
606
Russia
30
479 24
649
51
419
49
732
Singapore
336 29
386 22
16
14
274
382 + 19
276
Spain
433
482
142
14
358 24
496 15
759
UK
456 16
538 13
138
380 + 22
471 + 19
729
USA
263
243
28
381#33
11
120 + 25
238 31
-192
Mcan
27
22
531
26 19
295 23
468 21
478
Table 4: Comparison between CLUTR and other UED algorithms. Boldface denotes SOTA among UED algorithms, while italic in the Attention Agent colum means, CLUTR is compara- ble/outperforms the attention agent on that track. CLUTR outperforms PAIRED by big margin with 18x bigger mean return on the entire FI Dataset. CLUTR also outperforms Domain Random- ization, PLR, and REPAIRED and only falls short to Robust PLR. Nonetheless, CLUTR shows competitive results compared to Robust PLR, showing comparable results in seven out of the test tracks and outperforming in the Netherlands track. CLUTR also outperforms the non-UED SOTA on the full FI dataset. CLUTR outperforms the Attention Agent on 9 out of the 20 tracks and shows comparabbeeerformance in one. must be noted, all the UED methods except CLUTR was trained for 5M timesteps where CLUTR was run for 2M timesteps.
Moar Mergrtt withiwhhouu ruxble Hegrot
Hatums- IN withiethout leeble
2.5
20
Emvichmonn Rotime
600
1.6
400
1.0
CLUTR wio flerible 'ightt Agent
08
CLUTR wio floibta vgrus Anlagoniat
CLUTR Besble
Angent
CLUTR
TimeStap
Timeškip
(a) Mean Regret- Car Racing with vs without flexible (b) Returns UED generated Car Racing tracks with without flexible regret loss
regret loss
Figure 12: Mean Regret and agent returns during training CLUTR (with flexible regret) vs CLUTR with standard PAIRED regret approximation.
CLUTR with VAE trained non-sorted data up 5M timesteps. CLUTR-shuflee shows inferior performance and also signs of unlearning compared to CLUTR. Figure[66showw detailed experiment results.
19

---

Under review as a conference paper at ICLR 2023
500
Envronmont Roture
Roger
800
200
CLUTR wub fler PAITTD
05
Antagenee
TimeStop
TimeStop
(a) Mean Regret- Car Racing
Returns UED generated Car Racing tracks
Figure 13: Mean Regret and agent returns during training CLUTR with standard PAIRED regret loss (i.e., without the flexible regret). CLUTR shows smaller regret value(i.e.. closely competing agent and antagonist), indicating a better UED curriculum.
Neble on
Sca
Singooee
CLITR PARED
CLUTR-PAANED
CAITRIARED
CLUTA
CLUTR
Figure 14: Zero-shot generalization of both PAIRED and CLUTR (with the standard regret loss) agents after 5M timesteps the full FI benchmark. CLUTR with the standard regret loss outper- forms PAIRED on every track. For each track, we the agents on different episodes and the error bar denotes the standard error.
000
200
CLmM fedilll Wii
TraeBtes
Figure 15: Test Returns on Selected Tracks (Vanilla, Singapore, Germany, and Italy) of CLUTR with standard PAIRED regret loss alongside PAIRED performance.
20

---

Under review as a conference paper at ICLR 2023
0001
400
Enircceent
400
Toot Rotumo
300
200
CLUIH CLUTR Shuffed PAIRE
Stop
(a) During training CLUTR agent achieves higher turns while, CLUTR- shufited agent shows lower re- (b) CLUTR achieves higher and more stable mean re- turns. CLUTTR-huuflee agent's return also less sta- turns on selected tracks. CLUTR-Shufflc shows ble showing a decrease and increase.
signs of unlearning.
Analysis of sorting training data for VAE. Trained on shuffled data, CLUTR-Shufled performs inferior compared to CLUTR and shows signs of unlearning.
Figure 16:
D.4 IMPACT OF TASK REPRESENTATION LEARNING
In this section, discuss the impact of the learned task representation on performance. In Section 5.4 showed that if we finetune the
120
Devieten in Perferracee
133
VAE decoder during curriculum learning, the overall performance drops significantly (Fig- ure 6). To get better understanding, in Fig-
ure 17 we plot how much the performance deviates as the VAE decoder changes during the training process. The curve in red shows
Sec altin for Pntrased Decoca
the deviation of the decoder from its pretrained weights as fine-tuned during the training. We estimate the deviation as the L2 distance be- tween the finetuned and the pretrained decoder weights. The green curve shows the perfor- mance drop from CLUTR (with standard loss). To estimate the performance drop, we period- ically evaluate both CLUTR and CLUTR with Finetuned VAE, the selected test tracks dur- ing training. From the figure, we observe that, as the decoder weights are finetuned, they become increasingly different from the initial pretrained weights. At the same time, the overall performance gap from CLUTR also increases. This suggests that the pretrained VAE weights are crucial for better performance.
Figure 17: Impact of pretrained decoder weights on performance. The red curve plots the devia- tion of the decoder from its pretrained weights as is finetuned. The green curve shows the perfor- mance drop from CLUTRR with the standard loss. These curves suggest that pretrained weights are crucial for performance.
Furthermore, the quality of the learned representation depends on the quality of the data they are trained on. In section 5.3 we showed that VAE trained on a non-sorted dataset significantly de- teriorates the performance (Figure (). This further suggests that the learned representation has significant impact on performance. We also want to note that both of these variations (CLUTR with Finetuned VAE and the CLUTR with Shuffled VAE) perform much better than PAIRED, which suggests that, though CLUTR's performance depends on the representation, with a reasonable rep- resentation, it can still perform better than PAIRED.
21

---

Under review as a conference paper at ICLR 2023
E DETAILED RESULTS ON MINIGRID
E.1 CURRICULUM ANALYSIS
Figure 19ssows 3D Histograms showing the frequency of the generated grids against the total num- ber of obstacles they contain. PAIRED starts with high number of obstacles and then degenerates quickly into grids with very few numbers of obstacles and stays similar for a significant number of steps. Eventually, the number of obstacles increases sharply, converging into band of around 20 to 40 obstacles on average. On the other hand, CLUTR, the number of obstacles starts flat, centers around peak around the middle but still with wide interval for some number of steps, and the peak drops slightly while the interval stays almost the same. After the 'convergence', PAIRED rarely generates grids with fewer more obstacles than the band it converges to. On the contrary, CLUTR still generates grids with few or many blocks, which might help to address unlearning or improve the agents on grids with more obstacles, respectively. The above observations illustrate that we can achieve a more efficient curriculum learning without making the problem too easy early or without focusing on narrow interval with flat distribution later. Instead, we can start with wide interval and gradually focus on a peak around the middle without making the interval very narrow.
Number of Trainin
Narbbr Trainin Gends
Number of Cibstacies
Numbbrr Chittccees
Training Progress
Trannig Frogrecs
(a) CLUTR
(b) PAIRED
Figure 18: 3D Histograms showing the frequency of the generated grids against the total number of blocks they contain. Both PAIRED and CLUTR converge to a similar band of grids. However, CLUTR converges much faster.
Figure 19a shows the average episode lengths of both CLUTR and PAIRED. The curves show both methods start with long gpisodes-indicating at the beginning, the agents do not solve the training grids consistently, and many of the episodes end due to timeout. As the agents learn, the episodes become shorter for both methods until they converge to small value. However, CLUTR converges sooner than PAIRED.
We also compare the average solution length of the solved training grids. Both PAIRED and CLUTR show a similar pattern. However, PAIRED converges to a larger value than CLUTR. This might in- dicate that CLUTR is solving the environments more efficiently. This might also mean that CLUTR solving some easier tasks (e.g., fewer obstacles, as we noticed from Figure 19 even after conver- gence lowering its average solved path length slightly.
E.2 CLUTR CURRICULUM DOMAIN RANDOMIZED CURRICULUM THE LATENT SPACE: DOES CLUTR TEACHER DEGENERATES INTO A RANDOMIZED POLICY?
To answer whether CLUTR teacher actually learns something or degenerates into a randomized policy, we compare the curriculum generated by CLUTR with random uniform (i.e., Domain Randomization) curriculum. We generate the DR curriculum by repeatedly sampling the trained VAE (the same VAE used by CLUTR) with uniform random distribution. Figure 20 shows the comparison characterizing the grids by the number of obstacles they contain similarly as the previous section. As expected, see that the DR curriculum generates grids with obstacles ranging from
22

---

Under review as a conference paper at ICLR 2023
120
20.0
CLUTR
PAIRED
Average Episade Length
100
17.5
CLUTR
15.0
80
Solved Length
22.5
60
10.0
20
5.0
2.5
0
1000
2000
3000
4000
5000
6000
Training Timesteps
0.0
0
1000
2000
3000
4000
5000
6000
Logging Timestep
(a) Average length of the training episodes. CLUTR converges sooner than PAIRED shorter episode (h) Average solution length of the solved training length.
tasks.
Figure 19: Comparison of CLUTR and PAIRED curriculum based on properties of the generated grids.
50. The histograms clearly visualize the significant differences in the curricula, implying the CLUTR teacher indeed learns useful curriculum as suggested by the empirical result.
of
Number of Training
Number Obolacles
Nember of chatecino
Training Progross
Herrainn
CLUTR
(b) Domain Randomization of the Latent Space
Figure 20: 3D Histograms showing the frequency of the CLUTR generated grids against the total number blocks they contain vs. randomly generated grids.
E.3 ANALYSIS OF THE LATENT TASK MANIFOLD
To grow sense of the latent task manifold, linearly interpolate in the latent space between an empty grid and a 15x version of the FourRoom grid (shown in Figure 21.. Figure 22 visualizes the interpolation results. We first get the latent vectors of the empty grid and the target FourRoom equidistant
task using the VAE encoder. We then linearly interpolate points between them. At last, we reconstruct the grids from these vectors using our decoder. From Figure 22 we see that, as we interpolate in the latent space, the reconstructed grid incrementally adds more obstacles and the grids start to look more like the FourRoom target grid. We note that the reconstruction is not perfect. We also note that the increase in the number of obstacles is not uniform, e.g., the first reconstructed grids are all empty grids, and more obstacles added near the target
Figure 21: 15X15 Four- Rooms
23

---

Under review as a conference paper at ICLR 2023
Figure 22: A linear interpolation between an empty grid and 15x15 version of the Four-Room grid (Figure 21 in the latent space. The grids organized from top-left to bottom-right in row-major order.
point. Overall, this experiment provides an insight that the latent space holds a useful structure, which CLUTR teacher utilizes to generate the curriculum.
24


Reviews:
[{"review_id": "xZ5HWnkJMH", "reviewer": "Reviewer_NPYm", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates learning a task representation space for use in unsupervised environment design (UED), with an algorithm called PAIRED. PAIRED trains an RL agent, the adversary, to configure the environment parameters throughout training, in tandem with two RL agents that learn to solve these configurations (each configuration is termed a \"task\"). The adversary is allied with one of the agents, called the antagonist, and seeks to maximize the margin of return by which the antagonist outperforms the other agent, called the protagonist. Thus, PAIRED produces a curriculum that maximizes a lower bound on the regret experienced by the protagonist, leading to a robust minimax-regret policy at equilibrium. This paper then proposes CLUTR, a method that first trains a recurrent VAE over a large number of randomized tasks in a specific domain (e.g. mazes or car racing tracks) and then performs PAIRED over the learned latent space of the VAE. Their results show that PAIRED in such latent-spaces produces more robust protagonist policies than standard PAIRED, which designs tasks in the direct task space.", "summary_of_the_review": "This paper provides an interesting extension of UED, in which a PAIRED adversary, which generates regret-maximizing tasks for an RL agent, searches for such tasks in a learned latent space. The idea in itself is interesting, but the current paper lacks important experimental details and makes several ambiguous or unjustified claims. Therefore, I cannot recommend this paper for acceptance.", "strength_and_weaknesses": "### Strengths\n- The benefits of performing UED on a learned latent manifold of the task space is well-motivated.\n- The connection between CLUTR and prior works is clearly discussed.\n- The CarRacing experiments compare against a comprehensive set of baselines.\n\n### Weaknesses\nDespite what seem like promising results showing CLUTR outperforms PAIRED, there are several important points that are left unclear and claims that seem to be incorrect:\n\n**The regret objective used by CLUTR is unclear**\n\n- It is not clear whether CLUTR uses the flexible PAIRED objective or the standard PAIRED objective. Appendix D.2 seeks to compare CLUTR with the _standard PAIRED regret objective_, implying the main results use the flexible PAIRED objective.\n- If CLUTR uses the flexible regret objective, then the main experiment results should compare to PAIRED with the flexible regret objective, also known as \"flexible PAIRED\" in Dennis et al, 2020. It must be shown that CLUTR's gains over PAIRED with the standard regret objective are not in fact due to the choice of regret objective, rather than the task representation, as claimed.\n- If CLUTR does in fact use the flexible regret objective, then Theorem 1 from Dennis et al, 2020, which guarantees a minimax-regret policy for the protagonist at Nash equilibrium of the PAIRED game, is no longer directly applicable—since the flexible regret objective changes the nature of the underlying game between the three agents.\n- Moreover, the flexible regret objective was not introduced in Gur et al, 2021 as stated in the first paragraph of Section 3.2, but clearly detailed and evaluated in the appendix of Dennis et al, 2020.\n\n**CLUTR does not have access to the full task space**\n\nWhile PAIRED, in principle, has the ability to generate any 50-block maze, this is not necessarily true for CLUTR. This is because CLUTR is trained on a limited subset of all possible tasks, and the learned latent space is not guaranteed to contain all possible mazes. Therefore, Property 3 of CLUTR, stated on Page 5, is incorrect.\n\n**Interpretation of Figure 4 and 5 is flawed**\n\n- In Figure 5, CLUTR's adversary achieves lower regret than that of PAIRED. This result implies that the latent task representation space makes the optimization of the regret objective a harder problem for the teacher, rather than easier, as claimed in the paper. \n- Figure 4 shows the solved rate of the agent on the _training_ levels proposed by the CLUTR and PAIRED adversaries. Since the goal of UED is to produce an adversarial curriculum, it seems that the higher solved rates achieved by the agent under the CLUTR adversary implies the CLUTR adversary is less effective and does a worse job at optimizing for agent's regret. \n\n**The results do not provide evidence for whether the VAE and CLUTR adversary learn anything**\n\n- While the paper argues that CLUTR has the advantage of avoiding sequential credit-assignment, CLUTR must pay the additional cost of a much larger action space—based on the latent-space dimensionality. Thus, CLUTR may in fact face a much more difficult learning problem than PAIRED. Further, the VAE must also learn a useful representation of the task space, the success of which is not shown in the paper.\n- The paper should provide visualizations of how tasks are distributed within the VAE's latent space to provide evidence that meaningful latent structure is being learned via their method. The training curves for the VAE would also be useful to see.\n- The paper should provide evidence that CLUTR's metrics in Figures 3, 4, and 5 are different from that of domain randomization—which would show that CLUTR's adversary is learning to exploit the structure in the latent space, rather than resorting to a randomized policy.\n\nThese figures should compare to domain randomization (DR), and importantly, show that CLUTR is doing something meaningfully different from randomizing over the latent space (which would be case if the adversary policy has difficulty learning to design in this space).\n\n**Curriculum analysis is lacking**\n\n- The main emergent complexity and curriculum results can benefit from thorough analysis. In particular, the authors should plot the solution path length and number of blocks in training maze tasks proposed by the CLUTR adversary, in comparison to other methods. \n- Since the CLUTR tasks are generated by decoding latent representations using the pre-trained VAE, it is possible that CLUTR produces mazes with more obstacle blocks than PAIRED, which is strictly limited to at most 50 blocks. Prior works (Jiang et al, 2021) show that going from 25 to 50 blocks improves the OOD transfer performance of all methods compared, so it seems that this factor is a confounder.\n- Importantly, this curriculum analysis should compare to DR to show that CLUTR learns a meaningful curriculum.\n\n**Missing several key details**\n\nIn addition to the lack of clarity around CLUTR's regret objective, a few other important details seem to be missing. (See the Clarity section for details.)\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "iK_ip0H82R", "reviewer": "Reviewer_Ui8Z", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes CLUTR, a curriculum learning algorithm based on PAIRED, which decouples\ntask representation and curriculum learning. The task representation is learned \nusing a LSTM-based recurrent VAE, after which the teacher updates the curriculum and \nthe protagonist and antagonist policies are updated according to the regret\nof the generated tasks. The experiments showed empirically that CLUTR outperforms\nstate-of-the-art UED methods in terms of sample efficientyc and generalization\n on a range of tasks, and performs compratably to the non-UED state-of-the-art on the CarRacing task. \n", "summary_of_the_review": "The paper tackles the problem of sample inefficiency in RL problems by pre-training a VAE to embed the tasks in to a manifold and update policies based on sampled trajectories from the tasks, however, the novelty of the paper is a bit limited. Having said that. given the empirical performance on the extensive sets of experiments, the proposed method may be helpful for the community in practice. \n", "strength_and_weaknesses": "\nstrength: The paper is well written.\nThe experiments showed promising results and covered a range of hypothesis claimed in the paper. \n\n\nweakness: the novalty of the paper is somewhat limited. The main techniques used in CLUTR are based on VAE and PAIRED, which are existing methods. \n\nI am curious to know the effect of learning the representation of the tasks. \nFor example, how sensitive does the performance depend on the fitting of the VAE representation? Is the VAE represntation is mis-specified, how much does it worsen the performance?\n\n\nMinor:\n* E is not defined in 4.1 \n* equations in 4.1 are better written with $\\log q(z)$ \n* missing space in section 4: Section4.4 -> Section 4.4\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "d7uC4y_8dR", "reviewer": "Reviewer_eq4e", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper targets the topic of unsupervised environment design (UED) in curriculum reinforcement learning. In particular, it modifies PAIRED by using the decoder of a VAE as the task generator. With this improvement, the model decouples task representation and curriculum learning into a two-stage optimization. Thus, the authors observe that the training is more stable and the final reward is higher than existing UED methods.", "summary_of_the_review": "Due to my concerns about the clarification, I suggest rejecting this paper. The authors are welcome to address my questions.", "strength_and_weaknesses": "### **Strength**\n\n* **Interesting topic.** Unsupervised Environment Design is an important direction in curriculum reinforcement learning. Previous works usually suffer from unstable training problems. This paper targets at solving this problem by using the decoder of VAE to generate tasks.\n\n* **Comprehensive experiment.** The experiment part looks good to me. Some important factors are explored respectively.\n\n### **Weaknesses**\n\n* **Things to be clarified.** \n    * The proposed method heavily depends on PAIRED but details of PAIRED are missing in Section 4.3. \n    * In the 4th line of Algorithm 1, what does it mean by “Use adversary to sample latent task vector”? If the adversary is a parametrized model, what distribution is used for sampling z?\n    * What is single-step RL? It seems not a widely used term but the definition is missing. I guess the authors mean that their method does not do sequential generation with the interaction with the environment.\n    * The second difference between CLUTR and PAIRED is hard to follow. More explanation is needed. Why does the state space of PAIRED rely on the underlying POMDP?\n    * No example of permutation parameters is given in the experiment environment. The authors say that “for a navigation task, a set of obstacles corresponds to factorially different permutations of the parameters.” What is the obstacle in the Car racing experiment? \n    * I don’t think VAE(z, E) in equation (1) is a standard notation. It should be clarified.\n\n\n* **Formulation of CLUTR.** The derivation in Appendix B seems redundant to me. According to Figure 1, variable R actually does not involve the VAE inference since it only depends on another observable variable E. Therefore, Equation (1) is just the ELBO of VAE plus an arbitrary regularization term that relates R and E. I think the authors can directly say that rather than taking a detour.\n\n* **Motivation.** If I understand correctly, this paper proposes to use VAE to generate tasks because of two advantages, i.e., long-horizon credit assignment and permutation invariant. However, searching in the latent space itself is not an easy task since the smoothness of latent space heavily depends on the training and quality of the dataset. So, does this method always outperform PAIRED in general? For example, when most of the uniformly sampled environment parameters are invalid.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject