Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper investigates learning a task representation space for use in unsupervised environment design (UED), with an algorithm called PAIRED. PAIRED trains an RL agent, the adversary, to configure the environment parameters throughout training, in tandem with two RL agents that learn to solve these configurations (each configuration is termed a "task"). The adversary is allied with one of the agents, called the antagonist, and seeks to maximize the margin of return by which the antagonist outperforms the other agent, called the protagonist. Thus, PAIRED produces a curriculum that maximizes a lower bound on the regret experienced by the protagonist, leading to a robust minimax-regret policy at equilibrium. This paper then proposes CLUTR, a method that first trains a recurrent VAE over a large number of randomized tasks in a specific domain (e.g. mazes or car racing tracks) and then performs PAIRED over the learned latent space of the VAE. Their results show that PAIRED in such latent-spaces produces more robust protagonist policies than standard PAIRED, which designs tasks in the direct task space.
strength_and_weaknesses: ### Strengths
- The benefits of performing UED on a learned latent manifold of the task space is well-motivated.
- The connection between CLUTR and prior works is clearly discussed.
- The CarRacing experiments compare against a comprehensive set of baselines.

### Weaknesses
Despite what seem like promising results showing CLUTR outperforms PAIRED, there are several important points that are left unclear and claims that seem to be incorrect:

**The regret objective used by CLUTR is unclear**

- It is not clear whether CLUTR uses the flexible PAIRED objective or the standard PAIRED objective. Appendix D.2 seeks to compare CLUTR with the _standard PAIRED regret objective_, implying the main results use the flexible PAIRED objective.
- If CLUTR uses the flexible regret objective, then the main experiment results should compare to PAIRED with the flexible regret objective, also known as "flexible PAIRED" in Dennis et al, 2020. It must be shown that CLUTR's gains over PAIRED with the standard regret objective are not in fact due to the choice of regret objective, rather than the task representation, as claimed.
- If CLUTR does in fact use the flexible regret objective, then Theorem 1 from Dennis et al, 2020, which guarantees a minimax-regret policy for the protagonist at Nash equilibrium of the PAIRED game, is no longer directly applicable—since the flexible regret objective changes the nature of the underlying game between the three agents.
- Moreover, the flexible regret objective was not introduced in Gur et al, 2021 as stated in the first paragraph of Section 3.2, but clearly detailed and evaluated in the appendix of Dennis et al, 2020.

**CLUTR does not have access to the full task space**

While PAIRED, in principle, has the ability to generate any 50-block maze, this is not necessarily true for CLUTR. This is because CLUTR is trained on a limited subset of all possible tasks, and the learned latent space is not guaranteed to contain all possible mazes. Therefore, Property 3 of CLUTR, stated on Page 5, is incorrect.

**Interpretation of Figure 4 and 5 is flawed**

- In Figure 5, CLUTR's adversary achieves lower regret than that of PAIRED. This result implies that the latent task representation space makes the optimization of the regret objective a harder problem for the teacher, rather than easier, as claimed in the paper. 
- Figure 4 shows the solved rate of the agent on the _training_ levels proposed by the CLUTR and PAIRED adversaries. Since the goal of UED is to produce an adversarial curriculum, it seems that the higher solved rates achieved by the agent under the CLUTR adversary implies the CLUTR adversary is less effective and does a worse job at optimizing for agent's regret. 

**The results do not provide evidence for whether the VAE and CLUTR adversary learn anything**

- While the paper argues that CLUTR has the advantage of avoiding sequential credit-assignment, CLUTR must pay the additional cost of a much larger action space—based on the latent-space dimensionality. Thus, CLUTR may in fact face a much more difficult learning problem than PAIRED. Further, the VAE must also learn a useful representation of the task space, the success of which is not shown in the paper.
- The paper should provide visualizations of how tasks are distributed within the VAE's latent space to provide evidence that meaningful latent structure is being learned via their method. The training curves for the VAE would also be useful to see.
- The paper should provide evidence that CLUTR's metrics in Figures 3, 4, and 5 are different from that of domain randomization—which would show that CLUTR's adversary is learning to exploit the structure in the latent space, rather than resorting to a randomized policy.

These figures should compare to domain randomization (DR), and importantly, show that CLUTR is doing something meaningfully different from randomizing over the latent space (which would be case if the adversary policy has difficulty learning to design in this space).

**Curriculum analysis is lacking**

- The main emergent complexity and curriculum results can benefit from thorough analysis. In particular, the authors should plot the solution path length and number of blocks in training maze tasks proposed by the CLUTR adversary, in comparison to other methods. 
- Since the CLUTR tasks are generated by decoding latent representations using the pre-trained VAE, it is possible that CLUTR produces mazes with more obstacle blocks than PAIRED, which is strictly limited to at most 50 blocks. Prior works (Jiang et al, 2021) show that going from 25 to 50 blocks improves the OOD transfer performance of all methods compared, so it seems that this factor is a confounder.
- Importantly, this curriculum analysis should compare to DR to show that CLUTR learns a meaningful curriculum.

**Missing several key details**

In addition to the lack of clarity around CLUTR's regret objective, a few other important details seem to be missing. (See the Clarity section for details.)

clarity,_quality,_novelty_and_reproducibility: ### Clarity
The paper does a good job of explaining the high-level details of the method, as well as providing background on related topics and the experimental setup. However, there are several important details that are missing:
- As previously mentioned, it is not clear whether the main results are based on running CLUTR with the flexible regret objective or the standard PAIRED regret objective.
- The maze navigation results should report the number of training seeds used.
- It is not clear why the CarRacing experiments compare to the additional Robust PLR and ACCEL baselines, but the navigation experiments do not.
- In Section 5.4, the method of fine-tuning the decoder throughout training on the agent's regret is not provided anywhere in the paper. Could the authors provide details on how this fine-tuning is performed? It is unclear, as it seems the VAE was not trained to predict the regret values—which are not available during pre-training.

### Quality
- Overall, the paper has several rough edges and could benefit from copy editing.
- There are several claims that are incorrect or unjustified by experimental evidence, which are detailed in the Weaknesses section of this review.

### Novelty
- The idea of latent-space task design has been explored in other works such as Florensa et al, 2017, which the authors should also cite. 
- The idea of learning a task representation for UED has not been previously explored in detail.

### Reproducibility
- Due to the lack of clarity around key details described in this review, and the lack of experimental code from the authors, the results of this study are not reproducible based on the information available.

**References**
Florensa, Carlos, et al. "Automatic goal generation for reinforcement learning agents." International conference on machine learning. PMLR, 2018.
summary_of_the_review: This paper provides an interesting extension of UED, in which a PAIRED adversary, which generates regret-maximizing tasks for an RL agent, searches for such tasks in a learned latent space. The idea in itself is interesting, but the current paper lacks important experimental details and makes several ambiguous or unjustified claims. Therefore, I cannot recommend this paper for acceptance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes CLUTR, a curriculum learning algorithm based on PAIRED, which decouples
task representation and curriculum learning. The task representation is learned 
using a LSTM-based recurrent VAE, after which the teacher updates the curriculum and 
the protagonist and antagonist policies are updated according to the regret
of the generated tasks. The experiments showed empirically that CLUTR outperforms
state-of-the-art UED methods in terms of sample efficientyc and generalization
 on a range of tasks, and performs compratably to the non-UED state-of-the-art on the CarRacing task. 

strength_and_weaknesses: 
strength: The paper is well written.
The experiments showed promising results and covered a range of hypothesis claimed in the paper. 


weakness: the novalty of the paper is somewhat limited. The main techniques used in CLUTR are based on VAE and PAIRED, which are existing methods. 

I am curious to know the effect of learning the representation of the tasks. 
For example, how sensitive does the performance depend on the fitting of the VAE representation? Is the VAE represntation is mis-specified, how much does it worsen the performance?


Minor:
* E is not defined in 4.1 
* equations in 4.1 are better written with $\log q(z)$ 
* missing space in section 4: Section4.4 -> Section 4.4

clarity,_quality,_novelty_and_reproducibility: In general, the paper is well presented and well written. The authors claim they will release the code upon acceptance. 

summary_of_the_review: The paper tackles the problem of sample inefficiency in RL problems by pre-training a VAE to embed the tasks in to a manifold and update policies based on sampled trajectories from the tasks, however, the novelty of the paper is a bit limited. Having said that. given the empirical performance on the extensive sets of experiments, the proposed method may be helpful for the community in practice.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper targets the topic of unsupervised environment design (UED) in curriculum reinforcement learning. In particular, it modifies PAIRED by using the decoder of a VAE as the task generator. With this improvement, the model decouples task representation and curriculum learning into a two-stage optimization. Thus, the authors observe that the training is more stable and the final reward is higher than existing UED methods.
strength_and_weaknesses: ### **Strength**

* **Interesting topic.** Unsupervised Environment Design is an important direction in curriculum reinforcement learning. Previous works usually suffer from unstable training problems. This paper targets at solving this problem by using the decoder of VAE to generate tasks.

* **Comprehensive experiment.** The experiment part looks good to me. Some important factors are explored respectively.

### **Weaknesses**

* **Things to be clarified.** 
    * The proposed method heavily depends on PAIRED but details of PAIRED are missing in Section 4.3. 
    * In the 4th line of Algorithm 1, what does it mean by “Use adversary to sample latent task vector”? If the adversary is a parametrized model, what distribution is used for sampling z?
    * What is single-step RL? It seems not a widely used term but the definition is missing. I guess the authors mean that their method does not do sequential generation with the interaction with the environment.
    * The second difference between CLUTR and PAIRED is hard to follow. More explanation is needed. Why does the state space of PAIRED rely on the underlying POMDP?
    * No example of permutation parameters is given in the experiment environment. The authors say that “for a navigation task, a set of obstacles corresponds to factorially different permutations of the parameters.” What is the obstacle in the Car racing experiment? 
    * I don’t think VAE(z, E) in equation (1) is a standard notation. It should be clarified.


* **Formulation of CLUTR.** The derivation in Appendix B seems redundant to me. According to Figure 1, variable R actually does not involve the VAE inference since it only depends on another observable variable E. Therefore, Equation (1) is just the ELBO of VAE plus an arbitrary regularization term that relates R and E. I think the authors can directly say that rather than taking a detour.

* **Motivation.** If I understand correctly, this paper proposes to use VAE to generate tasks because of two advantages, i.e., long-horizon credit assignment and permutation invariant. However, searching in the latent space itself is not an easy task since the smoothness of latent space heavily depends on the training and quality of the dataset. So, does this method always outperform PAIRED in general? For example, when most of the uniformly sampled environment parameters are invalid.

clarity,_quality,_novelty_and_reproducibility: ### **Clarity**
There are some clarification problems (as mentioned in the weakness part) that need to be solved. Although the general idea of the proposed method is easy to understand, a lot of details are missing.

### **Quality**
The quality of this paper is limited by the presentation. A clearer organization would improve the quality.

### **Novelty**
The novelty of this paper is ok but maybe around the average level. If I understand correctly, the only modification of this paper is replacing the task generator with a pre-trained VAE.

### **Reproducibility**
Not sure. Code is not provided.



summary_of_the_review: Due to my concerns about the clarification, I suggest rejecting this paper. The authors are welcome to address my questions.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes CLUTR, a curriculum-learning algorithm based on PAIRED which decouples task representation and curriculum learning. The experimental results show that CLUTR performs better that other existing methods.

One of the main criticisms of this paper is about the curriculum learned by the CLUTR. Reviewer NPYm argues that CLUTR is only matching the domain randomization baseline and it needs more analysis to understand what is going on. Given that this paper still requires some work, I recommend rejection and encourage the authors to incorporate the reviewers' comments and suggestions for their future submission.
justification_for_why_not_higher_score: The paper has no strong reasons to be accepted given the concerns about the validity of the proposed improvements. Authors definitely need to have convincing arguments why their method is better than the simple domain randomization baseline.
Decision: Reject