Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper studies the problem of automatically designing a distribution of environments that adjusts to the learning agent, called unsupervised environment design. Specifically, they propose Dual Curriculum Design in which two co-evolving teachers, i.e. one is a generator and another one is a replay teacher, introduce new levels/environments to the learning agent (a.k.a student) to learn a policy that can be effectively generalized to new and unseen environments. Besides, this paper proposes a new theory about PLR and REPAIRED robustness guarantees. The paper evaluates zero-shot generalization, emergent complexity, and scalability of their proposed method in multiple environments and scenarios.
main_review: While this paper studies an important and interesting problem of automatically designing the distribution of environments for a learning agent, unfortunately, the proposed method(s) is confusing and the main contribution of the paper is not clear and I don't fully understand what are the takeaways from this paper. Namely:

       1. The main idea of the paper is built on top of PAIRED where a replay teacher is added to it and called REPAIRED. Although REPAIRED results are not good and not consistent across all experiments, this could have been the main story of the paper. 

       2. PLR and PLR+ ( for simplicity, I use PLR+ instead of PLR^\perp): The discussions, theory, changes to PLR, and finally introduction of PLR+ are quite out of place in this paper. If the PLR+ is the contribution of this paper, then the paper should be about this not REPAIRED. It seems the authors decided to include both since neither one shows good results in all experiments and each doesn't have a significant contribution alone.

Writing, results, and presentation: It is quite hard to follow the paper and get a clear picture of the experiments:

       3. What does robustness mean here? Are you referring to generalization? or something else?  

       4. Why are some methods used in some experiments but not in others? For instance, PLR+ is not in Figure 3, or part of Figure 4? 

       5. While I applauded authors to call out their bug in Figure 6, I am not sure what can I conclude about this paper since sometimes REPAIRED shows better results and some other time PLR+., take Figure 6 as an example, REPAIRED is better in F1-Italy and PLR+ is good in F1-Singapore and F1-Germany (both methods shows similar performance for CarRacing-v0 though) 

       6. What is B in line 164? 

       7.  What is the relationship between \pi* and \pi in Eq1? What is \pi*?

       8. What does context mean in 145? "However, to formalize this game, we must first formalize the single-teacher context. Suppose the UPOMDP is clear from context."

       9. What is \theta_A in line 106? Is it a learnable parameter? Is it \pi_A parameter? or something else?

       10. Why maximum-regret objective (U^R) and uniform objective (U^U) are defined in line 152?   They are not referred to or used in section 3 and only referred later in section 4.1

       11. Where is Figure 7 in the main text? (lines 264 and 283) Figure 7 in the appendix talks about something else and doesn't tell anything about replay rate of 0.5 for example.

       12. What are the free parameters of an underspecified environment? what underspecified means (line 4)? 

       13. When this method generates a level, what are the mechanisms in place to make sure that there exists a solution and that level can be solved? 


In summary, the significance and technical contributions of this work are not clear, the paper needs significant revision, and it is not ready for this conference. I'd highly recommend focusing on one concrete story in the paper rather than try to run two different narratives.   


limitations_and_societal_impact: This paper discusses an approach for unsupervised environment design based on existing methods for toy/simulated environments ​which doesn't have a social impact. As such, I don't think this work has any negative societal impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes to theoretical analysis of the Prioritized Level Replay (PLR) from prior work and proposes an environment generation framework named Dual Curriculum Design (DCD). The proposed framework is composed of teacher agents that keep generating new environments and a student agent that learns to solve the generated environments. The teacher agents are co-adapted to generate new environments by maximizing the student's agent's regret. In a grid world domain and the CarRacing domain, the proposed method outperforms previous baselines such as PAIRD.
main_review: Strengths:

-  Based on prior work, this paper proposes novel theoretical analysis as well as algorithmic solutions for generating environments as curricula. 

- The proposed technical solution and the theoretical analysis seem to be reasonable to me. 

-  The paper is clearly written and easy to follow. 

- This paper could provide interesting insights to future work on environment generation. However, the significance of the proposed method is not entirely convincing due to missing several important baselines. 

Weaknesses:

- Several state-of-the-art curriculum learning methods should be compared as baselines, including  [Portelas et al. CoRL 2019], [Florensa et al. ICML 2018], and [Zhang et al. NeurIPS 2020]. It is also important to include the baseline that uniformly samples environments as curricula.

- In Figure 4 and Figure 5, it seems that the learning curves often suffer from noticeable performance regressions during the later stage of training. Could the authors provide more detailed explanations about this phenomenon?

- In some subfigures in Figure 4 (e.g. "SixteenRooms" and "Maze"), the learning curves are cut off before convergence is reached. 

- The GridWorld and the CarRacing domains are a little toyish. It would be better to include more challenging task domains for continuous space control as in [Portelas et al. CoRL 2019] and [Florensa et al. ICML 2018].
limitations_and_societal_impact: It would be better if the authors could provide more discussions and explanations about why the proposed method cannot be evaluated in continuous domains in this paper.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper considers a recent proposed problem, unsupervised environment design, that provides an environment selection scheme to enable the policy being trained under a curriculum learning setting. The proposed method combines two new approaches in this direction, PAIRED and PLR, where the former one learns to generate an environment while the latter one does not include a generator but instead gives a sampling scheme to choose previously encountered environments. The combined method is explained as a two-teacher dual curriculum game and explained theoretically. Experiments demonstrate the effectiveness of the combined method compared to PAIRED and PLR, respectively.
main_review: Some comments:

1. The proposed method combines two new approaches, PAIRED and PLR, in the scope of environment designed and replay sampling, where the former one learns to generate an environment while the latter one does not include a generator but instead gives a sampling scheme to choose previously encountered environments. The combined method is explained as a two-teacher dual curriculum game and explained theoretically. I think such an explanation is reasonable and also the experiments are well-designed and sufficiently demonstrate the effectiveness of the combined method to me.

2. The algorithm in Algorithm 1 implies that the combined method alternatively chooses PAIRED and PLR, relying on the sampled d. How is P_D(d) obtained? Moreover, PAIRED is chosen only when d=0. What is the distribution of P_D(d)?

3. In section 6.1, the teacher works by placing one obstructing block per time step and finally places the agent and the goal. This is similar to the implementation of PAIRED, but in which the agent and the goal are placed first. Are there any differences of the designed orders for the teacher? Moreover, it is not very clear to me that as the placement is sequentially decided by the teacher, is this sequential decision progress treated as an MDP process and solved by RL? If so, it is a bit confused for me that in this sequential decision progress, there is no interaction between the teacher and the agents, and therefore the entire designment of the maze can be completed offline, such that the sequential decision problem can be transferred to an one-step decision (generating the entire maze). Of course, generating the entire maze in one step is impractical to be encoded in an action space in RL, and then the problem is cast to train a maze (environment) generator. I am not sure if my understanding is correct, so would the authors provide more explanations on this?

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper considers two forms of curriculum design, unsupervised environment design and prioritized experience replay, and highlight their complementary nature.
It then considers the games induced by both these approaches and combines them into a dual curriculum game where the agent faces one teacher or the other with certain probability. Further, the paper analyzes the equilibria of this joint game and the equilibria of the two component games.

The paper then proposes two new algorithms, a robust version of PLR that does not learn on experienced trajectories but only on trajectories sampled by the PLR teacher, and a replay-enhanced PAIRED that improves the UED technique with PLR.

The various approaches are then compared on mazes and car racing domains.
main_review: ## Technical Positives:
* This paper show how UED/PAIRED and PLR can be seen as dual approaches and also how they can be combined.
* These approaches fall into the larger space of automatic curriculum design for agents, and explore how different approaches can be combined effectively.
* The game theoretic analysis of the equilibria of the combined game compared with the equilibria of the base games is shown to be an effective approach to analyze such a combination.
* Notably, analyzing PLR through this lens allows principled modifications to the prioritization scheme and the agent update scheme.
* REPAIRED seems to actually be the more explicit combination of the two approaches considered, and the successful combination of PLR and PAIRED is encouraging.
* Despite the other errors alluded to in the supplementary material, overall the experiments on the maze and car racing domains show a fair bit of success for the above combinations. Both PAIRED and PLR seem to benefit from the modifications suggested.

## Questions, Remarks and Drawbacks:
* It seems the supplementary material and the main text are out of sync. On page 7 the main text repeatedly refers to a Figure 7 and the related text does not seem to match up with the discussion. Similarly in Section 6.2 Figure 11 is referred to as a visualization of the test tracks, which should probably refer to Figure 12.
* The assertion that PLR and UED are dual problems seemed unclear and imprecise.
* This "duality" also seems to not have much bearing on the game that is formulated for further analysis.

limitations_and_societal_impact: Societal impact is not discussed. The checklist asserts that this discussion is not applicable to their research. However some discussion of curriculum design and its impact on learning agents might have been useful.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: In this paper, the authors propose a common framework,  Dual Curriculum Design, that augments PAIRED with a PLR-based replay mechanism, named as REPAIRED. The theory also suggests convergence to Nash Equilibrium should be assisted by training on fewer data when using PLR—namely by only taking policy-gradient updates from data that originates from the PLR buffer, and only using samples from the environment distribution to populate the buffer. The authors further conducted experiments on a maze environment and on car-racing tracks. Results showed that the proposed approaches outperformed the PLR and other baselines in general.
main_review: Strengths:
The paper provides theoretical analysis on the robustness guarantees for PLR method and REPAIRED.
The performance of the new proposed algorithms in both experiments is better than the state-of-the-art benchmarks.

Concerns:
The analysis is based on approximate Nash equilibria. The robustness is based on the accuracy of this approximation. However, the theoretical guarantee shows $\pi$ is $4Bp$ close to having optimal worst-case regret. How significant is this distance in the landscape of the policy? It seems to me, both $B$ and $p$ could be big so I have concerns of the effectiveness of the analysis. It will be nice to analyze the sensitivity of this distance to the effectiveness for robustness analytically and experimentally.

Minor things: Some notations are confusing:
1. Line 107, in the left of the equation, the regret is on $\pi_A$ and $\pi_B$, on the right of the equation, the values become $\pi_P$ and $\pi_A$. Actually, throughout the whole paper, the mix of using $\pi_P$ and $\pi_B$ might confuse the readers. It will be great if the notation could be consistent.
2. It will be nice if the authors could give further qualitative and numerical insights to the various performance with different maps. Particularly, why DR is the best among all the methods. Could this unveil some limitations of the proposed method/all PLA-based methods?
limitations_and_societal_impact: No negative social concerns.
review_title: null

Meta-review: The paper proposes a unified approach for environmental design in RL. 

The reviewers positively evaluated the paper and recommended for acceptance. 

AC. 
Decision: Accept (Poster)