Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper addresses the problem that RL is bad at learning the compositional rules guiding compositional problems. It proposes a two pronged approach: first, it proposes a task (de)composition strategy that allows building up tasks (or POMDPs) compositionally. Second, it proposes a regret-driven curriculum strategy that is able to construct compositional tasks of increasing complexity without supervision, which, in turn, can be used to train a learner that is then able to pick up on the rules of compositionality guiding the domain.
main_review: Pros:
- The paper is very well written and easy to follow.
- The goal of the paper is extremely important: very many real-world scenarios are compositional, and RL is severely lacking in this regard.
- The individual design choices are well motivated.
- The core idea, guiding the complexity of the domains given to the learning agent by a measure of regret, is intuitively plausible.
- The authors test on existing domains (which appear to be solved) and design new harder domains that they release as open source.

Cons:
- The first contribution of the paper, the hierarchical and compositional decomposition of tasks using petrinets, is only explained in the appendix. Given the relevance to the overall goals of the paper, I think that at least a paragraph or two should be spend on that in the main paper.
- There is a discrepancy in the complexity of the two tasks: it looks as if the website domain is vastly more complex than the gridworld domain. Given that the gridworld domain would be more general and thus more relevant to the research community, it would have been nice to have this also scale to high complexity.
- I find myself doubting the value of the new domains the authors are releasing along with the paper. Not because I am not convinced of their relevance (in fact, I do believe that the domains could be highly relevant), but because the suggested approach in the paper solves these domains (success > 95%), effectively making them redundant before they were even released to the public.

Originality:
To my knowledge, the proposed regret-based curriculum training strategy is novel and a creative solution to a fundamental problem.

Quality:
The quality of the submission is very high. The language is clear, the paper is well structured, and the illustrations are good (though the text on the illustrations could be larger and sharper).

Clarity:
The clarity of the submission is high. Each argument made is easy to follow, and the overall structure makes it easy to understand each point in context.

Significance:
The core problem of the paper - RL's difficulties in generalizing over compositional domains - is highly significant. The significance of the entire paper is somewhat reduced by their choice of narrow domain.
limitations_and_societal_impact: not applicable
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a framework to generate compositional tasks as automated curricula using a graph representation. The proposed method jointly trains a task generator and the RL agent by maximizing the regret between the trained agents and the best-performing agent and the difficulty of the environment. Experiments are conducted in two domains: MiniGrid and MiniWob.
main_review: Strengths:

- Compared to prior work, the use of Petri Net graph formalism and the formulation of regret and difficulty are original in this paper. The proposed CoDE algorithm has enough novelty. 

- The proposed graph formalism and the CoDE algorithm seem to be reasonable. Detailed ablation studies are provided to analyze the performance of the proposed algorithm. However, a few important baselines are missing in the experiments and there is no ablation study to analyze the design choices in the CoDE algorithm. 

- The paper is mostly well written and easy to follow. 

- This paper address an important problem of generating tasks as curricula and proposes an interesting solution. However, the significance of the proposed method was not fully justified due to missing comparisons with several state-of-the-art methods. 

Weaknesses:

- One of my major concerns is that many state-of-the-art curriculum learning methods are not compared in the experiments, including ALP-GMM (Portelas et al. CoRL 2019), GoalGAN (Florensa et al. ICML 2018), and VDS (Zhang et al. NeurIPS 2020). Without comparisons with these recent works, it would be hard to justify the effectiveness of the proposed method. 

- The state/action spaces and the parameterization of the environments are not clearly explained in Sec. 6. Such information is important for understanding the complexity of the problems and should be mentioned in the paper. 

- The analysis in Sec. 4 is really helpful for the readers to better understand the problem. However, it would be better if more experimental results are provided to prove the hypothesis.

- (Minor) The pargraph of the generator model in Sec. 5 is a bit confusing since some of the descriptions are specific to gMiniWob domain (e.g. "page index" in line 231). It would be better to describe the model design in a domain-agnostic manner. 

- Some of the citations on curriculum learning and task generation are missing: 

    -- M. Svetlik, Matteo Leonetti, J. Sinapov, Rishi Shah, Nick Walker, and P. Stone. Automatic curriculum graph generation for reinforcement learning agents. In AAAI Conference on Artificial Intelligence, 2017.

    -- B. Peng, J. MacGlashan, R. Loftin, M. Littman, D. Roberts, and Matthew E. Taylor. Curriculum design for machine learners in sequential decision tasks. IEEE Transactions on Emerging Topics in Computational Intelligence, 2:268–277, 2018.

    -- K. Fang, Y. Zhu, A. Garg, A. Kuryenkov, V. Mehta, L. Fei-Fei, and S. Savarese. Learning task-oriented grasping for tool manipulation from simulated self-supervision. Robotics: Science and Systems, 2018.

    -- Wojciech Czarnecki, Siddhant M. Jayakumar, Max Jaderberg, Leonard Hasenclever, Yee Whye Teh, Nicolas Manfred Otto Heess, Simon Osindero, and Razvan Pascanu. Mix&match - agent curricula for reinforcement learning. In International Conference on Machine Learning, 2018.
limitations_and_societal_impact: The authors provide discussions on the limitation of applying the difficulty budget to tasks without a goal state. It would be even better if the authors could suggest alternative solutions for such scenarios.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper presents a method, Compositional Design of Environments (CoDE), for automatically constructing a curriculum over a complex domain of environments. The algorithm is framed as a multi-agent game: a generator constructs an environment using a sequence of actions, and afterwards a population of agents attempts to solve the constructed environment. Agents are trained to maximize the extrinsic reward they receive when attempting to solve the constructed environments. The generator reward, the main contribution of this paper, has two components: the first is a population-based regret computed as the difference between the average reward of the best performing agent in the population and the average reward across all agents in the population. The second component is a "difficulty" reward that encourages the generator agent to only complexify its constructed environments so long as one agent in the population remains capable of solving the task to a sufficient degree (given by a success threshold). If all agents fail the environment by another degree (the failure threshold), then the generator receives a negative reward, encouraging it to simplify its environment distribution when necessary.
main_review: The paper presents an interesting solution to automated curriculum design where the environments are themselves constructed by a reinforcement learning agent. The paper is most similar to previous work PAIRED, which also used regret as a learning signal for an environment generator agent. However, the main novelty presented in the paper is (1) a better regret estimate that uses a population of agents rather than only 2 (protagonist / antagonist) and (2) a difficulty reward signal that provides denser learning signal then regret only. Both improvements make sense intuitively and are clearly motivated in the main text.

The results and ablations seem to support that both of these modifications provide significant improvements over the PAIRED baseline. The results on gMiniWoB and the real website evaluation seem especially significant, and the real website dataset could be an interesting benchmark by itself for future research.

However, there are some confusing aspects in the experiment section:
1) On L295-296, it claims that CoDE only uses a population size of 2 for fair comparison to PAIRED. Can this then be understood that the CoDE results in the experiments section are PAIRED but with the added difficulty reward?
2) Given that information, what is the difference between PopRegret and CoDE in e.g. Figure 6?
3) How does population size affect final performance? This ablation seems important to motivating using |P| > 2.
4) There is a mention of "Flexible PAIRED" and "budget enforcement" in Figure 6, but no additional information. Can you explain what is meant here?
5) The success/failure thresholds of the difficulty reward seem like potentially important hyperparameters that can largely affect the final results of CoDE. I can't easily find what values these were set to in the text, can you describe this in more detail in the experiments section, including how you ended up at the values you chose?
6) It would be interesting to see an ablation of these thresholds, and the performance of CoDE for various values. This would determine how critical these thresholds are to performance, and how easy they are to tune for agent designers.
7) If the reward of an agent is unbounded, would using static thresholds still make sense? Or would you also need to increase the threshold with increasing/decreasing agent returns?
8) It's mentioned that agents are optimized using a policy gradient method. Given the highly non-stationary environment that the population is trained under, was there any observation that agent optimization is affected by the shifts in environment distributions over training (e.g. catastrophic forgetting or learning speed reductions due to activations saturating during training)?
9) In Figure 11, is there any intuition on why the CL baseline increases rapidly in performance but eventually collapses?

Additionally, can the authors comment on whether it is possible the generator and agent population fall into some unwanted local minima and how these solutions can be avoided? I think there is the possibility on very complex domains that the learning process gets stuck in a part of the game which might not overlap with the test distribution, e.g. in the space of creating any website, the generator can start designing extremely complex websites where even humans can not easily solve them.

In conclusion, I believe that the method is well-motivated and sufficiently empirically validated, and remains a useful contribution to the field of automated environment generation despite there being some limitations.
limitations_and_societal_impact: Limitations are discussed in section 8 in the main text and in the appendix. 

A potentially major limitation of this work is that it requires both a way to represent tasks efficiently so that a generator can produce a valid task description in a small number of actions, as well as a way to transform simple task descriptions into environments that can be executed by reinforcement learning agents. Constructing these components could conceivably be more expensive in some domains than hand-designing a curriculum of training tasks.
review_title: null

Meta-review: This paper makes several solid contributions — 1) a method to decompose tasks in a compositional manner, so as to enable RL agents to generalize to unseen, complex test scenarios, and 2) new open-source benchmarks for testing compositional generalization in RL agents, on gridworld and web-based domains. The reviewers appreciated the novelty of ideas presented and agree that the paper is solving an important problem. The authors in their responses promise to add in ablation studies and additional clarifications on the framework (state space, actions, etc.), better comparison to related work on curriculum/self-supervised learning, and adding more complex environments to the open source benchmarks. With these changes, I think this paper will be quite valuable to the community and enable future research on compositional generalization in RL. 
Decision: Accept (Poster)