Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work introduces a method called Environment-Adversarial Sub-Task Curriculum  (EAT-C) wjocj automatically generates a curricuulm of task-environment couples for efficient RL. To this end, EAT-C trains two policies, in addition to the main decision-making policy. One policiy recursively decomposes hard task to coarse-to-fine subtasks, while the other performs adversarial modifications of the environment in each of the subtask. The authors show empirically that EAT-C leads to efficient training of the policy and superior generalization capabilities compared to existing baselines.
main_review: ### Pros:

1. The authors address an important problem: a curriculum learning of challenging tasks under the sparse reward setting where current RL techniques are known to be inefficient. To me, the problem is real and practical.
2. The paper includes comprehensive experiments, including both qualitative analysis and quantitative results, to show the effectiveness of the proposed method. Thorough ablation studies are also performed.

### Cons:

1. Although the results look good on the two environments provided by the authors, providing additional experiments using more challenging benchmarks, such as 3D continuous control tasks, would strengthen the paper. For example, the 7DoF robotic arm problem from [1] seems quite relevant.
2. The think the writing can be improved. First, certain details are missing (e.g. how exactly the environment generation was implemented). Second, certain details (e.g. Algorithm pseudocode, model architecture, etc) can be moved to the appendix to make space for a Conclusion and Future work Section, which is noticeably missing. Third,, the figures 3 and 4 include texts and line names that is very small, and difficult to read (for someone who printed the paper using standard size). Please fix these.

[1] T. Jurgenson, Orly Avner, E. Groshev, and Aviv Tamar. Sub-goal trees - a framework for goal-based reinforcement learning. ArXiv, abs/2002.12361, 2020.
summary_of_the_review: The authors address an important problem and provide comprehensive experimental details, including qualitative analysis, quantitative results and thorough ablations. However, further experiments on more challenging domains would strengthen the paper. Also, the writing can be improved.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method for generating curriculum for goal-conditioned RL policies based on having one agent hierarchically generate sub-goals, and have the target agent train from rollouts on each level of this hierarchy, while a third agent adversarially modifies the environment to improve robustness and transfer.  The main contributions are introducing a mutual training regime between the path planer and the RL agent, and introducing adversarial environment design to training in each of the sub goals.
main_review: The biggest strength of this paper is in its combination of two important and productive lines of research from HRL and environment design.  The combination of the two is relatively natural because they solve different aspects of sparse reward problem, and to the extent there are synergies between approaches, this is a productive direction of research.  

However, it is not clear if what is achieved by this approach could be achieved by running existing HRL techniques in the context of adversarial environment design. The one thing that makes this papers approach distinct from this direct combination is that the adversarial design is done independently for every sub-goal.  However, it is not clear that this distinction is necessary for the approach to work.  In fact, the ablation study in Figure2 (b) leads me to believe that the adversarial environment design component is relatively unimportant, so I wouldn't be surprised if the somewhat marginal benefits could also be achieved by adversarial environment design separate from the hierarchy.  If this is the case, then it is difficult for me to see the benefits of combining the approaches, given the added complexity, added dependencies, and marginal benefits. 

In terms of the benefits, the empirical results are also inconclusive.  One concern is that SAC by itself beats some of the baselines, which makes it seem like the baselines may not be configured properly.  In particular, POET has environment-dependent parameters which need to be tuned, so it is not obvious to me how this baseline works, or how it was ensured that these parameters were set correctly.  Even without these concerns, it seems that most of the results do not seem to be completely statistically significant.

The next major concern is that it seems that it would be easy for many of the components to fall into undesirable equilibrium.  For instance, the environment adversary could make unsolvable environments, and it seems like the action space of the environment adversary had to be restricted to avoid this.  Also if the agent does not have some capability, and the planner would not ask the agent to solve those goals since it is cooperative, and thus the agent would not gain experience for those goals, which could also halt progress.

Finally, it is unclear to me if the hierarchy of subgoals being mutually trained with the RL agent is novel.  It appears similar to a few of the following and there should at least be a discussion of how these differ, if that is to be taken as a contribution:
"Long-Horizon Visual Planning with Goal-Conditioned Hierarchical Predictors" Pertsch K and Rybkin O, et. al.
"Generating Adjacency-Constrained Subgoals in Hierarchical Reinforcement Learning" Zhang, T and Guo, S et.al 
"Feudal reinforcement learning" Dayan, P and Hinton, G et.al
"Planning simple trajectories using neural subgoal generators" Schmidhuber, J and Wahnsiedler, R et.al

Minor point: 
* The paper ends suddenly, I was expecting a conclusion section
* I didn't realize the legends where different for Figure 2(a) and Figure 2(b), it would be less confusing if the reuse of colors could be avoided
* The caption for Figure 2 blends into the text.
* All references for Figure 3 are before all references to Figure 2, but they are displayed in the other order
* I do not know what is meant by a "mutual boosting scheme", the term is used several times but does not seem to be defined.
* The second to last paragraph of page 3 there is a typo: "requires to collect"
* In the first paragraph of page 6, you first mention that the RL agent is trained on one rollout-per-layer rather than one rollout just for the leaves of the hierarchy.  This should be made clearer earlier, as it is a significantly different method with this change.
* In the second paragraph of page 6 there is the claim that the sub-goal curricula is "an imitation of human learning that repeated practice the same complicated task in different ways".  I do not believe this claim, though I am not sure I understand it, but I do not think it is necessary to understand the method.
* There is also a claim at the end of that paragraph that they adversary generates diversity.  I do not know why that would be true, it seems that the adversary could easily fixate on a particular type of modification.
*The generic idea that environment design techniques make environments more reward sparse (for instance, at the end of the first paragraph of page 1), is true of fully-adversarial environment design techniques, but it is not true for techniques like:
"Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play" Sukhbaatar, S et al.
"Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design" Dennis, M, et al. 
both of which are designed to densify the reward structure by designing hard but solvable levels, and would likely serve as better baselines.  In particular, the method used for the environment designer in this work appears to correspond to the "maximin adversary" in the second paper, which was used as a baseline.
summary_of_the_review: I will be weakly recommending rejection since it is unclear to what extent the method works because of the integration of HRL and adversarial environment design, and to what extent the same results could be achieved through the simpler direct combination of the existing approaches in both fields.  In addition, it seems that the baselines could be made stronger, and I am not convinced by the empirical results.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces an auto-curriculum generation approach to train better low-level policies conditioned on the subgoals generated by a path-planning high-level policy. The core idea is to use a planning policy to decompose a task into subgoals via a tree structure (multi-level subgoal decomposition) and use an environment generation policy to adversarially make the environment harder for the subgoal policy to perform. Thanks to the tree structure, the adversarial training can follow a bottom-up process, i.e., in an easy-to-hard order. The evaluation on a 2D object using task and a 2D grid world environment shows the effectiveness of this approach.
main_review: ======Strengths======

1. The main idea of this approach is easy to follow and seems intuitive. The approach also shows promising results in the experiments. Though evaluated in simple tasks, applications in harder environments and tasks might be plausible if the subgoal decomposing and the environment generation could be adapted to more difficult environments and tasks. 

2. The figures and the qualitative examples are very helpful for readers to understand the core idea as well as how the auto-curriculum really works in the experiments.

======Weaknesses======

1. While the main idea of the algorithm is clearly presented, implementation details are missing. E.g., I could not find information about how the environment generation was implemented (e.g., action space). A more detailed description and/or code would be very helpful.

2. The performance of the algorithm may heavily depend on the tree structure generation approach. The current work adopts a specific kind of approach, which may not work for other kinds of tasks. It would be good to discuss the limit/generalizability of the overall algorithm.

3. The conclusion section is missing. It would be good to summarize the main contributions and findings at the end of the paper.

Some suggested references:

[1] Kaelbling, L., & Lozano-Perez, T. (2011). Hierarchical task and motion planning in the now. In Ieee international conference on
robotics and automation. (Classic work on task and motion planning, a well-known type of hierarchical planning.)

[2] Shu, T., Xiong, C., & Socher, R. (2018). Hierarchical and interpretable skill acquisition in multi-task reinforcement learning. In ICLR. (Human-designed curriculum for learning the composition of subgoals via a tree structure.)

[3] Baker, B., Kanitscheider, I., Markov, T., Wu, Y., Powell, G., McGrew, B., & Mordatch, I. (2019). Emergent tool use from multi-agent autocurricula. In ICLR. (Autocurricula emerged from adversarial multi-agent RL training.)

summary_of_the_review: The proposed algorithm presents an interesting and novel way to adversarially generate a curriculum for training low-level RL policies. The algorithm seems general to some extent and performed well in two tasks. However, the writing needs improvement (a lack of details, missing conclusion, etc.). There should also be a discussion on the limit of the current algorithm, which relies on the performance of a specific plan-planning method that provides the true structure. I am willing to raise my score if these problems are addressed in the revision.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes an RL method targeted at long-horizon tasks in environments with perturbed dynamics or configurations. By combining a recently proposed goal-based RL method with an adversarial approach to domain randomization, the authors obtain a method that allows them to learn robust policies in sparse long-horizon settings. The method is compared with different baselines form hierarchical- or curriculum RL, demonstrating promising performance.
main_review: In general, I enjoyed reading the paper. It is clearly written and does not try to oversell its contribution - the combination of a recent hierarchical goal-based RL approach and adversarial environment generation. My main questions/critiques revolve around the experimental section. I can imagine that some of them may have been already addressed in the appendix that has been mentioned in the main paper but seems to be missing in the current revision:

* As I already mentioned, I think that the authors are in general objective in their assessments of the novelty of their method.
Only one statement on page 2 caught my eye: "Unlike previous methods, training these two policies does not require external supervision or prior knowledge, we instead collect the time costs and rewards of the RL agent on previous sub-tasks to train them towards generating better curricula adaptive to RL progress, which will be used in the next episode." I think that there already exist hierarchical RL approaches that do not need additional feedback or priors in addition to the MDP specification, such as e.g. [1].
* The authors compared their method to curriculum RL methods such as ALP-GMM. For these curriculum RL algorithms, an appropriately defined parameterization of the MDPs is crucial to obtain good learning performance. Unfortunately, details regarding this parameterization were not included in the main paper. For the curriculum RL methods, I would expect the parameterization to include both the parameters that the EG in EAT-C can control as well as the goal state to be reached. In this setting, to comparison is the most fair and consequently, the benefit of the particular structure of the proposed approach becomes the deciding factor in the performance difference. Could the authors clarify on the environment parameterization for the curriculum RL methods? If it differs from the aforementioned one, an additional investigation with this particular parameterization would clearly improve the paper.
* Another point of confusion is the very poor performance of the hierarchical RL baseline, which is unexpected given the clear benefit that a reader would expected from hierarchical RL in long horizon tasks. Although I can imagine that this poor performance may be caused by the randomization over environment configurations, the authors should consider providing additional evidence for this hypothesis. This could be done by:
	* Evaluating the hierarchical RL method also in the non-randomized environment used for the ablations of EAT-C.
	* Evaluating the ablation of EAT-C without the EG in randomized environments.
	* Clarify details regarding the observability of the environment configuration (see next point).
* Is the configuration of the environment hidden from the RL agent? Or is the agent aware of e.g. obstacle/object positions via observations? I am asking because compared to the experiments by Yamada et al in the 2D pusher task, much more samples are required to reach acceptable performance. So it seems that the generalization over environment configurations induces large challenges. Hence it would be important to carefully specify how the environment randomization is perceived by the agent.
* A more "realistic" experiment could further improve the paper. Such an experiment could be e.g. the 7-DoF planning environment with obstacles done in the sub-goal trees paper by Jurgenson et al.
* Could the EG in theory create infeasible tasks? Does this need to be prevented by the appropriate parameterization of the environment/EG actions?
* In algorithm 2: Could the REACH procedure possibly run into an infinite recursion with a very poor planning policy $\pi_p$ that always predicts infeasible goals? If so, could this be avoided with some naive pre-training? If this infinite recursion cannot happen, what prevents the possibly infinite recursion?

[1] Kulkarni, Tejas D., et al. "Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation." Advances in neural information processing systems 29 (2016): 3675-3683.
summary_of_the_review: Despite its rather incremental nature, I like the overall paper. Mainly the questions w.r.t. experimental details and the smaller scale evaluation tasks prevent me from putting it above the acceptance threshold. If some of my issues/questions are addressed, I am happy to improve my score.
review_title: null

Meta-review: I thank the authors for their submission and active participation in the discussions. This papers is borderline with three reviewers leaning towards acceptance [3c96,7T33,Zhvq] and one leaning towards rejection [o38w]. Reviewer o38w's main concerns are around the lack of details about how the baselines were tuned and missing training details (specifically the connectivity test used to reject candidate environments). During discussion both, reviewers Zhvq and 7T33, agree that the paper requires substantial restructuring/rewriting to properly address the reviewer's feedback which is currently mostly addressed in the appendix. Based on the discussion with reviewers, my assessment is that this paper is not ready for publication at this point and that it will benefit greatly from another iteration. I want to very strongly encourage the authors to further improve their paper based on the reviewer feedback.
Decision: Reject