Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper mainly focuses on improving RL policy performance on long-horizon tasks, including learning efficiency and generalization. The proposed framework has a high-level objective to adversarially choose tasks to minimize the reward achieved by the low-level policies. The empirical evaluation demonstrate the method works across different domains.
strengths_and_weaknesses: **Strengths**

1. This paper is well-written and structured. I enjoy reading it. The authors demonstrate motivations clearly and are easy to accept. 

2. The theoretical explanation in this paper is sufficient and correct, good.

3. The results of the empirical evaluation are present clearly. 

 
**Weakness**

1. The first paragraph in section 3 misses the definition of the difference between $A_1$ and $A_2$. The action of agent 2 is causing task shifting, which is one of the main differences between the proposed method, and should make it more straightforward.

2. The discussion in related work can be more substantial. It would be better to summarize the difference between ROSAC with previous work rather than only listing the related research. On the other hand, there is a line of recent work that apply planning to improve the robustness of RL policy on long-horizon tasks (e.g., [1]-[4]). It is better to include them in the discussion.

3. It would be better to include more analysis of the experiment results in section 5. It is necessary to have some in-middle results to demonstrate how two agents work together to affect the training.
questions: **Questions**

This paper is over the qualified line, and I don't have many questions. I only have two concerns :

1. Considering that the proposed method is based on the setting of multi-tasks, I was expected to find more complex tasks in the experiments. Based on the experiments, I think it is not necessary to train sub-policies for diverse sub-tasks, but a goal-conditioned RL policy is enough. What if using a goal-conditioned RL policy instead of the set of policies $\pi_1$?

2. How does ROSAC work in a more realistic task? Refer to Line.13 in Algorithm 2, what if the agent performs badly on all sub-tasks and has relatively low improvement even as the training processing. In other words, what if the initial sub-task is very difficult to learn (e,g., a 7 DoF robotic arm control)?
limitations: 
**Reference**

1. Benjamin Eysenbach, Ruslan Salakhutdinov, Sergey Levine. Search on the Replay Buffer: Bridging Planning and Reinforcement Learning, Arxiv, abs/1906.05253.

3. Shuang Ao, Tianyi Zhou, Guodong Long, Qinghua Lu, Liming Zhu, Jing Jiang. CO-PILOT: COllaborative Planning and reInforcement Learning On sub-Task curriculum, NeurIPS 2021.

2. R. Gieselmann, Florian T. Pokorn, Planning-Augmented Hierarchical Reinforcement Learning, IEEE Robotics and Automation Letters, 2021.

4. A. Srinivas, A. Jabri, P. Abbeel, S. Levine, and Chelsea Finn. Universal planning networks. ArXiv, abs/1804.00645.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: The paper aims to learn robust policies for solving long-horizon tasks that are composed of sequences of subtasks. To accomplish this, the problem is framed as a zero-sum, two-player game: the agent aims to maximize its worst-case performance against an adversary who chooses the sequence of subtasks. Two algorithms are introduced to solve this multi-agent problem which demonstrate superior performance compared to a number of baselines. 
strengths_and_weaknesses: **Originality**: The authors present an interesting framework for training robust options, but this framework appears closely related to previous work where an adversary designs environments which challenge the current agent (e.g. [1,2]). The main difference here seems to be that the adversary can act many times within the *same episode* to affect segments of the environment that have not yet been generated. More generally, there is a plethora of work aimed at learning robust or safe policies (not necessarily options) using a worst-case minimax objective (e.g. [3,4]). I think these areas of research should be discussed in the related work, as I'm currently not convinced that this work differs in a significant way.

**Quality**: The proposed algorithms are well-motivated, exploiting the structure of the problem to improve the learning of the adversary. They are further justified via rigorous convergence guarantees. The experiments are conducted on complex environments and demonstrate faster learning compared to a number of baselines, but I have some concerns. 

- The learning curves are only reported over a small number of frames (500k) and this may not show the full picture. I'm interested in seeing whether ROSAC converges to a better solution than NAIVE/DAGGER under adversarial subtask selection. Also, I'm curious if there is any performance trade-off to training adversarially when you evaluate on uniformly random tasks.
- I think there needs to be a baseline which simply performs flat RL (preferably SAC, for a fair comparison) on the full task. If I'm understanding correctly, this is different from the NAIVE/DAGGER baselines, which instead independently try to solve each "segment" as quickly as possible. 

**Clarity**: Overall the prose is very well-written but the math can be hard to follow due to the amount of notation. 
The following parts could also be improved in terms of clarity:
- I think the start of Section 3 could better emphasize the properties of the problem you will be exploiting to outperform general methods for solving games (e.g. MADDPG). 
- It's not clear what the NAIVE and DAGGER baselines are just from the descriptions in the experiments section.  
- It's not stated what the error bars in Figs 3 and 4 are reporting.

**Significance**: Learning options that are robust and that generalize under various conditions is an important problem for RL. However, after reading the paper, I (unfortunately) remain skeptical of the impact of this particular work. First, there is an assumption that any sequence of options corresponds to a meaningful task, which is only true in a limited number of settings. Even in the racetrack example, many sequences of turns don't seem to correspond to configurations that are physically possible. The environment must also dynamically change based on the selected sequence of subtasks, requiring a great degree of control in the environment generation process. Again, I'm having trouble thinking of other settings where such a framework would be applicable. 

[1]: Dennis, Michael, et al. "Emergent complexity and zero-shot transfer via unsupervised environment design." Advances in Neural Information Processing Systems 33 (2020): 13049-13061.
[2]: Gur, Izzeddin, et al. "Environment generation for zero-shot compositional reinforcement learning." Advances in Neural Information Processing Systems 34 (2021): 4157-4169.
[3]: Pinto, Lerrel, et al. "Robust adversarial reinforcement learning." International Conference on Machine Learning. PMLR, 2017.
[4]: Campero, Andres, et al. "Learning with AMIGo: Adversarially Motivated Intrinsic Goals." International Conference on Learning Representations. 2020.
questions: - Can you elaborate on the key differences between your work and the previously mentioned areas of unsupervised environment design and robust (minimax) RL? 
- Can you elaborate on other real-world settings where your framework would be beneficial? 
- Why does the MADDPG baseline perform so poorly, even when compared to NAIVE? I found this surprising, since even a random adversary should result in options of decent quality. 
limitations: As mentioned above, I believe the authors should discuss any assumptions of control over the environment generation process (if my conclusion that this is a limitation is indeed correct). This would limit the types of tasks and settings where such a method would be applicable.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: The paper proposes an approach to learning option policies such that they can be composed sequentially, regardless of the next subtask/option to be performed. The main idea is to reframe the problem as a two-player game, where an adversarial environment arranges the order of the subtasks, and the agent learns one policy for each subtask to maximise the sequential completion of as many as possible in the worst case. The convergence of value iteration to an optimal policy in this setting is proven, while two algorithms based on Q-learning and soft actor-critic are provided. Results in two continuous state/action domains demonstrate that the proposed algorithms significantly outperform the baselines in practice.
strengths_and_weaknesses: \+ The two-player formulation is, to my knowledge, a novel way of analysing the problem 

\+ Practical algorithms are backed by theoretical guarantees, while empirically the proposed algorithms vastly outperform the given baselines

\+ The introduction of the racing car domain early on really helped to ground the ideas, making the paper much easier to understand

\- Much of the "heavy lifting" is encoded into the system. For example, much of the challenge of option discovery is simply waved away here by pre-specifying the subtasks, reducing the problem to only learning good policies. This is a much easier problem than many other approaches are concerned with (see comments in next section), which makes the contribution fairly incremental. 

\- It is unclear how this work fits in with other option-discovery approaches and there are potentially more appropriate baselines to compare against (see next section)

\- I found parts of the paper difficult to read because of the dense notation. In particular, Section 3.1 was slow going, particularly when reading the part about the asynchronous value iteration algorithm. 

questions: 1. The main aim of this paper is to learn robust option policies. As I mentioned in the previous section, much of the difficulty in option discovery is in identifying the subgoals options should reach in the first place. The formulation here sidesteps this problem by simply specifying what those should be. Nonetheless, my larger concern is how this fits into the existing literature. In particular, the line of work revolving around skill chaining seems to tackle the exact problem here: learn options such that they can be sequentially chained together reliably. These works go further though, in that they also do not assume that the subtasks are given. Quoting from [2]: "The chainability of the discovered skills, i.e, the property that successful execution of one permits the execution of another, allows the agent to build a graph that is suitable for deterministic planning". If my understanding is correct, then that line of work is tackling the same problem here and more. In that case, it's unclear how this paper improves upon that line of work. One argument potentially is that these options are reusable, but that is only because of the change of coordinates/relativisation of state variables, which is defined by the system itself. I'm thus unclear how this work advances over the skill chaining work, but I would be interested to hear the authors' thoughts on this. Even if there is a difference, I think that one of the skill chaining approaches should certainly be implemented as a baseline comparison. 

2. On the results, one thing I found very confusing is the performance of everything when the adversary is MCTS or random. It seems like a random adversary does just as well as MCTS, so does that not suggest that there is no need to phrase the problem as a 2-player game? The random adversary doesn't seek to minimise anything, so why frame it as a game if it does just as well as one that does? Why not simply always randomise over the subtask order and have the agent train on the randomised sequences?

3. The formulation of the multi-task MDP relies on jump probabilities, change of coordinates, etc, which I've never encountered prior. The paper cites [17], but I'm curious to understand whether this is a common formulation of such problems? It seems to be injecting an awful lot of knowledge and domain expertise into the system and essentially provides the agent with transferrable skills. 

4. I found the following sentence in the abstract confusing: "We aim to maximize the worst-case performance..." This put me in mind of other work [4-7] where the aim is to learn a policy or set of policies that are useful for subsequent tasks drawn from some distribution (as opposed to a given set of tasks that must be sequentially executed). I would recommend this be reworded.

5. What does the shading on the results graphs represent? Is it variance?

[1] Bagaria, Akhil, and George Konidaris. "Option discovery using deep skill chaining." International Conference on Learning Representations. 2019.

[2] Bagaria, Akhil, Jason K. Senthil, and George Konidaris. "Skill discovery for exploration and planning using deep skill graphs." International Conference on Machine Learning. PMLR, 2021.

[3] Bagaria, Akhil, et al. "Robustly Learning Composable Options in Deep Reinforcement Learning." IJCAI. 2021. 

[4] Alver, Safa, and Doina Precup. "Constructing a Good Behavior Basis for Transfer using Generalized Policy Updates." International Conference on Learning Representations. 2022.

[5] Nemecek, Mark W., and Ron Parr. "Policy caches with successor features." International Conference on Machine Learning. PMLR, 2021.

[6] Jinnai, Yuu, et al. "Discovering options for exploration by minimizing cover time." International Conference on Machine Learning. PMLR, 2019.

[7] Abel, David, et al. "Policy and value transfer in lifelong reinforcement learning." International Conference on Machine Learning. PMLR, 2018.
limitations: N/A
review_title: null

Meta-review: metareview: This paper proposes a robust option learning algorithm that learns subtask policies to maximise the worst-case performance. The results on a 2D navigation environment and a simulated car racing environment show that the proposed algorithm achieves a better and more robust performance compared to alternative option learning approaches. Although the reviewers found the idea interesting, all of them ended up sharing several major concerns during the discussion period. First, the problem setting is not fully justified. Specifically, the assumption about the "jump transition" was not fully justified, and the problem reduces to an existing line of work on skill chaining without such an assumption. Besides, the empirical results are not convincing enough. The subtasks in the environments were so simple that it is unclear how much benefit we can get from the proposed adversarial training. In fact, the performance degradation from the random adversary to the MCTS adversary was indeed quite marginal, which also raises the same question. A good justification of the problem setting and evaluation on more complex environments would strengthen the paper. Thus, I recommend rejecting the paper. 
confidence: Certain
Decision: Reject