Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: The authors identify several challenges of cooperative multi-agent reinforcement learning (MARL): sparse rewards, non-stationarity, and the more general requirement compared to existing works, of supporting a varying number of agents or agents with varying existing capabilities in the task space of interest. To tackle these challenges, the authors propose COST, a learning algorithm combining several components: a bandit-based task selection technique, a hierarchical skill-based MARL policy, and a self-attention-based message-passing system.
strengths_and_weaknesses: Strengths:

- The paper provides a detailed discussion of the challenges of learning complex task spaces in cooperative MARL, and the authors propose several ideas for addressing these issues.
- The self-attention message-passing architecture is an interesting method for supporting an a flexible number of participating agents. This approach, however, is quite similar to that taken by deep coordination graphs [1], and so a discussion and comparison with respect to this line of work seems necessary.

Weaknesses:

- While there are good ideas in this paper, the current presentation of the method and results is disorganized and unfocused. The result is that while the authors identify several important challenges for improving cooperative MARL, they fail to provide much insight into whether any of their proposed solutions are at all effective. This paper is trying to do too much at once, and as a result, focuses on too many problems simultaneously with independent methods for each, while not clearly demonstrating the benefits of their method for solving any one of them.
- The experiments show ablations only on the Google Research Football environment, and only for the curriculum and hierarchical skill-based components of the method. Meanwhile, the results compare to baselines that seem to use a completely different policy architecture, as the baselines do not seem to use the self-attention message-passing components. As a result, it seems that the self-attention architecture acts as a confounder for the results, making it hard to draw any conclusions from the presented results.
- In procedurally-generated environments, it is important to test for generalization to held-out test configurations of the environment. Some methods that use a return-maximizing curriculum, as performed by COST's multi-arm bandit, may result in overfitting to the training tasks. This aspect of their problem setting is completely ignored, and fully missing from the paper.
- Some design decisions seem entirely arbitrary without clear motivation, for example, the choice of BIRCH for task clustering and the decision to train the higher-level skill policies independently, while the low-level policies with parameter sharing across all agents.
- The curriculum visualization results seem to suggest no real curriculum is being learned. This also seems supported by the large overlapping error bars between COST and the COST without curriculum ablation in the Google Research Football learning curves.
- The authors miss crucial related work from the unsupervised environment design subfield of reinforcement learning [1,2,3,4] as wells prior work on Teacher-Student Curriculum Learning [5].
- The authors should consider moving the multi-armed bandit section in Section 2, their related works section to Section 3, which serves as their background section.
- Overall, the paper presents a large number of ultimately disparate ideas for improving cooperative MARL, without providing sufficient experiments teasing apart the effect of each newly proposed component. Each of the proposed solution components also already exists in a similar form in prior works (see cited references below), while, judging from the experimental results, no clear emergent benefits seem to result from their combination. This suggests that the novelty of this paper is limited.

References:

[1] Böhmer et al, 2020. Deep Coordination Graphs.

[2] Dennis et al, 2020. Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design.

[3] Jiang et al, 2021. Prioritized Level Replay.

[4] Jiang et al, 2021. Replay-Guided Adversarial Environment Design.

[5] Parker-Holder et al, 2022. Evolving Curricula with Regret-Based Environment Design.

[6] Matiisen et al, 2019. Teacher-Student Curriculum Learning
questions: - What exactly is the task parameterization that is used for varying the environment in each episode and as input to BIRCH for clustering? This is a core detail that does not seem to be clearly detailed anywhere.
- Similarly, why was VACL chosen as a baseline? Further, why was VACL only used as a baseline for the MPE environment and not Google Research Football?
- Can the authors elaborate on both why BIRCH was chosen for clustering and how this clustering method works? These details would provide missing justification for this essential design choice in the curriculum-learning component of their method.
- Why does the starting number of agents for VACL and COST differ so drastically in Figure 4?
- Are the differences between the learning curves in Figure 5 statistically significant?
- Do the baseline methods also make use of a message-passing self-attention architecture? If not, isn’t this a confounder for the experimental results?
limitations: COST assumes the ability to use a cheap talk channel both at training and at evaluation, which may often be infeasible in practice. Furthermore, it is not clearly discussed in the paper how scalable this method is to larger numbers of agents or what population size is ideal for applying this method. The authors should clarify these points in a future version of the paper.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes a curriculum learning framework for MARL to solve large-scale and more difficult multi-agent tasks. To deal with the problem of varying agent numbers, sparse reward, and the non-stationarity from changing student policies, the authors combine several techniques including the Transformer-based communication structure, hierarchical network design, and contextual bandit for better multi-task learning performance. The proposed method shows good performance on several tasks from MPE and GRF and the visualization results prove that the teacher model can learn a good curriculum for students. 
strengths_and_weaknesses: Strengths:
1. This paper attempts to solve the curriculum learning problem in MARL in a multi-task learning way, which is very interesting and intuitive. While most other works only focus on generating the curriculum or learning from a manually selected task sequence, the proposed method applies a teacher-student structure to solve these two problems concurrently.
2. Although there are some previous works utilizing the techniques like the self-attention module or the hierarchical structure of agents in MARL, it is natural to implement them under the curriculum learning setting in this paper and yield good results.
3. The experiment results on selected tasks outperform other baselines on two benchmarks. The ablation study and visualization results are also helpful for understanding the impact of each part.

Weaknesses:
1. The problem of varying numbers of agents in MARL has been widely discussed in MARL and several methods have been proposed and achieved good performance. This paper did not review these works or set them as baselines to compare.
2. The tasks selected in MPE and GRF are mostly 16 agents and 4 agents, which is either not very difficult or really large-scale, making the significance of this work not very good.
3. The baselines didn't include recent works and are clearly not enough for evaluation. The final performance is not impressive and only three scenarios are presented.
questions: 1. What does the forgetting problem mentioned in Line 40 mean since I thought curriculum learning does not require solving old tasks as continual learning
2.  I didn't quite understand the experimental settings in GRF since only the 5_vs_5 scenario is trained, what is the teacher agent selecting from?
3. Is the modification of baselines and environments like adding the shooting reward necessary? Since the VACL is the only curriculum-based method but it's modified and the original 5_vs_5 scenario can already be solved by some works.



limitations: Limitations should be discussed more adequately in the paper. Societal impact is likely minimal.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes a contextual bandit policy based curriculum learning method and applies it to MARL problems. To handle the changing input size induced by the changing numbers of agents, they utilize a self-attention mechanism in the communication channel. To mitigate the non-stationarity, they propose the imitation function to represent the policies and use a cluster algorithm to discretize and reduce the contextal space. The paper is well written and the experiments are introduced clearly.
strengths_and_weaknesses: Strengths:
1. The researched problem regarding curriculum learning on MARL is interesting and can be beneficial for the community.
2. The proposed using a contextual bandit policy on curriculum learning is promising.
3. The experiments are also well designed.

Weaknesses:
1. The self-attention based population-invariant architecture is similar to the architecture proposed by EPC, which might not be claimed as one of the main contributions.
2. The motivation and benefits of (hierarchical reinforcement learning) HRL in this work are not well explained.
3. The performance of the method seems not significant, especially on MPE tasks. And on both MPE and GRF tasks, the generated task distribution is hard to explain and connected to their corresponding performance. 
questions: 1. Section 5.2: it says “The results illustrate that in a simple environment where the student can directly learn to complete the task, there is no need to apply curriculum learning.”, however, the coverage rate in Figure 3 is pretty low (around 0.35 on SimpleSpread, and 0.30 on Push ball). It seems the ACL methods do not help very much to boost the performance, which is also consistent with the generated curriculum in Figure 4 where the curriculum from COST is only around the target task. 
2. Figure 6a shows the task distribution generated by COST, which is close-uniform. Why does it produce a very different performance from the uniform curriculum in Figure 5(a)?
3. How much does the HRL policy used in COST help to improve the performance?
limitations: 1. The motivation and advantage of the hierarchical policy and clustering methods are not analyzed clearly. How can they help?
2. The performance on MPE tasks does not show the effectiveness of the proposed method; and the performance on GRF cannot be connected well to the generated task distribution.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper proposes a multiagent curriculum method where a "Teacher" proposes tasks to a "Student" to learn to solve. The teacher follows a bandit algorithm to propose tasks taking into account the non-stationarity comes from the student learning algorithm, and the student uses a sophisticated attention-based learning architecture.
strengths_and_weaknesses: Albeit the paper explores a very relevant research topic, the experimental evaluation is very limited for a mainly-empirical approach. Moreover, there are a number of aspects and assumptions in the algorithm that are not made clear. Finally, there is a huge group of curriculum for RL papers that are not even mentioned (and hence it's no surprise that those methods are not included in the experimental evaluation). Therefore, I am not entirely convinced the method performs better than other state-of-the-art works (or even classical ones). More details on the reasons in the "Limitations" field.
questions: - How is the "distribution of tasks" defined? How is the original (final) task modified to build this distribution?
- How are the policies transferred/adapted across tasks? How similar should the tasks be for the method to work?
limitations: - The paper has an interesting choice of works added to its references. Arxiv papers [5,6,7] are added to cite the general concepts "reward shaping", "curriculum learning", and "learning from demonstrations", where there are plenty of very comprehensive peer-reviewed surveys in all of those areas. In special, the main surveys of the area of the paper (Curriculum Learning for RL) are not even mentioned (the first is probably the most comprehensive survey while the second is probably the first one to explicitly survey on Curriculum Learning for RL)

Narvekar, Sanmit, et al. "Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey." Journal of Machine Learning Research 21.181 (2020): 1-50.

Da Silva, Felipe Leno, and Anna Helena Reali Costa. "A survey on transfer learning for multiagent reinforcement learning systems." Journal of Artificial Intelligence Research 64 (2019): 645-703.

Perhaps due to an insufficient coverage of the literature, the authors have not added any heuristic-based curriculum learning method in ter experimental evaluation (e.g., one of the many surveyed on those two papers), and also failed to explicitly address many important aspects of a curriculum learning algorithm.

- One of the most important aspects is how the set of subtasks is defined from the final task. If it is assumed that the "distribution of tasks" is manually defined somehow, there is a very big number of heuristic-based curriculum methods that could be applicable for the empirical evaluation and some of them should be included. If this is somehow autonomously defined, there is a smaller group of applicable methods, and the one that comes to my mind at this moment is the Object-oriented curriculum:

Silva, Felipe Leno Da, and Anna Helena Reali Costa. "Object-oriented curriculum generation for reinforcement learning." Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems. 2018.

Another aspect that is not made clear is exactly how to interrupt the training in one task and proceed to the other. This seems to have been defined in a very ad hoc manner for this paper, but the procedure to do this should have been explicitly added in the algorithm.

Moreover, there are some assumptions followed in the work given that this is a multiagent procedure. It is assumed that all agents require constant and perfect communication between agents, which is not true for most practical multi agent scenarios. This should have been made very clear in the paper.

Finally, it is not very clear to me how the knowledge is transferred across the tasks.

- There should be a little more of evidence that the curriculum method is better than not using curriculum. You could add a baseline where exactly the same learning algorithm is used but without switching the task (training directly on the target task).
review_title: null

Meta-review: metareview: This paper addresses some important problems in curriculum learning. Reviewers were generally happy with the approach. The main criticisms are around the evaluation. There is a lot of prior and similar work in the the literature, and reviewers felt that the paper lacked strong baselines against state of the art curriculum learning methods (reviewer 1MvZ and mgEj). Reviewer Vwt6 also pointed out issues with evaluation. For example, it was unclear if difference in reported performance was due to differences in algorithm, or in neural net sizes. Including stronger evaluation, and more references to related methods from the literature, would make the paper much stronger. Our conclusion is that the paper is not yet ready for publication.
confidence: Certain
Decision: Reject