Towards Skilled Population Curriculum for MARL
Authors: Rundong Wang
Longtao Zheng
Wei Qiu
Bowei He
Bo An
Zinovi Rabinovich
Yujing Hu
Yingfeng Chen
Tangjie Lv
Changjie Fan
Venue: Submitted to ICLR 2023 (2023)
Keywords: multi-agent reinforcement learning
multi-agent cooperation

Abstract: Recent advances in multi-agent reinforcement learning (MARL) allow agents to coordinate their behaviors in complex environments. However, common MARL algorithms still suffer from scalability and sparse reward issues. One promising approach to resolve them is automated curriculum learning (ACL), where a student (curriculum learner) train on tasks of increasing difficulty controlled by a teacher (curriculum generator). Unfortunately, in spite of its success, ACL’s applicability is restricted due to: (1) lack of a general student framework to deal with the varying number of agents across tasks and the sparse reward problem, and (2) the non-stationarity in the teacher’s task due to the ever-changing student strategies. As a remedy for ACL, we introduce a novel automatic curriculum learning framework, Skilled Population Curriculum (SPC), adapting curriculum learning to multi-agent coordination. To be specific, we endow the student with population-invariant communication and a hierarchical skill set. Thus, the student can learn cooperation and behavior skills from distinct tasks with a varying number of agents. In addition, we model the teacher as a contextual bandit conditioned by student policies. As a result, a team of agents can change its size while retaining previously acquired skills. We also analyze the inherent non-stationarity of this multi-agent automatic curriculum teaching problem, and provide a corresponding regret bound. Empirical results show that our method improves scalability, sample efficiency, and generalization in multiple MARL environments. The source code and the video can be found at https://sites.google.com/view/marl-spc/.

Under review as a conference paper at ICLR 2023
TOWARDS SKILLED POPULATION CURRICULUM FOR MULTI-AGENT REINFORCEMENT LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
Recent advances in multi-agent reinforcement learning (MARL) allow agents coordinate their behaviors in complex environments. However, common MARL algorithms still suffer from scalability and sparse reward issues. One promising approach to resolve them is automatic curriculum learning (ACL), where student (curriculum learner) train on tasks of increasing difficulty controlled by teacher (curriculum generator). Unfortunately, in spite of its success, ACL's applicability is restricted due to: (1) lack of general student framework to deal with the varying number of agents across tasks and the sparse reward problem, and (2) the non- stationarity the teacher's ' task due to the ever-changing student strategies. As a remedy for ACL, we introduce novel automatic curriculum learning framework, Skilled Population Curriculum (SPC), adapting curriculum learning to multi-agent coordination. To be specific, we endow the student with population-invarian munication and a hierarchical skill set. Thus, the student can learn cooperation and behavior skills from distinct tasks with a varying number of agents. In addition, we model the teacher as contextual bandit conditioned by student policies. As a result, a team of agents can change its size while retaining previously acquired skills. We also analyze the inherent non-stationarity of this multi-agent automatic curriculum teaching problem, and provide corresponding regret bound. Empirical results show that our method improves scalability, sample efficiency, and general- ization in multiple MARL environments. The source code and the video can be found at /httts://sssteeeee google cow/ wtom/view//e/ooooooooo..o/..ooee mar marl-spc/
com-
INTRODUCTION
Multi-agent reinforcement learning (MARL) has long been go-to tool in complex robotic and strategic domains (RoboCup) 2019 OpenAl 2019). However, learning effective policies with sparse reward from scratch for large-scale multi-agent systems remains challenging. One of the challenges is that the joint observation-action space grows exponentially with varying numbers of agents. Meanwhile, the sparse reward signal requires a large number of training trajectories. Hence, applying existing MARL algorithms directly to complex environments with a large number of agents not effective. In fact, they may produce agents that do not collaborate with each other even when it. is of significant benefit (hhng et al. 2021 Yang & Wang 2020).
There several lines of work related to the large-scale MARL problem with sparse reward, including: reward shaping (Hu et al. 2020). curriculum learning (Chen et al. 2021). and learning from demonstrations (Huang al. 2021 Among these approaches, the curriculum learning paradigm, in which the difficulty of experienced tasks and the population of training agents progressively grow, shows particular promise. automatic curriculum learning (ACL), a teacher (curriculum generator) learns to adjust the complexity and sequencing of tasks faced by a student (curriculum learner). Several works have even proposed multi-agent ACL algorithms, based on approximate or heuristic approaches to teaching, such as DyMA-CL (Wang et al.) 2020c), EPC (Long et al.) 2020), and VACL (Chen et al. )02I). However, DyMA-CL and EPC rely on a framework an off-policy student with replay buffer, and ignore the forgetting problem that arises when the agent population size grows. ACL relies on the strong assumption that the value of the learned policy does not change when agents switch to a different task. Moreover, the teacher in these approaches is still facing unmitigated non-stationarity problem due to the ever-changing student strategies. In addition, if we somewhat expand the ACL paradigm and presume that the teacher may have another purpose for the sequence

---

Under review as a conference paper at ICLR 2023
of tasks performed by the student, another class of larger-scale MARL solutions should be mentioned. Namely, hierarchical MARL, which learns temporal abstraction with more dense rewards, including: skill discovery (Yang et al. 2019), option response (Vezhnevets et al. 2019), base-based MARL (Wang et al.] 2020b), and two levels of abstraction (Pang et al.) 2019). Alas, hierarchical MARL mostly focuses on specific task with fixed number of agents and does not consider the transfer ability of learned complementary skills. In this paper, we informally give our answer to question:
Can an elaborate combination of ACL and hierarchical principles learn complex cooperation with sparse reward in MARL?
Specifically, present novel automatic curriculum learning algorithm, Skilled Population Curricu- lum (SPC), which learns cooperative behaviors from scratch. The core idea of SPC is to encourage the student to learn skills from tasks with different numbers of agents. Motivation from the real world is team sports, where players often train their skills by gradually increasing the difficulty of tasks and the number of coordinating players. In particular, we implement SPC with three key components with the teacher-student framework. First, to solve the final complex cooperative tasks, we model the teacher as a contextual bandit, where we utilize an RNN-based (Hochreiter & Schmidhuber 1997) imitation model to represent student policies and use this to generate the bandit's 's context. Second, to handle the varying number of agents across these tasks, motivated by the transformer (Vaswani -invaiiiionnnaaiiant
ct al. 2017). which can process sentences of varying lengths, we implement communication by treating each agent's 's message as a word. Thus, a self-attention communication channel is used to support arbitrary number of agents sharing their messages. Third, learn transferable skills in the sparse reward setting, we utilize the skill framework in the student, Agents communicate on the high level about set of shared low-level policies. Empirical results show that our method achieves state-of-the-an performance in several tasks in the multi-particle environment (MPE) (Lowe et al. 2017) and the challenging 5vs5 competition in Google Research Football (GRF) (Kurach et al. 2019).
PRELIMINARIES
Dec-POMDP. A cooperative MARL problem can be formulated as a decentralized partially observ- able Markov decision process (Dec-POMDP) (Bernstein al., (002). which is described tuple (n, S, A. P, R. O. 2. 7), where represents the number of agents. $ represents the space of global (A,):=1..
states.
denotes the space of actions of all agents. O {0:)=e,.-. n denotes the space of observations of all agents. P S x A + S denotes the state transition probability function. All agents share the same reward as a function of the states and actions the agents R: S x A - R. Each agent i receives a private observation og E O2 according to the observation function (ss i): : S Oz. E [0, 1] denotes the discount factor.
Multi-armed Bandit. repeatedly makes decisions under uncertainty. In an MAB, a learner performs a sequence of actions. After every action, the learner immediately observes the reward corresponding to its action. Given set of K actions and a time horizon T, the objective is to maximize its total reward over T rounds. The regret used measure the gap between the cumulative reward of an MAB algorithm and the best-arm benchmark. An representative work is the Exp3 algorithm (Auer et al., 2002), which is proposed to increase the probability of pulling good arms and achieves a regret of Oly logl log(K)) under time-varying reward distribution. Another related work is the contextual bandit problem (Hazan & Megiddo 2007). where the learner makes decisions based on some prior information as the context.
bandits (MABs) are simple but very powerful framework that
3 SKILLED POPULATION CURRICULUM
In this section, we first provide a formal definition of the curriculum-enhancee Dec-POMDD frame- work, which formulates the MARL with curriculum problem under the Dec-POMDP framework. We then present our multi-agent automatic curriculum learning algorithm named Skilled Population Curriculum (SPC) shown in Fig. [ In the following subsections, we first formulate the curriculum learning framework in 3.1 then propose a contextual multi-armen bandit algorithm as the teacher to deal with the non-stationarity in 3.2 and last introduce the student with a skill and population- invariant communication framework to tackle the varying number of agents and the sparse reward problem in 3.3
2

---

Under review as a conference paper at ICLR 2023
Teacher
Student
Population-invariant
skill
Contexts
an.
4j
TE Low
hidden state
RNN
my
view of agent
Prob.
o,r
a
1 m
self-att channel
1711
-i2g
Env
Tasks
no S A41, ,,,,,,,,,,,,, r
Φ1 42
k--1 0k
teacter
The overall framework of SPC. It consists of three parts: configurable environments, a teacher, and a student. Left. The teacher is modeled as a contextual multi-armed bandit. At each teacher timestep, the teacher chooses a training task from the distribution of bandit actions. Mid. The student endowed with appulationinvarrian communication and a skill framework, and trained with MARL algorithms on the training task. The student returns to the teacher not only the hidden state of RNN imitation model as contexts but also the average discounted cumulative rewards on the testing task. Right. The student learns hierarchical policies. The population-invarian communication is on the high level, and implemented with a self-attention communication channel to handle the messages from varying number of agents. The agents in the student share the same low-level policy.
Figure I:
3.1 PROBLEM FORMULATION
We consider environments from multi- agget automatic curriculum learning problems are equipped with parameterized task spaces and thus can be modeled as curriculum curicullu-eehhanee enhanced Dec-POMDPs.
Rec-POM is defined by a tuple MM) where & and M represent a task space and a DecDPO respectively. Given the task Φ, the Dec-POMDP M(r) is presented as (n, A Pd,
Definition 3.1 (Curriculum-enhancee Dec-POMDP), A curriculum-enhance
A &
r O+ ;2,;-") )²). The superscript
& denotes that the Dec- PecMPOM elements are determined by the task ø. Note that task o can be a few parameters of the environment or task IDs in a finite task space. In a curriculum-enhunce Dec-POMDP, the objective is to improve the student's 's performance the target tasks through the sequence of training tasks given by the teacher.
Let denote trajectory whose unconditional distribution (r. under policy and a task with initial state distribution ((so) is (r) = μ (so) Ei ग (ag | st) pd ($1+1 | &t, ae). We use p(o) to represent the distribution of target tasks and ((σ) to represent the distribution of training tasks at each task sampling step. We consider the joint agents' policies <o(als) and qu (() parameterized by 8 and v respectively. The overall objective to maximize in a curriculum-enhanced DeMMP is:
p(p) (5(
J(0, 1/ = -Ep-p(d))rrPe [R) (x)
V (e) To)
where H (T) = -4 (st, aci so) and V (Ø, To) represent the value function of Tae in Decc-eMDD M(Ø). However, when optimizing (Ø). we cannot get the partial derivative V0, v) V E, To Ro) Pr 0 (r) since the reward function and the transition probability function w.r.t number of agents are parammeeee,i, non-differentiable, and discontinuous in most MARL scenarios.
Thus, we use the non- dindiffeerentable method, i.c., numti-armed bandit algorithms, to optimize qe (d). and use RL algorithm (the student) in alternating periods to optimize To(als). However, there three key challenges in solving this problem: (1) There is a lack of a general student framework to deal with the varying number of agents across tasks and the sparse reward problem. (2) The teacher
(3) The student will forget the old tasks and need re-learning them. Some tasks can be the prerequisites of other tasks, while some can be inter-independent and parallel.
is facing a non-stationarity problem due to the ever-changing student's 's strategies.
p(()) is not in the partial derivative since it is fixed distribution.
3

---

Under review as a conference paper at ICLR 2023
3.2 TEACHER MODELED AS A VON-STATIONARY CONTEXTUAL BANDIT
As mentioned in the previous subsection, the teacher is facing non-stationarity problem due to the ever-changing student's strategies, since the student learns across different tasks process. That is, in different stages of student learning. the teacher will observe different student's 's performances when giving the same task to the student, thus leading to a time-varying reward distribution of the teacher. In addition, the student might forget the learned policy during the learning process. To avoid this problem, the teacher should offer some trained tasks to the student. It can be seen as the exploitation and exploration problem of the teacher. The teacher is encouraged to give the training tasks that benefit the student's performance on the target tasks; however, there is still need for sufficient exploration various training tasks that may not directly facilitate the student's learning.
Fortunately, we notice that the non- staniioation stems from the student, which can be mitigated with contextual bandit which embeds the student policy into the context. As shown in Fig. Left, the teacher takes the student's policy representation as the context and chooses a task from the distribution of training tasks. Specifically, we extend the Exp3 algorithm (Auer et al.] (002) with context by two-step online clustering (see Alg Zhang et al. 1996)). The context æ is the student's policy representation, the teacher's action is certain task Φ, and the teacher's s reward is the return of the student in the target tasks. The teacher's 's algorithm is listed below. In steps 1-4, the teacher samples task for the student's 's training. In steps 6-7, the teacher would update the parameters based on the evaluation reward of the student. So, we need to learn a good representation for the student's 's policy as the context.
Algorithm Teacher Sampling and Training
Input:
Context x, the number of Clusters No. No instances of Exp3 with task distribution
w(ok, e) for k .,.... K and for e I,..., No. learning rate a, buffer maintaining the his-
torical contexts
Output:
M(Ø) = n A p0, r 0+, ,,9,,..,,,..,,,, }.). , the teacher bandit parameters
Sampling
1. Get the the context x, and save it to the buffer
2. Run the online cluster algorithm and get the index of the cluster center c(x)
3. Let the active Exp3 instance be the instance with index c(x)
4. Set the probability p(ox, c(x)) (1 (1-a)w((2.e(x))
for each task Φ
Σx= w((x.c(xx) K
5. Sample a new task according to the distribution of Pox.c Training
6. Get the return (discounted cumulative rewards) from student testing
7. Update the active Exp3 instance by setting w(ds. c(x)) = w(ckk, c(rr))
3.2.1 CONTEXT REPRESENTATION
straightforward representation is to directly use the student parameters as the context. However, the number of parameters is too large to be used as the input of neural network if we change the student's architecture. Thus, we turn to an alternative method.
A principle to learn good representation of policy is predictive representation, that is, the representation should be accurate to predict policy actions given states. According to the principle, we utilize an imitation function through supervised learning. Supervised learning does not require direct access to reward signals, making it an attractive approach for reward-agnoste representation learning. Intuitively, the imitation function attempts to mimic low-level policy based on historical behaviors. In practice, we use an RNN-based imitation function fam S x A + J0, 1]. Since recurrent neural networks are theoretically Turing complete (Hyötyniemi) 1996), its internal states can be used as the representation of the student's 's policy. Regarding the training of this imitation function, we use the negative cross entropy objective Ellog Jem (s.a). a)).
3.2.2 REGRET ANALYSIS
In this subsection, we show the regret bound of the proposed teacher algorithm E[R(T)] O (T2/3 log 1/3), where the number of total rounds, L is the Lipschitz constant, and K is the number of arms (the number of the teacher's actions). The regret analysis is used to justify the
4

---

Under review as a conference paper at ICLR 2023
usage of the bandit algorithm in the non-stationary setting. Since the teacher's reward is the return of the student the target tasks, the regret bound shows the optimality of the proposed method.
First, we introduce the Lipschitz assumption about the generalization ability of the task space. Assumption 3.2 (Lipschitz continuity w.r.t the context). Without loss of generality, the contexts are mapped into the [0, 1] interval, that the expected rewards for the teacher are Lipschitz with respect
to the context.
Ir( |)) - ( x²)l S L. |x- - x'1
for any arm Φ E and any pair of contexts a, x' E R where L is the Lipschitz constant, and X' is the context space.
This assumption suggests that, for any policy that is trained on set of tasks, the rate of performance change is not faster than the rate of policy change. It is realistic assumption since we cannot expect the student to achieve a dramatic improvement on a given task when the student is represented by new context via a few training steps.
Then, we borrow contextual bandit algorithm for small number of contexts (Auer et al. 2002 (see Appendix Alg- and the lemma as stepping stone for the proof of Theorem 3.4.
Lemma 3.3. Alg. has regret B[R(T)) = C(oOOV////) log K).
Lemma 3.3 introduces square root dependence on |. if running separate copy of Exp3 for each context Auer et al. 2002). It motivates us to handle large context space by discretization.
Theorem 3.4. Consider the Lipschitz contextual bandit problem with contexts in [0, 1]. The Alg. yields regret [E[R(T)] (TK T)13)))
Proof. See Appendix B for the proof.
In practice, the contextual space high-dimensional instead of in [0, 1]. and uniform mesh used to discretize the context space in the proof. Since we cannot have such a uniform mesh, without loss of generality, we utilize the Balanced Iterative Reducing and Clustering using Hierarchies (BIRCH) online clustering algorithm (Zhang et al. (999) to generate and discretize the context space. BIRCH summarizes large datasets into a smaller tree with clustered leaf nodes. The clustering is efficient and easy to update for newly added data. Thus, it is suitable for clustering the RNN-based policy representation. At the end of the training, the cluster can be seen as an approximation of the uniform mesh.
3.3 STUDENT WITH IPPULATION-INVARIANY SKILLS
Here, to tackle the varying number of agents and the sparse reward problem, we propose a population- invariant skill framework where agents can communicate via self-attention channel. In this framework, agents can learn the skills that can be transferred between different tasks. The student module MARL algorithm-agnostic.. It is orthogonal to any state-of-the-a MARL algorithm. Although there are few researches (Iqbal et al. 2021 Hu et al. [02T trying to address the varying number agents, they heavily rely on the prior knowledse of the environments.
Population- Invarian Teamwork Communication. Instead of learning independent policies for agents in the student, we introduce communication to enable the population iopulation-invariann property and learn tactics among agents. Motivated by the fact that the transformer (Vaswani et al. (017)) in natural language processing can handle varying lengths of sentences, we use the self-attention mechanism in our communication. As shown in Fig. [ Right, each agent j receives an observation oj. each round of communication, each agent j sends a message vector my (o;) to a self-attention channel, where f is an observation encoder function.
The channel aggregates all messages and sends the new message vector my through the self-attention mechanism. Concretely, given the input of the channel M = [m] ....... ,,,] E x=xdm and the trainable weight of the channel W Q: W Wr Wy E xdm we can obtain three different representations Q MW Q. K MW K, V MWv. Then, the output messages are
QKT
M = Attention K, V) = softmax
= [ma mag ' m
(3)
ddm
5

---

Under review as a conference paper at ICLR 2023
Simple Spread
Push Ball
Agents
Landmark
(a)
(b)
Figure 2:
The environments. (a): Multi-particle Environment. (b): Google Reaserach Football
where dm the dimension of the messages. Since the dimensions of the trainable weight are irrelevant to the number of agents, the student can take advantage of the population-invan invariant property to learn tactics.
Transferable Hierarchical Skills. As shown in the dotted box in Fig. Right, after receiving the new messages vi from the channel, each agent takes the high-level action (skill) an. = . (oj. my to execute the low-level policy aj = noow (oj, ah.j). In this work, we generalize the high-level action (skill) an.j to a continuous embedding space, so that the skill can be either latent continuous vector as in DIAYN (Eysenbach et al. 2018), or a categorical distribution for sampling discrete options
(Bacon et al. 2017).
Implementation. We implement the high- and low-level policies in the student with Proximal Policy Optimization (PPO) (Schulman et al. (017). Following the common practice proposed in (Fu et al. 2022). the high-level policy for each agent is learned independently. whereas the low-level policies share parameters, since the most basic action pattern should be the same within different agents. The low-level agent is rewarded by the environment. The high-level policy takes actions given fixed interval during training. Within this interval, a cumulative low-level reward is used as a high-level reward. When the categorical distribution is used to enable discrete skill, we would sample an "option" from the categorical distribution and feed the corresponding one-hot embedding to the low-level policy.
4 FURTHER RELATED WORK
Automatic Curriculum Learning in MARL. Curriculum learning is a training strategy inspired by the human learning process, mimicking how humans learn new concepts in an orderly manner, usually based on the difficulty level of the problems (Portelas al. 2020b). The selection of tasks formulated as a Curriculum Markov Decision Process (CMDP) (Narvekar & Stone 2018). Automatic Curriculum Learning mechanisms aim to learn a task selection function based on information about past interactions, such as ADR (Akkaya et al. 2019 Mehta et al. 2020), ALP-GMM (Portelas et al., 2020a), SPCL (Jiang et al. 2015), GoalGAN (Florensa et al. 2018), PLR (Jiang et al.) 202 lb a), SPDL Klink al. 2020), CURROT (Klink et al. (2022). and graph hhuph-eurricllum (Svetlik al.) 2017]. Inspired by the mechanism of biodiversity in nature, a series of MARL curriculum learning frameworks have recently been proposed with remarkable empirical success. These include open- ended evolution (Banzhaf et al. [216,]Lemman Lehman et al. 2008 [Standish] 2003).
population- based training
(Jaderberg al. 2019 Liu al. 2019), meta- meta-nearning Gupta et al.
2021
Portelas al. 2020c and
training with emergent curriculum (Baker et al. 2019 Leibo et al.
2019
Portelas et al., 2020b). In
general, these frameworks can be unitied under the idea of an automatic curriculum that automatically generates an endless procession of better performing agents by exerting selection pressure among many self-optimizing agents.
Hierarchical MARL and Communication. Hierarchical reinforcement learning (HRL) has been extensively studied to address the sparse reward problem and to facilitate transfer learning. Single- agent HRL focuses on learning the temporal decomposition of tasks, either by learning subgoals
(Nachum et al.) 201 8b a Sukhbaatar et al.) 20188 Nair & Finn 2019 Wang et al. 202 or by discovering reusable skills (Daniel et al. 2012 Gregor et al. 2016 Shankar & Gupta| 2020 Sharma et al., (020). Recent works about hierarchical MARL have been discussed in the Introduction. In multi-agent settings, communication has demonstrated success in multi-agent cooperation (Foerster
et al. 2016: Das et al. 2019 Sukhbaatar et al.) 2016 Singh et al. 2018: Jiang & Lu 2018:) Kim et al.
2019
Wang et al. 2020a). However, existing approaches that extend HRI to multi-agent systems

---

Under review as a conference paper at ICLR 2023
utilize communication are limited to fixed number of agents and are hard to transfer with different number of agents.
Google Research Football (Kurach et al. 2019). Recent works attempt to tackle multi-agent scenarios in GRF by using a containerized learning framework (Wu et al. 202I). learning from demonstration (Huang al., (021), individuality (Jiang & Lu (021), and diversity (Li et al. 2021).
However, they mainly focus on single-agent control, or train relatively easy academy tasks in GRF, or use offline expert data to train agents.
5 EXPERIMENTS
We consider several tasks in two environments, Simple-Spread and Push-Ball in the Multi-agent Particle-world Environment (MPE) (Lowe et al. (017)) and the challenging 5vs5 task of GRF (Kurach ct al. 2019), to further demonstrate the performance of our approach.
We aim to answer the following three research questions. Q1:
Is curriculum learning needed in the complex large-scale MARL problem? (Sec. (.2) Q2: Can our SPC outperform previous curriculum- based MARL methods? If so, which components in SPC contributes the most to performance gains?
(Sec. 5.3
Q3: Can SPC learn a good curriculum for the sludent? (Sec. 5.4
5.1 ENVIRONMENTS, BASELINES AND METRIC
Environments. In the GRF 5vs5 scenario, we need to control 4 agents (except the goalkeeper) to compete with the opponent built-in AI. Each agent would observe a compact encoding, which consists of 1 15-dimensional vector summarizing many aspects of the game, such as player coordinates, ball possession and direction, active player, and game mode. The action set available to an individual agent consists of 19 discrete actions such as idle, movement, passing, shooting, dribbling, or sliding. The GRF provides two types of reward: scoring and checkpoints, to encourage the agent to move the ball forward and have a successful shot.
In MPE, we investigate Simple-Spres and Push-Ball (see Fig. 2ap. In Simple-Spread there are n agents that need to cover all landmarks. Agents are penalized for collisions and only receive positive reward when all the landmarks are covered. Push-Ball, there are n agents, n balls, and n landmarks. The agents need push the balls to cover every landmark. A success reward is given after all the landmarks have been covered.
Baselines. We evaluate the following approaches as baseline in Table
We compare MARL algorithms to justify curriculum learning in the complex large-
Table 1: Baseline algorithms.
Categories
Methods
scale MARL problem. Also, we modify VACL by removing the centralized critic for a fair comparison of the MPE. Due to the difficulty of the GRF, we include a shooting reward to encourage the student shoot.
MARL (Q1)
QMIX (Rashid et 2018
IPPO (de Witl et al. 2020
Curriculum-basse
IPPO with uniform task sampling
(Q2)
VACL (Chen et al. 2021
Ablation Study (Q3)
SPC with uniform task sampling
Metric. Even if we use the reward to opti- mize various algorithms, the mean episode reward in such environments cannot show the performance of the agents. Therefore, for GRF scenar- ios, we plot the win rate and the average goal difference, which is the number of goals scored by the MARL agents minus the number of goals scored by the other team.
SPC without HRL and COM
The experiments are carried out on 30 nodes, one of which has 128-core CPU and 4 A100 GPUs. Each experiment trial is repeated over 5 seeds and runs for 1-2 days.
5.2 THE NECESSITY OF CURRICULUM LEARNING
First, we describe experiments using MPE, In contrast to the fully observable setting and the centralized critic in VACL, we consider individual PPO in partially observable environments with default rewards. We randomly pick a starting state, and the episode ends after a fixed number of maximum steps. To be specific, the task space consists of n agents, where {2, 4,8, 16). We set the maximum allowed steps to 25. All evaluations are performed on the target task, where n = 16. IPPO is trained and evaluated directly the target task. In Fig. we can see that IPPO performs
7

---

Under review as a conference paper at ICLR 2023
(a) Cover Hate on Simple Spread
[b] Cover Rate on Push Bell
(e) Simple Spread
Push Ball
VACL
Retelin 1
0.35
0.50
0.25
MoatRumber
0
6
0.20
Vicc SPC
0.3
0.
2,2
3.5
Thiining Timesteps
Training Timestapa
Thariig Timestsps
Tharring Timestops
Figure 3: The evaluation performance of vari- ous methods on MPE.
Figure 4: The changes in the number of agents on MPE.
SPC
SPC SPC urilorm
0.8
SPC or Leiform
SPC with HILL
SPC wis, HELL
w/th CON
ain.
20.
INTO unilorm
INFO
boacler
0.2
Training Timeeluss
Training Timestops
(a) Win Rate
(b) Goal Difference
Figure 5: The evaluation performance of various methods on 5vs5 football competition.
nearly VACL. SPC achieves higher coverage rate than the baseline methods, but the improvement not significant. Furthermore, we experimentally investigate the probability variation of different population sizes in Fig. 4 We observe that the curriculum afforded by SPC is approaching the target task. The results illustrate that in a simple environment where the student can directly learn to complete the task, there is need to apply curriculum learning.
Then we show the performance comparison with the baselines in GRF. We also run CDS (Li et al. 2021) and CMARL (Wu al. 2021), however, we did not include their performances, since the goal difference reported in CMARL (Wu et al. 2021 is relatively low compared to our method. In Fig. 5a we can that without the curriculum learning scheme, QMi and IPPO cannot perform well in the 5vs5 scenario. However, IPPO is slightly better than QMiX in the scope of MARL algorithms in this scenario. In Fig. 5b we omit the lines of QMix since the mean score low, affecting the presentation of the figure. The reason could be that QMix is an off- off--poicy MARL algorithm, which would rely heavily on the replay buffer. However, in such sparse reward scenarios, the replay buffer has much less efficient samples for QMix to learn. For example, the replay buffer would contain tons of zero-score samples, leading to a non-promising performance. Meanwhile, IPPO with a shared actor and critic, on-policy algorithm, would utilize the samples more efficiently. Therefore, curriculum learning is promising solution to the complex large-scale MARL problem.
During our experiments, we found that IPPO shared parameter PPO can easily achieve good performance in most academic scenarios in GRF However, 5vs5 is an obstacle for agents to more complex scenarios. Due the limitation of computational resources, tested SPC in the 1lvsl1 scenario. The result can be seen the Appendix((
handle
5.3 PERFORMANCE AND ABLATION STUDY
In the experiments on MPE, In both environments, SPC performs better than VACL. Instead of training with continuous relaxation of the categorical distribution of population size in VACI., our bandit teacher achieves a higher success rate at test time, since the population size discrete variable in nature. Also, in Fig. we observe that the curriculum provided by SPC is effective in exploring the task space as agents become increasingly competent.
In the experiments on GRF, we do not include VACL in our baselines in the GRF, since the imple- mentation in the source code of VACL heavily based on prior knowledge of specific scenarios, such as the threshold to divide the learning process. We see that SPC has higher win rate and goal difference than IPPO with uniform task sampling in the 5ys5 football competition. The experiments MPE and GRF show that when the teacher is rewarded by the student's performance,
8

---

Under review as a conference paper at ICLR 2023
5. 5_v5_5
0.8
academy Bacll_aas__saa shoot withwwwhhppppppppe keeper
0.5
Task Prob
0.6
ecademy_empaa htwpyyeepttyyoyyoeee Serrattygoallloseec
Dimension's
0.4
0.2
M
40
60 M Training Timesteps
80 M. 100 M 120 140 M
-05
0.0
0.5
Dimension
(a) The task distribution of SPC during training.
(h) The visualization of contexts
Figure 6: Visualization of Learned Curriculum.
the bandit-based teacher can exploit the student learning stage and give the suitable training tasks to the student.
For ablation study, we replace our contextual multi-armed bandit teacher with uniform task sampling and remove the hierarchical part in the student framework. As shown in Figs. 5a 5bl we can clearly see that SPC can achieve a higher win rate and a greater score difference than SPC with uniform and SPC w/o. HRL. Also, SPC with uniform task sampling outperforms IPPO with uniform task sampling. The difference between these two methods is only the introduction of HRL. It shows the contribution of HRL in the 5vs5 football competition. When removing HRL and contextual multi-armed bandit, the performance degradation w.r.t. SPC are similar. It shows that HRI and the contextual multi-armed bandit seem to contribute equally. This can again justify the need for a curriculum learning scheme. However, we can see that SPC w. uniform has a larger variance in performance than SPC w/o. HRL. means that uniform sampling might introduce more undesired tasks for student training.
5.4 VISUALIZATION OF LEARNED CURRICULUM
We visualize the distribution of task sampling of SPC during training based on a selected trial as shown Fig. 66a] An interesting observation that the task probability seems nearly uniform. We interpret this into anti-forgettin mechanism. We can that the beginning of training, the task probability seems to be near-uniform, since the teacher should explore the task space and try to keep track of the student's learning status. During training, the probabilities vary at about 80-100 million timesteps, we can see a sudden drop in academy empty cmppty-gco yoall 1 cllllllll oss and academy 3______________ wi wwwithhkeee keepee, since the student almost handles the skills learned in such scenarios. However, when training is continued, we can still observe that agents are trained on these tasks more frequently. We also visualize the distribution of contexts in Fig. 66 using t-SNE (Van der Maaten & Hinton| 2008). The contexts are collected and stored in a buffer. We divide the contexts into four classes according to the index. We can clearly see different parts in terms of different contexts about the final student policy representation.
time steps. For example,
DISCUSSION
Conclusion. We introduce novel ACL algorithm, Skilled Population Curriculum (SPC). SPC addresses the scalability and sparse reward issue in current multi-agent system and learns complex behaviors from scratch. Specifically, to handle the varying number of agents, we incorporate population-invarian multi-agent communication framework and exploit a hierarchical scheme for each agent to learn skills to deal with sparse rewards. Moreover, to mitigate the non-stationarity, we model the teacher representation. Empirical results show that our method achieves wate-of-the-an performance on several tasks in the multi martice environment and the challenging 5vs5 competition in GRF.
contextual bandit, where the context is represented by the student's policy
Limitations. We acknowledge some limitations our algorithm. First, since the objective of SPC is to solve difficulty tasks, SPC is desr-ded when applied to some simple tasks. Second, the set of subtasks is still closely related to the environment. Lastly, remains unclear how much the number of agents can affect the dynamics of the environment since treat it black-box optimization.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Igge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et Solving rubik's cube with 13, 2019.
robot hand. arXiv preprint
Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, and Robert E Schapire. The nonstochastic multiarmed bandit problem. SIAM Journal on Computing, 32(1):48-77, 2002.
Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent tool use from multi-agent autocurricula. arXiv preprint 1rXiv:190..07588, 2019.
Wolfgang Banzhaf, Bert Baumgaertner, Guillaume Beslon, René Doursat, James Foster, Barry Mc- Mullin, Vinicius Veloso De Melo, Thomas Miconi, Lee Spector, Susan Stepney, et al. Defining and simulating open-ended novelty: Requirements, guidelines, and challenges. Theory in Biosciences, 135(3):131-161, 2016.
Daniel S Bernstein, Robert Givan, Neil Immerman, and Shlomo Zilberstein. The complexity of decentralized control of Markov Decision Processes. Mathematics of Operations Research, 27(4): 119-840, 2002.
Jiayu Chen, Yuanxin Zhang, Yuanfan Xu, Huimin Ma, Huazhong Yang. Jiaming Song, Yu Wang, and Yi Wu. Variational automatic curriculum learning for sparse-reward cooperative multi-agent problems, Advances in Neural Information Processing Systems, 34, 2021.
Christian Daniel, Gerhard Neumann, and Jan Peters. Hierarchical relative entropy policy search. In Artificial Intelligence and Statistics, pp. 273-281, 2012.
Abhishek Das, Théophile Gervet, Joshua Romoff, Dhruv Batra, Devi Parikh, Mike Rabbat, and Joelle Pineau. Tarmac: Targeted multi-agent communication. In International Conference on Machine Learning, pp. 1538-1546. PMLR, 2019.
Christian Schroeder de Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip HS Torr, Mingfei Sun, and Shimon Whiteson. Is independent learning all you need in the StarCraft multi-agent challenge? arXiv preprint arXiv:2011.0955: 2020.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv: /00.vv/000000000 2018.
Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. In International Conference Machine Learning, pp. 1515-1528. PMLR, 2018
Jakob Foerster, Ioannis Alexandros Assacl, Nando De Freitas. and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. Advances neural information processing systems, 29, 2016.
Wei Fu, Chao Yu, Zelai Xu, Jiaqi Yang, and Yi Wu. Revisiting some common practices in cooperative multi-agent reinforcement learning, arXiv preprint wwXXi:2206.0700.. 2022.
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv
preprint 161 I. 1.05507, 2016.
Abhinav Gupta, Marc Lanetot, and Angeliki Lazaridou. Dynamic population-based meta-learning for multi-agemt communication with natural language. Advances in Neural Information Processing Systems, 34:16899-1699 12, 2021.
Elad Hazan and Nimrod Megiddo. Online learning with prior knowledge. In International Conference Computational Learning Theory, pp. 499-513. Springer, 2007.
10

---

Under review as a conference paper at ICLR 2023
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Siyi Hu, Fengda Zhu, Xiaojun Chang, and Xiaodan Liang. Updet: Universal multi-agent rein- forcement learning via policy decoupling with transformers. arXiv preprint arXiv:2101.0800 2021.
Yujing Hu, Weixun Wang, Hangtian Jia, Yixiang Wang, Yingfeng Chen, Jianye Hao, Feng Wu, and Changjie Fan. Learning utilize shaping rewards: new approach of reward shaping. arXiv preprint arXiv:201 1.02669, 2020.
Shiyu Huang, Wenze Chen, Longfei Zhang, Ziyang Li. Fengming Zhu, Deheng Ye, Ting Chen, and Jun Zhu. Tikick: Toward playing multi--geett football full games from single-agent demonstrations. arXiv preprint arXi:221 2021.
Heikki Hyötyniemi. Turing machines recurrent neural networks. In STeP '96/Publications of the Fimnish Artificial Intelligence Sociery, 1996.
Shariq Iqbal, Christian A Schroeder De Witt, Bei Peng, Wendelin Böhmer, Shimon Whiteson, and Fei Sha. Randomized entity-wise factorization for multi-agent reinforcement learning. In International Conference on Machine Learning, pp. 1596-4606, PMLR, 2021.
Max Jaderberg, Wojciech M Czarnecki, lain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castaneda, Charles Beattie, Neil C Rabinowitz, Ari S Morcos, Avraham Ruderman, et al. Human- level performance in 3d multiplayer games with population-bes based reinforcement learning. Science, 364(6443):859-388 865, 2019.
Jiechuan Jiang and Zongqing Lu. Learning attentional communication for multi-agent cooperation. Advances in neural information processing systems, 31, 2018.
Jiechuan Jiang and Zongqing Lu. The emergence of individuality. In International Conference on Machine Learning, pp. 4992-5990000 5001. PMLR, 2021.
Lu Jiang, Deyu Meng, Qian Zhao, Shiguang Shan, and Alexander G Hauptmann. curriculum learning. Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015.
Minqi Jiang, Michael Dennis, Jack Parker- Holker, Jakob Foerster, Edward Grefenstette, and Tim Rocktäschel. Replay-guided adversarial environment design. Advances in Neural Information Processing Systems, 14:1888-1899 2021a.
Minqi Jiang, Edward Grefenstette, and Tim Rocktäschel. Prioritized level replay. In International Conference on Machine Learning, pp. 4940-4950. PMLR, 2021b.
Daewoo Kim, Sangwoo Moon, David Hostallero, Wan Ju Kang, Taeyoung Lee, Kyunghwan Son, and Yung Yi. Learning to schedule communication in multi- reinforcement learning, arXiv preprint arXiv:1902.06 1554.0 2019.
Pascal Klink, Carlo D' D'Eramo, Jan R Peters, and Joni Pajarinen. Self-paced deep reinforcement learning. Advances in Neural Information Processing Systems, 33:9216-9222 (0666922,70220 2020.
Pascal Klink, Haoyi Yang, Carlo D'Eramo, Jan Peters, and Joni Pajarinen. Curriculum reinforcement learning via constrained optimal transport. International Conference on Machine Learning, pp. 11341-11358. PMLR, 2022.
Karol Kurach, Anton Raichuk, Piotr Stańczyk. Michal Zajkac, Olivier Bachem, Lasse Espeholt, Carlos Riquelme, Damien Vincent, Marcin Michalski, Olivier Bousquet, et al. Google research football: novel reinforcement learning environment. arXiv preprint arXiv: /80, 2019.
Joel Lehman, Kenneth O Stanley, et al. Exploiting open-endedness solve problems through the search for novelty. In ALIFE, pp. 329-336. Citeseer, 2008.
Joel Z Leibo, Edward Hughes, Mare Lanctot, and Thore Graepel. Autocurricula and the emergence of innovation from social interaction: A manifesto for multi-agent intelligence research. arXiv preprint /903 2019.
11

---

Under review as a conference paper at ICLR 2023
Chenghao Li, Chengjie Wu, Tonghan Wang. Jun Yang. Qianchuan Zhao, and Chongjie Zhang. Cele- brating diversity in shared aulti-agent reinforcement learning. arXiv preprint NDV::106602195, 2021.
Siqi Liu, Guy Lever, Josh Merel, Saran Tunyasuvunakool, Nicolas Heess, and Thore Graepel. Emergent coordination through competition. arXiv preprint arXiv: 19Xi::/022.0755: 2019.
Qian Long, Zihan Zhou, Abhibav Gupta, Fei Fang, Yi Wu, and Xiaolong Wang. Evolutionary popula- tion curriculum for scaling multi-agent reinforcement learning. arXiv preprint arXiv:2003.10423, 2020.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Picter Abbeel, and Igor Mordatch. Multi-agent actor- critic for mixed cooperative-compettive environments, Advances Neural Information Processing Systems, 2017.
Bhairav Mehta, Manfred Diaz, Florian Golemo, Christopher J Pal, and Liam Paull. Active domain randomization. Conference on Robot Learning, pp. 1162-1176. PMLR, 2020.
Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Near-optimal representation learning for hierarchical reinforcement learning. arXiv preprint arXiv: 20.
Ofr Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. Advances in neural information processing systems, 31, 2018b.
Suraj Nair and Chelsea Finn. Hierarchical foresight: Self-supervised learning long-horizon tasks
via visual subgoal generation. arXiv preprint 1r09v:190.005580 2019.
Sanmit Narvekar and Peter Stone. Learning curriculum policies for reinforcement learning, arXiv preprint arXiv:1812.00288 2018.
OpenAl. OpenAl Five. http:///pperaa..cm//llll/gggggn ofie://-iaaaaiieae/eeee 2019. Accessed
March 4, 2019.
Zhen-Jia Pang, Ruo-Ze Liu, Zhou-Yu Yu Meng, Yi Zhang, Yang Yu, and Tong Lu. On reinforcement learning for full-length game of starcraft. In Proceedings of the AAAI Conference on Artificial Intelligence, 2019.
Rémy Portelas, Cédric Colas, Katja Hofmann, and Pierre- Yver Oudeyer. Teacher algorithms for curriculum learning of deep RI in continuously parameterized environments. In Conference on Robot Learning, pp. 835-853, 2020a.
Rémy Portelas, Cédric Colas, Lilian Weng, Katja Hofmann, and Pierre- Yves Oudeyer. Automatic curriculum learning for deep RL: A short survey. arXiv preprint vrXiv:2003.04664 2020b.
Rémy Portelas, Clément Romac, Katja Hofmann, and Pierre Yves Oudeyer. Meta automatic curricu- lum learning. arXiv preprint arXiv:2011. 1004663, 2020c.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shi- mon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. International Conference on Machine Learning, pp. 4295-4304, 2018.
RoboCup. Robocup Federation Official Website. https::/wwwe hwtp::wwwbbouggg. www.:w..booop..oooooo 2019. Ac- cessed April 10, 2019.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06337, 2017.
Tanmay Shankar and Abhinav Gupta. Learning robot skills with temporal variational inference. In Proceedings of the 37th International Conference Machine Learning. JMLR. org, 2020.
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-awar unsupervised discovery of skills. In International Conference on Learning Representations, 2020.
Amanpreet Singh, Tushar Jain, and Sainbayar Sukhbuatar. Learning when to communicate at scale in multiagent cooperative and competitive tasks. arXiv preprint arXiv: 18/.. 829055555 2018.
12

---

Under review as a conference paper at ICLR 2023
Russell K Standish. Open-ended artificial evolution. Intelligence and Applications, 3(02):167-175, 2003.
International Journal of Computational
Sainbayar Sukhbaatar, Rob Fergus, et al. Learning multiagent communication with backpropagation. Advances in neural information processing systems, 29, 2016.
Sainbayar Sukhbaatar, Emily Denton, Arthur Szlam, and Rob Fergus. Learning goal embeddings via self-play for hierarchical reinforcement learning, arXiv preprint
2018.
Maxwell Svetlik, Matteo Leonetti, Jivko Sinapov, Rishi Shah, Nick Walker, and Peter Stone. Auto- matic curriculum graph generation for reinforcement learning agents. In Proceedings of the AAAI Conference Artificial Intelligence, volume 31, 2017.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is you need. Advunces in Neural Information Processing Systems, 30, 2017.
Alexander Sasha Vezhnevets, Yuhuai Wu, Remi Leblond, and Joel Z Leibo. Options as responses: Grounding behavioural hierarchies multi-agent arXiv preprint 1rXiv:1906.014700 2019.
Rundong Wang, Xu He, Runsheng Yu, Wei Qiu, Bo An, and Zinovi Rabinovich. Learning efficient multi-agent communication: An information bottleneck approach. In International Conference on Machine Learning, pp. 9908-9918 18. PMLR, 2020a.
Rundong Wang, Runsheng Yu, Bo An, and Zinovi Rabinovich. 12hrl: Interactive fefluence-base hierarchical reinforcement learning. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pp. 3131-3138, 2021.
Tonghan Wang. Tarun Gupta, Anuj Mahajan, Bei Peng, Shimon Whiteson, and Chongjie Zhang. Rode: Learning roles to decompose multi-agent tasks. arXiv preprint arXiv:2010.01523, 2020b.
Weixun Wang, Tianpei Yang. Yong Liu, Jianye Hao, Xiaotian Hao, Yujing Hu, Yingfeng Chen, Changjie Fan, and Yang Gao. From few more: Large-scale dynamic multiagent curriculum learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 7293- 7300, 2020c.
Siyang Wu, Tonghan Wang, Chenghao Li, and Chongjie Zhang. Containerized distributed value-based multi-agent reinforeement learning, arXiv preprint arXiv:2110.08169, 2021.
Jiachen Yang, Igor Borovikov, and Hongyuan Zha. Hierarchical cooperative multi-agent reinforce- ment learning with skill discovery. arXiv preprint arXiv: 1912.0335 2019.
Yaodong Yang and Jun Wang. An overview of multi-agent reinforcement learning from game theoretical perspective. arXiv preprint 0rXiv:2011.0083 2020.
Kaiqing Zhang, Zhuoran Yang, and Tamer Başar. Multi-agent reinforcement learning: A selective overview of theories and algorithms. Handbook of Reinforcement Learning and Control, pp. 321-384, 2021.
Tian Zhang, Raghu Ramakrishnan, and Miron Livny. Birch: An efficient data clustering method for very large databases. ACM Aigmod Record, 25(2):103-11) 1996.
13

---

Under review as a conference paper at ICLR 2023
A ALGORITHM
Algorithm 2 A contextual bandit algorithm for small number of contexts
1: Initialization: For each context 2, create an instance Exp3, of algorithm Exp3
2: for round do
3: Invoke algorithm Exp3,. with x
4: Play the action chosen by Exp3,
5: Return reward r. Exp3,
6: end for
B PROOF OF THEOREM 3.4
Theorem 3.4. Consider the Lipschitz contextual bandüt problem with contexts in [0, 11. The Alg. yields regret [[[[[T]] 0 (T2/3 I))s).)..
Proof. Let Sm be the e-uniform mesh on [0, 1], that is, the set of all points [0, 1] that are integer multiples of e. We take = 1/(d - where the integer d is the number of points in Sm, which will be adjusted later the analysis.
We apply Alg. 2 to the context space Sm- Let fs.. (ar) be mapping from context to the closest point in Sm:
fs.. (x) min argmin |xc a') w'ES
In each round t. we replace the context I, with fsm (xt) and call Exp3 The regret bound will have two components: the regret bound for Exp3s and (a suitable notion of) the discretiza-
tion error. Formally, let us define the "discretized best response" T 2 - T: T (x) = n (fs.. (x)) for each context æ X.
We define the total reward of an algorithm Alg is Reward(Alg) -E and the discretization error are defined as:
Then the regret of Exp3
Rs( = Reward (TT) Reward (Exp3g)
DE(S) Reward (x²*) Reward (Tg)).
It follows that regret is the sum R(TT) (T( DE(S). We have E [Rs(T)] OVVT log K) from Lemma 3.3 so remains to upper bound the discretization error and adjust the discretization step
For each round and the respective context x = æt, r (TT(x) | fs(z)) 2 r (T*( fs(x)) 2 (x* (x=) I) cL. The first inequality is determined by the optimality of nt' and the second is determined by Lipschitzness. Summing this up over all rounds f, we obtain E [Reward (mş)) 2
Reward [x*] eLT.. -eLT.
Thus, the regret that
( logT O (12/3 ( log 2/3333
E(R(T)) S ELT SELT+O
(4)
For the last inequality, we choose
K log 11/3
C I1vsl1 FULL GAME ON GRF
We further conduct experiments on the 1lvsl1 scenario of GRF. As shown in Fig. about 50% win rate after training with 200 million timesteps.
SPC achieves
14

---

Under review as a conference paper at ICLR 2023
SPC
0.5
0.
Wh
0.3
0.1
0.0
M
50.00
M. 300 150 175 M 200
Thaiinng Timestaps
Figure 7: The performance of SPC on the 11v11 scenario.
D QUALITATIVELY ANALYSIS ABOUT LOW-LEVEL SKILLS
We show some statistics, for example, shooting, passing and running, in GRF in terms of different fixed low-level policies (defined by set of fixed high-level actions). We evaluate the statistics by only fixing one agent's 's high-level action, maintaining other agents with trained SPC and testing times. We show the results in Table
shooting per game
passing per game
running per game
skill
7.9 times
0.5 times
2254 time steps
skill 2
2.3 times
26.4 times
2149 time steps
skill 3
1.6 times
3.9 times
2875 time steps
Table 2: Statistics analysis about low-level skills.
E CORNER-5 ON GRF
To further illustrate the effectiveness of the SPC teacher module, we conduct experiments on the corner-5 scenario GRF, where the target task is control five of the eleven players to obtain goal in the GRF aca demy corner scenario. The experiments are designed to determine whether not the contextual bandit in SPC outperforms alternative curriculum learning methods to schedule the number of agents in training. We compare SPC teacher against non-curricllum training (None), uniform task sampling (Uniform), ALP-GMM, and VACL. The training task space consists of agents, where n {1.3, 5}. All teachers have the same base architecture without transformer architecture and hierarchical framework. We also investigate the effect of RNN-based contexts (see Contextual Bandit and Bandit). Fig, showw the benefit of SPC contextual bandit over other ACL methods after training with one million timesteps.
Hone
Soree
of
Cineeciad Ranii
Cendit
-005
-0.0
(a) Win Rate
(b) Goal Difference
Figure 8: The evaluation performance of various teacher algorithms the GRF corner-5 scenario.
15

---

Under review as a conference paper at ICLR 2023
Table SPC hyper-parameters used in GRE
Name
Value
Discount rate
0.99
GAE parameter
1.0
KL coefficient
0.2
Rollout fragment length
1000
Training batch size
100000
SGD minibatch size
10000
# of SGD iterations
60
Learning rate
le-4
Entropy coefficient
0.0
Clip parameter
0.3
Value function clip parameter
10.0
Table 4: SPC hyper-parameters used in MPE.
Name
Value
Discount rate
0.99
GAE parameter
1.0
KL coefficient # of SGD iterations Learning rate
0.5
10
le-4
Entropy coefficient
0.0
Clip parameter
0.3
Value function clip parameter
10.0
F IMPLEMENTATION DETAILS
Here we describe the SPC framework. We use the open-sourced Ray RLlib implementation of Proxi- mal Policy Optimization (PPO), which scales out using multiple workers for experience collection. This allows to use a large amount of rollouts from parallel workers during training to ameliorate high variance and aid exploration. We multiple rollouts parallel with distributed workers and parameter sharing for each agent. The trainer broadcasts new weights to the workers after their synchronous sampling. We now turn our attention to enviromment-specife settings.
F.1 GOOGLE RESEARCH FOOTBALL
We set five tasks for training the 5vs5 scenario. They are academyeepty--raaaclooss www.sa.a...shhtt.wt....e.e......
3_vs_3, 5.vs.5. 5. In all scenarios, we do not control our team's goalkeeper.
In the scademm.emppy..ggal...... www.eetyyyyya.llsee..eeomo one agent need move forward and shoot with an empty goal. In academy academy.ppas.aandee
and 3_vs_3, two agents are controlled to play against goalkeeper and 3 players respectively. academy 3 3.vs wi keeper, three agents are controlled to play against center-back and goalkeeper. In 5 vs 5, 4 agents are controlled to play against players. Without loss of generality, we initialize player with fixed positions and roles center midfielders.
We use both MLP and self-attention mechanism for the high-level policy, and use MLP for the low-level policy. For high-level policy, the input is first projected to embedding using 2 hidden layers with 256 units each and ReLU activation, which is then fed into multi-head self-attention (8 heads, 64 units each). The output is then projected to the actions and values using another fully connected layer with 256 units. For low-level policy, we use MLP with hidden layers with 256 units each, i.e., the default configuration of policy network in RLlib.
16

---

Under review as a conference paper at ICLR 2023
F.2 MPE
In this environment, agents must cooperate through physical actions to reach set of landmarks. Agents observe the relative positions of other agents and landmarks, and collectively rewarded based on the proximity of any agent to each landmark. In other words, the agents have to 'cover" all of the landmarks. Further, the agents occupy significant physical space and are penalized when colliding with each other. The agents need to infer the landmark to cover and move there while avoiding other agents.
The hyper- parameters of SPC in MPE are shown in Table 4 In MPE, apper-parameters such as rollout fragment length, training batch size and SGD minibatch size are adjusted according to horizon of the scenarios so that policy are updated after episodes are done. We use the same network as GRF, but with 128 units for MLP hidden layers. Other omitted hyper-parameters follow the default configuration in RLlib PPO implementation.
17


Reviews:
[{"review_id": "fcIXgEMZulZ", "reviewer": "Reviewer_HPXF", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a learning framework that aims to address the change in the number of agents in an environment as a student agent is trained through a curriculum of tasks as proposed by a separate teacher agent. The teacher agent is modeled as a contextual bandit that aims to resolve the inherent non-stationarity in student policies. The results show that using the proposed changes, the agents are able to outperform previous work on competitive benchmarks.", "summary_of_the_review": "The paper proposes novel changes to the popular ACL framework addressing key challenges such as non-stationarity in student policies and variable agents in the populations.", "strength_and_weaknesses": "The paper does an extensive study on the current issues in Automatic Curriculum Learning framework. The schematic diagram helps in better understanding the contribution and the proposed changes to the ACL framework. The MAB correction to train the teacher also helps in computing an approximate regret bound of its policy.\n\nThe paper misses out on some prior work that also use variable agents in a population as a means to learn diverse strategies [1, 2]. I am curious to know how the does the hierarchical setup affect in training diverse student strategies. Could a setup using meta-RL help in learning more diverse strategies rather than using bandit-based Exp3 algorithm?\n\nBesides, some more ablation studies on analyzing the various student policies would also help in understanding the effect of the population invariant method?\n\n[1] Gupta et al. 2021. Dynamic population-based meta-learning for multi-agent communication with natural language.\n\n[2] Portelas et al. 2020. Meta Automatic Curriculum Learning.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "YQ4TLizxETS", "reviewer": "Reviewer_cbi7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes Skilled Population Curriculum (SPC), a framework for cooperative multi-agent reinforcement learning based on automatic curriculum learning. In particular, the student is equipped with a hierarchical skill set and designed to be population-invariant. The teacher is a contextual bandit that uses a representation of the student’s policy as its context. To implement population invariance, SPC uses self-attention to combine agent messages. In the experimental evaluation, SPC is compared to QMIX, IPPO, and VACL on Google Research Football and the Multi-agent Particle-world Environment, which they solve from (relatively) sparse rewards.", "summary_of_the_review": "The experimental results are very strong, showing the improved performance achieved by SPC. There are also useful ablations and visualizations that illustrate how SPC achieves this performance. I am recommending to accept this paper, but believe it could be strengthened (see Weaknesses).\n", "strength_and_weaknesses": "Strengths:\n- This paper tackles a particularly difficult problem, which is cooperative multi-agent RL from sparse rewards. While other methods struggle on the evaluated tasks, especially Google Research Football, SPC makes significant improvements in terms of performance.\n- The visualizations and ablation study help readers understand the importance of each component of SPC.\n\nWeaknesses:\n- The experiments are only performed on Multi-agent Particle-world Environment and Google Research Football, which is relatively narrow compared to prior works that also study other environments like the StarCraft II benchmark.\n- The method is quite complex. It’s shown, though, that the HRL structure and contextual bandit teacher are important. What about the self-attention channel?\n- It would be useful to understand qualitatively what the different learned skills are doing.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0iABY8VJgM3", "reviewer": "Reviewer_WEWH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes the SPC framework that combines automatic curriculum learning (ACL) and hierarchical MARL to learn cooperative behaviors in complex multi-agent domains. Specifically, this paper leverages a multi-armed bandit algorithm for ACL while addressing the following challenges: 1) the varying number of agents across tasks based on the transformer architecture, 2) non-stationarity based on the contextual bandit framework, and 3) reward sparsity based on the skills. Evaluations in MPE and GRF demonstrate the effectiveness of SPC. ", "summary_of_the_review": "I initially vote for 5 due to the weaknesses and concerns. I will make a final decision on the recommendation after the authors' response.", "strength_and_weaknesses": "**Strength:**\n1. The paper develops the theoretical contribution: the regret analysis of the contextual bandit problem for solving ACL (Section 3.2.2). \n2. Evaluations are performed using the complex domain of GRF.\n\n**Weaknesses, Questions, and Comment:**\n1. Overall, this paper combines multiple methods (e.g., transformer, skills, contextual bandits) for tackling too many challenges simultaneously, resulting in a relatively complicated framework. \n2. While SPC ablation studies are provided, it is still challenging to understand the effectiveness of SPC. For example, SPC uses the transformer architecture, but baselines do not use the transformer. As such, it is unclear whether SPC performs better than baselines due to ACL and skill or due to the base architecture difference. \n3. The paper's clarity can be improved. In particular, the paper states that multi-armed bandit algorithms are used due to the non-differentiable objective. However, there are other methods, including REINFORCE, and it will be helpful to justify why multi-armed bandit algorithms are selected over other methods. \n4. Regarding hierarchical skills, the paper states that the categorical distribution is used to enable the option-style skill by passing the one-hot embedding to low-level policies. However, each option consists of a tuple of the initiation set, intra-option policy (low-level policy), and termination function. As such, it is unclear whether simply passing the one-hot embedding to the low-level policy enables the option. \n5. Regarding the VACL baseline, the paper states that the centralized critic is removed for a fair comparison. However, my understanding is that the use of a transformer architecture can be effectively viewed as centralized training because my understanding is that gradient is passed across agents during training. If this view is correct, then it is unclear why the centralized critic is removed in VACL. \n6. In Section 5.4, the paper states that Figure 6b shows the shift in student policy representation throughout the training, but this is unclear because Figure 6b does not include the time notion. Could you clarify this? \n7. Minor: There is a typo: \"In Fig. 6, we can see that without the curriculum learning scheme\" -> \"In Fig. 5, we can see that without the curriculum learning scheme\"", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0XwUcS-npmB", "reviewer": "Reviewer_qKoe", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a feasible way to perform curriculum learning in multi-agent reinforcement learning (MARL). It describes well why curriculum learning cannot be applied to MARL seamlessly and proposes sound solutions to these problems. The experimental evaluation is conducted in a satisfactory way, although the results are not outstanding. There are additional theoretical results on the regret analysis of the proposed approach.", "summary_of_the_review": "This paper is a possibly good contribution to MARL, but has currently some limitations that I described above and that I would like to see addressed in the revision. I am willing to increase my score in case of an improved version of this paper.\n\nPost-rebuttal feedback\n------------------------------\nI thank the authors for the improved version of their submission. Most of my concerns have been addressed and I am happy to increase my score, as previously promised.", "strength_and_weaknesses": "Strengths\n-----------\n- The paper considers the important problem of enabling the use of curricula in multi-agent reinforcement learning (MARL).\n- The proposed solution is sound and backed up by satisfactory intuitive motivations.\n- The empirical analysis is conducted on challenging RL experiments comparing with a satisfactory number of baselines. I think it would be a nice addition to have comparisons with more than the single QMIX method for value function factorization (e.g., QTRAN [1], QPLEX [2]), although I do not expect great improvements.\n\nWeaknesses\n--------------\n- The paper is not very well written, and the structure is quite chaotic. Especially Section 3 is split in different subsections, not well connected, and the whole flow is not fluent. \n- The regret bound could be an interesting theoretical results, but it is not complemented by any remark or empirical study, which makes the analysis a bit pointless.\n- The method seems not easy to implement and relies on complex self-supervised mechanisms which may be not easy to use in generic problems. To this end, at least a hint of a computational and time complexity analysis of the proposed method is a needed addition.\n- The ability to transfer seems an important feature of the proposed method, but experiments do not provide clear evidence of effective skills transfer.\n- The paper does not cite important works in automatic curricula generation such as [3], [4], [5].\n\n[1] Son, Kyunghwan, et al. \"Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning.\" International conference on machine learning. PMLR, 2019.\n\n[2] Wang, Jianhao, et al. \"Qplex: Duplex dueling multi-agent q-learning.\" arXiv preprint arXiv:2008.01062 (2020).\n\n[3] Klink, Pascal, et al. \"Self-paced deep reinforcement learning.\" Advances in Neural Information Processing Systems 33 (2020): 9216-9227.\n\n[4] Klink, Pascal, et al. \"Curriculum Reinforcement Learning via Constrained Optimal Transport.\" International Conference on Machine Learning. PMLR, 2022.\n\n[5] Svetlik, Maxwell, et al. \"Automatic curriculum graph generation for reinforcement learning agents.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 31. No. 1. 2017.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject