Title: Towards Skilled Population Curriculum for Multi-Agent Reinforcement Learning
Venue: NeurIPS 2023
Accepted: no
Keywords: multi-agent reinforcement learning

multi-agent curriculum

skill learning
Abstract: Recent advances in multi-agent reinforcement learning (MARL) allow agents to coordinate their behaviors in complex environments. However, common MARL algorithms still suffer from scalability and sparse reward issues. One promising approach to resolving them is automatic curriculum learning (ACL). ACL involves a student (curriculum learner) training on tasks of increasing difficulty controlled by a teacher (curriculum generator). Despite its success, ACL's applicability is limited by (1) the lack of a general student framework for dealing with the varying number of agents across tasks and the sparse reward problem, and (2) the non-stationarity of the teacher's task due to ever-changing student strategies. As a remedy for ACL, we introduce a novel automatic curriculum learning framework, Skilled Population Curriculum (SPC), which adapts curriculum learning to multi-agent coordination. Specifically, we endow the student with population-invariant communication and a hierarchical skill set, allowing it to learn cooperation and behavior skills from distinct tasks with varying numbers of agents. In addition, we model the teacher as a contextual bandit conditioned by student policies, enabling a team of agents to change its size while still retaining previously acquired skills. We also analyze the inherent non-stationarity of this multi-agent automatic curriculum teaching problem and provide a corresponding regret bound. Empirical results show that our method improves the performance, scalability and sample efficiency in several MARL environments. The source code and a video demonstration can be found at https://sites.google.com/view/marl-spc/.

1

Abstract

Recent advances in multi-agent reinforcement learning (MARL) allow agents to coordinate their behaviors in complex environments. However, common MARL algorithms still suffer from scalability and sparse reward issues. One promising approach to resolving them is automatic curriculum learning (ACL). ACL involves a student (curriculum learner) training on tasks of increasing difficulty controlled by a teacher (curriculum generator). Despite its success, ACL's applicability is limited by (1) the lack of a general student framework for dealing with the varying number of agents across tasks and the sparse reward problem, and (2) the nonstationarity of the teacher's task due to ever-changing student strategies. As a remedy for ACL, we introduce a novel automatic curriculum learning framework, Skilled Population Curriculum (SPC), which adapts curriculum learning to multiagent coordination. Specifically, we endow the student with population-invariant communication and a hierarchical skill set, allowing it to learn cooperation and behavior skills from distinct tasks with varying numbers of agents. In addition, we model the teacher as a contextual bandit conditioned by student policies, enabling a team of agents to change its size while still retaining previously acquired skills. We also analyze the inherent non-stationarity of this multi-agent automatic curriculum teaching problem and provide a corresponding regret bound. Empirical results show that our method improves the performance, scalability and sample efficiency in several MARL environments. The source code and the video can be found at https://sites.google.com/view/marl-spc/ .

1 Introduction

Multi-agent reinforcement learning (MARL) has long been a go-to tool in complex robotic and strategic domains [1, 2]. However, learning effective policies with sparse reward from scratch for large-scale multi-agent systems remains challenging. One of the challenges is the exponential growth of the joint observation-action space with an increasing number of agents. In addition, sparse reward signal requires a large number of training trajectories, posing difficulties in applying existing MARL algorithms directly to complex environments. As a result, these algorithms may produce agents that do not collaborate with each other, even when it would be of significant benefit [3, 4].

There are several lines of research related to the large-scale MARL problem with sparse reward, 30

including reward shaping [5], curriculum learning [6], and learning from demonstrations [7]. Among 31

these approaches, the curriculum learning paradigm, in which the difficulty of experienced tasks 32

and the population of training agents progressively grow, shows particular promise. In automatic 33

curriculum learning (ACL), a teacher (curriculum generator) learns to adjust the complexity and 34

sequencing of tasks faced by a student (curriculum learner). Several works have even proposed multi35

agent ACL algorithms, based on approximate or heuristic approaches to teaching, such as DyMA-CL 36

[8], EPC [9], and V ACL [6]. However, these approaches rely on a framework of an off-policy student 37 with a replay buffer that is hard to decide the size of the replay buffer since the proportion of different 38 tasks matters. Also, they make a strong assumption that the value of the learned policy does not 39 change when agents switch to a different task. For example, In the football environment, when we 40 treat the score as the reward, the same state-action pairs of the team agents in different tasks might 41 lead to different returns. 3 learned agents could get more scores in a 3v1 match, while the same 42 three agents could get fewer scores in a 4v11 match with an unlearned random teammate. When 43 decomposing at the same state-action pairs, agents get different credit assignments. Moreover, the 44 teacher in these approaches still faces a non-stationarity problem due to the ever-changing student 45 strategies. Another class of larger-scale MARL solutions is hierarchical learning, which utilizes 46 temporal abstraction to decompose a task into a hierarchy of subtasks. This includes skill discovery 47 [10], option as response [11], role-based MARL [12], and two levels of abstraction [13]. However, 48 these approaches mostly focus on one specific task with a fixed number of agents and do not consider 49 the transferability of learned skills. In this paper, we provide our insight into this question: 50

51

52

Whether an elaborate combination of principles from ACL and hierarchical learning can enable complex cooperation with sparse reward in MARL ?

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

Specifically, we present a novel automatic curriculum learning algorithm, Skilled Population Curriculum (SPC), that addresses the challenges of learning effective policies for large-scale multi-agent systems with sparse reward. The core idea behind SPC, motivated by real-world team sports where players often train their skills by gradually increasing the difficulty of tasks and the number of coordinating players, is to encourage the student to learn skills from tasks with different numbers of agents, akin to how team sports players train by gradually increasing the difficulty of tasks and the number of coordinating players. To achieve this, SPC is implemented with three key components. First, to solve the final complex cooperative tasks, we equip the contextual bandit teacher with an RNN-based [14] imitation model to represent student policies and generate the bandit's context. Second, to handle the varying number of agents across these tasks and bypass the limitation of the related studies, we utilize population-invariant communication in the student module is implemented to handle varying number of agents across tasks. By treating each agent's message as a word and using a self-attention communication channel [15], SPC supports an arbitrary number of agents to share messages. Third, to learn transferable skills in the sparse reward setting, a hierarchical skill framework is used in the student module to learn transferable skills in the sparse reward setting, where agents communicate on the high-level about a set of shared low-level policies. Empirical results show that our method achieves state-of-the-art performance in several tasks in Multi-agent Particle Environment (MPE) [16] and the challenging 5vs5 competition in Google Research Football (GRF) [17].

2 Preliminaries

Dec-POMDP. A cooperative MARL problem can be formulated as a decentralized partially observable Markov decision process (Dec-POMDP) [18], which is described as a tuple ⟨ n, S , A , P, R, O , Ω , γ ⟩ , where n represents the number of agents. S represents the space of global states. A = { A i } i =1 , ··· ,n denotes the space of actions of all agents. O = { O i } i =1 , ··· ,n denotes the space of observations of all agents. P : S × A → S denotes the state transition probability function. All agents share the same reward as a function of the states and actions of the agents R : S × A → R . Each agent i receives a private observation o i ∈ O i according to the observation function Ω ( s, i ) : S → O i . γ ∈ [0 , 1] denotes the discount factor.

Multi-armed Bandit. Multi-armed bandits (MABs) are a simple but very powerful framework that repeatedly makes decisions under uncertainty. In this framework, a learner performs a sequence of actions and immediately observes the corresponding reward after each action. The goal is to maximize the total reward over a given set of K actions and a specific time horizon T . The measure of success in MABs is often determined by the regret, which is the difference between the cumulative reward of an MAB algorithm and the best-arm benchmark. One well-known MAB algorithm is the Exp3 algorithm [19], which aims to increase the probability of selecting good arms and achieves a regret of O ( √ KT log( K )) under a time-varying reward distribution. Another related concept is the contextual bandit problem [20], where the learner makes decisions based on prior information as the context.

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

3 Skilled Population Curriculum

In this section, we first provide a formal definition of the curriculum-enhanced Dec-POMDP framework, which formulates the MARL with curriculum problem under the Dec-POMDP framework. We then present our multi-agent ACL algorithm, Skilled Population Curriculum (SPC), as shown in Fig. 1. In the following subsections, we establish the curriculum learning framework in Sec. 3.1, and then present a contextual multi-armed bandit algorithm as the teacher to address the non-stationarity in Sec. 3.2. Lastly, we introduce the student with transferable skills and population-invariant communication to tackle the varying number of agents and the sparse reward problem in Sec. 3.3.

3.1 Problem Formulation

We consider environments from multi-agent automatic curriculum learning problems are equipped with parameterized task spaces and thus can be modeled as curriculum-enhanced Dec-POMDPs.

Definition 3.1 (Curriculum-enhanced Dec-POMDP) . Acurriculum-enhanced Dec-POMDP is defined by a tuple ⟨ Φ , M⟩ , where Φ and M represent a task space and a Dec-POMDP, respectively. Given the task ϕ , the Dec-POMDP M ( ϕ ) is presented as { n ϕ , S ϕ , A ϕ , P ϕ , r ϕ , O ϕ , Ω ϕ , γ ϕ } . The superscript ϕ denotes that the Dec-POMDP elements are determined by the task ϕ . Note that task ϕ can be a few parameters of the environment or task IDs in a finite task space. In a curriculum-enhanced Dec-POMDP, the objective is to improve the student's performance on the target tasks through the sequence of training tasks given by the teacher. .

Let τ denote a trajectory whose unconditional distribution Pr π,ϕ µ ( τ ) (under a policy π and a task ϕ with initial state distribution µ ( s 0 ) ) is Pr π,ϕ µ ( τ ) = µ ( s 0 ) ∑ ∞ t =0 π ( a t | s t ) P ϕ ( s t +1 | s t , a t ) . We use p ( ϕ ) to represent the distribution of target tasks and q ( ϕ ) to represent the distribution of training tasks at each task sampling step. We consider the joint agents' policies π θ ( a | s ) and q ψ ( ϕ ) parameterized by θ and ψ , respectively. The overall objective to maximize in a curriculum-enhanced Dec-POMDP is:

where R ϕ ( τ ) = ∑ t γ t r ϕ ( s t , a t ; s 0 ) and V ( ϕ, π θ ) represents the value function of π θ in DecPOMDP M ( ϕ ) . However, when optimizing q ψ ( ϕ ) , we cannot get the partial derivative ∇ ψ J ( θ, ψ ) = ∇ ψ ∑ τ 1 q ψ ( ϕ ) R ϕ ( τ ) Pr π,ϕ µ ( τ ) 1 since the reward function and the transition probability function w.r.t number of agents are non-parametric, non-differentiable, and discontinuous in most MARL scenarios.

Thus, we use the non-differentiable method, i.e., multi-armed bandit algorithms, to optimize q ψ ( ϕ ) , and use an RL algorithm (the student) in alternating periods to optimize π θ ( a | s ) . However, there are three key challenges in solving this problem: (1) The teacher is facing a non-stationarity problem due to the ever-changing student's strategies. (2) The student will forget the old tasks and need to re-learn them. Some tasks can be the prerequisites of other tasks, while some can be inter-independent and parallel. (3) There is a lack of a general student framework to deal with the varying number of agents across tasks and the sparse reward problem.

3.2 Teacher as a Non-Stationary Contextual Bandit

As previously discussed, the teacher faces a non-stationarity problem due to the ever-changing student's strategies during the learning process. Specifically, as the student learns across different tasks in different learning stages, the teacher will observe varying student performance when providing the same task, resulting in a time-varying reward distribution for the teacher. In addition, the student may forget previously learned policies. To mitigate this problem, the teacher should balance the exploitation of tasks that have been found to benefit the student's performance on the target tasks, with the exploration of tasks that may not directly facilitate the student's learning.

Fortunately, we notice that the non-stationarity stems from the student, which can be mitigated with a contextual bandit which embeds the student policy into the context. As shown in Fig. 1 Left, the teacher utilizes the student's policy representation as the context and chooses a task from the

144

…

…

෥

𝒎𝒋

skill

𝝓𝟏

…

Tasks

Prob.

…

Contexts

Teacher

Env

𝑛 𝜙𝑖

, 𝑺

𝜙𝑖 , 𝑨

𝜙𝑖 , 𝑃

𝜙𝑖 , 𝑟

𝜙𝑖

𝝓𝟐

𝝓𝒌-𝟏 𝝓𝒌

Student

𝒐, 𝒓

𝒂

𝝅𝒉

RNN

𝝅𝒍

𝒐

𝒂

hidden

state

𝝅𝒉,𝒋

𝒐𝒋

𝒂𝒉,𝒋

𝒎𝒋

self-att

channel

𝒎𝟏

𝒎𝟐

𝒎𝒏 𝝓 𝒊

෥

𝒎𝟏

𝝅𝒍𝒐𝒘

𝒂𝒋

෥

𝒎𝟐

෥

𝒎 𝒏 𝒊

𝝓

Population-invariant

view of agent j

𝒓𝒕𝒆𝒂𝒄𝒉𝒆𝒓

distribution of training tasks. Specifically, we extend the Exp3 algorithm [19] by incorporating 137

contexts through a two-step online clustering process [21]. The context, represented by x , is the 138

student's policy representation. The teacher's action is a specific task, denoted by ϕ , and the teacher's 139

reward is the return of the student in the target tasks. The teacher's algorithm is outlined in Alg. 1. 140

During the sampling stage (steps 1-5), the teacher selects a task for the student's training. In the 141

training stage (steps 6-7), the teacher adjusts the parameters based on the evaluation reward received 142

from the student. 143

Algorithm 1 Teacher Sampling and Training

Input: Context x , the number of Clusters N c , N c instances of Exp3 with task distribution w ( ϕ k , c ) for k = 1 , . . . , K and for c = 1 , . . . , N c , learning rate α , a buffer maintaining the historical contexts

Output: M ( ϕ ) = { n ϕ , S ϕ , A ϕ , P ϕ , r ϕ , O ϕ , Ω ϕ , γ ϕ } , the teacher bandit parameters Sampling

Get the the context x , and save it to the buffer

Run the online cluster algorithm and get the index of the cluster center c ( x )

Let the active Exp3 instance be the instance with index c ( x )

Set the probability p ( ϕ k , c ( x )) = (1 -α ) w ( ϕ k ,c ( x )) ∑ K j =1 w ( ϕ k ,c ( x )) + α K for each task ϕ k

Sample a new task according to the distribution of p ϕ k ,c

Training

Get the return (discounted cumulative rewards) from student testing r

Update the active Exp3 instance by setting w ( ϕ k , c ( x )) = w ( ϕ k , c ( x )) e αr/K

3.2.1 Context Representation

Upon analysis, it is essential to learn an effective representation for the student's policy as the context. 145

One straightforward representation is to use the student parameters θ directly as the context. However, 146

the number of parameters is too large to be used as the input of neural network if we change the 147

student's architecture. Therefore, we propose an alternative method. 148

A principle for learning a good representation of a policy is predictive representation , which means 149

the representation should be accurate to predict policy actions given states. In accordance with this 150

principle, we utilize an imitation function through supervised learning. Supervised learning does 151

representation learning. Intuitively, the imitation function attempts to mimic low-level policy based 153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

on historical behaviors. In practice, we use an RNN-based imitation function f im : S × A → [0 , 1] . Since recurrent neural networks are theoretically Turing complete [22], their internal states can be used as the representation of the student's policy. We train this imitation function by using the negative cross entropy objective E [log f im ( s, a )] .

3.2.2 Regret Analysis

In this subsection, we demonstrate that the proposed teacher algorithm has a regret bound of E [ R ( T )] = O ( T 2 / 3 ( LK log T ) 1 / 3 ) , where T is the number of total rounds, L is the Lipschitz constant, and K is the number of arms (the number of the teacher's actions). The regret analysis is used to justify the usage of the bandit algorithm in the non-stationary setting. The regret bound represents the optimality of SPC, as the teacher's reward is the return of the student in the target tasks.

First, we introduce the Lipschitz assumption about the generalization ability of the task space.

Assumption 3.2 (Lipschitz continuity w.r.t the context) . Without loss of generality, the contexts are mapped into the [0 , 1] interval, so that the expected rewards for the teacher are Lipschitz with respect to the context.

for any arm ϕ ∈ Φ and any pair of contexts x, x ∈ X

where L is the Lipschitz constant, and X is the context space.

This assumption suggests that for any policy trained on a set of tasks, the rate at which performance improves is not faster than the rate at which the policy changes. This is a realistic assumption, as we cannot expect the student to achieve a significant improvement on a task with only a few training steps under a new context. We use an existing contextual bandit algorithm for a limited number of contexts [19] (see Appendix A) and Lemma 3.3 as a foundation for proving Theorem 3.4.

Lemma 3.3. Alg. 2 has a regret bound of E [ R ( T )] = O ( √ TK |X| log K ) .

Lemma 3.3 introduces a square root dependence on |X| if separate copies of Exp3 are run for each context [19]. This motivates us to address the large context space by utilizing discretization techniques.

Theorem 3.4. Consider the Lipschitz contextual bandit problem with contexts in [0 , 1] . The Alg. 1 yields regret E [ R ( T )] = O ( T 2 / 3 ( LK ln T ) 1 / 3 ) .

In practice, the high-dimensional context space cannot be discretized using a uniform mesh in [0 , 1] as in the proof of Theorem 3.4. To address this issue, we utilize the Balanced Iterative Reducing and Clustering using Hierarchies (BIRCH) online clustering algorithm [21] to discretize the context space. BIRCH is an efficient and easy-to-update algorithm that can effectively cluster large datasets. In this case, it is used to cluster the high-dimensional RNN-based policy representation. The resulting clusters can be seen as an approximation of a uniform mesh.

3.3 Student with Population-Invariant Skills

We propose a population-invariant skill framework to address the challenges of varying number of agents and sparse reward problem. This framework allows agents to communicate via a self-attention channel, enabling them to learn transferable skills across different tasks. The student module is designed to be algorithm-agnostic and is orthogonal to any state-of-the-art MARL algorithm. While there have been some efforts in the literature to address the varying number of agents [23, 24], these approaches heavily rely on prior knowledge of the environments.

Population-Invariant Teamwork Communication. In order to enable the population-invariant property and learn tactics among agents, we introduce communication. Leveraging the transformer architecture's capability to process inputs of varying lengths [15], we incorporate self-attention into our communication mechanism. As illustrated in Fig. 1 Right, each agent j receives an observation o j and encodes it into a message vector m j = f ( o j ) which is then sent through a self-attention channel, where f is an observation encoder function.

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

Agents

Landmark

Simple Spread

(a)

(b)

The channel aggregates all messages and sends the new message vector, ˜ m j , through the self-attention mechanism. Concretely, given the channel input M = [ m 1 , m 2 , · · · , m n ] ∈ R n × d m , and the trainable weight of the channel W Q , W K , W V ∈ R d m × d m , we obtain three distinct representations: Q = MW Q , K = MW K , V = MW V . Then the output messages are

where d m is the dimension of the messages. As the dimensions of the trainable weight are independent of the number of agents, our student models can leverage the population-invariant property to effectively learn tactics.

Transferable Hierarchical Skills. As depicted in the dotted box in Fig. 1 Right, after receiving the new messages ˜ m j from the channel, each agent employs a high-level action (skill) a h,j = π h,j ( o j , ˜ m j ) to execute the low-level policy a j = π low ( o j , a h,j ) . In this work, we generalize the high-level action (skill) a h,j to a continuous embedding space, so that the skill can be either a latent continuous vector as in DIAYN [25], or a categorical distribution for sampling discrete options [26].

Implementation. We implement the high- and low-level policies in the student with Proximal Policy Optimization (PPO) [27]. Following the common practice proposed in [28], the high-level policy for each agent is learned independently, whereas the low-level policies share parameters, as the fundamental action pattern should be consistent among different agents. The low-level agents are rewarded by the environment, while the high-level policy is trained to take actions at fixed intervals. Within this interval, the cumulative low-level reward is used as the high-level reward. When using a categorical distribution to enable discrete skills, we sample an 'option' from the distribution and provide the corresponding one-hot embedding to the low-level policy.

4 Related Work

Automatic Curriculum Learning in MARL. Curriculum learning is a training strategy that mimics the human learning process by organizing tasks based on their difficulty level [29]. The selection of tasks is formulated as a Curriculum Markov Decision Process (CMDP) [30]. Automatic Curriculum Learning mechanisms aim to learn a task selection function based on past interactions, such as ADR [31, 32], ALP-GMM [33], SPCL [34], GoalGAN [35], PLR [36, 37], SPDL [38], CURROT [39], and graph-curriculum [40]. Recently, several MARL curriculum learning frameworks have been proposed, such as open-ended evolution [41-43], population-based training [44, 45], meta-learning [46, 47] and training with emergent curriculum [48, 49, 29]. In summary, these frameworks share a common principle of an automatic curriculum that continually generates improved agents through selection pressure among a population of self-optimizing agents.

Hierarchical MARL and Communication. Hierarchical reinforcement learning (HRL) has been extensively studied to address the issue of sparse reward and facilitate transfer learning. Single-agent HRL focuses on learning the temporal decomposition of tasks, either by learning subgoals [5054] or by discovering reusable skills [55-58]. Recent developments in hierarchical MARL have been discussed in Sec. 1. In multi-agent settings, communication has been effective in promoting cooperation among agents [59-65]. However, current approaches that extend HRL to multi-agent systems or utilize communication are limited to a fixed number of agents and lack the ability to transfer to different agent counts.

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

5 Experiments

To demonstrate the effectiveness of our approach, we conduct experiments on several tasks in two environments: Simple-Spread and Push-Ball in the Multi-agent Particle Environment (MPE) [16], and the challenging 5vs5 task of the Google Research Football (GRF) environment [17]. We aim to investigate the following research questions:

Q1 : Is curriculum learning necessary in complex large-scale MARL problems? (Sec. 5.2)

Q2 : Can SPC outperform previous curriculum-based MARL methods? If so, which components of SPC contribute the most to performance gains? (Sec. 5.3)

Q3 : Can SPC effectively learn a curriculum for the student? (Sec. 5.4)

5.1 Environments, Baselines and Metric

Environments. In the GRF 5vs5 scenario, we control four agents, excluding the goalkeeper, to compete against the built-in AI opponents. Each agent observes a compact encoding, consisting of a 115-dimensional vector that summarizes various aspects of the game, such as player coordinates, ball possession and direction, active players, and game mode. The available action set for an individual agent includes 19 discrete actions, such as idle, move, pass, shoot, dribble, etc. The GRF provides two types of rewards: scoring and checkpoints, to encourage agents to move the ball forward and make successful shots. Additionally, we include a shooting reward in the challenging GRF 5vs5 task. We select several basic scenarios in GRF, including 3vs3, Pass-Shoot, 3vs1, and Empty-Goal as curriculum.

In MPE, we investigate Simple-Spread and Push-Ball (see Fig. 2a). In Simple-Spread, there are n agents that need to cover all n landmarks. Agents are penalized for collisions and only receive a positive reward when all the landmarks are covered. In Push-Ball, there are n agents, n balls, and n landmarks. The agents must push the balls to cover each landmark. A success reward is given after all the landmarks have been covered.

Baselines. We compare our approach to the following methods in Table 1 as baselines 2 :

Metric. To evaluate the performance of our approach in the GRF 5vs5 scenario, we use metrics beyond just the mean episode reward, as this alone may not accurately reflect the agents' performance. Specifically, we use the win rate and the average goal difference, which is calculated as the number of goals scored by the MARL agents minus the number of goals scored by the opposing team.

We evaluate the performance of MARL algorithms to justify the need for curriculum learning in complex large-scale MARL problems. To ensure a fair comparison, we modify VACL by removing the centralized critic for MPE tasks. Centralized Training Decentralized Execution methods is not included as baselines since they are not suitable for varying numbers (e.g., MADDPG/MAPPO's critic requires a fixed size of input or QMIX's mixing network also fixed size of the input).

In all experiments, we use individual Proximal Policy Optimization (IPPO) as the backend MARL algorithm. To ensure the robustness of our results, we conduct experiments on a 30-node cluster, with one node containing a 128-core CPU and four A100 GPUs. Each trial of the experiment is repeated over five seeds and runs for 1-2 days.

5.2 The Necessity of Curriculum Learning

Our experiments first show that in simple environments, such as MPE, students can directly learn to complete the task without the need for curriculum. For MPE experiments, we randomly select a starting state and the episode ends after a fixed number of maximum steps. Specifically, the task

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

0.0

0.5

1.0

1.5

Training Timesteps

1e6

0.10

0.15

0.20

0.25

0.30

0.35

0.40

(a) Cover Rate on Simple Spread

IPPO

VACL

SPC

2

4

Training Timesteps

1e5

0.20

0.25

0.30

(b) Cover Rate on Push Ball

IPPO

VACL

SPC

0.0

0.5

1.0

1.5

Training Timesteps

1e6

2

4

6

8

10

12

14

16

Mean Number of Agents

(a) Simple Spread

VACL

SPC

0

2

4

6

Training Timesteps

1e5

2

4

6

8

10

12

14

16

VACL

SPC

0 M

20 M

40 M

60 M

80 M

100 M

Training Timesteps

0.0

0.2

0.4

0.6

0.8

SPC

SPC w. uniform

SPC w/o. HRL

SPC w/o. COM

IPPO w. uniform

IPPO w/o. teacher

QMix w/o. teacher

(a) Win Rate

0 M

20 M

40 M

60 M

80 M

100 M

Training Timesteps

-3

-2

-1

0

1

2

3

4

SPC

SPC w. uniform

SPC w/o. HRL

SPC w/o. COM

IPPO w. uniform

IPPO w/o. teacher

space consists of n agents, where n ∈ { 2 , 4 , 8 , 16 } , and the maximum allowed steps is set to 25. All evaluations are performed on the target task, with n = 16 . IPPO is trained and evaluated directly on the target task, and results in Fig. 3 demonstrate that it performs similarly to the V ACL algorithm. We plot the performance within a sliding window so that the starting point is not exactly from 0 timestep. VACL uses entity progression, which is a rule-based curriculum update mechanism so it lacks the flexibility to switch the curriculum when relatively easy tasks can be learned quickly. The reason for the performance jump is that SPC can switch to the largest population rapidly, which we consider one advantage of SPC. Additionally, we observe that the SPC approach only achieves a slightly higher coverage rate than the baseline methods. Furthermore, we investigate the probability variation of different population sizes, shown in Fig. 4. We observe that the curriculum provided by SPC is approaching the target task. These results suggest that in simple environments where the student can learn to directly complete the task, curriculum learning may not be necessary.

When it comes to more complex scenarios, such as the 5vs5 task in GRF, our results demonstrate that curriculum learning is a promising solution. As shown in Fig. 5a, without curriculum learning, QMix and IPPO cannot perform well in the 5vs5 scenario, and IPPO is slightly better than QMix. In Fig. 5b, we omit the curve of QMix as its mean score is low and affects the presentation of the figure. The reason could be that QMix is an off-policy MARL algorithm, which would rely heavily on the replay buffer. However, in such sparse reward scenarios, the replay buffer has much less effective samples for QMix to learn. For example, the replay buffer would contain tons of zero-score samples, leading to a non-promising performance. Meanwhile, IPPO, with its on-policy nature, is able to achieve better sample efficiency and outperform off-policy algorithms like QMix in such scenarios. Though MARL methods can achieve good performance in basic scenarios in GRF, they fail to solve complex scenarios such as the 5vs5 task. Therefore, curriculum learning is a promising solution to the complex large-scale MARL problem.

5.3 Performance and Ablation Study

Our study demonstrates that SPC outperforms VACL in MPE tasks. Instead of training with a continuous relaxation of the population size variable as in V ACL, our bandit teacher achieves a higher success rate at test time, since the population size is a discrete variable in nature. Furthermore, the curriculum provided by SPC is effective in exploring the task space and converge to the target task when the task is relatively simple and curriculum is not necessary, as shown in Fig. 4.

In GRF experiments, we do not include VACL in our baselines in the GRF, as its implementation relies heavily on prior knowledge of specific scenarios, such as the thresholds to divide the learning

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

20 M

40 M

60 M

80 M

100 M 120 M 140 M

Training Timesteps

0.0

0.2

0.4

0.6

0.8

1.0

Task Prob.

5_vs_5

3_vs_3

academy_pass_and_shoot_with_keeper

academy_3_vs_1

academy_empty_goal_close

(a) The task distribution of SPC during training.

-0.5

0.0

0.5

Dimension 1

-0.5

0.0

0.5

1.0

Dimension 2

0

1

2

3

(b) The visualization of contexts

process. Fig. 6 indicates that SPC has higher win rate and goal difference than IPPO with uniform task sampling in the 5vs5 competition. These experiments demonstrate that when the teacher is rewarded by the student's performance, a bandit-based teacher can exploit the student's learning stage and provide suitable training tasks.

In our ablation study, we examine the impact of two key components of our SPC algorithm: the contextual multi-armed bandit teacher and the hierarchical structure of the student framework. By replacing the former with uniform task sampling and removing the latter, As shown in Fig. 5a and Fig. 5b, SPC can achieve a higher win rate and a greater score difference than SPC with uniform and SPC without HRL. Furthermore, SPC with uniform task sampling outperforms IPPO with uniform task sampling. This highlights the importance of HRL in the 5vs5 football competition, and suggests that both the contextual multi-armed bandit and the hierarchical structure contribute equally to the performance of SPC. When removing HRL and bandit, the performance degradation w.r.t. SPC are similar. However, it should be noted that SPC with uniform task sampling has a larger variance in performance than SPC without HRL, indicating that uniform sampling may introduce more undesired tasks for student training. Overall, these results further justify the necessity of SPC in complex large-scale MARL problems 3 .

5.4 Visualization of Learned Curriculum

We visualize the distribution of task sampling of SPC during training based on a selected trial as shown in Fig. 6a. At the beginning of training, the task probability appears to be near-uniform, as the teacher explores the task space and keeps track of the student's learning status, acting as an anti-forgetting mechanism. As training progresses, the probabilities change over time. For example, the proportions of 3vs1 and Empty-Goal tasks gradually drop as the student becomes proficient in these scenarios. We also visualize the distribution of contexts in Fig. 6b using t-SNE [70], where the contexts are collected and stored in a buffer. We divide the contexts into four classes according to the index, and different parts represent different contexts of the final student policy representation.

6 Discussion

Conclusion. We present Skilled Population Curriculum (SPC), a novel multi-agent ACL algorithm that addresses scalability and sparse reward issues in multi-agent systems. SPC learns complex behaviors from scratch by incorporating a population-invariant multi-agent communication framework and using a hierarchical scheme for agents to learn skills. Moreover, SPC mitigates non-stationarity by modeling the teacher as a contextual bandit, where the context is represented by the student's policy representation. Though our design choices focus on solving the GRF 5vs5 task, we believe that analyzing and addressing these issues is crucial for further development in multi-agent ACL algorithms. While SPC may be complex to implement due to its various components, we provide clean and well-organized code for ease of use.

Limitations. We acknowledge that there are limitations of our algorithm. SPC is over-designed for simple tasks since our objective is to solve difficult tasks. Also, it would be interesting to understand the impact of varying number of agents on the dynamics of the environment.

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

References

RoboCup. Robocup Federation Official Website. https://www.robocup.org/ , 2019. Accessed April 10, 2019.

OpenAI. OpenAI Five. https://openai.com/blog/openai-five/ , 2019. Accessed March 4, 2019.

Kaiqing Zhang, Zhuoran Yang, and Tamer Ba¸ sar. Multi-agent reinforcement learning: A selective overview of theories and algorithms. Handbook of Reinforcement Learning and Control , pages 321-384, 2021.

Yaodong Yang and Jun Wang. An overview of multi-agent reinforcement learning from game theoretical perspective. arXiv preprint arXiv:2011.00583 , 2020.

Yujing Hu, Weixun Wang, Hangtian Jia, Yixiang Wang, Yingfeng Chen, Jianye Hao, Feng Wu, and Changjie Fan. Learning to utilize shaping rewards: A new approach of reward shaping. arXiv preprint arXiv:2011.02669 , 2020.

Jiayu Chen, Yuanxin Zhang, Yuanfan Xu, Huimin Ma, Huazhong Yang, Jiaming Song, Yu Wang, and Yi Wu. Variational automatic curriculum learning for sparse-reward cooperative multi-agent problems. Advances in Neural Information Processing Systems , 34, 2021.

Shiyu Huang, Wenze Chen, Longfei Zhang, Ziyang Li, Fengming Zhu, Deheng Ye, Ting Chen, and Jun Zhu. Tikick: Toward playing multi-agent football full games from single-agent demonstrations. arXiv preprint arXiv:2110.04507 , 2021.

Weixun Wang, Tianpei Yang, Yong Liu, Jianye Hao, Xiaotian Hao, Yujing Hu, Yingfeng Chen, Changjie Fan, and Yang Gao. From few to more: Large-scale dynamic multiagent curriculum learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 7293-7300, 2020.

Qian Long, Zihan Zhou, Abhibav Gupta, Fei Fang, Yi Wu, and Xiaolong Wang. Evolutionary population curriculum for scaling multi-agent reinforcement learning. arXiv preprint arXiv:2003.10423 , 2020.

Jiachen Yang, Igor Borovikov, and Hongyuan Zha. Hierarchical cooperative multi-agent reinforcement learning with skill discovery. arXiv preprint arXiv:1912.03558 , 2019.

Alexander Sasha Vezhnevets, Yuhuai Wu, Remi Leblond, and Joel Z Leibo. Options as responses: Grounding behavioural hierarchies in multi-agent rl. arXiv preprint arXiv:1906.01470 , 2019.

Tonghan Wang, Tarun Gupta, Anuj Mahajan, Bei Peng, Shimon Whiteson, and Chongjie Zhang. Rode: Learning roles to decompose multi-agent tasks. arXiv preprint arXiv:2010.01523 , 2020.

Zhen-Jia Pang, Ruo-Ze Liu, Zhou-Yu Meng, Yi Zhang, Yang Yu, and Tong Lu. On reinforcement learning for full-length game of starcraft. In Proceedings of the AAAI Conference on Artificial Intelligence , 2019.

Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation , 9(8): 1735-1780, 1997.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems , 30, 2017.

Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent 397 actor-critic for mixed cooperative-competitive environments. Advances in Neural Information 398 Processing Systems , 2017. 399

Karol Kurach, Anton Raichuk, Piotr Sta´ nczyk, Michal Zajkac, Olivier Bachem, Lasse Espeholt, 400 Carlos Riquelme, Damien Vincent, Marcin Michalski, Olivier Bousquet, et al. Google research 401 football: A novel reinforcement learning environment. arXiv preprint arXiv:1907.11180 , 2019. 402

Daniel S Bernstein, Robert Givan, Neil Immerman, and Shlomo Zilberstein. The complexity of 403 decentralized control of Markov Decision Processes. Mathematics of Operations Research , 27 404 (4):819-840, 2002. 405

Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, and Robert E Schapire. The nonstochastic 406 multiarmed bandit problem. SIAM Journal on Computing , 32(1):48-77, 2002. 407

408

409

Elad Hazan and Nimrod Megiddo. Online learning with prior knowledge. In International Conference on Computational Learning Theory , pages 499-513. Springer, 2007.

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

Tian Zhang, Raghu Ramakrishnan, and Miron Livny. Birch: An efficient data clustering method for very large databases. ACM Aigmod Record , 25(2):103-114, 1996.

Heikki Hyötyniemi. Turing machines are recurrent neural networks. In STeP '96/Publications of the Finnish Artificial Intelligence Society , 1996.

Shariq Iqbal, Christian A Schroeder De Witt, Bei Peng, Wendelin Böhmer, Shimon Whiteson, and Fei Sha. Randomized entity-wise factorization for multi-agent reinforcement learning. In International Conference on Machine Learning , pages 4596-4606. PMLR, 2021.

Siyi Hu, Fengda Zhu, Xiaojun Chang, and Xiaodan Liang. Updet: Universal multi-agent reinforcement learning via policy decoupling with transformers. arXiv preprint arXiv:2101.08001 , 2021.

Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv:1802.06070 , 2018.

Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 31, 2017.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Wei Fu, Chao Yu, Zelai Xu, Jiaqi Yang, and Yi Wu. Revisiting some common practices in cooperative multi-agent reinforcement learning. arXiv preprint arXiv:2206.07505 , 2022.

Rémy Portelas, Cédric Colas, Lilian Weng, Katja Hofmann, and Pierre-Yves Oudeyer. Automatic curriculum learning for deep RL: A short survey. arXiv preprint arXiv:2003.04664 , 2020.

Sanmit Narvekar and Peter Stone. Learning curriculum policies for reinforcement learning. arXiv preprint arXiv:1812.00285 , 2018.

Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al. Solving rubik's cube with a robot hand. arXiv preprint arXiv:1910.07113 , 2019.

Bhairav Mehta, Manfred Diaz, Florian Golemo, Christopher J Pal, and Liam Paull. Active domain randomization. In Conference on Robot Learning , pages 1162-1176. PMLR, 2020.

Rémy Portelas, Cédric Colas, Katja Hofmann, and Pierre-Yves Oudeyer. Teacher algorithms for curriculum learning of deep RL in continuously parameterized environments. In Conference on Robot Learning , pages 835-853, 2020.

Lu Jiang, Deyu Meng, Qian Zhao, Shiguang Shan, and Alexander G Hauptmann. Self-paced curriculum learning. In Twenty-Ninth AAAI Conference on Artificial Intelligence , 2015.

Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. In International Conference on Machine Learning , pages 1515-1528. PMLR, 2018.

Minqi Jiang, Edward Grefenstette, and Tim Rocktäschel. Prioritized level replay. In International Conference on Machine Learning , pages 4940-4950. PMLR, 2021.

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

Minqi Jiang, Michael Dennis, Jack Parker-Holder, Jakob Foerster, Edward Grefenstette, and Tim Rocktäschel. Replay-guided adversarial environment design. Advances in Neural Information Processing Systems , 34:1884-1897, 2021.

Pascal Klink, Carlo D'Eramo, Jan R Peters, and Joni Pajarinen. Self-paced deep reinforcement learning. Advances in Neural Information Processing Systems , 33:9216-9227, 2020.

Pascal Klink, Haoyi Yang, Carlo D'Eramo, Jan Peters, and Joni Pajarinen. Curriculum reinforcement learning via constrained optimal transport. In International Conference on Machine Learning , pages 11341-11358. PMLR, 2022.

Maxwell Svetlik, Matteo Leonetti, Jivko Sinapov, Rishi Shah, Nick Walker, and Peter Stone. Automatic curriculum graph generation for reinforcement learning agents. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 31, 2017.

Wolfgang Banzhaf, Bert Baumgaertner, Guillaume Beslon, René Doursat, James A Foster, Barry McMullin, Vinicius Veloso De Melo, Thomas Miconi, Lee Spector, Susan Stepney, et al. Defining and simulating open-ended novelty: Requirements, guidelines, and challenges. Theory in Biosciences , 135(3):131-161, 2016.

Joel Lehman, Kenneth O Stanley, et al. Exploiting open-endedness to solve problems through the search for novelty. In ALIFE , pages 329-336. Citeseer, 2008.

Russell K Standish. Open-ended artificial evolution. International Journal of Computational Intelligence and Applications , 3(02):167-175, 2003.

Max Jaderberg, Wojciech M Czarnecki, Iain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castaneda, Charles Beattie, Neil C Rabinowitz, Ari S Morcos, Avraham Ruderman, et al. Human-level performance in 3d multiplayer games with population-based reinforcement learning. Science , 364(6443):859-865, 2019.

Siqi Liu, Guy Lever, Josh Merel, Saran Tunyasuvunakool, Nicolas Heess, and Thore Graepel. Emergent coordination through competition. arXiv preprint arXiv:1902.07151 , 2019.

Abhinav Gupta, Marc Lanctot, and Angeliki Lazaridou. Dynamic population-based metalearning for multi-agent communication with natural language. Advances in Neural Information Processing Systems , 34:16899-16912, 2021.

Rémy Portelas, Clément Romac, Katja Hofmann, and Pierre-Yves Oudeyer. Meta automatic curriculum learning. arXiv preprint arXiv:2011.08463 , 2020.

Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent tool use from multi-agent autocurricula. arXiv preprint arXiv:1909.07528 , 2019.

Joel Z Leibo, Edward Hughes, Marc Lanctot, and Thore Graepel. Autocurricula and the emergence of innovation from social interaction: A manifesto for multi-agent intelligence research. arXiv preprint arXiv:1903.00742 , 2019.

Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. Advances in neural information processing systems , 31, 2018.

Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Near-optimal representation learning for hierarchical reinforcement learning. arXiv preprint arXiv:1810.01257 , 2018.

Sainbayar Sukhbaatar, Emily Denton, Arthur Szlam, and Rob Fergus. Learning goal embeddings via self-play for hierarchical reinforcement learning. arXiv preprint arXiv:1811.09083 , 2018.

Suraj Nair and Chelsea Finn. Hierarchical foresight: Self-supervised learning of long-horizon tasks via visual subgoal generation. arXiv preprint arXiv:1909.05829 , 2019.

Rundong Wang, Runsheng Yu, Bo An, and Zinovi Rabinovich. I2hrl: Interactive influencebased hierarchical reinforcement learning. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence , pages 3131-3138, 2021.

Christian Daniel, Gerhard Neumann, and Jan Peters. Hierarchical relative entropy policy search. 496 In Artificial Intelligence and Statistics , pages 273-281, 2012. 497

Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv 498 preprint arXiv:1611.07507 , 2016. 499

Tanmay Shankar and Abhinav Gupta. Learning robot skills with temporal variational inference. 500 In Proceedings of the 37th International Conference on Machine Learning . JMLR. org, 2020. 501

Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics502 aware unsupervised discovery of skills. In International Conference on Learning Representa503 tions , 2020. 504

505

506

507

Jakob Foerster, Ioannis Alexandros Assael, Nando De Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. Advances in neural information processing systems , 29, 2016.

508

509

510

511

512

513

514

515

516

517

518

519

520

521

522

523

524

525

526

527

528

529

530

531

532

533

534

535

536

Abhishek Das, Théophile Gervet, Joshua Romoff, Dhruv Batra, Devi Parikh, Mike Rabbat, and Joelle Pineau. Tarmac: Targeted multi-agent communication. In International Conference on Machine Learning , pages 1538-1546. PMLR, 2019.

Sainbayar Sukhbaatar, Rob Fergus, et al. Learning multiagent communication with backpropagation. Advances in neural information processing systems , 29, 2016.

Amanpreet Singh, Tushar Jain, and Sainbayar Sukhbaatar. Learning when to communicate at scale in multiagent cooperative and competitive tasks. arXiv preprint arXiv:1812.09755 , 2018.

Jiechuan Jiang and Zongqing Lu. Learning attentional communication for multi-agent cooperation. Advances in neural information processing systems , 31, 2018.

Daewoo Kim, Sangwoo Moon, David Hostallero, Wan Ju Kang, Taeyoung Lee, Kyunghwan Son, and Yung Yi. Learning to schedule communication in multi-agent reinforcement learning. arXiv preprint arXiv:1902.01554 , 2019.

Rundong Wang, Xu He, Runsheng Yu, Wei Qiu, Bo An, and Zinovi Rabinovich. Learning efficient multi-agent communication: An information bottleneck approach. In International Conference on Machine Learning , pages 9908-9918. PMLR, 2020.

Chenghao Li, Chengjie Wu, Tonghan Wang, Jun Yang, Qianchuan Zhao, and Chongjie Zhang. Celebrating diversity in shared multi-agent reinforcement learning. arXiv preprint arXiv:2106.02195 , 2021.

Siyang Wu, Tonghan Wang, Chenghao Li, and Chongjie Zhang. Containerized distributed value-based multi-agent reinforcement learning. arXiv preprint arXiv:2110.08169 , 2021.

Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference on Machine Learning , pages 4295-4304, 2018.

Christian Schroeder de Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip HS Torr, Mingfei Sun, and Shimon Whiteson. Is independent learning all you need in the StarCraft multi-agent challenge? arXiv preprint arXiv:2011.09533 , 2020.

Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research , 9(11), 2008.

A Contextual Bandit for Limited Number of Contexts 537

Algorithm 2 A contextual bandit algorithm for a small number of contexts

1: Initialization: For each context x , create an instance Exp3 x of algorithm Exp3

2: for round do

3: Invoke algorithm Exp3 x with x = x t

4: Play the action chosen by Exp3 x

5: Return reward r t to Exp3 x

6: end for

B Proof of Theorem 3.4

Theorem 3.4. Consider the Lipschitz contextual bandit problem with contexts in [0 , 1] . The Alg. 1 yields regret E [ R ( T )] = O ( T 2 / 3 ( LK ln T ) 1 / 3 ) .

Proof. Let S m be the ϵ -uniform mesh on [0 , 1] , that is, the set of all points in [0 , 1] that are integer multiples of ϵ . We take ϵ = 1 / ( d -1) where the integer d is the number of points in S m , which will be adjusted later in the analysis.

We apply Alg. 2 to the context space S m . Let f S m ( x ) be a mapping from context x to the closest point in S m :

In each round t , we replace the context x t with f S m ( x t ) and call Exp3 S . The regret bound will have two components: the regret bound for Exp3 S and (a suitable notion of) the discretization error. Formally, let us define the 'discretized best response" π ∗ S m : X → Φ : π ∗ S m ( x ) = π ∗ ( f S m ( x )) for each context x ∈ X .

We define the total reward of an algorithm Alg is Reward( Alg ) = ∑ T t =1 r t . Then the regret of Exp3 S and the discretization error are defined as:

It follows that regret is the sum R ( T ) = R S ( T ) + DE( S ) . We have E [ R S ( T )] = O ( √ TK log K ) from Lemma 3.3, so it remains to upper bound the discretization error and adjust the discretization step ϵ .

For each round t and the respective context x = x t , r ( π ∗ S ( x ) | f S ( x )) ≥ r ( π ∗ ( x ) | f S ( x )) ≥ r ( π ∗ ( x ) | x ) -ϵL . The first inequality is determined by the optimality of π ∗ S and the second is determined by Lipschitzness. Summing this up over all rounds t , we obtain E [Reward( π ∗ S )] ≥ Reward[ π ∗ ] -ϵLT .

Thus, the regret is that

C SPC on GRF 11vs11 Full Game

538

539

540

541

542

543

544

545

546

547

548

549

550

551

552

553

554

555

556

557

558

559

560

561

562

563

We also conduct experiments on the GRF 11vs11 full game scenario with sparse reward. As shown in 564 Fig. 7, SPC achieves about 50% win rate against built-in AI in the target task after training with 200 565

575

576

577

578

579

580

581

582

0 M

25 M

50 M

75 M

100 M

125 M

150 M

175 M

200 M

Training Timesteps

0.0

0.1

0.2

0.3

0.4

0.5

0.6

SPC

million timesteps. This is non-trivial as this is one of the most challenging benchmarks for MARL 566 community, and most current MARL methods struggle to achieve progress without hand-crafted 567 engineering. 568

569

D Qualitatively Analysis On Low-Level Skills

We demonstrate game statistics under different high-level actions. For example, the times of shooting, 570 passing and running actions per game in GRF. These different low-level policies are induced by 571 the high-level actions. We evaluate these statistics by fixing one agent's high-level actions and 572 maintaining other agents with SPC. The results in Table 2 are averaged over five runs in the 5vs5 573 scenario. 574

E Comparing Different Teacher Algorithms on GRF Corner-5

To further illustrate the effectiveness of the SPC teacher module, we conduct experiments on the corner-5 scenario on GRF, where the target task is to control five of the eleven players to obtain a goal in the GRF Corner scenario. The experiments are designed to determine whether or not the contextual bandit in SPC outperforms alternative curriculum learning methods to schedule the number of agents in training. We compare SPC teacher against non-curriculum training (None), uniform task sampling (Uniform), a state-of-the-art curriculum learning method (ALP-GMM), and a

0.0 M

0.2 M

0.4 M

0.6 M

0.8 M

1 M

Training Timesteps

0.00

0.05

0.10

0.15

0.20

0.25

0.30

ALP-GMM

None

Uniform

Contextual Bandit

Bandit

VACL

(a) Win Rate

0.0 M

0.2 M

0.4 M

0.6 M

0.8 M

1 M

Training Timesteps

-0.15

-0.10

-0.05

0.00

0.05

0.10

0.15

0.20

0.25

ALP-GMM

None

Uniform

Contextual Bandit

Bandit

VACL

583

584

585

586

587

588

589

590

591

592

593

594

595

596

597

598

599

600

601

602

603

604

605

606

607

608

609

610

611

612

613

614

multi-agent curriculum learning method (VACL). The training task space consists of n agents, where n ∈ { 1 , 3 , 5 } . All teachers have the same base architecture without transformer architecture and HRL. We also investigate the ablation of the RNN-based contexts (see Contextual Bandit and Bandit). Fig. 8 shows the benefit of SPC contextual bandit over other ACL methods after training with one million timesteps.

F Implementation Details

We use the default implementation of Proximal Policy Optimization (PPO) in Ray RLlib, which scales out using multiple workers for experience collection. This allows us to use a large amount of rollouts from parallel workers during training to ameliorate high variance and aid exploration. We do multiple rollouts in parallel with distributed workers and use parameter sharing for each agent. The trainer broadcasts new weights to the workers after their synchronous sampling.

F.1 Google Research Football

We set five tasks for training the GRF 5vs5 scenario, including 5vs5, 3vs3, Pass-Shoot, 3vs1, and Empty-Goal. In the Empty-Goal, one agent need to move forward and shoot with an empty goal. In Pass-Shoot and 3vs3, two agents are controlled to play against a goalkeeper and three players, with different position initialization. In 3vs1, three agents are controlled to play against a center-back and a goalkeeper. In 5vs5, four agents are controlled to play against five players. Without loss of generality, we initialize all player with fixed positions and roles as center midfielders.

We use both MLP and self-attention mechanism for the high-level policy, and use MLP for the low-level policy. For high-level policy, the input is first projected to an embedding using two hidden layers with 256 units each and ReLU activation, which is then fed into multi-head self-attention (8 heads, 64 units each). The output is then projected to the actions and values using another fully connected layer with 256 units. For low-level policy, we use MLP with two hidden layers with 256 units each, i.e., the default configuration of policy network in RLlib.

(a) SPC hyper-parameters used in GRF.

(b) SPC hyper-parameters used in MPE.

F.2 MPE

In MPE tasks, agents must cooperate through physical actions to reach a set of landmarks. Agents observe the relative positions of other agents and landmarks, and are collectively rewarded based on the proximity of any agent to each landmark. In other words, the agents have to cover all of the landmarks. Further, the agents are penalized when colliding with each other. The agents need to infer the landmark to cover and move there while avoid colliding with other agents.

The hyper-parameters of SPC in MPE are shown in Table 3b. In MPE, hyper-parameters such as rollout fragment length, training batch size and SGD minibatch size are adjusted according to horizon

of the scenarios so that policy are updated after episodes are done. We use the same neural network 615

architecture as in GRF, but with 128 units for all MLP hidden layers. Other omitted hyper-parameters 616

follow the default configuration in RLlib PPO implementation. 617

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a new approach to automatic curriculum learning designed specifically for multi-agent coordination problems.\n\nSTRENGTHS:\n- The main strength of the paper, in my opinion, is the well-formulated approach to the curriculum learning problem. To the best of my knowledge of the related literature, the non-stationary contextual bandit as the teacher and the population-invariant skills for the students are both original and useful contributions to the literature.\n\nWEAKNESSES:\n- My general problem with this paper is that I am finding it hard to evaluate the significance of the work in the automatic curriculum learning sphere without an adequate baseline provided for GRF. Whilst the authors do argue that VACL is not used on GRF due to requiring prior knowledge, it seems unreasonable to therefore provide no baselines that are actually designed for these larger settings. For example, if VACL was unusable, then I would have maybe liked to have seen a comparison to population-based approaches in MARL or any of the other automatic curriculum learning approaches mentioned in Sec. 4. Overall, it is hard to properly evaluate the gains from this automatic curriculum learning framework without seeing the performance of baselines in an environment that actually requires automatic curriculum learning (MPE does not need it according to line 297-298).\n\nI am happy to update my score if the authors can make a reasonable argument against the lack of other baselines in the work.\n\nQUESTIONS:\n- Line 37-39, 'However,..., tasks matters' - I am very confused by this sentence, could the authors please clarify?\n\n- Upon inspection of the results, the final improved performance does not seem to be massively impacted by the hierarchical RL element of the framework. I was wondering if the authors could discuss a little more on this, in terms of its necessity in the framework and when they believe it would provide greater gains in performance?\n\nLIMITATIONS:\nThe authors briefly make mention to the limitations of the work. I agree with the over-design of the framework for simple tasks, so would definitely like to see its performance in more difficult environments that it is designed for.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents Skilled Population Curriculum (SPC) which is a method for learning a curriculum to help a team of agents complete a complex task. SPC models the problem of choosing tasks for agents as a contextual bandit problem, and builds on top of the Exp3 algorithm to solve this bandit problem. SPC also uses an attention-based communication approach, and a hierarchical policy framework. Experiments are performed in the Multi-agent Particle Environment (MPE) and Google Research Football environment (GRF). While MPE does not seem to benefit much from SPC, in the more complex GRF domain the authors show using their SPC approach can accelerate training relative to MARL baselines.\n\nSTRENGTHS:\n- The paper is clear: it is well-structured, and provides a good balance of intuition and detail. It is clearly motivated, and addresses an interesting problem.\n\n- The presented results show clear benefits to the authors' approach.\n\n- The authors use suitable baselines approaches, and suitable environments.\n\nWEAKNESSES:\n- SPC seems to add significant computational complexity vs. baselines like IPPO. While the authors can justify focusing on sample complexity, for completeness they should also record information about the wall-clock time / computational resources needed to train their different baselines.\n\n- In their research question stated on lines 52–53, the authors highlight their desire to consider complex sparse-reward settings. However, the MPE domains are a sparse setting, though fairly simple, and the results here show little benefit to using SPC. On the other hand, the GRF experiments appear to have a somewhat dense reward (the GRF checkpoint reward, which while not necessarily active every timestep, it could be argued is 'somewhat dense'). It seems like absent the checkpoint reward, SPC would struggle because there would be little information in the returns for the teacher agent to use — and I expect this would be the case in most complex (very) sparse reward environments\n\n- Line 277 the authors state MADDPG/MAPPO would not be suitable in these experiments. This might be true in general, but for GRF specifically the critic input size actually would be the same across all tasks (as it pads observations if agents are absent). But since GRF is fully observable, MAPPO is equivalent to IPPO so this is not an issue for this work — though the authors may wish to revise their statement.\n\n- Though the GRF environment is complex and difficult to solve, its level of complexity is somewhat deceptive, as evidenced by the video on the project website. The rollouts show that the agents have learned a simple \"force an offside and run in a straight at the goal line\" strategy which exploits deficiencies in the GRF bots. This behaviour has been observed before by Song et. al (http://arxiv.org/abs/2305.09458). However, this is not a fault of the authors, and is more broadly an issue in the MARL research community. Because of this, it's not clear what skills the agents learn in the training tasks that are useful in the target task. It would be interesting to see a plot of training task performance throughout training.\n\n- It's unclear why the IPPO baselines have a sharp step change in performance around 80 and 90 million timesteps. The authors should investigate this, and perhaps make a comment (at least in the appendix) about why it occurs. In my experience, things like this sometimes occur when training runs stop unexpectedly before the full 100M timesteps, and so the remaining timesteps are aggregating over fewer seeds with lower performance. I would encourage the authors to produce plots reporting the interquartile mean of their results, and produce a plot showing the disaggregated training curves for each seed. These can go in the appendix.\n\n- The authors state (line 301): \" InFig. 5b, we omit the curve of QMix as its mean score is low and affects the presentation of the figure\". I don't expect QMix to perform worse than the presumably near-uniform policies at the start of training for the other agents. So it's not clear how including QMix would disrupt the graph. Is it the case that QMix has a worse average goal difference than -2?\n\n- Can the authors clarify: the target distribution for GRF is \"100% 5vs5\"? What is the target distribution for the MPE tasks? (I see now that these are mentioned later in the text: they should be mentioned when introducing the environments)\n\n- It doesn't seem like there's a pattern to the task distribution (Fig. 6a) beyond \"academy_pass_and_shoot_with_keeper becomes less common\". It would be good to see the same plot for other trials. This possibly explainable by academy_pass_and_shoot_with_keeper requiring coordinated passing and shooting, whereas the 5vs5 rollouts (see video on project website) show a very simple GRF-bot exploiting strategy which does not closely resemble the behaviour required in academy_pass_and_shoot_with_keeper.\n\n\t- Where the authors claim \"For example, the proportions of 3vs1 and Empty-Goal tasks gradually drop as the student becomes proficient in these scenarios\", it is difficult to support this by looking at Fig. 6a.\n\n- In my opinion this approach is over-engineered, but the authors do acknowledge this. Stripping some components (e.g the hierarchical RL) and focusing on deeply investigating the remaining components would improve this work\n\n- Minor writing fixes:\n\n\t- line 42/43 \"more scores\" → \"more goals\"\n\n\t- line 43: \"4v11\" → \"4v1\" (I assume)\n\n\t- line 305: \"tons of\" → \"many\" (more formal tone)\n\nQUESTIONS:\n- Why did you decide to add the shooting reward for 5vs5? Does it make a big difference?\n\n- What causes the step-change drop in IPPO performance?\n\n- How does including QMix in 5b disrupt the graph?\n\nLIMITATIONS:\n- The limitations section is quite limited, and limitations and assumptions could be more clearly stated throughout.\n\n- The authors recognise that their approach is complex and computationally intensive, so might not be applicable in simple environments. Testing in a complex environment like Google Research Football is a good choice, although due to issues with Google Research Football (such as the exploitability of the built-in AI) it is perhaps not as complex as the authors may hope, even though it has presented a challenge to past MARL research. However, this is a broader issue within the MARL community and the authors of this paper cannot fairly be singled out for this.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a new automatic curriculum learning framework, Skilled Population Curriculum (SPC), for multi-agent reinforcement learning. The algorithm includes three major components: (1) a contextual bandit conditioned by student-policies representation for automatic curriculum learning; (2) An attention-based communication architecture for policies to learn cooperation and behavior skills from distinct tasks with varying numbers of agents; (3) A hierarchical policy architecture to help agents to learn transferable skills between different tasks. The experiments are conducted in Google Research Football environment and Multi-agent Particle environments, which demonstrate the efficiency of the proposed method to IPPO and VACL.\n\nSTRENGTHS:\n1. The proposed method is simple yet efficient in the complex Google Research Football environment. \n2. The motivation of the components are also clear and make sence. \n3. In exepriment, several ablation studies demonstrate the effectiveness of the proposed components; \n4. Also, the paper is overall easy to follow to me. The key idea is easy to understand.\n\nWEAKNESSES:\nThis paper could benefit from further improvements in the following aspects:\n\n1. It seems that the manuscript introduces various components. While each one appears to be intuitive and rational in isolation, I recommend that the authors should provide a unifying theme or framework to better connect these components. Presently, it appears as if these components are addressing three discrete issues: a) efficient curriculum learning, b) policy architecture development, and c) communication in varying agent scenarios. It is noteworthy that a paper does not necessarily need to devote substantial attention to the innovative aspects of each introduced components. In the case of this paper, the hierarchical structure, for instance, appears to be a standard approach with limited novelty. The authors can highlight how they design efficient automatic curriculum learning in the context of variable agent scenarios.\n\n2. In the section discussing related work (Line 221), the authors mention various curriculum learning mechanisms without a detailed discussion. Could the authors provide an expanded explanation on how these works conduct curriculum learning and how they relate to or differ from the proposed methodology?\n\n3. There is room for improvement in the experiments section. Specific recommendations are detailed in the questions section.\n\n4. The paper could be further polished, for instance:\n    - There are several instances where a capital letter follows a comma, such as in line 40: \"For example, In the football environment, when we…\"\n    - The legend of Figure 6(b) lacks clarity. It would be beneficial if the authors could provide a detailed explanation of what the labels 0,1,2,3 represent.\n\nQUESTIONS:\n1. Could the authors clarify why maximizing rewards should be the objective of a teacher in curriculum learning? Intuitively, it seems a teacher policy that aims to maximize performance given student policies would tend to recommend simpler tasks to learn, which is not what we would like to see.\n\n2. Ignoring the previous question, could the authors explain why the Bandit model necessitates using the representation of students' policies as input of the teacher policy? It seems that providing an optimal course distribution based on the current student's behavior would be sufficient. Why is there a need to consider course distributions under the representations of other (mainly comes from the historical) student policies?\n\nRegarding the experimental section:\n1. At Line 293, the authors mention that the SPC can switch to the largest population rapidly. Could the authors provide further explanation as to why this is possible and why it represents an advantage?\n2. The second and third paragraphs of Section 5.3 are unclear; the authors could rephrase them for clarity. You could try presenting your point as follows: \"From figure X (or the comparison of X and Y), we can observe XX, which indicates XX.\"\n3. In Appendix C, a more challenging 11 vs. 11 experiment was introduced, with the authors claiming superior performance of the SPC, which is great to see. But this claim raises questions as there are no baselines for comparison. Could the authors consider adding some baselines to this task?\n\nLIMITATIONS:\nNAN", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces the Skilled Population Curriculum (SPC), an automated curriculum learning algorithm designed for Curriculum-enhanced Dec-POMDP. The goal of SPC is to enhance the student's performance on target tasks via a sequence of training tasks provided by the teacher. The SPC functions as a nested-HRL method, where the teacher serves as the upper-level policy and is modeled as a contextual multi-armed bandit. At each teacher timestep, the teacher selects a training task from the distribution of bandit actions, with the context derived from the student policy's hidden state. The teacher's bandit is optimized using the student policy's test reward. The lower-level policy, also known as the \"student\", is in itself a hierarchical policy. The high-level policy implements population-invariant communication using a self-attention communication channel to manage messages from a number of agents, and all students share the same low-level policy.\n\nSTRENGTHS:\n- This paper is well-presented. Figure 1 is well-designed. I can get a good understanding of this paper's method just by reading this figure.\n- The algorithm is implemented with Ray RLlib, though the code is not currently available.\n\nWEAKNESSES:\n- **This study seems to be an overcomplicated amalgamation of pre-existing methods.** SPC stacks three layers of hierarchical policies (teacher 1 + student 2), the teacher is modeled as a multi-arm bandit with a fixed output dimension (number of tasks), and the lower-level control policies of the students are shared. The intricacy of this pipeline leads me to question its generalizability and practical applicability.\n- **More rigorous comparison with current MARL algorithms, and need benchmark results on SMAC**, which is de facto the most standard benchmark for MARL algorithms. Please consider adding [MAPPO](https://github.com/marlbenchmark/on-policy), [HARL](https://github.com/PKU-MARL/HARL), and their multi-agent communication variant as your baselines.\n- Line 236-238, “However, current approaches that extend HRL to multi-agent systems or utilize communication are limited to a fixed number of agents and lack the ability to transfer to different agent counts”, this is an inaccurate claim because it has been done in the ICLR 2022 publication, [*ToM2C*](https://arxiv.org/pdf/2111.09189.pdf), which similarly uses the HRL with a population-invariant multi-agent communication mechanism. AFAIK this cannot be treated as \"communication limited to a fixed number of agents\". Please consider citing this work and changing your statement regarding the previous work.\n\nQUESTIONS:\n- **I need more convincing results for proving \"The Necessity of Curriculum Learning\".** Why is an in-depth analysis of the teacher-student framework necessary? Despite its significantly increased implementation complexity compared to the original MAPPO algorithm, their performances appear roughly equivalent. Moreover, the MAPPO algorithm, provided sufficient exploration and large batch size, has already demonstrated state-of-the-art performance on both SMAC and GFR benchmarks. I presume an advantage of combining the ACL and the teacher-student framework lies in handling more challenging scenarios through incremental learning. To underscore the superiority of SPC over traditional multi-agent PPO methods, could you present performance data from the SMAC Super-Hard Map difficulty? This could include instances like 3s5z_vs_3s6z, where MAPPO previously underperformed significantly.\n- I am interested in understanding the implementation of multi-agent communication in Ray RLlib. It appears that the agents are exchanging messages before outputting their current actions. It's somewhat challenging for me to envision how this process is technically executed within the RLlib framework. I wish the code is available.\n\nLIMITATIONS:\nThe limitations of this paper are only briefly mentioned in the last section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the multi-agent RL problem with sparse reward and a varying number of agents. The authors propose a novel automatic curriculum learning strategy to solve complex cooperation tasks in this setting. Their curriculum strategy involves a teacher component and a student component. The teacher component selects the sequence of training tasks for the student component using the contextual bandit algorithm with predictive representation of the student’s current policy as context. The student component is endowed with a hierarchical skill framework and population-invariant communication. They empirically investigate their proposed strategy in two environments (MPE and GRF).\n\nSTRENGTHS:\nThe paper is overall well-written, and the related work is extensively discussed.\n\nThe theoretical results in this paper seem correct; I haven’t checked the details of the proofs.\n\nThe population invariant communication module is an interesting contribution to dealing with the varying number of agents across tasks. It would be interesting to compare its effectiveness (on its own) against the existing methods to deal with varying numbers of agents [23, 24].\n\nWEAKNESSES:\nI am unsure about the broader applicability of the contextual representation of the student policy using an online clustering algorithm. How much information will be lost in this process for a high-dimensional policy (e.g., that operates on image inputs)?  \n\nPresented experimental results are not sufficient to validate the effectiveness of the proposed curriculum strategy (specifically the teacher component) in complex scenarios, given that in the MPE environment, the impact/necessity of curriculum is negligible.\n\nQUESTIONS:\nWhy EPC [9] is not used as a baseline in the experiments?\n\nThe authors mention, \"To ensure a fair comparison, we modify VACL by removing the centralized critic for MPE tasks.” Please explain why.\n\nIn Figure 5, including the results for SPC w/o “both” HRL and COM would be good. Then, we can see the effectiveness of the teacher component (or curriculum). \n\nIn Figure 3 (MPE environment), including the ablation study results (similar to Figure 5) would be good. \n\nIt is also important to discuss/report the proposed strategy's computational cost/overhead (run time) compared to the baselines like VACL.\n\nLIMITATIONS:\nThe paper is of an algorithmic nature and does not have any direct potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-31"}