Learning Progress Driven Multi-Agent Curriculum
Authors: Wenshuai Zhao
Zhiyuan Li
Joni Pajarinen
Venue: ICML 2025 poster (2025)
Keywords: Curriculum learning
Temporal difference
Sparse reward
Primary area: reinforcement_learning->multiagent
TL;DR: We propose learning progress based self-paced curriculum learning for MARL tasks, addressing issues of existing return-based curriulum learning methods when applied to MARL.

Abstract: The number of agents can be an effective curriculum variable for controlling the difficulty of multi-agent reinforcement learning (MARL) tasks. Existing work typically uses manually defined curricula such as linear schemes. We identify two potential flaws while applying existing reward-based automatic curriculum learning methods in MARL: (1) The expected episode return used to measure task difficulty has high variance; (2) Credit assignment difficulty can be exacerbated in tasks where increasing the number of agents yields higher returns which is common in many MARL tasks. To address these issues, we propose to control the curriculum by using a TD-error based *learning progress* measure and by letting the curriculum proceed from an initial context distribution to the final task specific one. Since our approach maintains a distribution over the number of agents and measures learning progress rather than absolute performance, which often increases with the number of agents, we alleviate problem (2). Moreover, the learning progress measure naturally alleviates problem (1) by aggregating returns. In three challenging sparse-reward MARL benchmarks, our approach outperforms state-of-the-art baselines.

Learning Progress Driven Multi-Agent Curriculum
Wenshuai Zhao' Zhiyuan Li' Joni Pajarinen
Abstract
the number of agents potential curriculum in addition to the environmental parameters in single-agent CRL, as the number of agents is critical for the difficulty of MARL tasks. However, prior work is limited to controlling the number of agents manually or heuristically, for example, FewToMore (Wang et al., 2020) and EPC (Long et al., 2020) simply train the agents starting from tasks with fewer agents and gradually move to the target task in a predefined way. Variational automatic curriculum learning (VACL) (Chen et al., 2021) proposes general automatic CRL method that can control both the initial states and the number of agents. However, the number of agents is still manually set in a linear scheme.
The number of agents can be an effective curricu- lum variable for controlling the difficulty of multi- agent reinforcement learning (MARL) tasks. Ex- isting work typically uses manually defined cur- ricula such as linear schemes. We identify two po- tential flaws while applying existing reward-based automatic curriculum learning methods in MARL: (1) The expected episode return used to measure task difficulty has high variance; (2) Credit assign- ment difficulty can be exacerbated in tasks where increasing the number of agents yields higher re- turns which is common in many MARL tasks. To address these issues, we propose to control the curriculum by using& TD-error based learning progress measure and by letting the curriculum proceed from an initial context distribution to the final task specific one. Since our approach main- tains a distribution over the number of agents and measures learning progress rather than absolute performance, which often increases with the num- ber of agents, we alleviate problem (2). Moreover. the learning progress measure naturally alleviates problem (1) by aggregating returns. In three chal- lenging sparse-reward MARL benchmarks, our approach outperforms state-of-the-art baselines.
Existing works that assume a curriculum proceeding from fewer to more agents (Wang et al., 2020; Long al., 2020) are typically based on the intuition that coordinating larger number of agents is inherently more challenging than coor- dinating fewer agents to complete a task (Lowe et al., 2017; Rashid et al., 2018; Chao et al., 2021). However, this as- sumption does not always hold. For example, in the MPE Simple-Spread (Looe et al., 2017) task shown in Figure 1, several agents (blue circles) try cover as many landmarks (red circles) as possible. Intuitively, if the number of agents increases sufficiently, the most naive policy such as random moving can work well and achieve high episode returns. On the contrary, with fewer agents, the agents have to learn a complicated cooperation strategy to accomplish the task. Moreover, while an increased number of agents can lead to higher returns, this may also exacerbate credit assignment challenges in policy learning. We argue in this paper that more sophisticated control of the number of agents is needed in multi-agent curriculum reinforcement learning.
1. Introduction
Curriculum reinforcement learning (CRL) (Portelas et al., 2020b) consists of teacher generating a sequence of tasks of varying difficulty to train agents in order to bypass the difficulty of exploration in target tasks with sparse rewards. In single-agent CRL, the teacher usually controls the initial states or environmental parameters change the task diffi- culty. When considering curriculum learning in multi-agent reinforcement learning (MARL) settings (Li et al., 2025; Zhao et al., 2024; Li et al., 2024), it is natural to employ
Therefore, we first directly apply the state-of-the-art single- agent automatic curriculum reinforcement learning (ACRL) method, self-paced reinforcement learning (SPRL) (Klink et al., 2021), to adaptively control the number of agents. We name this method as SPRLM. SPRLM explores range of tasks with different contexts (Schaul et al., 2015a) and seeks easier ones with higher performance while making progress towards the target task, hence generating a reasonable task sequence as a curriculum. In our experiments, SPRLM outperforms heuristic baselines by successfully generating effective task distributions without being restricted pre- defined task sequences. However, current ACRL methods,
Department of Electrical Engineering and Automation, Aalto University, Espoo, Finland. Correspondence to: Wenshuai Zhao bienshuai.zhaoo @hao@aalto.f... fi>.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Learning Progress Driven Multi-Agent Curriculum
serve as an effective curriculum variable to facilitate explo- ration. SPRLM outperforms heuristic baselines most tasks, while SPMARL consistently surpasses all baselines, including other ACRL methods applied to MARL.
Our contributions are threefold: (1) SPRLM extends single- agent SPRL to a multi-agent context and shows that a princi- pled curriculum based on the number of agents outperforms manually designed baselines; (2) We identify two flaws in the straightforward extension and propose SPMARL to ad- dress these issues; (3) Our experiments on three distinct benchmarks demonstrate that SPMARL outperforms base- line methods'.
(a) 8 agents
(b) 20 agents
Figure 1. In Simple-Spp Spread task, agents (blue circles) need to cover as many landmarks (red circles) as possible. With the number of landmarks fixed, 20 agents shown on the right can casily complete the task and achieve higher returns compared to 8 agents on the left. However, a higher number of agents exacerbates the credit assignment problem in policy learning.
2. Related Work
In this section, we discuss general automatic curriculum gen- eration methods and existing works that apply curriculum learning to multi-agent tasks, especially those employing the number of agents as a curriculum variable in order to scale up the current MARL methods.
including SPRL, typically evaluate task difficulty based on expected episode returns, which introduces two issues in multi-agent settings. First, the episode returns of a task c, G(c) Ei=o ''r,, are usually sparse and can be only evaluated once per episode, leading to high variance in the estimated task difficulty (Sutton & Barto, 2018). Second, a curriculum focused on maximizing rewards may result in tasks that are ineffective for policy improvement. For instance, the extreme case of the Simple-Spread task with an infinite number of agents, the initial random policy may suffice to be optimal, resulting in zero policy gradient. In many Dec-POMDP (Amato et al., 2013) tasks where agents share the same reward, increasing the number of agents can lead to higher returns while also complicating credit assignment.
Automatic Curriculum Reinforcement Learning: Cur- riculum learning has been extensively studied in the single- agent domain and sometimes in the context of unsupervised environment design (UED) (Teoh et al., 2024). A set of auto- matic curriculum generation methods are proposed (Portelas et al., 2020b) to control initial states (Florensa et al., 2017), goal positions (Florensa et al., 2018) or environment dy- namies (Matiisen et al., 2019; Klink et al., 2021). Common objectives in the literature aim to optimize controllable con- texts based on criteria such as reward (Colas et al., 2019; Klink et al., 2021) and difficulty (Florensa et al., 2018) 8). Reward- based methods usually generate tasks with high re- turns, while the latter seeks tasks of intermediate difficulty rather than the easiest tasks. Several studies (Florensa et al., 2017; Mysore et al., 2019; Portelas et al., 2020a) employ the concept of learning progress to maximize training ef- ficiency, similar to our approach. However, we note that these methods still measure the learning progress based on the sparse episode return increases, while our method estimates learning progress based on the critic loss from the underlying MARL updates which is much more stable. More importantly, the extensive literature on single-agent ACRL largely overlooks the credit assignment problem that emerges when using the number of agents as a curriculum variable in multi-agent settings. In the following, sim- ply refer to these ACRL methods as reward-basel ACRL methods and build our work primarily on one representative ACRL method, SPRL (Klink et al., 2021). We argue that the improvements we propose in SPMARL can also be ap- plied other reward-based ACRL methods in the context of MARL tasks.
Inspired by the above findings, we further introduce self- paced multi-agent reinforcement learning (SPMARL). SP- MARL tackles the two issues of high variance returns and complex credit assignment with many agents by optimiz- ing a novel objective based on learning progress instead of expected episode returns. We use the expected temporal difference (TD) error to measure learning progress, that is, the value loss in multi-agent reinforcement learning. This new approach leverages the advantageous properties of the value function (s). As V*(() (s) reflects the performance of the current policy, the associated value loss w.r.t. different tasks naturally indicates the extent of policy changes on these tasks, given that converged value estimates typically imply no further policy updates. Moreover, the value loss is evaluated across all state transitions, significantly reducing estimation variance compared to episode returns.
We evaluate methods three benchmarks including MPE Simple-Spreud (Lowe et al., 2017), the XOR matrix game (Fu et al., 2022), and four SMAC-v2 Protoss tasks (El- lis et al., 2022). Each benchmark is modified to introduce severe sparse rewards, posing significant exploration chal- lenges. The results show that the number of agents can
Soorce Code: https//giitbbbcom//eensssssaaaap
2

---

Learning Progress Driven Multi-Agent Curriculum
Multi-Agent Curriculum Reinforcement Learning: Com- pared to extensive study in the single-agent realm, only few works have explored multi-agent curriculum reinforce- ment learning. Dynamic Multi-Agent Curriculum Learning (DyMA-CL.) (Wang et al., 2020) solves large-scale problems by learning from a small-size multi-agent scenario and pro- gressing to the target number of agents, where the number of agents is manually chosen. EPC (Long et al., 2020) expands the number of agents in the order of N + 2N, training multiple parallel agents at each stage and selecting the best ones for the next stage through crossover process. Varia- tional Automatic Curriculum Learning (VACL) (Chen et al., 2021) presents a principal method for curriculum learning, framing the optimization variational inference problem. Although VACL provides general framework for controlling environment parame- ters, it adheres predetermined sequence k' = k + 1 or k' = 2k for adjusting the number of agents. Consequently, we categorize these curriculum MARL approaches as a Lin- ear baseline, incorporating both increasing (from fewer to more agents) and decreasing (from more to fewer agents) curricula in our experiments. Recently, (Wu et al., 2024) in- troduced a method for accounting for both performance and similarity target tasks. However, the similarity is assessed using state visitation distributions, which are challenging to apply directly to tasks with differing state spaces as the number of agents varies.
the joint action at time step sampled from decentralized policies (-(1))).
3.2. Contextual Reinforcement Learning
Different from a typical Markov decision process (MDP) with fixed transition properties M = (S,A,P.r; Pn), contextual reinforcement learning (Neumann et al., 2011; Schaul et al., 2015a) parameterizes MDPs by contextual parameter с E C S Ih' which can be certain environ- mental parameters, goals or initial states, while assuming a shared state-action space over these MDPs, M(c) (S, S..AA Pe. Pa,e). The objective of contextual RL is de- fined as: maxg [(0, μ1) max, where we have the target context distribution C ~ juc) and initial state distribution s ~ (o. The value function Vo c) denotes the expected discounted return in states s and under the context с following the conditioned policy I (a|s, C, 0). Generally, contextual RL aims to generalize behavior over different tasks by exploiting the continuation between MDPs. In curriculum learning. we usually set the target context distribution μu() Dirac delta function, since we are interested in solving one specific task with fixed context while exploiting other tasks from different distributions v/(c) as curriculum.
(Vo c)],
curriculum distribution as a
4. SPRL for Multi-Agent Curriculum
SPRLM directly applies self-paced reinforcement learning (SPRL) (Klink et al., 2021) to control the number of agents as curriculum to address hard exploration problems in sparse-reward tasks.
3. Problem Formulation
In this section, we introduce the multi-agent reinforcement learning framework Dec-POMDP and the curriculum learn- ing framework contextual reinforcement learning.
Similar to general homotopy optimization methods (Allgo- wer & Georg, 2012), CRL works by assuming the contin- uation between tasks with varying contexts, i.e. the policy learned from one task can be good initialization for an- other task. In tasks where directly learning from the target context p(c) is difficult, i.c. the maximization of the follow- ing objective is hard due to sparse reward under the target contexts,
3.1. Dec-POMDP
We study the decentralized partially observable Markov decision process (Dec-POMDP) problem (Amato et al., 2013). which can be formulated as a tuple: (S, (O') )iN,, r, P, 7), where N' == (1, (1,...,, denotes a set of agents. At time step of, each agent observes local observation of from the full state st in the state space S of the environment and performs an action af in the action space A' based on its policy (-I1"), where h encodes the history information of partial observations and actions. The joint policy consists of all the individual policies (-1s+) = x² x ....... x n" The environment takes the joint action of all agents a, = (at, az') changes its state following the dynamics function P:S Ax S [0, 1] and generates common reward r S x A + R for all the agents. n E 1) is reward discount factor. The agents learn their individual policies and maximize the expected return: r" = arg max, Eg P P r a,)], where a, is
max E Eet [J c)] = Pgx (V) c)], (I)
SPRL solves the hard exploration problem by first train- ing the agent on easier tasks in contextual MDPs and then progressing to the target task. Formally, SPRL can be for- mulated constrained optimization problem
min DKL(p(cle) u(c))
s.t. [p(c c)) Z ViB, (Rc(p(c)vx) l| p(c|v*+1) S,
(2)
where μu(e) is the target context distribution, usually set as a Dirac delta distribution. p(c|p) denotes the context distribu-
3

---

Learning Progress Driven Multi-Agent Curriculum
tion parameterized by v such the mean and variance of Gaussian distribution. #k+1 represents the context distribu- tion to be updated. This distribution is supposed to generate easier tasks compared to the target task. We manually set a performance threshold Vín in order to find task distribu- tions with satisfying performance. In the first constraint in Equation 2, we maximize [p(c|e) (0, c)] by estimating the expected values over the new context distribution via importance sampling,
Algorithm Self-Paced Multi-Agent Reinforcement Learn- ing (SPMARL)
Initial context with number of agents distribu- tion vo, initial policy parameters Og, target context with number of agents distribution ju(c), expected perfor- mance threshold Ví.,, number of iterations K, rollouts M for each policy update, relative entropy bound
Input:
2: for k = 1 to K do
3: Policy Learning:
for i = 1 M do
M Σ
p(C.)/Rt)) V Vo(s.,oc.), ==1 p(csllk))
Sample context, i.c., the number of agents, C₁ p(c|1+)
max PAL1 M
(3)
6: Rollout trajectory p(ci, 0k)
in which M is the number of rollouts we collect. (g c,) represents the initial state value under current context C₁ and indicates the difficulty of current task with C,. Usually, Vo c) is estimated by the sparse episode returns.
7:
end for
8:
Obtain 0x+1 by the chosen MARL method and col- lected trajectories Dr. = {(c,, T.) I [1, M]}
Estimate Vo..1 c+) for context c;; i E [1, M]
10:
Context Distribution Update:
11:
Stage 1: Optimize the learning progress
Teacher
12:
if 3T
i= Ve (s). c²) < VL then
A
Stage
13:
Obtain vk+1 from Equation 5 under the KL diver-
Maximize [J e)] ي/ Vin s.t. (piii() و(جン) se
gence constraint
14:
Stage 2: Progress to the target
Prob.
Return
Student
15:
else
16: Obtain uk+1 from Equation 2
Agent
Agant
1ig Phange
Lat
17:
end if
date
Environmant
18:
end for
Stage
sidk)
Mirimize Dau(p(eie) I p(e))
Prob.
5. Self-Paced MARL
x1 (6, Vin
Agent Num.
While SPRLM is supposed to be able to successfully find easier tasks with high performance and eventually miti- gate the hard exploration problem due sparse reward, the reward-based objective in SPRLM could lead to slow learn- ing progress and unstable estimation. We improve SPRLM by proposing self-paced MARL (SPMARL) to optimize a new objective measuring the learning progress (LP) instead of directly optimizing task performance.
Phagge
Figure 2. SPRL involves a two-stage optimization.
In practice the constrained optimization problem in Equation 2 can be solved in a two-stage paradigm as shown in Figure 2. At first, when the expected per- formance Eptecjy [J((, c)] is lower than VLB maxi- mize [1,, c)] under the constraint DKL (p(clx)) || p(c)pe+1) & e and obtain the new context distribution p(c|p)41), which can be done by existing trust-region opti- mizer (Virtanen et al., 2020). Secondly, once the perfor- mance achieves the threshold Ví.R, we start to progress to the target context distribution μ(c) by minimizing DKL (p(c)v) ) plc)) while still constrained by the KL diver- gence threshold e. Note that in this stage, SPRL maintains the performance always above Vi by setting the context distribution unchanged until the performance is higher than the threshold again. In a multi-agent setting, SPRLM adap- tively controls the number of agents as the context variable c to generate curriculum.
The primary requirement for the proposed concept of learn- ing progress is that it should be easily estimable measure of policy improvement. To this end, we draw inspiration from the advantageous properties of value function V (s). which estimates the expected return of current policy my. Specifically, value loss inherently measures the extent of pol- icy change over a particular task and can be approximated by the expected temporal difference (TD) error across all state transitions. Therefore, we posit that value loss can serve as effective instantiation of learning progress, ad- dressing the two key issues identified in reward-based CRL methods. In the SPMARL framework, the value loss is com- puted following the underlying MARL algorithm MAPPO (see appendix) defined as:

---

Learning Progress Driven Multi-Agent Curriculum
Linear: The linear scheme can be seen as an abstract of existing multi-agent curriculum learning works (Wang al., 2020; Long et al., 2020; Chen et al., 2021), In Simple-Spreat and XOR matrix game, we set linearly increasing curriculum while in SMAC-v2 Protoss 5 vs. 5 task we use a linearly decreasing scheme in order to diversify the baselines.
1
LP(c)= = [aa.:
V(s)
(4)
where R(s,aa a) is the discounted return since state action pair a). Thanks to the CTDE framework allowing us to access the full state information during training, the es- timation sufficiently accurate. In SPMARL, we follow the two-stage optimization scheme in SPRL,, but replace the reward objective of Equation 3 in the first stage with our new learning progress measurement
Absolute learning progress with Gaussian mixture models (ALPGMM) (Portelas et al., 2020a) uses Guassian mixture models to sample tasks with maximum absolute learning progress. However, the estimation of learning progress in ALPGMM still based on differences in rewards, which may be prone to the same high variance issues that affect other reward- based curriculum learning methods.
ALPGMM:
p(ca|va+ LPoC..). p(c:)lu)
max 24-+1 M
(5)
We keep the second stage optimization the same as SPRL, Le. keeping the same performance threshold VLB- This is reasonable because even though we do not directly opti- mize returns over different contexts, the optimized learning progress objective will implicitly result in higher perfor- mance. As validated in our experiments, the curriculum generated by SPMARL even triggers a faster performance increase than SPRL. The detailed SPMARL is shown in Algorithm
VACL: Variational Automatic Curriculum Learning (VACL) utilizes Stein Variational Gradient Descent (SVGD) to update the task distribution, prioritizing tasks associated with higher returns, akin to SPRL. However, unlike SPRL, VACL does not enforce strict convergence to the target task distribution, which can potentially hinder performance on the target tasks.
Intuition on Learning Progress: The idea of using TD error the objective generate tasks in SPMARL is analogous the concept of Prioritized Experience Re- play (PER) (Schaul et al., 2015b) in Deep Q-Networks (DQN) (Mnih et al., 2015), where samples with higher TD error are prioritized. However, unlike PER, SPMARL esti- mates the averaged TD error across different contexts and selects tasks with higher TD error, which are expected to enhance the policy learning progress.
W/O teacher: This is the default MARL algorithm MAPPO trained directly on the target task without any curriculum. In experiments, due to the challenging sparse reward settings across all benchmarks, W/O teacher consistently fails on all tasks, yielding zero rewards. Consequently, we exclude its performance from the figures for clarity.
Note that ALPGMM was originally designed for single- agent tasks to optimize environmental parameters. In this work, we adapt it to dynamically control the number of agents. Similarly, while VACL employs a linear scheme for adjusting the number of agents its original formulation, we extend its variational inference framework enable adaptive agent control.
6. Experiments
We evaluate our method on three challenging benchmarks with severe sparse rewards, including (1) MPE Simple- Spread task (Lowe et al., 2017) with 8 agents, (2) XOR game (Fu et al., 2022) with 20 agents, and (3) four SMAC- v2 Protoss tasks (Ellis et al., 2022). The specified number of agents pertains to the target task, while the curriculum leamn ing methods generate set of tasks with varying numbers of agents to train the policy for optimal performance on the target tasks. The hyper-parameters used in our experiments are listed in the appendix. We compare SPMARL with both SPRLM and several baselines:
Player 2
B
Player 1
1
0
SPRLM:
Our first algorithm, which directly applies SPRL (Klink et al., 2020) to MARL settings to adap- tively control the number of agents. SPRLM also rep- resents a set of general automatic CRL methods based on rewards (Portelas et al., 2020b) that can be applied to multi-agent settings.
(a) XOR game.
(b) Protoss 5 vs. 5.
Figure 3. (a): The payoff matrix of 2-player XOR game. (b): Scenario from Protoss 5 vs. 5 SMACv2 showing agents battling the built-in AI.
5

---

Learning Progress Driven Multi-Agent Curriculum
SirppeSpread
SirpleSpread
SimpleSpread
80
Evaluation Rotum
60
16
Training Epicads Relurn
40
Number ofAgents
60
20
vad
10
alogmm
10
8
spmarl (ours)
1.0
Environment Sample
16/
Timestep
1e8
TimeStep
1e5
Figure 4. Comparison on the Simple-Spread task, where the target is set with 8 agents and landmarks. The plots averaged over 5 random seeds and the shadow area denotes the 95% confidence intervals. left figure shows evaluation returns on the target task with 8 agents. Note that the x-axis represents the samples collected from the environment, which proportional to the number of agents, The middle figure presents the generated curriculum from different methods, where SPMARL and SPRLM first generate more agents and then converge the larget agents while ALPGMM and VACL always generates agents. The right figure shows the episode returns on the training tasks. The ALPGMM algorithm achieves the highest because samples tasks with more than 14 agents.
XOR
XOR
XOR
Epiada Roaurn
WW
Agants Agants
Eploade
100
Number
vacl
50
alggmmm
5
spmart (ours)
0.8
1.0
1.2
0.0
0.2
Environment Sample
168
Timesteg
1a7
TimaStop
187
Figure 5. Comparison the 20-player XOR game where each agent needs to output different actions to succeed. While the linear curriculum from few to more (linear) and alpgmm successfully achieve optima eventually, SPRLM and SPMARL demonstrate a faster convergencc.
6.1. MPE Simple-Spread
other baselines. SPMARL further surpasses SPRLM by achieving faster convergence, attributed to its more rapidly updated context distribution. ALPGMM generates larger number of agents during training, leading to the highest training rewards. However, unlike the SPRL framework, it does not enforce convergence of the context distribution to the target distribution, resulting lower evaluation per- formance. is reasonable that Linear completely fails learn any valid policies because of the severe sparse rewards. Linear generating curriculum from few more agents exacerbates the problem as in this task few agents have less chance to receive any non-zero rewards.
As shown Figure Simple-Spreade involves several agents trying cover all the landmarks original Simple-Spread task is designed with a dense reward function denoted by the negation of summed distances to each landmark. However, we test our method on a modi- fied version featuring sparse rewards which becomes chal- lenging. this modified version, agents can only receive rewards when least landmarks are covered and the re- ward is the number of landmarks successfully covered at each timestep. We set the target task with 8 agents and landmarks.
as possible. The
Analysis of the generated curriculum: is noteworthy to observe the different curricula generated by SPMARL and SPRLM the middle figure. SPMARL rapidly updates the context distribution to include more agents, whereas SPRLM continues to explore for longer duration. As result, SPMARL starts early to converge to the target task after achieving the performance threshold Vr of 50.
Analysis of the evaluation performance: The experimen- tal results are presented in Figure 4, demonstrating that the number agents serves as an effective curriculum variable for addressing the sparse reward problem. In the left figure, which depicts episode returns evaluated the target task during training, both SPMARL and SPRLM outperform
6

---

Learning Progress Driven Multi-Agent Curriculum
5v5
5v5
5v5
14
0.6
Number of Agents
12
Evaluatin
0.4
Ineer vaci
10
Training
0.8
alpgmm
(ours)
spmart (ours)
0.0
0.2
0.4
0.8
1.0
0.0
0.2
0.4
0.8
167
167
16/
6v6
6v6
6v6
1.0
0.6
Number of Agents
Training Rate
12
Evalutin Win
0.4
0.8
lineer vacl
10
0.2
sprim (ours)
0.2
spmarl (ours)
0.0
0.0
7v7
7v7
7v7
Evaluaiio Rote
Number Agantss
0.8
0.6
lineer
vasl
alpgmm
(ours)
spmarl (ours)
0.9
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
167
1e7
10/
8v8
8v8
8v8
0.5
14
Win Rate
Number ofAgents
Training Win
0.3
12
0.6
linear vadi alogmim
10
sprim (ours)
spmarl (ours)
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Sample
1e9
Timestop
Figure Comparison on SMACv2 Protoss tasks. From top to bottom row, the tasks are vs. 5, vs. 6, vs. and vs. Across four tasks, SPMARL achieves performance that is comparable to better than all baseline methods.
The faster context updates in SPMARL highlight two key advantages provided by our proposed learning progress objective compared to the performance objective in SPRIM: more stable estimation and more effective task generation for policy improvement.
ALPGMM achieves the highest training episode returns, it consistently samples tasks with larger number of agents without considering the target task. However, de- spite the apparent advantage of training with more agents, ALPGMM does not lead to improved performance on the target task. Similarly, VACL fails to converge to the target context distribution.
Analysis of the training process: The episode returns dur- ing training are closely influenced by the curriculum gen- erated by different methods. As shown in the right figure,

---

Learning Progress Driven Multi-Agent Curriculum
6.2. XOR Matrix Game
fails to show better evaluation performance on the target task despite much higher training performance in the right figure. These can be attributed to the exacerbated credit assignment difficulty due to too many agents. On the contrary, our method SPMARL. learns to decrease the number of agents when the current context distribution has a sufficient number of agents to meet the performance threshold.
In this subsection, we conduct experiments on a cooperative task called N -lplayer XOR game. In this game, each player has N possible actions and they receive positive reward only if they each select different actions. A simple example of 2-player XOR game is given by the payoff' matrix shown in Figure 3()). While learning the optimal policy for a 2-player XOR game is relatively simple, the complexity increases dramatically with the number of players due to the exponentially expanding joint policy space. Consequently, curriculum learning approach becomes essential. In our experiments, we set the target task for 20 players.
6.4. Comparison of the Objective Variances
We visualize the standard deviation (std.) of the objectives for SPMARL and SPRLM, specifically the std. of the TD error and episode return. As shown in Figure SPMARL exhibits significantly lower variance compared to SPRLM in both Protoss 7v7 and 8v8. This observation further supports our hypothesis that lower estimation variance in the TD error facilitates more effective curriculum generation. More results and significance analysis can be found in Section A.3.
As shown in Figure 5, VACI. to learn an effective policy. The Linear baseline follows the human prior assumption that tasks with fewer agents are easier, gradually increasing the number of agents over time. While Linear eventually converges to the optimal policy, its need to explore the entire context space results in inefficiency and instability, as shown in the fluctuating training curves in the right figure.
ALPGMM also converges to the optimal policy but only after extensive exploration, as its generated curriculum fails
to converge to the target task.
84
9.2
In contrast, our methods automatically identify effective curricula. As shown in the middle figure, both SPRLM and SPMARL effccRt.ffcccietly explore the context space and ul- timately converge to the target 20 agents task. Notably, SPMARL further outperforms SPRLM by achieving faster convergence.
80
05
Tereshop
147
Timastie
(a) Protoss 7v7.
(b) Protoss 8v8.
Figure 7. Comparison of the std. of the TD error objective used in SPMARL and the std. the episode return used in SPRLM.
6.3. SMAC-v2
7. Discussion
We further test our method on SMAC-v2 (Ellis et al., 2022), a new version of the StarCraft Multi-Agent Challenge (SMAC) (Samvelyan et al., 2019), which increases the dif- ficulty by imposing higher stochasticity. Specifically, we evaluate SPMARL the Protoss tasks (Figure 3(b)) with sparse reward. In this setting, the agents can only receive 1 if win or 1 if lose at the end of the game.
We explore the potential of employing a controllable number of agents to alleviate the challenges inherent to multi-agent systems. Our second proposed method, SPMARL, enhances the convergence speed of the first straightforward algorithm SPRLM by introducing a novel learning progress objective but retains the two-stage optimization paradigm under the KL constraint, as shown in Figure 2. Although the KL con- straint was initially proposed to maintain performance con- tinuity during context updates in reward-based SPRL, SP- MARL demonstrates superior performance improvements when not directly optimizing performance. While the KI. constraint aids in stabilizing training and benefits the initial phase of this work, its necessity can be reconsidered when utilizing our learning progress as the new optimization ob- jective. We leave this investigation our future work.
As illustrated in Figure 6, baseline methods incorporating curriculum learning successfully learn effective policies, highlighting the significance of curriculum learning for this task. Among these methods, SPMARL and ALPGMM outperform other curriculum learning approaches, namely Linear and VACL, in terms of both convergence speed and final performance. In contrast, SPRLM fails to generate an effective curriculum except for the 5v5 task, likely due to the extreme reward setting {-1,0, 13, which hinders the optimization of the curriculum distribution from converging to viable solution.
8. Conclusion
It is notable that in the Linear bascline, even though with superior prior knowledge designing curriculum from suffi- ciently as high as 15 agents to the target number of agents, it
In this paper, we investigate the control of agent numbers as an effective curriculum strategy. Existing works are typ- ically limited to predefined curriculum that rely on heuris-
8

---

Learning Progress Driven Multi-Agent Curriculum
ties, such linear scheme. To address this limitation, we first directly extend the state-of-the-ar single-agent currieu- lum method SPRL to the multi-agent setting. Our analysis reveals two potential flaws in general swward-based cur- riculum methods for MARL: unstable estimation based on sparse episode returns (Suttonn & Barto, 2018), and the in- creased credit assignment difficulty in tasks where more agents tend to yield higher returns. These problems can eventually lead to slow learning progress. Therefore, we further propose SPMARL that prioritizes tasks based on learning progress instead of the episode returns. SPMARL optimizes value loss over the context distribution. Impor- tantly, value loss naturally indicates policy improvement as tasks with higher value loss represents more significant policy change. Moreover, the expected value loss can be accurately estimated w.r.t. all the state transitions rather than sparse episode returns. Consequently, SPMARL generates tasks that benefit policy learning much as possible. Al- though SPMARL does not focus directly on performance, it implicitly increases episode returns more effectively by im- proving learning progress. Evaluation on three challenging benchmarks demonstrates the effectiveness of SPMARL in addressing diffieult sparse-reward problems. In experiments, SPMARL outperforms comparison methods.
2405, 2013.
Bettini, M., Prorok, A., and Moens, V. Benchmarl: Bench- marking multi-agent reinforcement learning. Journal of Machine Learning Research, 25(217):1-10, 2024.
Chao, Y., VELU, A., VINITSKY, E., et al. The surprising effectiveness of ppo in cooperative, multi-agent games. arXiv preprint arXiv:2103.0/955, 2021.
Chen, J., Zhang, Y., Xu, Y., Ma, H., Yang, H., Song, J., Wang. Y., and Wu, Y. Variational automatic currieu- lum learning for sparse-reward cooperative multi-agent problems. Advances in Neural Information Processing Systems, 34, 2021.
Colas, C.,, Fournier, P., Chetouani, M., Sigaud, O., and Oudeyer, P.-Y. Curious: intrinsically motivated modular multi- reinforcement learning. In Proceedings of International Conference on Machine Learning, pp. 1331- 1340, 2019.
Ellis, B., Moalla, S., Samvelyan, M., Sun, M., Mahajan, A., Foerster, J. N., and Whiteson, S. Smacv2: An im- proved benchmark for cooperative multi-agent reinforce- ment learning. arXiv preprint arXiv:2212.07485 2022.
Florensa, C., Held, D., Wulfmeier, M., Zhang, M., and Abbeel, P. Reverse curriculum generation for reinforce- ment learning. In Proceedings of Conference on robot learning, pp. 482-495. PMLR, 2017.
Acknowledgements
We acknowledge CSC Center for Science, Finland, for awarding this project access to the LUMI supercomputer, owned by the EuroHPC Joint Undertaking, hosted by CSC (Finland) and the LUMI consortium through CSC. We also acknowledge the computational resources provided by the Aalto Science-IT IT project and funding by Research Coun- cil of Finland (357301). Zhiyuan Li is supported by the Research Council of Finland from the Flagship program: Finnish Center for Artificial Intelligence (FCAI).
Florensa, C., Held, D., Geng, X., and Abbeel, P. Auto- matic goal generation for reinforcement learning agents. In Proceedings of International Conference on Machine Learning, pp. 1515-1523 2018.
Fu, W., Yu, C., Xu, Z., Yang. J., and Wu, Y. Revisiting some common practices in cooperative multi-agent re- inforcement learning. In Proceedings of International Conference on Machine Learning, pp. 6877, 2022.
Impact Statement
Klink, P., Abdulsamad, H., Belousov, B., and Peters, J. Self- paced contextual reinforcement learning. In Proceedings of Conference on Robot Learming, pp. 513-529, 2020.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Klink, P., Abdulsamad, H., Belousov, B., D' D'Eramo, C., Peters, J., and Pajarinen, J. A probabilistic interpretation of self-paced learning with applications to reinforcement learning. arXiv preprint
References
2021.
Allgower. E. L. and Georg, K. Numerical continuation methods: an introduction, volume 13. Springer Science & Business Media, 2012.
Li, Z.. Zhao, W., Wu, L., and Pajarinen. J. Backpropagation through agents. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 13718 13726, 2024.
Amato, C., Chowdhary, G., Geramifard, A., Üre, N. K., and Kochenderfer, M. J. Decentralized control of partially observable markov decision processes. In Proceedings of IEEE Conference on Decision and Control, pp. 2398-
Li, Z., Zhao, W., Wu, L., and Pajarinen, J. Agentmixer: Multi-agent correlated policy factorization. In Proceed- ings of the AAAI Conference on Artificial Intelligncee, pp. 2025.

---

Learning Progress Driven Multi-Agent Curriculum
Long, Q. Zhou, Z., Gupta, A., Fang, F., Wu, Y., and Wang, X. Evolutionary population curriculum for seal- ing multi-agent reinforcement learning, arXiv preprint arXiv:2003.10423, 2020.
Schaul, T., Horgan, D., Gregor, K., and Silver, D. Universal value function approximators. In Proceedings of Interna- tional Conference on Machine Learning, pp. 1312-1320, 2015a.
Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, P., and Mordatch, I. Multi-agent actor-critic for mixed cooperative-compettive environments. arXiv preprint arXiv:1706.02275. 2017.
Schaul, T., Quan, J., Antonogiou, I., and Silver, D. Priori- tized experience replay. arXiv preprint arXiv:1511.0500 2015b.
Schulman, J., Moritz, P., Levine, S., Jordan, M., and Abbeel, P. tigh-diimessooal continuous control using generalized advantage estimation. arXiv preprint 1rx.v:1506002435 2015.
Mann, H. B. and Whitney, D. R. On test of whether one of two random variables is stochastically larger than the other. The annals of mathematical statisties. pp. 50-60, 1947.
Schulman, J., Wolski, F,, Dhariwal, P., Radford, A., and
Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXivv:107.0660 2017.
Matiisen, T., Oliver, A., Cohen, T., and Schulman, Teacher-student curriculum learning. IEEE transactions neural networks and learning systems, 31:3732-3740, 2019.
Sutton, R. S. and Barto, A. G. Reinforcement learning: An introduction. MIT press, 2018.
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Ve-
Teoh, J., Li, W., and Varakantham, P. Improving environ- ment novelty quantification for effective unsupervised environment design. Proceedings of Advances in Neu- ral Information Processing Systems, pp. 135299-135333, 2024.
ness, J., Bellemare, M. G., Graves, A., Riedmiller, M.,
Fidjeland, A. K., Ostrovski, G., et al. Human-level con- trol through deep reinforcement learning. Nature, 518: 529-533, 2015.
Mysore, S., Platt, R., and Saenko, K. Reward-guided cur- riculum for robust reinforcement learning. In Workshop on Multi-task and Lifelong Reinforcement Learning at ICML, 2019.
Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., et al. Scipy 1.0: fundamental algorithms for scientific computing in python. Nature Methods, 17:261-277 2020.
Neumann, G. et al. Variational inference for policy search in changing situations. In Proceedings of International Conference on Machine Leurning, pp. 817-824, 2011.
Wang, W., Yang, T., Liu, Y., Hao, J., Hao, X., Hu, Y., Chen, Y., Fan, C., and Gao, Y. From few to more: Large-scale dynamic multiagent curriculum learning. Proceedings of the AAAI Conference on Artificial Intelligence, pp. 7293-7300, 2020.
Portelas, R., Colas, C., Hofmann, K., and Oudeyer, P.-Y. Teacher algorithms for curriculum learning of deep rl in continuously parameterized environments. In Pro- ceedings of Conference on Robot Learning, pp. 833-853, 2020a.
Wu, J., Hao, J., Yang, T., Hao, X., Zheng, Y., Wang, W., and Taylor, M. E. Portal: Automatic curricula generation for multiagent reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 15934- 15942, 2024.
Portelas, R., Colas, C., Weng, L., Hofmann, K., and
Oudeyer, P.-Y. Automatic curriculum learning for deep rl: short survey. Proceedings of International Joint Con- ference Artificial Intelligence, pp.
Zhao, W., Rantala, E.-A., Pajarinen, J., and Queralta, P. Less is more: Robust robot learning via partially observ- able multi-agent reinforcement learning. arXiv preprint 2309h:2309.47999 2023.
2020b.
Rashid, T., Samvelyan, M., Schroeder, C., Farquhar, G., Foerster, J., and Whiteson, S. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In Proceedings of International Conference on Machine Learning, pp. 4304, 2018
Zhao, W., Zhao, Y., Li, Z., Kannala, J., and Pajarinen, J. Optimistic multi-agent policy gradient. In Proceedings of International Conference on Machine Learning, pp. 61 61186-6 61186-61202. 61202, 2024.
Samvelyan, M., Rashid, T., De Witt, C. S., Farquhar, G., Nardelli, N., Rudner, G., Hung, C.-M,, Torr, P. H., Foerster, J., and Whiteson, S. The starcraft multi-agent challenge. arXiv preprint arXiv: / /902 04043, 2019.
10

---

Learning Progress Driven Multi-Agent Curriculum
A. Appendix
A.1. Implementation Factors
Several implementation details need to be considered when applying curriculum learning to control the number of agents. First, the changing number of agents usually leads varying vector lengths of agents' observation and we need to handle the new coming agents. Second, the context distribution in SPRL usually implemented as Gaussian distribution which is continuous while we need to optimize the discrete number of agents. Third, the existing MARL benchmarks rarely support setting different numbers of agents in a convenient way.
In SPMARL, we tackle the first difficulty by simply setting fixed-length observation vector for each agent when it doesn't t impact the performance much (Zhao et al., 2023) or padding zeros to the state vectors. To handle the newly joined agents, we adopt parameter sharing for all the agents so that the new agents can use the learned policy. For the second problem, we retain the Gaussian distribution and simply discretize the sampled float contexts, which works well in practice. the third problem, we create set of environment wrappers for existing MARL benehmarks to easily change the number of agents.
In addition, VLB is an important of the expected return on the target task. We perform an ablation in Section A.3.5.
in SPRL and SPMARL. In our experiments, VLB can be usually set 80%
A.2. MAPPO
Multi-agent PPO (MAPPO) (Chao al., 2021) is popular MARL algorithm with strong performances on various benchmarks. MAPPO works in a straightforward way by applying single-agent PPO (Schulman et al., 2017) to multi-agent learning while using centralized critic with additional full-state information. In MAPPO, each agent learns a centralized state value function (s), and the individual policy is updated by maximizing the following objective
max 1 (ar)))=) (min(m(0)A(ss)))))) a²), clipp))))) ( =)
(6)
where r(0) is the importance ratio between the current policy and the previous policy used to generate the data,
Toy (a hi)
r(0)
Tos (a)
The advantage A(s,a') is usually estimated by the generalized advantage estimator (GAE) (Schulman et al., 2015) defined as Equation 8, where we use the full state information thanks to the centralized training and decentralized execution (CTDE) (Lowe et al., 2017) framework,
A, GAE(A,7)
Σ (7A)'644 1-
(8)
and 6, denotes the TD error
+ V(s.).
(9)
In our implementation, in order to transfer to different numbers of agents, we use parameter sharing (Chao et al., 2021)
among agents.
A.3. More Results
In this section, we present additional experimental results to provide a comprehensive evaluation and analysis of our method. Section 1 reports a significance analysis of the experiments presented the main text. We extend the empirical comparison four additional SMAC-v2 tasks in Section A.3.2, and two BenchMARL benchmarks (Bettini al., 2024) in Section A.3.3. In Section A.3.4, further compare the variance of SPMARL and SPRLM across four tasks. Finally, Section A.3.5 provides an ablation study on the hyperparameter VLB using two SMAC-v2 tasks.
1. SIGNIFICANCE ANALYSIS
In addition to the learning curves presented in the main text, we report the statistical results of the experiments in the following tables. Table summarizes the mean evaluation performance of various algorithms during training, showing that SPMARL. consistently outperforms the baselines or achieves comparable best-case results.

---

Learning Progress Driven Multi-Agent Curriculum
Table Mean and standard deviation of the evaluation performance during training on all tasks. Results for methods compared SPMARL with p-value higher than 0.05 according to the Mann- Whitney U test (see Table 2) are highlighted in bold.
Algorithm
SimpleSpread
XOR
5v5
6v6
7v7
8v8
W/O Teacher
0.00 (0.00)
0.00 (0.00)
0.00 (0.00)
0.00 (0.00)
0.00 (0.00)
0.00 (0.00)
Linear
0.00 (0.00)
26.80 (1.16)
0.25(0.02)
0.25 (0.02)
0.25 (0.01)
0.25 (0.01)
VACL
24.43 (4.53)
0.00 (0.00)
0.47 (0.02)
0.37 (0.02)
0.28 (0.01)
0.22 (0.01)
ALPGMM
24.65 (1.98)
104,80 (14.73)
0.55 (0.02)
0.45 (0.01)
0.36 (0.01)
0.28 (0.01)
SPRLM
27.48 (5.38)
141.45 (16.98)
0.54 (0.05)
0.13 (0.15)
0.08 (0.11)
0.11 (0.07)
SPMARL
33.36 (1.84)
143.08 (7.72)
0.55 (0.01)
0.45 (0.01)
0.38 (0.02)
0.31 (0.01)
To further assess the significance of these differences, perform one-sided Mann-Whitney 1947) to evaluate whether the evaluation returns of SPMARI. are stochastically greater than those of the baselines. As shown in Table 2, the test yields p-values below 0.05 in most cases.
Whitney U test (Mann & Whitney,
Table 2. Statistical significance (p-values) of SPMARI compared to baseline methods.
Algorithm
SimpleSpread
XOR
5v5
6v6
7v7
8v8
Linear
0.0060
0.0085
0.0079
0.0079
0.0040
0.0040
VACL
0.0040
0.0052
0.0040
0.0079
0.0079
0.0079
ALPGMM
0.0040
0.0097
0.7222
0.4524
0.0079
0.0317
SPRLM
0.0159
0.8395
0.4206
0.0079
0.0022
0.0040
A.3.2. EXPERIMENTS ON MORE SMAC-v2 TASKS
We conduct additional experiments on four S.MAC-22 tasks: Terran vs. Terran 6 vs. 6, Zerg 5 vs. 5, and Zerg vs. 6. As shown in Figure 8, SPMARL consistently outperforms baseline methods and generates coherent and effective curricula across all tasks.
A.3.3. EXPERIMENTS ON BenchMARL TASKS
We further evaluate our method on two additional tasks from the recent BenchMARL benchmark (Bettini et al., 2024), beyond the already-included SMAC-v2 and Simple-Spread environments. As shown in Figure 9, SPMARL achieves performance comparable to other baselines. dense-reward setting as the original BenchMARL, which may reduce the presence of exploration challenges and thus diminish the benefits of curriculum learning. Nonetheless, SPMARL remains the only method that generates curricula which converge to the desired target distributions.
important to note that the two new tasks, Balance and Wheel, employ the same
A. VARIANCE COMPARISON ON MORE TASKS
,......,. cn). R 25, the How to compute the variance: Mathematically. assume we have collected a set of contexts {c1, c2,.,,
corresponding episode returns {R1, R2...., R, ). the TD-errors (TD,, TD2,..., TD,) are computed as TD; = LP((c)= =
[|][[, a) [()] Note that the return used in computing TD-errors is usually estimated as the critic in RL, R(s,, a Ti + 7 further reducing the estimation variance (Schulman al., 2015). The standard deviation
then computed as / Σ (I; 2)2, where can be R2 or TD, and & represents the mean. Since TD-error on each context has been averaged over the samples of the whole episode, it usually shows lower estimation variance.
We present additional results in Figure 10 comparing the standard deviation of objective estimations between SPMARL and SPRLM on four more tasks, further highlighting the stability of our proposed approach.
A.3.5. ABLATION ON VLB
The hyperparameter VLB plays crucial role in SPRL methods (Klink et al., 2020; 2021). As our approach adopts the two-stage optimization framework from SPRL, we follow a similar strategy for setting VLD Specifically, our experiments,
12

---

Learning Progress Driven Multi-Agent Curriculum
Evaluation
Curiculum
Training
0.8
0.6
Number Agents
10
0.4
vacl
0.2
alggrmm
(ours)
spmari (ours)
Environment Sample
1e7
Timestep
1e8
TimeSlep
1e8
Evaluation
Curiculum
Training
0.8
Number of
Epiaade Retur
0.8
0.4
10
0.2
slpgrm
sprim (ours)
0.2
spmeri (ours)
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.5
2.0
2.5
3.0
0.0
0.5
1.5
2.0
2.5
3.0
Envtronmant Sample
107
Timestop
168
TimeStap
1eli
Evaluation
Curriculum
Training
12
0.8
Reaum
Number Agents
0.8
0.3
0.2
vacl
0.4
alpgmm
sprim (ours)
spmari (ours)
0.0
0.5
1.5
2.0
2.0
2.5
3.0
2.0
2.5
Environment Semple
Timestep
TimeSlep
Evaluation
Curiculum
Training
0.8
vacl
0.6
alpgmm
12
sprim (ours)
Number Agantis
0.6
(ours)
Epizoda
Epioode
0.4
0.2
1.0 Environment Sample
2.5 to7
2.5
Timastop
106
TimeStap
166
Figure 8. Comparison SMACv2 Terran and Zerg tasks. From top to hottom row, the tasks are Terman vs. 5 Terran 6 vs. 6, Zerg vs. 5, and Zerg 6 vs. Across tasks, SPMARI. achieves performance that is comparable to or better than baseline methods.
we empirically set Vía to approximately 80% of the final task performance, For instance, in the SMACv2 Protoss 5 vs. 5 task, where the final performance around 0.8, we choose Ví. 0.6. To evaluate the sensitivity of our method to this hyperparameter, we conduct an ablation study. The results, shown in Figure 11, demonstrate that our method performs robustly across wide range of VLB values.
A.4.
We list the syper-parameters of both the underlying multi-agent agent reinforvement learning algorithm MAPPO and our curriculum learning algorithm SPMARI.. Note that we always set the same
for both SPMARL, and SPRLM. The
13

---

Learning Progress Driven Multi-Agent Curriculum
Evaluation
Curriculum
Training
126
100
ofAgerts
Epicode Retum
75
75
6
25
alpgrmm
4
sprim (ours)
0
sprnaet (ours)
Environment Sample
te/
Timestop
1e6
TimeStop
Evaluation
Curriculum
Training
5
Number of Agents
-5
Epicods
-7
sprim (ours)
spmari (ours)
0.00
0.25
0,50
0.75
1.00
1.25
1.50
0.0
0.5
1.0 Timestap
1.5
2.0
0.0
1,0
1.5
2,0
Environment Sample
TimeSlap
Figure 9. Comparison on BenchMARL Balance and Wheel tasks. From top to bottom, the tasks are Balance and Wheel. On these two non-sparse tasks, SPMARL shows comparable performance other baselines, which could be due that these tasks do present exploration challenges. However, note that SPMARL is the only method generating curriculum that converges to target distributions.
shared parameters across all the domains are listed in Table for MAPPO and Table 4 for SPMARL. Benchmark Benchmark-specifee parameters are listed in Table 5 for MPF, Table 6 for XOR game, Table 7 for SMAC v2, and Table 8 for BenchMARI.
Table 3, Common hyper-parameten of MAPPO across all domains
Parameter
Value
recurrent neural network
True
recurrent data chunk length
10
gradient clip norm
10.0
hidden layer dim
64
gae lambda
0.95
gamma
0.99
optimizer
Adam
optimizer epsilon
le-5
entropy coefficient
0.01
ppo-clip
0.2
14

---

Learning Progress Driven Multi-Agent Curriculum
1.0
1.0
0.8
0.8
Standard Devittio
Standard Deviation
0.4
0.4
0.2
sprim spmari
sprim spmart
Timestop
1a7
Timestop
107
(a) Protoss Sv5.
(b) Protoss 6v6.
2.0
Deviation
Standard Deviaton
1.5
1.0
0.5
spri spmart
sprim
0.0
0.0
0.5
1.5
2.0
0.0
0.5
1.0
1.5
2.0
Timeslop
Timestop
(c) Balance.
(d) Wheel.
Figure 10. Comparison of the standard deviation of the TD error objective used in SPMARL and that of the episode return used in SPRLM. The results show that the estimation variance of TD error used in SPMARL is usually lower than the variance returns used in SPRLM.
Evaluation
Cumiculum
Training
0.8
11
Number Agenis
Epleade Relurn
VLs
0.7
0.4
0.4
0.8
0.6
0.7
0.0
6
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Environmant Sample
167
Timastop
Timestop
1e7
Evaluation
Curiculum
Training
0.7
Number of Agents
10
Edledde Relum
0.3
0.4
0.5
0.3
0.6
Environment Semple
167
Timastop
Timestop
Figure 11. Ablation of Vía SMACv2 Protoss tasks. From to bottom, the tasks are Protoss 5 vs. 5 and Protoss 6 vs. 6. The results indicate that SPMARL performs robustly across a broad range of VL..
15

---

Learning Progress Driven Multi-Agent Curriculum
Table Common parametrrsss of SPMARL across all domains
Parameter
Value
max kl
0.05
context kl threshold
8000
target var
4e-3
std lower bound
0.2
Table 5. Hyper-parament for MPE
Parameter
Value
MAPPO
actor Ir
7e-4
critic Ir
7e-4
ppo epoch
10
hidden layer
1
episode length
25
number of mini batch
number of rollout threads
256
SPMARL
lower context bound upper context bound
6
20
init mean
10
init var
25
target mean
8
perf lb
50
Table 6. Hyper-parameters for XOR
Parameter
Value
MAPPO
actor Ir
5e-4
critic Ir
5e-4
ppo epoch
10
hidden layer
1
episode length
200
number of mini batch
number of rollout threads
50
SPMARL
lower context bound
2
upper context bound
20
init mean
6
init
16
target mean
20
perf lb
0.5
16

---

Learning Progress Driven Multi-Agent Curriculum
Table 7. Hyper-parameters SMAC v2 tasks
Parameter
Value
MAPPO
actor Ir
5e-4
critic Ir
5e-4
ppo epoch
5
hidden layer
1
episode length
400
number of mini batch
1
number rollout threads
25
SPMARL
lower context bound
5
upper context bound
15
init mean init var
10
25
target mean (5v5)
5
target mean (6v6)
6
target mean (7v7)
7
target mean (8v8)
8
perf lb (Protoss 5v5, 6v6, 7v7)
0.6
perf 1b (Protoss 8v8)
0.4
perf lb (Terran 5v5, 6v6)
0.5
perf lb (Zerg 5v5, 6v6)
0.5
Table Hyper-parametee for BenchMARL tasks
Parameter
Value
MAPPO
actor Ir
5e-4
critic Ir
5e-4
ppo epoch
5
hidden layer
1
episode length
200
number of mini batch
1
number of rollout threads
25
SPMARL
lower context bound
2
upper context bound
10
init mean
5
init
25
target mean (Balance)
6
target mean (Wheel)
6
perf lb (Balance)
80
perf lb (Wheel)
-6
17


Reviews:
[{"review_id": "yfrmNaWCBs", "reviewer": "Reviewer_ZwRM", "summary": "This paper looks at curriculum learning in multi-agent reinforcement learning (MARL) by using the number of agents as a dynamic context variable. The authors first adapt self-paced reinforcement learning (SPRL) to the multi-agent setting (SPRLM), then propose SPMARL - a more principled variant that measures learning progress via TD error rather than noisy episode returns. Across several sparse-reward MARL benchmarks, SPMARL shows faster convergence and stronger final performance than prior approaches. Benchmarks are sufficient but not extensive, leaving out some of the JaxMARL work and others that have appeared in the last couple of years. The simplicity of the method is very valuable, though, given the complexity of MARL environments and the MARL problem writ large.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer ZwRM\nWe thank the reviewer for the appreciation of the simplicity and generality of our method. We hope new experiments and clarifications help to address your concerns.\n\n### 1. Clarity on observability setting:\n>If I'm understanding the paper correctly, the setting is fully observable (albeit decentralized). Full observability feels like a potential hitch. In a rebuttal, would love to hear the authors' thoughts on that.\n\nWe agree with the reviewer that observability is important in MARL. In our experiments, we use partial observability for all the tasks except the *XOR* matrix game, since the *XOR* game is a stateless task. For example, in *Simple-Spread*, the agents can only observe $4$ nearest agents and landmarks, and in *SMACv2* tasks agents can only observe enemies and agents in a certain range. Partial observability also helps in transferring policies across tasks with different numbers of agents since the observation space is fixed instead of growing with the number of agents. We use RNN policies for all the methods.\n\n\n### 2. More benchmarks:\n> The simplicity of the method is very valuable. ... The framework also feels like it could be super general and applied across single-agent RL and MARL.\n> That said, the experimental analysis seems to leave off some recent tools for benchmarking like JaxMARL and others. \n\nWe appreciate the reviewer for kindly acknowledging the contribution of our simple yet effective method and thank you for suggesting new benchmarks. We found that our current benchmarks such as *MPE simple-spread* and *SMACv2* are also included in *JaxMARL*. Therefore, we mainly performed new experiments on more *SMACv2* tasks and two tasks from *BenchMARL* [1] suggested by reviewer ffyd.\n\nThe results in [link](https://sites.google.com/view/spmarl-icml2025/#h.bj45hu212vx7) show the results on four new *SMACv2* tasks, *Terran 5v5*, *Terran 6v6*, *Zerg 5v5*, and *Zerg 6v6*. We can see that our algorithm SPMARL consistently outperfoms the baselines and generates stable and intepretable curricula across all the tasks.\n\nFor the BenchMARL tasks, *Balance* and *Wheel*, our method achieves performance comparable to the baselines, as shown in this [link](https://sites.google.com/view/spmarl-icml2025/home#h.oawhzhwh26av). This may be due to the fact that the exploration challenge in these tasks is not particularly severe, reducing the necessity of curriculum learning. However, SPMARL is the only one generating curriculum that converges to the target task distribution.\n\n[1] Bettini, Matteo, Amanda Prorok, and Vincent Moens. \"Benchmarl: Benchmarking multi-agent reinforcement learning.\" Journal of Machine Learning Research 25.217 (2024): 1-10.", "claims_and_evidence": "If I'm understanding the paper correctly, the setting is fully observable (albeit decentralized).  Full observability feels like a potential hitch.  In a rebuttal, would love to hear the authors' thoughts on that.", "methods_and_evaluation_criteria": "Sparse exploration is hard, and I appreciate the authors focusing on that. The framework also feels like it could be super general and applied across single-agent RL and MARL.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "MARL research, for better or for worse, hinges on experimental results at the moment.  It's not a negative that there aren't theoretical results in this paper, that's the norm.  That said, the experimental analysis seems to leave off some recent tools for benchmarking like JaxMARL and others.  I am empathetic to how expensive these experiments can be to run, but it's still good to keep up to date on what's out there in a fast-moving field.", "relation_to_broader_scientific_literature": "See experimental section - there are outdated / missing references from the last eighteen or so months.", "essential_references_not_discussed": "See experimental section - there are outdated / missing references from the last eighteen or so months.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qSO0CoESlz", "reviewer": "Reviewer_p7uW", "summary": "The paper presents a curriculum learning method for MBRL, where the task difficulty is controlled by the number of agents, using TD error for learning progress measurement. The method is evaluated on three sparse-reward benchmarks and presents empirical advantages over baselines.\n\n### Update after rebuttals\nThank you authors for the rebuttals. I have two further concerns after reading the rebuttals and other reviewers' comments: \n1. In the rebuttal, the author mentioned that $V_\\text{LB}$ is chosen heuristically according to the converging performance, which is unknown before running the algorithm. It raises questions on how to apply these heuristics in a new environment. \n2. The additional experiments in response to reviewer ffyd show that the advantage of the proposed method compared to prior arts is much smaller compared to environments shown in the original manuscript. \n3. The authors also did not provide the action and state space specifications for these environments, making it hard for readers to interpret the advantage of the method in complex environments. \n\nFor the reasons above, while agent number adjustment is worth investigating for MBRL, I'd maintain my score for the paper in its current form.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer p7uW\nWe thank the reviewer for the appreciation of the novlty of our method. We hope our clarification and new experiments help to address your concerns.\n\n### 1. Clarity on the variance comparison:\n> However, the settings of these two figures can be further explained, including details such as how many samples the variance is computed over. Given that this is a core claim of the paper, analysis on more environments should be shown.\n\nIn Figures 6 and 8, we simply compute the standard deviation of the collected episode returns and TD errors in each iteration, which are collected with the same context samples and used to estimate the objectives of the curriculum learning methods. The number of samples equals to the number of episodes done in the iteration which is at least $25$ episodes in *SMACv2* since some episodes may terminate earlier. \n\nMathematically, assume we have collected a set of contexts $\\\\{c_1, c_2, \\dots, c_n\\\\}, n \\approx 25$, the corresponding episode returns $\\\\{R_{1}, R_{2}, \\dots, R_{n}\\\\}$, and the TD-errors $\\\\{TD_{1}, TD_{2}, \\cdots, TD_{n}\\\\}$ where $TD_{i} = LP(c_i) = \\\\frac{1}{2} E_{s, a \\sim \\pi(a \\vert s, c_i)} \\left[ \\lVert R(s, \\mathbf{a}) - V(s) \\rVert^2 \\right]$, the standard deviation is computed as $\\sigma = \\sqrt{\\frac{1}{n} \\sum_{i=1}^{n} (x_i - \\bar{x})^2}$, where $x_i$ can be $R_{i}$ or $\\text{TD}_i$ and $\\bar{x}$ represents the mean. Since TD-error on each context has been averaged over the samples of the whole episode, it shows lower estimation variance.\n\nWe ran additional experiments and performed similar analysis on two tasks from BenchMARL, i.e. *Balance* and *Wheel*. The results in [link](https://sites.google.com/view/spmarl-icml2025/home#h.cnlpmxrzpuhe) show that our method SPMARL demonstrates much lower standard deviation than the return estimation used in SPRL. In these experiments, the samples are also at least $25$ for each computation of the standard deviation, sometimes it can be more than $25$ when some episodes terminate earlier.\n\nThe variance reduction of using TD error compared to episode returns is also thoroughly analyzed in [1].\n\n[1] Schulman, John, et al. \"High-Dimensional Continuous Control Using Generalized Advantage Estimation.\" Proceedings of the International Conference on Learning Representations (ICLR), 2016.\n\n\n\n### 2. Clarity on choosing $V_{LB}$:\n>The key hyperparameter, $V_{LB}$, is not ablated. How to choose hyperperparameter for new environments?\n\nThanks for raising this issue. We are sorry for not explaining it clearly. $V_{LB}$ is an important hyperparameter in our method. We empirically choose $V_{LB}$ to be $80\\%$ of the final performance. For example, in Protoss 5v5, the converged win rate is around 0.8, we set $V_{LB}$ with 0.6. However, our ablation in [link](https://sites.google.com/view/spmarl-icml2025/home#h.g3dhqeny6kze) shows that $V_{LB}$ can be chosen over a larger range.\n\n\n### 3. Number of random seeds:\n> How many seeds are the experiments in Fig. 3-4 evaluated on?\n\nWe appreciate the reviewer's feedback in highlighting this ambiguity. All our experiments were conducted using five random seeds.\n\n### 4. Discussion on curiosity-driven learning:\n\nThank you for your insightful comment. Curiosity-driven exploration is a well-studied topic in RL, primarily focusing on developing intrinsic reward signals through techniques such as self-prediction, rather than modifying environments as in curriculum learning. However, we believe that general curiosity-driven exploration can be effectively integrated with curriculum design to further enhance exploration.\n\n### 5. Clarity on state, action space\n\nWe thank the reviewer for raising this issue. In the appendix, in the next paper version, we include a detailed introduction of all the tasks used in our experiments.", "claims_and_evidence": "The paper claims that using TD error for learning progress reduces variance of objective estimations, and supports the claim using Figure 8 and 10. However, the settings of these two figures can be further explained, including details such as how many samples the variance is computed over. Given that this is a core claim of the paper, analysis on more environments should be shown.", "methods_and_evaluation_criteria": "The method is evaluated on selected benchmarks.", "theoretical_claims": "No theoretical proofs.", "experimental_designs_or_analyses": "The method is evaluated across 3 benchmarks, each across several seeds.", "relation_to_broader_scientific_literature": "The paper presents novelty by leveraing the number of agents to control task difficulties and using TD errors for learning progress estimation.", "essential_references_not_discussed": "Prior works on intrinsic reward driven curriculum learning, e.g. curiosity-driven learning [1], are not discussed. \n\n[1] Pathak, D., Agrawal, P., Efros, A. A., & Darrell, T. (2017). Curiosity-driven Exploration by Self-supervised Prediction.", "other_strengths_and_weaknesses": "Weaknesses:\n* The key hyperparameter, $V_\\text{LB}$, is not ablated. How to choose hyperperparameter for new environments?", "other_comments_or_suggestions": "* Environment configurations, including state space and action space, should be specified.", "questions_for_authors": "1. How many seeds are the experiments in Fig. 3-4 evaluated on?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wYUn8T5Yz8", "reviewer": "Reviewer_ffyd", "summary": "The authors apply an automatic curriculum design method, SPRL, to multi-agent reinforcement learning, using the number of agents as a parameter to control task difficulty. They call this method SPRLM.\nThey further extend SPRLM to maximise temporal difference error, which they call \"learning progress\", and call this algorithm SPMARL.\nThey evaluate SPRLM and SPMARL on 3 benchmarks against 2 different baselines.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer ffyd\nWe thank the reviewer for the valuable comments. We note that the main concern comes from the comprehensiveness of the benchmarks and inclarity of the method description. In response, we have conducted additional experiments on new benchmarks, and we hope that the enhanced clarity in our revisions will effectively address these concerns.\n\n### 1. More SMACv2 tasks and BenchMARL:\n\nWe have conducted additional experiments on four new SMACv2 tasks: *Terran 5v5*, *Terran 6v6*, *Zerg 5v5*, and *Zerg 6v6*. The results, available at this [link](https://sites.google.com/view/spmarl-icml2025/#h.bj45hu212vx7), demonstrate that our algorithm, SPMARL, consistently outperforms the baselines and generates stable, interpretable curricula across all tasks.\n\nFor the BenchMARL tasks, *Balance* and *Wheel*, our method achieves performance comparable to the baselines, as shown in this [link](https://sites.google.com/view/spmarl-icml2025/home#h.oawhzhwh26av). This may be due to the fact that the exploration challenge in these tasks is not particularly severe, reducing the necessity of curriculum learning. However, we note that SPMARL is the only one generating curriculum that converges to the target task distribution.\n\n\n### 2. Clarity on Section 4, Algorithm 1:\nWe apologize for the lack of clarity in explaining our method. In Section 4, we primarily apply the existing single-agent curriculum learning method, SPRL, to control the number of agents, which we refer to as SPRLM. Our main contribution is the introduction of the TD-error-based learning progress, a novel approach to curriculum learning, as noted by reviewer ZwRM. \n\nIntuitively, our method first identifies tasks, defined by varying numbers of agents, that exhibit the highest learning progress. After training on these high-progress tasks and achieving the performance threshold $V_{LB}$, we adjust the task distribution towards the target by minimizing the $KL$ divergence between current task distribution and the target distribution. This process is constrained by the $KL$ divergence between the old and new task distributions to prevent rapid changes in the task distribution.\n\nWe appreciate the reviewer's valuable suggestions on the organization of the paper. We agree that moving the diagram and comparison table from the appendix to the main text will improve readability. We will revise the paper accordingly based on your helpful recommendations.\n\n### 3. Related literature\n\nWe appreciate the reviewer's suggestion regarding the literature on Unsupervised Environment Design (UED), which falls within the domain of single-agent curriculum learning. UED approaches, such as [1], primarily focus on general environment design using state-action coverage as the objective. In contrast, our work addresses the increased credit assignment difficulty in multi-agent reinforcement learning (MARL) when applying reward-based methods. We acknowledge the close relationship between these fields and will update our paper to include a discussion on this topic.\n\nCompared to single-agent RL, research on curriculum learning in MARL has seen significantly less progress. This is partly because many single-agent curriculum learning methods, such as SPRL and VACL, can be adapted to MARL to control the number of agents. However, our work specifically investigates the challenges that arise in MARL and proposes a TD-error-driven curriculum tailored for MARL.\n\n[1] Teoh, Jayden et al. \"Improving Environment Novelty Quantification for Effective Unsupervised Environment Design.\" Advances in Neural Information Processing Systems 37 (2024).", "claims_and_evidence": "The authors claim that their method \"outperforms state-of-the-art baselines\".\nHowever, the results on their benchmarks are mixed - on the XOR task, they significantly improve on the baselines, on the SimpleSpread task, they slightly improve upon baselines, and on the the SMAC-v2 baseline performance is inline with one of their baselines. They provide mean and std of these results in the appendix.", "methods_and_evaluation_criteria": "The authors present results on 3 benchmarks, SimpleSpread, XOR and the Protoss task from SMAC-v2. \nWhilst SimpleSpread is illustrative of a task in which more-agents does not make the task harder, it seems overly simple, as does XOR. SMAC-v2 has other tasks, Terran and Zerg, which are ignored. \nI am not very familiar with MARL, but it is my understanding that more comprehensive benchmarks exist, such as JaxMARL or MARL Bench.", "theoretical_claims": "No theoretical claims.", "experimental_designs_or_analyses": "See above, I have concerns about the comprehensiveness of the benchmarks.", "relation_to_broader_scientific_literature": "The authors cite 2 papers from 2022, 1 from 2023 and 1 from 2024. All other citations are pre-2022. Whilst I am not expert in MARL, I struggle to believe such little relevant work has been done on curriculum learning in MARL over the last few years.", "essential_references_not_discussed": "The authors miss a discussion of the field of Unsupervised Environment Design (UED) which is a prominent direction of research in curriculum design and reinforcement learning.", "other_strengths_and_weaknesses": "Strengths: Extending curriculum design in MARL to automatically set the number of agents is useful, and applying it a setting such a SimpleSpread that gets easier with more agents is clever. Additionally combining this method with TD error is a neat way to get around sparse rewards.\n\nWeaknesses: The discussion of prior and related work is confusing - it is not clear to what extent the method presented in Section 4 is background or novel to their approach. Further, the actual explanation of the method is extremely difficult to follow, despite Section 4, Algorithm 1 and the diagram in appendix A.  The work would benefit from more time clarifying the background to this method and clarifying section 4 and algorithm 1.\n\nAs above, I am concerned with the suitability of the benchmarks.", "other_comments_or_suggestions": "No further comments", "questions_for_authors": "Are there better benchmarks you can add to this?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper looks at curriculum learning in MARL.  Curriculum learning in the *single*-agent setting, roughly, breaks a task into subtasks and then generates a (fungible, changeable) curriculum over those subtasks to accomplish the task.  In the *multi*-agent setting, which is what this paper looks at, the problem is at a high level the same but the curriculum chosen by one agent might influence those chosen in later stages by other agents, and so on - that is, some form of adversarial reasoning enters into the picture.  This is a complex environment.  This paper posits to add the *number* of agents into the curriculum learning problem as a variable.  That feels natural to this AC (it’s sort of a proxy for the complexity of one important part of the environment), so is worth exploring.  Moving in deeper, the paper looks into credit assignment as the number of agents (that primary variable for CL) changes, and at reducing the variance in the estimate of the value of return ablated across number of agents.

Let this AC start by stating that experimental design in RL is hard, MARL is harder, and doing CL as a function of the number of agents in MARL is even harder.  Lots of empathy there.  That said, let’s bring up some very valid points from reviewers:
- Related work, and discussion therein (all reviewers, especially ffyd and [7uW) - there is not a lot of placement of this work in the current literature, and we’d all appreciate a better discussion there.  This AC is not an expert in CL for MARL, but is well-versed in CL for single-agent RL as well as broadly versed in MARL.  Is this *really* the full set of works that have been published in this space in recent years?  I find it hard to believe that like five papers total have been published in this space in recent years, but after some searching I am inclined to believe that is correct.  If this is the response a set of semi-experts in the area are having to the paper, it’d be good to fix that in the text.
- How do JaxMARL and BenchMARL fit into all of this, *explicitly*?  This AC sees the authors’ rebuttal to ZwRM bringing this up.  Clarify what’s going on, and if there are gaps in a future version of the paper definitely compare against those.  These are becoming standard MARL benchmark suites.
- Questions of generalization around the V_LB parameter - this AC encourages you to add further ablation studies here specifically looking across environments (i.e., generalization) in a future version of the paper.
Decision: Accept (poster)