Title: Accelerate Multi-Agent Reinforcement Learning in Zero-Sum Games with Subgame Curriculum Learning
Venue: NeurIPS 2023
Accepted: no
Keywords: multi-agent reinforcement learning

curriculum learning

zero-sum games
Abstract: Learning Nash equilibrium (NE) in complex zero-sum games with multi-agent reinforcement learning (MARL) can be extremely computationally expensive. Curriculum learning is an effective way to accelerate learning, but an under-explored dimension for generating a curriculum is the difficulty-to-learn of the *subgames* -- games induced by starting from a specific state. In this work, we present a novel subgame curriculum learning framework for zero-sum games. It adopts an adaptive initial state distribution by resetting agents to some previously visited states where they can quickly learn to improve performance. Building upon this framework, we derive a subgame selection metric that approximates the squared distance to NE values and further adopt a particle-based state sampler for subgame generation. Integrating these techniques leads to our new algorithm, *Subgame Automatic Curriculum Learning* (SACL), which is a realization of the subgame curriculum learning framework. SACL can be combined with any MARL algorithm such as MAPPO. Experiments in the particle-world environment and Google Research Football environment show SACL produces much stronger policies than baselines. In the challenging hide-and-seek quadrant environment, SACL produces all four emergent stages and uses only half the samples of MAPPO with self-play. The project website is at https://sites.google.com/view/sacl-neurips. 

Abstract

Learning Nash equilibrium (NE) in complex zero-sum games with multi-agent reinforcement learning (MARL) can be extremely computationally expensive. Curriculum learning is an effective way to accelerate learning, but an under-explored dimension for generating a curriculum is the difficulty-to-learn of the subgames -games induced by starting from a specific state. In this work, we present a novel subgame curriculum learning framework for zero-sum games. It adopts an adaptive initial state distribution by resetting agents to some previously visited states where they can quickly learn to improve performance. Building upon this framework, we derive a subgame selection metric that approximates the squared distance to NE values and further adopt a particle-based state sampler for subgame generation. Integrating these techniques leads to our new algorithm, Subgame Automatic Curriculum Learning (SACL), which is a realization of the subgame curriculum learning framework. SACL can be combined with any MARL algorithm such as MAPPO. Experiments in the particle-world environment and Google Research Football environment show SACL produces much stronger policies than baselines. In the challenging hide-and-seek quadrant environment, SACL produces all four emergent stages and uses only half the samples of MAPPO with self-play. The project website is at https://sites.google.com/view/sacl-neurips .

1 Introduction

Applying reinforcement learning (RL) to zero-sum games has led to enormous success, with trained agents defeating professional humans in Go [41], StarCraft II [46], and Dota 2 [5]. To find an approximate Nash equilibrium (NE) in complex games, these works often require a tremendous amount of training resources including hundreds of GPUs and weeks or even months of time. The unaffordable cost prevents RL from more real-world applications beyond these flagship projects supported by big companies and makes it important to develop algorithms that can learn close-toequilibrium strategies in a substantially more efficient manner.

One way to accelerate training is curriculum learning - training agents in tasks from easy to hard. 27 Many existing works in solving zero-sum games with MARL generate a curriculum by choosing 28 whom to play with. They often use self-play to provide a natural policy curriculum as the agents 29 are trained against increasingly stronger opponents [4, 2]. The self-play framework can be further 30 extended to population-based training (PBT) by maintaining a policy pool and iteratively training new 31 best responses to mixtures of previous policies [31, 23]. Such a policy-level curriculum generation 32 paradigm is very different from the paradigm commonly used in goal-conditioned RL [29, 35]. 33 Most curriculum learning methods for goal-conditioned problems directly reset the goal or initial 34 states for each training episode to ensure the current task is of suitable difficulty for the learning 35 agent. In contrast, the policy-level curriculum in zero-sum games only provides increasingly stronger 36

opponents, and the agents are still trained by playing the full game starting from a fixed initial state 37 distribution, which is often very challenging. 38

39

40

41

42

43

44

45

46

In this paper, we propose a general subgame curriculum learning framework to further accelerate MARL training for zero-sum games. It leverages ideas from goal-conditioned RL. Complementary to policy-level curriculum methods like self-play and PBT, our framework generates subgames (i.e., games induced by starting from a specific state) with growing difficulty for agents to learn and eventually solve the full game. We provide justifications for our proposal by analyzing a simple iterated Rock-Paper-Scissors game. We show that in this game, vanilla MARL requires exponentially many samples to learn the NE. However, by using a buffer to store the visited states and choosing an adaptive order of state-induced subgames to learn, the NE can be learned with linear samples.

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

A key challenge in our framework is to choose which subgame to train on. This is non-trivial in zero-sum games since there does not exist a clear progression metric like the success rate in goalconditioned problems. While the squared difference between the current state value and the NE value can measure the progress of learning, it is impossible to calculate this value during training as the NE is generally unknown. We derive an alternative metric that approximates the squared difference with a bias term and a variance term. The bias term measures how fast the state value changes and the variance term measures how uncertain the current value is. We use the combination of the two terms as the sampling weights for states and prioritize subgames with fast change and high uncertainty.

Instantiating our framework with the state selection metric and a non-parametric subgame sampler, we develop an automatic curriculum learning algorithm for zero-sum games, i.e., Subgame Automatic Curriculum Learning (SACL). SACL can adopt any MARL algorithm as its backbone and preserve the overall convergence property. In our implementation, we choose the MAPPO algorithm [52] for the best empirical performances.

We first evaluate SACL in the Multi-Agent Particle Environment and Google Research Football, where SACL learns stronger policies with lower exploitability than existing MARL algorithms for zero-sum games given the same amount of environment interactions. We then stress-test the efficiency of SACL in the challenging hide-and-seek environment. SACL leads to the emergence of all four phases of different strategies and uses 50% fewer samples than MAPPO with self-play.

2 Preliminary

2.1 Markov game

A Markov game [25] is defined by a tuple MG = ( N , S , A , P, R , γ , ρ ) , where N = { 1 , 2 , · · · , N } is the set of agents, S is the state space, A = Π N i =1 A i is the joint action space with A i being the action space of agent i , P : S × A → ∆ ( S ) is the transition probability function, R = ( R 1 , R 2 , · · · , R N ) : S × A → R n is the joint reward function with R i being the reward function for agent i , γ is the discount factor, and ρ is the distribution of initial states. Given the current state s and the joint action a = ( a 1 , a 2 , · · · , a N ) of all agents, the game moves to the next state s ′ with probability P ( s ′ | s, a ) and agent i receives a reward R i ( s, a ) .

For infinite-horizon Markov games, a subgame MG ( s ) is defined as the Markov game induced by starting from state s , i.e., ρ ( s ) = 1 . Selecting subgames is therefore equivalent to setting the Markov game's initial states. The subgames of finite-horizon Markov games are defined similarly and have an additional variable to denote the current step t .

We focus on two-player zero-sum Markov games, i.e., N = 2 and R 1 ( s, a ) + R 2 ( s, a ) = 0 for all 78 state-action pairs ( s, a ) ∈ S × A . We use the subscript i to denote variables of player i and the 79 subscript -i to denote variables of the player other than i . Each player uses a policy π i : S → A i 80 to produce actions and maximize its own accumulated reward. Given the joint policy π = ( π 1 , π 2 ) , 81 each player's value function of state s and Q-function of state-action pair ( s, a ) are defined as 82

89

round 1

0

round 2

!

!

wins

!

!

loses or draws

0

round n

0

1

……

!

!

wins

!

!

wins

!

!

wins

"

"

"

!

"

#$!

The solution concept of two-player zero-sum Markov games is Nash equilibrium (NE), which is a 83 joint policy where no player can get a higher value by changing its policy alone. 84

Definition 1 (NE) . A joint policy π ∗ = ( π ∗ 1 , π ∗ 2 ) is a Nash equilibrium of a Markov game if for all 85 initial states s 0 with ρ ( s 0 ) > 0 , the following condition holds 86

We use V ∗ i ( · ) to denote the NE value function of player i and Q ∗ i ( · , · ) to denote the NE Q-function of 87 player i , and the following equations hold by definition and the minimax nature of zero-sum games. 88

2.2 MARLalgorithms in zero-sum games

MARL methods have been applied to zero-sum games tracing back to the TD-Gammon project [45]. 90 A large body of work [54, 6, 42, 16] is based on regret minimization, and a well-known result is 91 that the average of policies produced by self-play of regret-minimizing algorithms converges to the 92 NE policy of zero-sum games [15]. Another notable line of work [25, 17, 23, 34] combines RL 93 algorithms with game-theoretic approaches. These works typically use self-play or population-based 94 training to collect samples and then apply RL methods like Q-learning [51] and PPO [39] to learn the 95 NE value functions and policies, and have recently achieved great success [41, 20, 46, 5]. 96

For the analysis in the next section, we introduce a classic MARL algorithm named minimax-Q 97 learning [25] that extends Q-learning to zero-sum games. Initializing functions Q i ( · , · ) with zero 98 values, minimax-Q uses an exploration policy induced by the current Q-functions to collect a batch 99 of samples { ( s t , a t , r t i , s t +1 ) } T t =0 and uses these samples to update the Q-functions by 100

101

102

103

where α is the learning rate. This sample-and-update process continues until the Q-functions converge. Under the assumptions that the state-action sets are discrete and finite and are visited an infinite number of times, it is proved that the stochastic updates by Eq. (6) leads to the NE Q-functions [43].

104

105

106

107

108

109

110

111

112

113

114

3 A motivating example

In this section, we show by a simple illustrative example that vanilla MARL methods like minimax-Q require exponentially many samples to derive the NE. However, if we can dynamically set the initial state distribution and induce an appropriate order of subgames to learn, the sample complexity can be substantially reduced from exponential to linear. Such an observation motivates our proposed algorithm described in later sections.

3.1 Iterated Rock-Paper-Scissors game

We introduce an iterated variant of the Rock-Paper-Scissor (RPS) game, denoted as RPS ( n ) . As shown in Fig. 1, P 1 and P 2 play the RPS game for up to n rounds. If P 1 wins all rounds, it gets a reward of 1 and P 2 gets a reward of -1 . If P 1 loses or draws in any round, the game ends immediately without playing the remaining rounds and both players get zero rewards. Note that the RPS ( n ) game

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

Algorithm 1: Subgame curriculum learning

Input:

state sampler oracle( · ) .

Initialize policy

π

;

repeat

Sample

s

0

∼

oracle(

S

)

;

Rollout

π

in

MG

(

s

0

)

;

Train

π

via MARL;

until π

converges ;

Output:

final policy π .

is different from playing the RPS game repeatedly for n times because players can play less than n rounds and they only receive a non-zero reward if P 1 wins in all rounds. We use s k to denote the state where players have already played k RPS games and are at the k +1 round. It is easy to verify that the NE policy for both players is to play Rock, Paper, or Scissors with equal probability at each state. Under this joint NE policy, P 1 can win one RPS game with 1 / 3 probability, and the probability for P 1 to win all n rounds and get a non-zero reward is 1 / 3 n .

Consider using standard minimax-Q learning to solve the RPS ( n ) game. With Q-functions initialized to zero, we execute the exploration policy to collect samples and perform the update in Eq. (6). Note all state-actions pairs are required to be visited to guarantee convergence to the NE. Therefore, in this sparse-reward game, random exploration will clearly take O (3 n ) steps to get a non-zero reward. Moreover, even if the exploration policy is perfectly set to the NE policy of RPS ( n ) , the probability for P 1 to get the non-zero reward by winning all RPS games is still O (1 / 3 n ) , requiring at least O (3 n ) samples to learn the NE Q-values of the RPS ( n ) game.

3.2 From exponential to linear complexity

An important observation is that the states in later rounds become exponentially rare in the samples generated by starting from the fixed initial state. If we can directly reset the game to these states and design a smart order of minimax-Q updates on the subgames induced by these states, the NE learning can be accelerated significantly. Note that RPS ( n ) can be regarded as the composition of n individual RPS (1) games, a suitable order of learning would be from the easiest subgame RPS (1) starting from state s n -1 to the full game RPS ( n ) starting from state s 0 . Assuming we have full access to the state space, we first reset the game to s n -1 and use minimax-Q to solve subgame RPS (1) with O (1) samples. Given that the NE Q-values of RPS ( k ) are learned, the next subgame RPS ( k +1) is equivalent to an RPS (1) game where the winning reward is the value of state s n -k . By sequentially applying minimax-Q to solve all n subgames from RPS (1) to RPS ( n ) , the number of samples required to learn the NE Q-values is reduced substantially from O (3 n ) to O ( n ) .

In practice, we usually do not have access to the entire state space and cannot directly start from the last subgame RPS (1) . Instead, we can use a buffer to store all visited states and gradually span the state space. By resetting games to the newly visited states, the number of samples required to cover the full state space is still O ( n ) , and we can then apply minimax-Q from RPS (1) to RPS ( n ) . Therefore, the total number of samples is still O ( n ) . The detailed analysis can be found in Appendix A.1. We validate our analysis by running experiments on RPS ( n ) games for n = 1 , · · · , 10 and the results averaged over ten seeds are shown in Fig. 2. It can be seen that the sample complexity reduces from exponential to linear by running minimax-Q over a smart order of subgames, and the result of using a state buffer in practice is comparable to the result with full access.

4 Method

The motivating example suggests that NE learning can be largely accelerated by running MARL 150 algorithms in a smart order over states. Inspired by this insight, we present a general framework to 151 accelerate NE learning in zero-sum games by training over a curriculum of subgames. We further 152 propose two practical techniques to instantiate the framework and present the overall algorithm. 153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

4.1 Subgame curriculum learning

The key issue of the standard sample-and-update framework is that the rollout trajectories always start from the fixed initial state distribution ρ , so visiting states that are most critical for efficient learning can consume a large number of samples. To accelerate training, we can directly reset the environment to those critical states. Suppose we have an oracle state sampler oracle ( · ) that can initiate suitable states for the current policy to learn, i.e., generate appropriate induced subgames, we can derive a general-purpose framework in Alg. 1, which we call subgame curriculum learning. Note that this framework is compatible with any MARL algorithm for zero-sum Markov games.

Adesirable feature of subgame curriculum learning is that it does not change the convergence property of the backbone MARL algorithm, as discussed below.

Proposition 1. If all initial states s 0 with ρ ( s 0 ) > 0 are sampled infinitely often, and the backbone MARL algorithm is guaranteed to converge to an NE in zero-sum Markov games, then subgame curriculum learning also produces an NE of the original Markov game.

The proof can be found in Appendix A.2. Note that such a requirement is easy to satisfy. For example, given any state sampler oracle ( · ) , we can construct a valid mixed sampler by sampling from oracle ( · ) for probability 0 < p < 1 and sampling from ρ for probability 1 -p .

Remark. With a given state sampler, the only requirement of our subgame curriculum learning framework is that the environment can be reset to a desired state to generate the induced game. This is a standard assumption in the curriculum learning literature [13, 29, 35] and is feasible in many RL environments. For environments that do not support this feature, we can simply reimplement the reset function to make them compatible with our framework.

4.2 Subgame sampling metric

A key question is how to instantiate the oracle sampler, i.e., which subgame should we train on for faster convergence ? Intuitively, for a particular state s , if its value has converged to the NE value, that is, V i ( s ) = V ∗ i ( s ) , we should no longer train on the subgame induced by it. By contrast, if the gap between its current value and the NE value is substantial, we should probably train more on the induced subgame. Thus, a simple way is to use the squared difference of the current value and the NE value as the weight for a state and sample states with probabilities proportional to the weights. Concretely, the state weight can be written as

where ˜ V 1 ( s ) = V 1 ( s ) and ˜ V 2 ( s ) = -V 2 ( s ) . The second equality holds because the game is zero-sum and V ∗ 2 ( s ) = -V ∗ 1 ( s ) . With random initialization and different training samples, { ˜ V i } 2 i =1 can be regarded as an ensemble of two value functions and the weight w ( s ) becomes the expectation over the ensemble. The last equality further expands the expectation to a bias term and a variance term, and we sample state with probability P ( s ) = w ( s ) / ∑ s ′ w ( s ′ ) . For the motivating example of RPS ( n ) game, the NE value decreases exponentially from the last state s n -1 to the initial state s 0 . With value functions initialized close to zero, the prioritized subgames throughout training will move gradually from the last round to the first round, which is approximately the optimal order.

However, Eq. (9) is very hard to compute in practice because the NE value is generally unknown. Inspired by Eq. (9), we propose the following alternative state weight

which takes a hyperparameter α and uses the difference between two consecutive value function 193 checkpoints instead of the difference between the NE value and the current value in Eq. (9). The first 194 term in Eq. (10) measures how fast the value functions change over time. If this term is large, the 195 value functions are changing constantly and still far from the NE value; if this term is marginal, the 196 value functions are probably close to the converged NE value. The second term in Eq. (10) measures 197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

the uncertainty of the current learned values and is the same as the variance term in Eq. (9) because V ∗ 1 ( s ) is a constant. If α = 1 , Eq. (10) approximates Eq. (9) as t increases. It is also possible to train an ensemble of value functions for each player to further improve the empirical performance. Additional analysis can be found in Appendix A.3.

Since Eq. (10) does not require the unknown NE value to compute, it can be used in practice as the weight for state sampling and can be implemented for most MARL algorithms. By selecting states with fast value change and high uncertainty, our framework prioritizes subgames where agents' performance can quickly improve through learning.

4.3 Particle-based subgame sampler

With the sample weight at hand, we can generate subgames by sampling initial states from the state space. But it is impractical to sample from the entire space which is usually unavailable and can be exponentially large for complex games. Typical solutions include training a generative adversarial network (GAN) [11] or using a parametric Gaussian mixture model (GMM) [35] to generate states for automatic curriculum learning. However, parametric models require a large number of samples to fit accurately and cannot adapt instantly to the ever-changing weight in our case. Moreover, the distribution of weights is highly multi-modal, which is hard to capture for many generative models.

We instead adopt a particle-based approach and maintain a large state buffer M using all visited states throughout training to approximate the state space. Since the size of the buffer is limited while the state space can be infinitely large, it is important to keep representative samples that are sufficiently far from each other to ensure good coverage of the state space. When the number of states exceeds the buffer's capacity K , we use farthest point sampling (FPS) [36] which iteratively selects the farthest point from the current set of points. In our implementation, we first normalize each dimension of the state and then use the deep graph library package to utilize GPUs for fast and stable FPS results.

4.4 Overall algorithm

Combining the subgame sampling metric and the particle-based sampler, we present a realization of the subgame curriculum learning framework, i.e., the Subgame Automatic Curriculum Learning (SACL) algorithm, which is summarized in Alg. 2.

When each episode resets, we use the particle-based sampler to generate suitable initial states s 0 for the current policy to learn. To satisfy the requirements in Proposition 1, we also reset the game according to the initial state distribution ρ ( · ) with 0 . 3 probability. After collecting a number of samples, we train the policies and value functions using MARL. The weights for the newly collected states are computed according to Eq. (10) and used to update the state buffer M . If the capacity of the state buffer is exceeded, we use FPS to select representative states-weight pairs and delete the others. An overview of SACL in the hide-and-seek game is illustrated in Fig. 3.

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

high

low

sample

weight

initial states

update

self-play

reset

Fort Building

Running and Chasing

Ramp Use

Ramp Defense

sample

weight

initial states

update

self-play

reset

low

high

agent

agent

agent

environment

agent

agent

agent

environment

A

5 Experiment

We evaluate SACL in three different zero-sum environments: Multi-Agent Particle Environment (MPE) [28], Google Research Football (GRF) [22], and the hide-and-seek (HnS) environment [2]. We use a state-of-the-art MARL algorithm MAPPO [52] as the backbone in all experiments.

In zero-sum games, because the performance of one player's policy depends on the other player's policy, the return curve throughout training is no longer a good evaluation method. One way to compare the performance of different policies is to use cross-play, which uses a tournament-style match between any two policies and records the results in a payoff matrix. However, due to the non-transitivity of many zero-sum games [3], winning other policies does not necessarily mean being close to NE policies, so a better way to evaluate the performance of policies is to use exploitability. Given a pair of policies ( π 1 , π 2 ) , the exploitability is defined as

Exploitability can be roughly interpreted as the 'distance' to the joint NE policy. In complex environments like the ones we use, the exact exploitability cannot be calculated because we cannot traverse the policy space to find π ′ i that maximizes the value. Instead, we compute the approximate exploitability by training an approximate best response ˜ π ′ i of the fixed policy π i using MARL.

5.1 Main results

We first compare the performance of SACL in three environments against the following baselines for solving zero-sum games: self-play (SP), two popular variants including Fictitious Self-Play (FSP) [17] and Neural replicator dynamics (NeuRD) [19], and a population-based training method policy-space response oracles (PSRO) [23]. More implementation details can be found in Appendix B.

Multi-Agent Particle Environment. We consider the predator-prey scenario in MPE, where three slower cooperating predators chase one faster prey in a square space with two obstacles. In the default setting, all agents are spawned uniformly in the square. We also consider a harder setting where the predators are spawned in the top-right corner and the prey is spawned in the bottom-left corner. All algorithms are trained for 40M environment samples and the curves of approximate exploitability w.r.t. sample over three seeds are shown in Fig. 4(a) and 4(b). SACL converges faster and achieves lower exploitability than all baselines in both settings, and its advantage is more obvious in the hard scenario. This is because the initial state distribution in corners makes the full game challenging to solve, while SACL generates an adaptive state distribution and learns on increasingly harder subgames to accelerate NE learning. More results and discussions can be found in Appendix C.

Google Research Football. We evaluate SACL in three GRF academy scenarios, namely pass and shoot , run pass and shoot , and 3 vs 1 with keeper . In all scenarios, the left team's agents cooperate

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

(a) MPE: exploitability.

(b) MPE hard: exploitability.

(c) HnS: number of samples.

to score a goal and the right team's agents try to defend them. The first two scenarios are trained for 50M environment samples and the last scenario is trained for 100M samples. Table 1 lists the approximate exploitabilities of different methods' policies over three seeds, and SACL achieves the lowest exploitability. Additional cross-play results and discussions can be found in Appendix C.

Hide-and-seek environment. HnS is a challenging zero-sum game with known NE policies, which makes it possible for us to directly evaluate the number of samples used for NE convergence. We consider the quadrant scenario where there is a room with a door in the lower right corner. Two hiders, one box, and one ramp are spawned uniformly in the environment, and one seeker is spawned uniformly outside the room. Both the box and the ramp can be moved and locked by agents. The hiders aim to avoid the lines of sight from the seeker while the seeker aims to find the hiders.

There is a total of four stages of emergent stages in HnS, i.e., Running and Chasing, Fort Building, Ramp Use, and Ramp Defense. As shown in Fig. 4(c), SACL with MAPPO backbone produces all four stages and converges to the NE policy with only 50% the samples of MAPPO with self-play. We also visualize the initial state distribution to show how SACL selects appropriate subgames for agents to learn. Fig. 5(a) depicts the distribution of hiders' position in the Fort Building stage. The probabilities of states with hiders inside the room are much higher than states with hiders outside, making it easier for hiders to learn to build a fort with the box. Similarly, the distribution of the seeker's position in the Ramp Use stage is shown in Fig. 5(b), and the most sampled subgames start from states where the seeker is close to the walls and is likely to use the ramp.

5.2 Ablation study

We perform ablation studies to examine the effectiveness of the proposed sampling metric and particle-based sampler. All experiments are done in the hard predator-prey scenario of MPE and the results are averaged over three seeds. More ablation studies on state buffer size, subgame sample probability, and other hyperparameters can be found in Appendix C.

Subgame sampling metric. The sampling metric used in SACL follows Eq. (10) which consists of a bias term and a variance term. We compare it with four other metrics including a uniform metric, a bias-only metric, a variance-only metric, and a temporal difference (TD) error metric. The last metric uses the TD error | δ t | = | r t + γ V ( s t +1 ) -V ( s t ) | as the weight, which can be regarded as an estimation of value uncertainty. The results are shown in Fig. 5(c) and the sampling metric used by SACL achieves the best results and outperforms both the bias-only metric and variance-only metric.

State generator and buffer update method. We substitute the particle-based sampler with other state generators including using GAN from the work [11] and using GMM from the work [35]. We also replace the FPS buffer update method with a uniform one that randomly keeps states and a greedy one that keeps states with the highest weights. Results in Fig. 5(c) show that our particle-based sampler with FPS update leads to the fastest convergence and lowest exploitability.

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

(a) Fort Building.

(c) Ablation on metric.

(d) Ablation on generator.

6 Related work

A large number of works achieve faster convergence in zero-sum games by playing against an increasingly stronger policy. The most popular methods are self-play and its variants [18, 1, 21, 34]. Self-play creates a natural curriculum and leads to emergent complex skills and behaviors [4, 2]. Population-based training like double oracle [31] and policy-space response oracles (PSRO) [23] extend self-play by training a pool of policies. Some follow-up works further accelerate training by constructing a smart mixing strategy over the policy pool according to the policy landscape [3, 33, 26, 12]. [30] extends PSRO to extensive-form games by building policy mixtures at all states rather than only the initial states, but it still directly solves the full game starting from some fixed states.

In addition to policy-level curriculum learning methods, other works to accelerate training in zerosum games usually adopt heuristics and domain knowledge like the number of agents [27, 49] or environment specifications [5, 40, 44]. By contrast, our method automatically generates a curriculum over subgames without domain knowledge and only requires the environments can be reset to desired states. Subgame-solving technique [7] is also used in online strategy refinement to improve the blueprint strategy of a simplified abstract game. Another closely related work to our method is [9] which combines backward induction with policy learning, but this method requires knowledge of the game topology and can only be applied to finite-horizon Markov games.

Besides zero-sum games, curriculum learning is also studied in cooperative settings. The problem is often formalized as goal-conditioned RL where the agents need to reach a specific goal in each episode. Curriculum learning methods design or train a smart sampler to generate proper task configurations or goals that are most suitable for training advances w.r.t. some progression metric [10, 14, 13, 37, 29, 35, 11]. Such a metric typically relies on an explicit signal, such as the goal-reaching reward, success rates, or the expected value of the testing tasks. However, in the setting of zero-sum games, these explicit progression metrics become no longer valid since the value associated with a Nash equilibrium can be arbitrary. A possible implicit metric is value disagreement [53] used in goal-reaching tasks, which can be regarded as the variance term in our metric. By adding a bias term, our metric approximates the squared distance to NE values and gives better results in ablation studies.

Our work adopts a non-parametric subgame sampler which is fast to learn and naturally multi-modal, instead of training an expensive deep generative model like GAN [13]. Such an idea has been recently popularized in the literature. Some representative samplers are Gaussian mixture model [50], Stein variational inference [8], Gaussian process [32], or simply evolutionary computation [47, 48]. Technically, our method is also related to prioritized experience replay [38, 14, 24] with the difference that we maintain a buffer [50] to approximate the uniform distribution over the state space.

7 Conclusion

We present SACL, a general algorithm for accelerating MARL training in zero-sum Markov games 333 based on the subgame curriculum learning framework. We propose to use the approximate squared 334 distance to NE values as the sampling metric and use a particle-based sampler for subgames generation. 335 Instead of starting from the fixed initial states, RL agents trained with SACL can practice more on 336 subgames that are most suitable for the current policy to learn, thus boosting training efficiency. We 337 report appealing experiment results that SACL efficiently discovers all emergent strategies in the 338 challenging hide-and-seek environment and uses only half the samples of MAPPO with self-play. We 339 hope SACL can be helpful to speed up prototype development and help make MARL training on 340 complex zero-sum games more affordable to the community. 341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

References

Yu Bai, Chi Jin, and Tiancheng Yu. Near-optimal reinforcement learning with self-play. Advances in neural information processing systems , 33:2159-2170, 2020.

Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent tool use from multi-agent autocurricula. In International Conference on Learning Representations , 2020.

David Balduzzi, Marta Garnelo, Yoram Bachrach, Wojciech Czarnecki, Julien Perolat, Max Jaderberg, and Thore Graepel. Open-ended learning in symmetric zero-sum games. In International Conference on Machine Learning , pages 434-443. PMLR, 2019.

Trapit Bansal, Jakub Pachocki, Szymon Sidor, Ilya Sutskever, and Igor Mordatch. Emergent complexity via multi-agent competition. In International Conference on Learning Representations , 2018.

Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemys ł aw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680 , 2019.

Noam Brown, Adam Lerer, Sam Gross, and Tuomas Sandholm. Deep counterfactual regret minimization. In International conference on machine learning , pages 793-802. PMLR, 2019.

Noam Brown and Tuomas Sandholm. Safe and nested subgame solving for imperfectinformation games. Advances in neural information processing systems , 30, 2017.

Jiayu Chen, Yuanxin Zhang, Yuanfan Xu, Huimin Ma, Huazhong Yang, Jiaming Song, Yu Wang, and Yi Wu. Variational automatic curriculum learning for sparse-reward cooperative multi-agent problems. Advances in Neural Information Processing Systems , 34:9681-9693, 2021.

Weizhe Chen, Zihan Zhou, Yi Wu, and Fei Fang. Temporal induced self-play for stochastic bayesian games. arXiv preprint arXiv:2108.09444 , 2021.

Xi Chen, Diederik P Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, Ilya Sutskever, and Pieter Abbeel. Variational lossy autoencoder. arXiv preprint arXiv:1611.02731 , November 2016.

Patrick Dendorfer, Aljosa Osep, and Laura Leal-Taixé. Goal-gan: Multimodal trajectory prediction based on goal position estimation. In Proceedings of the Asian Conference on Computer Vision , 2020.

Xidong Feng, Oliver Slumbers, Ziyu Wan, Bo Liu, Stephen McAleer, Ying Wen, Jun Wang, and Yaodong Yang. Neural auto-curricula in two-player zero-sum games. Advances in Neural Information Processing Systems , 34:3504-3517, 2021.

Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. In International conference on machine learning , pages 1515-1528. PMLR, 2018.

Carlos Florensa, David Held, Markus Wulfmeier, Michael Zhang, and Pieter Abbeel. Reverse curriculum generation for reinforcement learning. In Conference on robot learning , pages 482-495. PMLR, 2017.

Yoav Freund and Robert E Schapire. Game theory, on-line prediction and boosting. In Proceedings of the ninth annual conference on Computational learning theory , pages 325-332, 1996.

Audr¯ unas Gruslys, Marc Lanctot, Rémi Munos, Finbarr Timbers, Martin Schmid, Julien Perolat, Dustin Morrill, Vinicius Zambaldi, Jean-Baptiste Lespiau, John Schultz, et al. The advantage regret-matching actor-critic. arXiv preprint arXiv:2008.12234 , 2020.

Johannes Heinrich, Marc Lanctot, and David Silver. Fictitious self-play in extensive-form 387 games. In International conference on machine learning , pages 805-813. PMLR, 2015. 388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

Johannes Heinrich and David Silver. Deep reinforcement learning from self-play in imperfectinformation games. arXiv preprint arXiv:1603.01121 , 2016.

Daniel Hennes, Dustin Morrill, Shayegan Omidshafiei, Rémi Munos, Julien Perolat, Marc Lanctot, Audrunas Gruslys, Jean-Baptiste Lespiau, Paavo Parmas, Edgar Duéñez-Guzmán, et al. Neural replicator dynamics: Multiagent learning via hedging policy gradients. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems , pages 492-501, 2020.

Max Jaderberg, Wojciech M Czarnecki, Iain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castaneda, Charles Beattie, Neil C Rabinowitz, Ari S Morcos, Avraham Ruderman, Nicolas Sonnerat, Tim Green, Louise Deason, Joel Z Leibo, David Silver, Demis Hassabis, Koray Kavukcuoglu, and Thore Graepel. Human-level performance in first-person multiplayer games with population-based deep reinforcement learning. arXiv preprint arXiv:1807.01281 , July 2018.

Chi Jin, Qinghua Liu, Yuanhao Wang, and Tiancheng Yu. V-learning-a simple, efficient, decentralized algorithm for multiagent rl. arXiv preprint arXiv:2110.14555 , 2021.

Karol Kurach, Anton Raichuk, Piotr Sta´ nczyk, Micha ł Zaj ˛ ac, Olivier Bachem, Lasse Espeholt, Carlos Riquelme, Damien Vincent, Marcin Michalski, Olivier Bousquet, et al. Google research football: A novel reinforcement learning environment. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 4501-4510, 2020.

Marc Lanctot, Vinicius Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Pérolat, David Silver, and Thore Graepel. A unified game-theoretic approach to multiagent reinforcement learning. Advances in neural information processing systems , 30, 2017.

Yunfei Li, Tao Kong, Lei Li, and Yi Wu. Learning design and construction with varying-sized materials via prioritized memory resets. In 2022 International Conference on Robotics and Automation (ICRA) , pages 7469-7476, 2022.

Michael L Littman. Markov games as a framework for multi-agent reinforcement learning. In Proceedings of the eleventh international conference on machine learning , volume 157, pages 157-163, 1994.

Xiangyu Liu, Hangtian Jia, Ying Wen, Yujing Hu, Yingfeng Chen, Changjie Fan, Zhipeng Hu, and Yaodong Yang. Towards unifying behavioral and response diversity for open-ended learning in zero-sum games. Advances in Neural Information Processing Systems , 34:941-952, 2021.

Qian Long, Zihan Zhou, Abhinav Gupta, Fei Fang, Yi Wu, and Xiaolong Wang. Evolutionary population curriculum for scaling multi-agent reinforcement learning. In International Conference on Learning Representations , 2020.

Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actorcritic for mixed cooperative-competitive environments. In Proceedings of the 31st International Conference on Neural Information Processing Systems , 2017.

Tambet Matiisen, Avital Oliver, Taco Cohen, and John Schulman. Teacher-student curriculum learning. IEEE transactions on neural networks and learning systems , 2019.

Stephen McAleer, John B Lanier, Kevin A Wang, Pierre Baldi, and Roy Fox. Xdo: A double oracle algorithm for extensive-form games. Advances in Neural Information Processing Systems , 34:23128-23139, 2021.

H Brendan McMahan, Geoffrey J Gordon, and Avrim Blum. Planning in the presence of cost functions controlled by an adversary. In Proceedings of the 20th International Conference on Machine Learning (ICML-03) , pages 536-543, 2003.

Bhairav Mehta, Manfred Diaz, Florian Golemo, Christopher J Pal, and Liam Paull. Active domain randomization. In Conference on Robot Learning , pages 1162-1176. PMLR, 2020.

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

Nicolas Perez-Nieves, Yaodong Yang, Oliver Slumbers, David H Mguni, Ying Wen, and Jun Wang. Modelling behavioural diversity for learning in open-ended games. In International Conference on Machine Learning , pages 8514-8524. PMLR, 2021.

Julien Perolat, Bart de Vylder, Daniel Hennes, Eugene Tarassov, Florian Strub, Vincent de Boer, Paul Muller, Jerome T Connor, Neil Burch, Thomas Anthony, et al. Mastering the game of stratego with model-free multiagent reinforcement learning. arXiv preprint arXiv:2206.15378 , 2022.

Rémy Portelas, Cédric Colas, Katja Hofmann, and Pierre-Yves Oudeyer. Teacher algorithms for curriculum learning of deep rl in continuously parameterized environments. In Conference on Robot Learning , pages 835-853. PMLR, 2020.

Charles R. Qi, Li Yi, Hao Su, and Leonidas J. Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in Neural Information Processing Systems , 30, 2017.

Sebastien Racaniere, Andrew K Lampinen, Adam Santoro, David P Reichert, Vlad Firoiu, and Timothy P Lillicrap. Automated curricula through setter-solver interactions. arXiv preprint arXiv:1909.12892 , 2019.

Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint arXiv:1511.05952 , 2015.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Jack Serrino, Max Kleiman-Weiner, David C Parkes, and Josh Tenenbaum. Finding friend and foe in multi-agent games. Advances in Neural Information Processing Systems , 32, 2019.

David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature , 529(7587):484, 2016.

Eric Steinberger, Adam Lerer, and Noam Brown. Dream: Deep regret minimization with advantage baselines and model-free learning. arXiv preprint arXiv:2006.10410 , 2020.

Csaba Szepesvári and Michael L Littman. A unified analysis of value-function-based reinforcement-learning algorithms. Neural computation , 11(8):2017-2060, 1999.

Zhenggang Tang, Chao Yu, Boyuan Chen, Huazhe Xu, Xiaolong Wang, Fei Fang, Simon Du, Yu Wang, and Yi Wu. Discovering diverse multi-agent strategic behavior via reward randomization. arXiv preprint arXiv:2103.04564 , 2021.

Gerald Tesauro et al. Temporal difference learning and td-gammon. Communications of the ACM , 38(3):58-68, 1995.

Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature , 575(7782):350-354, 2019.

Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O Stanley. Poet: open-ended coevolution of environments and their optimized solutions. In Proceedings of the Genetic and Evolutionary Computation Conference , pages 142-151, 2019.

Rui Wang, Joel Lehman, Aditya Rawal, Jiale Zhi, Yulun Li, Jeffrey Clune, and Kenneth Stanley. Enhanced poet: Open-ended reinforcement learning through unbounded invention of learning challenges and their solutions. In International Conference on Machine Learning , pages 9940-9951. PMLR, 2020.

Weixun Wang, Tianpei Yang, Yong Liu, Jianye Hao, Xiaotian Hao, Yujing Hu, Yingfeng Chen, 483 Changjie Fan, and Yang Gao. From few to more: Large-scale dynamic multiagent curriculum 484 learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 485 7293-7300, 2020. 486

487

488

489

David Warde-Farley, Tom Van de Wiele, Tejas Kulkarni, Catalin Ionescu, Steven Hansen, and Volodymyr Mnih. Unsupervised control through non-parametric discriminative rewards. In International Conference on Learning Representations , 2019.

490

491

492

493

494

495

496

497

Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning , 8(3):279-292, 1992.

Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative, multi-agent games. arXiv preprint arXiv:2103.01955 , 2021.

Yunzhi Zhang, Pieter Abbeel, and Lerrel Pinto. Automatic curriculum learning through value disagreement. Advances in Neural Information Processing Systems , 33:7648-7659, 2020.

Martin Zinkevich, Michael Johanson, Michael Bowling, and Carmelo Piccione. Regret minimization in games with incomplete information. Advances in neural information processing systems , 20, 2007.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a general subgame curriculum learning framework to accelerate MARL training for zero-sum games. It adopts an adaptive initial state distribution by resetting agents to some previously visited states where they can quickly learn to improve performance. The author derives a subgame selection metric that approximates the squared distance to NE values and further adopt a particle-based state sampler for subgame generation.\n\nSTRENGTHS:\nThe paper presents a general framework to accelerate NE learning in zero-sum games by training over a curriculum of subgames. The author develops an automatic curriculum learning algorithm, i.e., Subgame Automatic Curriculum Learning, which can adopt any MARL algorithm as its backbone and preserve the overall convergence property. The paper is well written. A motivating example is also described to illustrate the main idea, which is beneficial for readers to understand the work.\n\nWEAKNESSES:\n1. As we can see, the convergent speed is accelerated by the proposed method. However, there is no evidence that the final performance is improved. \n2. The core part of this work is to compute the weight of the state. It would be better to compare with other metrics.\n\nQUESTIONS:\nI mainly have the following concerns.\n1.How about the performance improvement brought by the proposed method?\n2.Would it be better if other metrics, for example, the least visited times of a state, are used solely or combined with the proposed metric in Section4.2? It would be better the see more comparison results about this.\n3.It seems there are mistakes in Eq. (8) and Eq. (9). The index 2 is missing from (7) to (8). How the equation is transformed from (8) to (9).\n\nLIMITATIONS:\nThe author could make a discussion about the application of the proposed method on three or more players.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel subgame curriculum learning framework for accelerating multi-agent reinforcement learning (MARL) in zero-sum Markov games. The framework uses an adaptive initial state distribution to induce subgames of varying difficulty for agents to learn, and leverages a sampling metric that approximates the squared distance to Nash equilibrium (NE) values to prioritize subgames with fast value change and high uncertainty. The paper instantiates the framework with a particle-based state sampler and integrates it with any MARL algorithm, resulting in the Subgame Automatic Curriculum Learning (SACL) algorithm. The paper evaluates SACL in three zero-sum environments and show that it converges faster and achieves lower exploitability than existing methods.\n\nSTRENGTHS:\n- The paper addresses an important and challenging problem of reducing the computational cost of solving complex zero-sum games with MARL, which has many potential applications and implications.\n- The paper provides a illustrative motivating example to justify the effectiveness of the subgame curriculum learning framework, which leverages ideas from goal-conditioned RL and prioritized experience replay.\n- The paper is able to provide practical methods to approximately solve the hard parts of the theoretical work. For example, the paper introduces a novel subgame sampling metric that approximates the squared distance to NE values with a bias term and a variance term. The paper also adopts a particle-based state sampler that is compatible with most MARL algorithms.\n- The paper conducts extensive experiments on three different zero-sum environments and demonstrates that SACL can converge to NE policies with substantially fewer samples. The ablation study part is able to demonstrate how well the approximation methods in SACL work comparing to other alternatives.\n\nWEAKNESSES:\n- The paper assumes that the environment can be reset to any desired state to generate induced subgames, which may not be feasible or realistic in some settings.\n- In Section 5.1, it is not clear how the approximate exploitability is computed using MARL methods for best response training. What are the exact algorithms used for this purpose? How many samples are used for each best response training? How reliable are these estimates? Some details on these aspects would help evaluate the results more fairly.\n- In Section 5.2, it would be helpful to report some quantitative results on how different choices of state buffer size, hyperparameters alpha, subgame sample probability, or other hyperparameters affect SACL's performance or convergence.\n- It would also be helpful to have more details on how to implement FPS for buffer update and how to measure the distance between states.\n\nQUESTIONS:\n- In the subgame sampling metric, the difference between two consecutive value function checkpoints is used to approximate the difference between the NE value and the current value. What if the value function goes into a bad local minimum and stops moving? Has this been observed in any experiment?\n- In Section 5.2, the paper reports that the subgame sampling metric works better than the TD error in practice. Since the value function is often updated based on TD error between each iteration, what is the intuition of the subgame sampling metric being better?\n- How does SACL handle cases where there are multiple NEs in a game? Does it converge to one randomly?\n- How generalizable is SACL to other types of games such as cooperative games or general-sum games? What modifications or extensions are needed to adapt SACL to these settings?\n\nLIMITATIONS:\n- The paper requires access to full state information and reset function for each environment.\n- The paper relies on approximate exploitability as a proxy for measuring closeness to NE policies. However, approximate exploitability may not reflect the true performance gap between different policies due to sample variance or approximation errors.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a subgame curriculum learning framework to accelerate multi-agent reinforcement learning (MARL) training for zero-sum games.  The framework adopts an adaptive initial state distribution by resetting agents to some previously visited states where they can quickly learn to improve performance. The paper derives a subgame selection metric that approximates the squared distance to Nash equilibrium (NE) values and further adopts a particle-based state sampler for subgame generation.  Experiments in the particle-world environment, Google Research Football environment, and hide-and-seek show that SACL produces much stronger policies than baselines.\n\nSTRENGTHS:\n- The paper provides a detailed analysis of the proposed approach and justifies it through experiments and analysis of a simple iterated Rock-Paper-Scissors game.\n- The paper provides a clear and concise explanation of the proposed approach and its implementation.\n- SACL is shown to produce much stronger policies than baselines in experiments conducted in the particle-world environment and Google Research Football environment.\n\nWEAKNESSES:\n- There is a gap between the motivated iterated Rock-Paper-Scissors game and the experiments conducted in the particle-world environment and Google Research Football. Since state and action space is discrete and finite in RPS, while these spaces are continuous in MPE and GRF. Also, The state is not communicative in RPS while it is communicative in MPE or GRF. In one rollout, two states are communicative if and only if each is reachable from the other with nonzero probability in a finite number of steps. So it is ok to use samples in RPS since it is a trade of time and space. While the state in MPE or GRF is something like position or speed, the advantage shown in the motivated scenario is hard to extend to more realistic environments.\n- The link between eq.9 and eq.10 is weak, which should be the key to the theoretical contribution of the paper. I have several concerns about eq.10.\n\t- The estimated value function at different checkpoints represents different policies' values. Do the authors consider the missing $\\pi$ inside the value function?\n\t- Does the asymptotic convergence of the estimated value function of a few sampled states mean the policies converge to NE? The explanation in Appendix is not theoretical and less convincing by giving examples.\n- The subgame sampler is also essential, why the farthest point should be sampled first?\n- Go is a good baseline for SACL. Since it is a fair game. To my understanding, SACL can be seen as an alternative to MCTS. One can start from the arbitrary state of a Go board, rollout, and then train the policy and value function. From this point of view, only having one rollout in Alg.2 might cause a large variance in the value function.\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nThe paper does not explicitly mention any limitations of the proposed method. The paper has no potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper presents an algorithm (SACL) for accelerating MARL training in zero-sum Markov games based on the subgame curriculum learning framework. A sampling metric based on approximated squared distance to NE and a particle-based sampler are proposed to sample states for subgame generation. Experiment results show the superiority of SACL in producing stronger policies and boosting training efficiency.

STRENGTHS:
1. The paper is well-written and easy to follow. The general idea of the algorithm is well illustrated via a toy example: the iterated RPS games. 
2. The experiment results show both the efficiency and effectivity of SACL.

WEAKNESSES:
* I am not sure the game considered in this paper is perfect-information or imperfect-information. From the Rock-Paper-Scissor example and the baseline methods considered (NeuRD and PSRO), I am aussming the case of imperfect-information.  If it is the case of imperfect-information, the optimal policy for a subgame depends on the distribution of hidden information. Yet, I don't see any component of SACL that deals with this distribution.  For instance, the DeepStack (a poker AI) algorithm uses the agent's range and opponent counterfactual values when resolving a subgame in poker. 


* Also, the choice of the metric and oracle sampler appears ad-hoc to me.
  1. Why can we  use Equ 10 to approximate Equ 9? Is this estimator unbiased or with low variance?
  2. In Equ 10, is the algorithm stable for different choices of \alpha? It would be good to see some discussion or ablation study on it.

QUESTIONS:
How to define the distance between two points when using FPS to select the points?

LIMITATIONS:
Please see weaknesses.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-12

comments: SUMMARY:
This paper proposes a framework for learning Nash equilibria in zero-sum Markov games based on subgame curriculum learning. Novel sampling metrics for subgames generation are proposed. The proposed SACL algorithm is able to achieve equal performance at lower  sample complexity compared with self-play algorithms.

STRENGTHS:
- Proposed a novel framework for learning NE in zero-sum games. This can be combined with different RL algorithms to produce learning mechanisms with lower sample complexity.
- The presentation of the paper is clear.

WEAKNESSES:
- While the baseline methods require an exponential complexity, the proposed SACL costs approximately half, which is a limited practical improvement especially considering real-world applications as the authors suggested.

QUESTIONS:
- Are there potential ways to enlarge the complexity improvement? This could significantly improve the impact of the proposed method.

I am not familiar with this field.

LIMITATIONS:
A more thorough discussion of the limitations and future steps could be beneficial.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-12