Title: Co-Learning Empirical Games and World Models
Venue: NeurIPS 2023
Accepted: no
Keywords: Multiagent learning

Empirical Game Theory

Model-Based Reinforcement Learning

Transfer Learning
Abstract: Game-based decision-making involves reasoning over both world dynamics and strategic interactions among the agents. Typically, empirical models capturing these respective aspects are learned and used separately. We investigate the potential gain from co-learning these elements: a world model for dynamics and an empirical game for strategic interactions. Empirical games drive world models toward a broader consideration of possible game dynamics induced by a diversity of strategy profiles. Conversely, world models guide empirical games to efficiently discover new strategies through planning. We demonstrate these benefits first independently, then in combination as realized by a new algorithm, Dyna-PSRO, that co-learns an empirical game and a world model. When compared to PSRO---a baseline empirical-game building algorithm, Dyna-PSRO is found to compute lower regret solutions on partially observable general-sum games. In our experiments, Dyna-PSRO also requires substantially fewer experiences than PSRO, a key algorithmic advantage for settings where collecting player-game interaction data is a cost-limiting factor.

1

Abstract

Game-based decision-making involves reasoning over both world dynamics and strategic interactions among the agents. Typically, empirical models capturing these respective aspects are learned and used separately. We investigate the potential gain from co-learning these elements: a world model for dynamics and an empirical game for strategic interactions. Empirical games drive world models toward a broader consideration of possible game dynamics induced by a diversity of strategy profiles. Conversely, world models guide empirical games to efficiently discover new strategies through planning. We demonstrate these benefits first independently, then in combination as realized by a new algorithm, Dyna-PSRO, that co-learns an empirical game and a world model. When compared to PSRO-a baseline empirical-game building algorithm, Dyna-PSRO is found to compute lower regret solutions on partially observable general-sum games. In our experiments, DynaPSRO also requires substantially fewer experiences than PSRO, a key algorithmic advantage for settings where collecting player-game interaction data is a costlimiting factor.

1 Introduction

Even seemingly simple games can actually embody a level of complexity rendering them intractable to direct reasoning. This complexity stems from the interplay of two sources: dynamics of the game environment, and strategic interactions among the game's players. As an alternative to direct reasoning, models have been developed to facilitate reasoning over these distinct aspects of the game. Empirical games capture strategic interactions in the form of payoff estimates for joint policies [80]. World models represent a game's transition dynamics and reward signal directly [69, 19]. Whereas each of these forms of model have been found useful for game reasoning, typical use in prior work has focused on one or the other, learned and employed in isolation from its natural counterpart.

Co-learning both models presents an opportunity to leverage their complementary strengths as a 25 means to improve each other. World models predict successor states and rewards given a game's 26 current state and action(s). However, their performance depends on coverage of their training data, 27 which is limited by the range of strategies considered during learning. Empirical games can inform 28 training of world models by suggesting a diverse set of salient strategies, based on game-theoretic 29 reasoning [80]. These strategies can expose the world model to a broader range of relevant dynamics. 30 Moreover, as empirical games are estimated through simulation of strategy profiles, this same 31 simulation data can be reused as training data for the world model. 32

Strategic diversity through empirical games, however, comes at a cost. In the popular framework 33

of Policy-Space Response Oracles (PSRO) [38], empirical normal-form game models are built 34

iteratively, at each step expanding a restricted strategy set by computing best-response policies to 35

the current game's solution. As computing an exact best-response is generally intractable, PSRO 36

uses Deep Reinforcement Learning (DRL) to compute approximate response policies. However, 37

each application of DRL can be considerably resource-intensive, necessitating the generation of 38

a vast amount of gameplays for learning. Whether gameplays, or experiences, are generated via 39 simulation [48] or from real-world interactions [24], their collection poses a major limiting factor in 40 DRL and by extension PSRO. World models present one avenue to reduce this cost by transferring 41 previously learned game dynamics across response computations. 42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

World

Model

Empirical

Game

Dyna-PSRO

Strategic Diversity

Planning

Weinvestigate the mutual benefits of co-learning a world model and an empirical game by first verifying the potential contributions of each component independently. We then show how to realize the combined effects in a new algorithm, Dyna-PSRO , that co-learns a world model and an empirical game (illustrated in Figure 1). Dyna-PSRO extends PSRO to learn a world model concurrently with empirical game expansion, and applies this world model to reduce the computational cost of computing new policies. This is implemented by a Dyna-based reinforcement learner [67, 68] that integrates planning, acting, and learning in parallel. Dyna-PSRO is evaluated against PSRO on a collection of partially observable general-sum games. In our experiments, Dyna-PSRO found lower-regret solutions while requiring substantially fewer cumulative experiences.

The main points of novelty of this paper are as follows: (1) empirically demonstrate that world models benefit from the strategic diversity induced by an empirical game; (2) empirically demonstrate that a world model can be effectively transferred and used in planning with new other-players. The major contribution of this work is a new algorithm, Dyna-PSRO, that co-learns an empirical game and world model finding a stronger solution at less cost than the baseline, PSRO.

2 Related Work

Empirical Game Theoretic Analysis (EGTA). The core idea of EGTA [80] is to reason over approximate game models ( empirical games ) estimated by simulation over a restricted strategy set. This basic approach was first demonstrated by Walsh et al. [77], in a study of pricing and bidding games. Phelps et al. [51] introduced the idea of extending a strategy set automatically through optimization, employing genetic search over a policy space. Schvartzman & Wellman [58] proposed using RL to derive new strategies that are approximate best responses (BRs) to the current empirical game's Nash equilibrium. The general question of which strategies to add to an empirical game has been termed the strategy exploration problem [31]. PSRO [38] generalized the target for BR beyond NE, and introduced DRL for BR computation in empirical games. Many further variants and extensions of EGTA have been proposed, for example those using structured game representations such as extensive-form [43, 34]. Some prior work has considered transfer learning across BR computations in EGTA, specifically by reusing elements of policies and value functions [64, 65].

Model-Based Reinforcement Learning (MBRL). Model-Based RL algorithms construct or use 80 a model of the environment (henceforth, world model ) in the process of learning a policy or value 81 function [69]. World models may either predict successor observations directly (e.g., at pixel 82 level [76, 79]), or in a learned latent space [18, 17]. The world models can be either used for 83 background planning by rolling out model-predicted trajectories to train a policy, or by decision84 time planning where the world model is used to evaluate the current state by planning into the 85 future. Talvitie [71] demonstrated that even in small Markov decision processes (MDP) [52], model86 prediction errors tend to compound-rendering long-term planning at the abstraction of observations 87 ineffective. A follow-up study demonstrated that for imperfect models, short-term planning was 88 no better than repeatedly training on previously collected real experiences; however, medium-term 89 planning offered advantages even with an imperfect model [27]. Parallel studies hypothesized that 90 these errors are a result of insufficient data for that transition to be learned [36, 8]. To remedy 91 the data insufficiency, ensembles of world models were proposed to account for world model 92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

uncertainty [8, 36, 84], and another line of inquiry used world model uncertainty to guide exploration in state-action space [3, 59]. This study extends this problem into the multiagent setting, where now other-agents may preclude transitions from occurring. The proposed remedy is to leverage the strategy exploration process of building an empirical game to guide data generation.

Multiagent Reinforcement Learning (MARL). Previous research intersecting MARL and MBRL has primarily focused on modeling the opponent, particularly in scenarios where the opponent is fixed and well-defined. Within specific game sub-classes, like cooperative games and two-player zero-sum games, it has been theoretically shown that opponent modeling reduces the sample complexity of RL [73, 85]. Opponent models can either explicitly [46, 15] or implicitly [4, 29] model the behavior of the opponent. Additionally, these models can either construct a single model of opponent behavior, or learn a set of models [12, 21]. While opponent modeling details are beyond the scope of this study, readers can refer to Albrecht & Stone's survey [1] for a comprehensive review on this subject. Instead, we consider the case where the learner has explicit access to the opponent's policy during training, as is the case in empirical-game building. A natural example is that of Self-Play, where all agents play the same policy; therefore, a world model can be learned used to evaluate the quality of actions with Monte-Carlo Tree Search [60, 62, 72, 56]. Li et al. [41] expands on this by building a population of candidate opponent policies through PSRO to augment the search procedure. Krupnik et al. [35] demonstrated that a generative world model could be useful in multi-step opponent-action prediction. Sun et al. [66] examined modeling stateful game dynamics from observations when the agents' policies are stationary. Chockalingam et al. [11] explored learning world models for homogeneous agents with a centralized controller in a cooperative game. World models may also be shared by independent reinforcement learners in cooperative games [81, 86].

3 Co-Learning Benefits

We begin by specifying exactly what we mean by world model and empirical game. This requires defining some primitive elements. Let t ∈ T denote time in the real game, with s t ∈ S the information state and h t ∈ H the game state at time t . The information state s t ≡ ( m π,t , o t ) is composed of the agent's memory m π ∈ M π , or recurrent state, and the current observation o ∈ O . Subscripts denote a player-specific component s i , negative subscripts denote all but the player s -i , and boldface denote the joint of all players s . The transition dynamics p : H× A → ∆( H ) × ∆( R ) define the game state update and reward signal. The agent experiences transitions , or experiences , ( s t , a t , r t +1 , s t +1 ) of the game; where, sequences of transitions are called trajectories τ and trajectories ending in a terminal game state are episodes .

At the start of an episode, all players sample their current policy π from their strategy σ : Π → [0 , 1] , where Π is the policy space and Σ is the corresponding strategy space . A utility function U : Π → R n defines the payoffs/returns (i.e., cumulative reward) for each of n players. The tuple Γ ≡ ( Π , U, n ) defines a normal-form game (NFG) based on these elements. We represent empirical games in normal form. An empirical normal-form game (ENFG) ˆ Γ ≡ ( ˆ Π , ˆ U,n ) models a game with a restricted strategy set ˆ Π and an estimated payoff function ˆ U . An empirical game is typically built by alternating between game reasoning and strategy exploration. During the game reasoning phase, the empirical game is solved based on a solution concept predefined by the modeler. The strategy exploration step uses this solution to generate new policies to add to the empirical game. One common heuristic is to generate new policies that best-respond to the current solution [45, 57]. As exact best-responses typically cannot be computed, RL or DRL are employed to derive approximate best-responses [38].

An agent world model w represents dynamics in terms of information available to the agent. Specifically, w maps information states and actions to observations and rewards, w : O × A ×M w → O × R , where m w ∈ M w is the world model's memory , or recurrent state. For simplicity, in this work, we assume the agent learns and uses a deterministic world model, irrespective of stochasticity that may be present in the true game. Specific implementation details for this work are provided in Appendix C.2.

Until now, we have implicitly assumed the need for distinct models. However, if a single model could serve both functions, co-learning two separate models would not be needed. Empirical games, in general, cannot replace a world model as they entirely abstract away any concept of game dynamics.

Conversely, world models have the potential to substitute for the payoff estimations in empirical games by estimating payoffs as rollouts with the world model. We explore this possibility in an

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

auxiliary experiment included in Appendix E.4, but our findings indicate that this substitution is impractical. Due to compounding of model-prediction errors, the payoff estimates and entailed game solutions were quite inaccurate.

Having defined the models and established the need for their separate instantiations, we can proceed to evaluate the claims of beneficial co-learning. Our first experiment shows that the strategic diversity embodied in an empirical game yields diverse game dynamics, resulting in the training of a more performant world model. The second set of experiments demonstrates that a world model can help reduce the computational cost of policy construction in an empirical game.

3.1 Strategic Diversity

Aworld model is trained to predict successor observations and rewards, from the current observations and actions, using a supervised learning signal. Ideally, the training data would cover all possible transitions. This is not feasible, so instead draws are conventionally taken from a dataset generated from play of a behavioral strategy . Performance of the world model is then measured against a target strategy . Differences between the behavioral and target strategies present challenges in learning an effective world model.

We call the probability of drawing a state-action pair under some strategy its reach probability . From this, we define a strategy's strategic diversity as the distribution induced from reach probabilities. across the full state-action space. These terms allow us to observe two challenges for learning world models. First, the diversity of the behavioral strategy ought to cover the target strategy's diversity. Otherwise, transitions will be absent from the training data. It is possible to supplement coverage of the absent transitions if they can be generalized from covered data; however, this cannot be generally guaranteed. Second, the closer the diversities are, the more accurate the learning objective will be. An extended formal argument of these challenges is provided in Appendix C.3.

If the target strategy were known, we could readily construct the ideal training data for the world model. However the target is generally not known at the outset; indeed determining this target is the ultimate purpose of empirical game reasoning. The evolving empirical game essentially reflects a search for the target. Serendipitously, construction of this empirical game entails generation of data that captures elements of likely targets. This data can be reused for world model training without incurring any additional data collection cost.

Game. We evaluate the claims of independent co-learning benefits within the context of a commons game called 'Harvest'. In Harvest, players move around an orchard picking apples. The challenging commons element is that apple regrowth rate is proportional to nearby apples, so that socially optimum behavior would entail managed harvesting. Self-interested agents capture only part of the benefit of optimal growth, thus non-cooperative equilibria tend to exhibit collective over-harvesting. The game has established roots in human-behavioral studies [30] and in agent-based modeling of emergent behavior [53, 40, 39]. For our initial experiments, we use a symmetric two-player version of the game, where in-game entities are represented categorically [28]. Each player has a 10 × 10 viewbox within their field of vision. The possible actions include moving in the four cardinal directions, rotating either way, tagging, or remaining idle. A successful tag temporarily removes the other player from the game, but can only be done to other nearby players. Players receive a reward of 1 for each apple picked. More detailed information and visualizations are available in Appendix D.1.

Experiment. To test the effects of strategic diversity, we train a suite of world models that differ in the diversity of their training data. The datasets are constructed from the play of three policies: a random baseline policy, and two PSRO-generated policies. The PSRO policies were arbitrarily sampled from an approximate solution produced by a run of PSRO. We sampled an additional policy from PSRO for evaluating the generalization capacity of the world models. These policies are then subsampled and used to train seven world models. The world models are referred to by icons that depict the symmetric strategy profiles used to train them in the normal-form. Strategy profiles included in the training data of the world models are shaded black. For instance, the first (random) policy , or the first and third policies . Each world model's dataset contains 1 million total transitions, collected uniformly from each distinct strategy profile (symmetric profiles are not re-sampled). The world models are then evaluated on accuracy and recall for their predictions of both

observation and reward for both players. The world models are optimized with a weighted-average 199 cross-entropy objective. Additional details are in Appendix C.2. 200

Observation

0.27

±

0.04

0.75

±

0.02

0.80

±

0.02

0.58

±

0.05

0.83

±

0.02

0.62

±

0.05

0.68

±

0.04

Reward

0.73

±

0.08

0.52

±

0.10

0.53

±

0.10

0.68

±

0.08

0.50

±

0.10

0.68

±

0.08

0.69

±

0.08

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Profiles Sampled To Train World Model

Results. Figure 2 presents each world model's per-profile accuracy, as well as its average over all profiles. Inclusion of the random policy corresponds to decreases in observation prediction accuracy: 0 . 75 ± 0 . 02 → 0 . 58 ± 0 . 05 , 0 . 80 ± 0 . 02 → 0 . 62 ± 0 . 05 , and 0 . 83 ± 0 . 02 → 0 . 68 ± 0 . 04 . Figure 13 (Appendix E.1) contains the world model's per-profile recall. Inclusion of the random policy corresponds to increases in reward 1 recall: 0 . 25 ± 0 . 07 → 0 . 37 ± 0 . 11 , 0 . 25 ± 0 . 07 → 0 . 36 ± 0 . 11 , and 0 . 26 ± 0 . 07 → 0 . 37 ± 0 . 11 .

Discussion. The PSRO policies offer the most strategically salient view of the game's dynamics. Consequently, the world model trained with these policies yields the highest observation accuracy. However, this world model performs poorly on reward accuracy, scoring only 0 . 50 ± 0 . 10 . In comparison, the model trained on the random policy scores 0 . 73 ± 0 . 08 . This seemingly counterintuitive result can be attributed to a significant class imbalance in rewards. predicts only the most common class, no reward, which gives the illusion of higher performance. In contrast, the remaining world models attempt to predict rewarding states, which reduces their overall accuracy. Therefore, we should compare the world models based on their ability to recall rewards. When we examine again, we find that it also struggles to recall rewards, scoring only 0 . 26 ± 0 . 07 . However, when the random policy is included in the training data ( ), the recall improves to 0 . 37 ± 0 . 11 . This improvement is also due to the same class imbalance. The PSRO policies are highly competitive, tending to over-harvest. This limits the proportion of rewarding experiences. Including the random policy enhances the diversity of rewards in this instance, as its coplayer can demonstrate successful harvesting. Given the importance of accurately predicting both observations and rewards for effective planning, appears to be the most promising option. However, the strong performance of suggests future work on algorithms that can benefit solely from observation predictions. Overall, these results support the claim that strategic diversity enhances the training of world models.

3.2 Response Calculations

Empirical games are built by iteratively calculating and incorporating responses to the current solution. However, direct computation of these responses is often infeasible, so RL or DRL is used to approximate the response. This process of approximating a single response policy using RL is computationally intensive, posing a significant constraint in empirical game modeling when executed repeatedly. World models present an opportunity to address this issue. A world model can serve as a medium for transferring previously learned knowledge about the game's dynamics. Therefore, the dynamics need not be relearned, reducing the computational cost associated with response calculation.

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

Exercising a world model for transfer is achieved through a process called planning . Planning is any procedure that takes a world model and produces or improves a policy. In the context of games, planning can optionally take into account the existence of coplayers. This consideration can reduce experiential variance caused by unobserved confounders (i.e., the coplayers). However, coplayer modeling errors may introduce further errors in the planning procedure [21].

Planning alongside empirical-game construction allows us to side-step this issue as we have direct access to the policies of all players during training. This allows us to circumvent the challenge of building accurate agent models. Instead, the policies of coplayers can be directly queried and used alongside a world model, leading to more accurate planning. In this section, we empirically demonstrate the effectiveness of two methods that decrease the cost of response calculation by integrating planning with a world model and other agent policies.

3.2.1 Background Planning

The first type of planning that is investigated is background planning , popularized by the Dyna architecture [67]. In background planning, agents interact with the world model to produce planned experiences 1 . The planned experiences are then used by a model-free reinforcement learning algorithm as if they were real experiences (experiences generated from the real game). Background planning enables learners to generate experiences of states they are not currently in.

Experiment. To assess whether planned experiences are effective for training a policy in the actual game, we compute two response policies. The first response policy, serving as our baseline, learns exclusively from real experiences. The second response policy, referred to as the planner, is trained using a two-step procedure. Initially, the planner is exclusively trained on planned experiences. After 10 000 updates, it then transitions to learning solely from real experiences. Policies are trained using IMPALA [14], with further details available in Appendix C.1. The planner employs the world model from Section 3.1, and the opponent plays the previously held-out policy. In this and subsequent experiments, the cost of methods is measured by the number of experiences they require with the actual game. This is because, experience collection is often the bottleneck when applying RL-based methods [48, 24]. Throughout the remainder of this work, each experience represents a trajectory of 20 transitions, facilitating the training of recurrent policies.

0

.

0

0

.

5

1

.

0

Real Exp.

10

6

0

10

20

30

40

Return

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

Plan Exp.

Real Exp.

10

6

Baseline

Plan: Real

Plan: Model

×

×

Results. Figure 3 presents the results of the background planning experiment. The methods are 261 compared based on their final return, utilizing an equivalent amount of real experiences. The baseline 262 yields a return of 23 . 00 ± 4 . 01 , whereas the planner yields a return of 31 . 17 ± 0 . 25 . 263

Discussion. In this experiment, the planner converges to a stronger policy, and makes earlier gains 264 in performance than the baseline. Despite this, there is a significant gap in the planner's learning 265

curves, which are reported with respect to both the world model and real game. This gap arises due 266 to accumulated model-prediction errors, causing the trajectories to deviate from the true state space. 267 Nevertheless, the planner effectively learns to interact with the world model during planning, and 268 this behavior shows positive transfer into the real game, as evidenced by the planner's rapid learning. 269 The exact magnitude of benefit will vary across coplayers' policies, games, and world models. In 270 Figure 14 (Appendix E.2), we repeat the same experiment with the poorly performing world 271 model, and observe a marginal benefit ( 26 . 05 ± 1 . 32 ). The key take-away is that background planning 272 tends to lead towards learning benefits, and not generally hamper learning. 273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

3.2.2 Decision-Time Planning

The second main way that a world model is used is to inform action selection at decision time [planning] (DT) . In this case, the agent evaluates the quality of actions by comparing the value of the model's predicted successor state for all candidate actions. Action evaluation can also occur recursively, allowing the agent to consider successor states further into the future. Overall, this process should enable the learner to select better actions earlier in training, thereby reducing the amount of experiences needed to compute a response. A potential flaw with decision-time planning is that the agent's learned value function may not be well-defined on model-predicted successor states [71]. To remedy this issue, the value function should also be trained on model-predicted states.

Experiment. To evaluate the impact the decision-time planning, we perform an experiment similar to the background planning experiment (Section 3.2.1). However, in this experiment, we evaluate the quality of four types of decision-time planners that perform one-step three-action search. The planners differ in the their ablations of background planning types: (1) warm-start background planning (BG: W) learning from planned experiences before any real experiences, and (2) concurrent background planning (BG: C) where after BG: W, learning proceeds simultaneously on both planned and real experiences. The intuition behind BG: C is that the agent can complement its learning process by incorporating planned experiences that align with its current behavior, offsetting the reliance on costly real experiences. Extended experimental details are provided in Appendix C.

0

.

0

0

.

5

1

.

0

Real Exp.

10

6

0

10

20

30

40

50

Return

0

.

0

0

.

5

1

.

0

1

.

5

2

.

0

2

.

5

Real & Planned Exp.

10

6

DT

/check

/check

/check

/check

BG: W

/check

/check

BG: C

/check

/check

×

×

Results. The results for this experiment are shown in Figure 4. The baseline policy receives a final return of 23 . 00 ± 4 . 01 . The planners that do not include BG: W, perform worse, with final returns of 9 . 98 ± 7 . 60 (DT) and 12 . 42 ± 3 . 97 (DT & BG: C). The planners that perform BG: W outperform the baseline, with final returns of 44 . 11 ± 2 . 81 (DT & BG: W) and 44 . 31 ± 2 . 56 (DT, BG: W, & BG: C).

Discussion. Our results suggest that the addition of BG: W provides sizable benefits: 9 . 98 ± 7 . 60 (DT) → 44 . 11 ± 2 . 81 (DT & BG:W) and 12 . 42 ± 3 . 97 (DT & BG: C) → 44 . 31 ± 2 . 56 (DT, BG: W, &BG: C). We postulate that this is because it informs the policy's value function on model-predictive states early into training. This allows that the learner is able to more effectively search earlier into training. BG: C appears to offer minor stability and variance improvements throughout the training

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

procedure; however, it does not have a measurable difference in final performance. This result suggests using planning methods in combination to reap their respective advantages.

However, we caution against focusing on the magnitude of improvement found within this experiment. As the margin of benefit depends on many factors including the world model accuracy, the opponent policy, and the game. To exemplify, similar to the background planning section, we repeat the same experiment with the poorly performing world model. The results of this ancillary experiment are in Figure 15 (Appendix E.3). The trend of BG: W providing benefits was reinforced: 6 . 29 ± 5 . 12 (DT) → 20 . 98 ± 9 . 76 (DT & BG: W) and 3 . 64 ± 0 . 26 (DT & BG: C) → 33 . 07 ± 7 . 67 (DT, BG: W, & BG: C). However, the addition of BG: C now measurably improved performance 20 . 98 ± 9 . 76 (DT & BG: W) → 33 . 07 ± 7 . 67 (DT, BG: W, & BG: C). The main outcome of these experiments is the observation that multi-faceted planning is unlikely to harm a response calculation, and has a potentially large benefit when applied effectively. These results support the claim that world models offer the potential to improve response calculation through decision-time planning.

4

Dyna-PSRO

In this section we introduce Dyna-PSRO, Dyna -Policy-Space Response Oracles, an approximate game-solving algorithm that builds on the PSRO [38] framework. Dyna-PSRO employs co-learning to combine the benefits of world models and empirical games.

Dyna-PSRO is defined by two significant alterations to the original PSRO algorithm. First, it trains a world model in parallel with all the typical PSRO routines (i.e., game reasoning and response calculation). We collect training data for the world model from both the episodes used to estimate the empirical game's payoffs, and the episodes that are generated during response learning and evaluation. This approach ensures that the world model is informed by a diversity of data from a salient set of strategy profiles. By reusing data from empirical game development, training the world model incurs no additional cost for data collection.

The second modification introduced by Dyna-PSRO pertains to the way response policies are learned. Dyna-PSRO adopts a Dyna-based reinforcement learner [67, 68, 70] that integrates simultaneous planning, learning, and acting. Consequently, the learner concurrently processes experiences generated from decision-time planning, background planning, and direct game interaction. These experiences, regardless of their origin, are then learned from using the IMPALA [14] update rule. For all accounts of planning, the learner uses the single world model that is trained within Dyna-PSRO. This allows game knowledge accrued from previous response calculations to be transferred and used to reduce the cost of the current and future response calculations. Pseudocode and additional details for both PSRO and Dyna-PSRO are provided in Appendix C.4.

Games. Dyna-PSRO is evaluated on three games. The first is the harvest commons game used in the experiments described above, denoted 'Harvest: Categorical'. The other two games come from the MeltingPot [39] evaluation suite and feature rich image-based observations. 'Harvest: RGB' is their version of the same commons harvest game (details in Appendix D.2). 'Running With Scissors' is a temporally extended version of rock-paper-scissors (details in Appendix D.3). World model training and implementation details for each game are in Appendix C.2, likewise, policies in Appendix C.1.

Experiment. Dyna-PSRO's performance is measured by the quality of the solution it produces when compared against the world-model-free baseline PSRO. The two methods are evaluated on SumRegret (sometimes called Nash convergence ), which measures the regret across all players SumRegret ( σ , Π ) = ∑ i ∈ n max π i ∈ Π i ˆ U i ( π i , σ -i ) -ˆ U i ( σ i , σ -i ) , where σ is the method's solution and Π ⊆ Π denotes the deviation set. We define deviation sets based on policies generated across methods (i.e., regret is with respect to the combined game ): Π ≡ ⋃ method ˆ Π method , for all methods for a particular seed (detailed in Appendix C.5) [2]. We measure SumRegret for intermediate solutions, and report it as a function of the cumulative number of real experiences employed in the respective methods.

Results. Figure 5 presents the results for this experiment. For Harvest: Categorical, Dyna-PSRO found a no regret solution within the combined-game in 3 . 2e6 experiences. Whereas, PSRO achieves a solution of at best 5 . 45 ± 1 . 62 within 2e7 experiences. In Harvest: RGB, Dyna-PSRO reaches a

0

1

2

Real Exp.

×

10

7

0

10

0

10

1

10

2

SumRegret

Harvest: Categorical

0

1

2

Real Exp.

×

10

7

0

10

0

10

1

Harvest: RGB

0

1

2

Real Exp.

×

10

7

0

0.02

Running With Scissors

PSRO

Dyna-PSRO

solution with 0 . 89 ± 0 . 74 regret at 5 . 12e6 experiences. At the same time, PSRO had found a solution 352 with 6 . 42 ± 4 . 73 regret, and at the end of its run had 2 . 50 ± 2 . 24 regret. In the final game, RWS, 353 Dyna-PSRO has 2e -3 ± 5e -4 regret at 1 . 06e7 experiences, and at a similar point ( 9 . 6e6 experiences), 354 PSRO has 6 . 68e -3 ± 2 . 51e -3 . At the end of the run, PSRO achieves a regret 3 . 50e -3 ± 7 . 36e -4 . 355

356

357

358

359

360

361

362

Discussion. The results indicate that across all games, Dyna-PSRO consistently outperforms PSRO by achieving a superior solution. Furthermore, this improved performance is realized while consuming fewer real-game experiences. For instance, in the case of Harvest: Categorical, the application of the world model for decision-time planning enables the computation of an effective policy after only a few iterations. On the other hand, we observe a trend of accruing marginal gains in other games, suggesting that the benefits are likely attributed to the transfer of knowledge about the game dynamics. In Harvest: Categorical and Running With Scissors, Dyna-PSRO also had lower variance than PSRO.

363

364

365

366

367

368

369

370

371

372

373

374

375

5 Limitations

Although our experiments demonstrate benefits for co-learning world models and empirical games, there are several areas for potential improvement. The world models used in this study necessitated observational data from all players for training, and assumed a simultaneous-action game. Future research could consider relaxing these assumptions to accommodate different interaction protocols, a larger number of players, and incomplete data perspectives. Furthermore, our world models functioned directly on agent observations, which made them computationally costly to query. If the generation of experiences is the major limiting factor, as assumed in this study, this approach is acceptable. Nevertheless, reducing computational demands through methods like latent world models presents a promising avenue for future research. Lastly, the evaluation of solution concepts could also be improved. While combined-game regret employs all available estimates in approximating regret, its inherent inaccuracies may lead to misinterpretations of relative performance.

6 Conclusion

This study showed the mutual benefit of co-learning a world model and empirical game. First, we 376 demonstrated that empirical games provide strategically diverse training data that could inform a more 377 robust world model. We then showed that world models can reduce the computational cost, measured 378 in experiences, of response calculations through planning. These two benefits were combined and 379 realized in a new algorithm, Dyna-PSRO. In our experiments, Dyna-PSRO computed lower-regret 380 solutions than PSRO on several partially observable general-sum games. Dyna-PSRO also required 381 substantially fewer experiences than PSRO, a key algorithmic advantage for settings where collecting 382 experiences is a cost-limiting factor. 383

384

385

386

References

Stefano V Albrecht and Peter Stone. Autonomous agents modelling other agents: A comprehensive survey and open problems. Artificial Intelligence , 258:66-95, 2018.

David Balduzzi, Karl Tuyls, Julien Pérolat, and Thore Graepel. Re-evaluating evaluation. In 387 32nd Conference on Neural Information Processing Systems , 2018. 388

Philip Ball, Jack Parker-Holder, Aldo Pacchiano, Krzysztof Choromanski, and Stephen Roberts. 389 Ready policy one: World building through active learning. In 37th International Conference of 390 Machine Learning , 2020. 391

392

393

394

Nolan Bard, Michael Johanson, Neil Burch, and Michael Bowling. Online implicit agent modelling. In 12th International Conference on Autonomous Agents and Multiagent Systems , 2013.

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. In 28th Conference on Neural Information Processing Systems , pages 1171-1179, 2015.

James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018.

George W Brown. Iterative solution of games by fictitious play. In Activity analysis of production and allocation , volume 13, pages 374-376, 1951.

Jacob Buckman, Danijar Hafner, George Tucker, Eugene Brevdo, and Honglak Lee. Sampleefficient reinforcement learning with stochastic ensemble value expansion. In 22nd Conference on Neural Information Processing Systems , 2018.

Albin Cassirer, Gabriel Barth-Maron, Eugene Brevdo, Sabela Ramos, Toby Boyd, Thibault Sottiaux, and Manuel Kroiss. Reverb: A framework for experience replay, 2021.

Silvia Chiappa, Sébastien Racaniere, Daan Wierstra, and Shakir Mohamed. Recurrent environment simulators. In 5th International Conference on Learning Representations , 2017.

Valliappa Chockingam, Tegg Taekyong Sung, Feryal Behbanai, Rishab Gargeya, Amlesh Sivanantham, and Aleksandra Malysheva. Extending world models for multi-agent reinforcement learning in malmö. In Joint AIIDE 2018 Workshops co-located with the 14th AAAI conference on artificial intelligence and interactive digital entertainment , 2018.

Brian Collins. Combining opponent modeling and model-based reinforcement learning in a two-player competitive game. Master's thesis, University of Edinburgh, 2007.

B. Curtis Eaves. The linear complementarity problem. Management Science , 17(9):612-634, 1971.

Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, Shane Legg, and Koray Kavukcuoglu. IMPALA: Scalable distributed deep-RL with importance weighted actor-learner architectures. In 35th International Conference on Machine Learning , 2018.

Jakob N Foerster, Richard Y Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor Mordatch. Learning with opponent-learning awareness. In 17th International Conference on Autonomous Agents and MultiAgent Systems , 2018.

Kunihiko Fukushima. Cognitron: A self-organizing multilayered neural network. Biological Cybernetics , 20:121-136, 1975.

Carles Gelada, Saurabh Kumar, Jacob Buckman, Ofir Nachum, and Marc G. Bellemare. Deep427 MDP: Learning continuous latent space models for representation learning. In 36th International 428 Conference on Machine Learning , volume 97, pages 2170-2179, 2019. 429

David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In 31st 430 Conference on Neural Information Processing Systems , 2018. 431

David Ha and Jürgen Schmidhuber. World models. In arXiv preprint arXiv:1803.10122 , 2018. 432

Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with 433 discrete world models. In 9th International Conference on Learning Representations , 2021. 434

435

436

He He, Jordan Boyd-Graber, Kevin Kwok, and Hal Daumé III. Opponent modeling in deep reinforcement learning. In 33rd International Conference on Machine Learning , 2016.

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

Tom Hennigan, Trevor Cai, Tamara Norman, and Igor Babuschkin. Haiku: Sonnet for JAX, 2020.

Pablo Hernandez-Leal, Michael Kaisers, Tim Baarslag, and Enrique Munoz de Cote. A survey of learning in multiagent environments: Dealing with non-stationarity. arXiv preprint arXiv:1707.09183 , 2017.

Todd Hester and Peter Stone. Texplore: Real-time sample-efficient reinforcement learning for robots. In Machine Learning for Robotics (MLR) , 2012.

Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation , 9(8):1735-1780, 1997.

Matthew W. Hoffman, Bobak Shahriari, John Aslanides, Gabriel Barth-Maron, Nikola Momchev, Danila Sinopalnikov, Piotr Sta´ nczyk, Sabela Ramos, Anton Raichuk, Damien Vincent, Léonard Hussenot, Robert Dadashi, Gabriel Dulac-Arnold, Manu Orsini, Alexis Jacq, Johan Ferret, Nino Vieillard, Seyed Kamyar Seyed Ghasemipour, Sertan Girgin, Olivier Pietquin, Feryal Behbahani, Tamara Norman, Abbas Abdolmaleki, Albin Cassirer, Fan Yang, Kate Baumli, Sarah Henderson, Abe Friesen, Ruba Haroun, Alex Novikov, Sergio Gómez Colmenarejo, Serkan Cabi, Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Andrew Cowie, Ziyu Wang, Bilal Piot, and Nando de Freitas. Acme: A research framework for distributed reinforcement learning. arXiv preprint arXiv:2006.00979 , 2020.

G. Zacharias Holland, Erin Talvitie, and Michael Bowling. The effect of planning shape on dyna-style planning in high-dimensional state spaces. In FAIM workshop 'Prediction and Generative Modeling in Reinforcement Learning' , 2018.

HumanCompatibleAI. https://github.com/HumanCompatibleAI/multi-agent , 2019.

Pararawendy Indarjo. Deep state-space models in multi-agent systems. Master's thesis, Leiden University, 2019.

Marco A. Janssen, Robert Holahan, Allen Lee, and Elinor Ostrom. Lab experiments for the study of social-ecological systems. Science , 328(5978):613-617, 2010.

Patrick R. Jordan, L. Julian Schvartzman, and Michael P. Wellman. Strategy exploration in empirical games. In 9th International Conference on Autonomous Agents and Multi-Agent Systems , pages 1131-1138, 2010.

Gabriel Kalweit and Joschka Boedecker. Uncertainty-driven imagination for continuous deep reinforcement learning. In 1st Conference on Robot Learning , pages 195-206, 2017.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd International Conference for Learning Representations , 2015.

Christine Konicki, Mithun Chakraborty, and Michael P. Wellman. Exploiting extensive-form structure in empirical game-theoretic analysis. In Web and Internet Economics: 18th International Conference , 2022.

Orr Krupnik, Igor Mordatch, and Aviv Tamar. Multi-agent reinforcement learning with multistep generative models. In 4th Conference on Robot Learning , pages 776-790, 2020.

Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-ensemble 475 trust-region policy optimization. In 6th International Conference on Learning Representations , 476 2018. 477

478

479

480

481

482

483

Marc Lanctot, Edward Lockhart, Jean-Baptiste Lespiau, Vinicius Zambaldi, Satyaki Upadhyay, Julien Pérolat, Sriram Srinivasan, Finbarr Timbers, Karl Tuyls, Shayegan Omidshafiei, Daniel Hennes, Dustin Morrill, Paul Muller, Timo Ewalds, Ryan Faulkner, János Kramár, Bart De Vylder, Brennan Saeta, James Bradbury, David Ding, Sebastian Borgeaud, Matthew Lai, Julian Schrittwieser, Thomas Anthony, Edward Hughes, Ivo Danihelka, and Jonah Ryan-Davis. OpenSpiel: A framework for reinforcement learning in games. CoRR , abs/1908.09453, 2019.

484

485

486

487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

516

517

518

519

520

521

Marc Lanctot, Vinicius Zambaldi, Audr¯ unas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Pérolat, David Silver, and Thore Graepel. A unified game-theoretic approach to multiagent reinforcement learning. In 31st Conference on Neural Information Processing Systems , page 4193-4206, 2017.

Joel Z. Leibo, Edgar Duéñez-Guzmán, Alexander Sasha Vezhnevets, John P. Agapiou, Peter Sunehag, Raphael Koster, Jayd Matyas, Charles Beattie, Igor Mordatch, and Thore Graepel. Scalable evaluation of multi-agent reinforcement learning with melting pot. PMLR, 2021.

Joel Z. Leibo, Vinicius Zambaldi, Marc Lanctot, Janusz Marecki, and Thore Graepel. Multiagent reinforcement learning in sequential social dilemmas. In 16th International Conference on Autonomous Agents and Multiagent Systems , 2017.

Zun Li, Marc Lanctot, Kevin McKee, Luke Marris, Ian Gemp, Daniel Hennes, Paul Muller, Kate Larson, Yoram Bachrach, and Michael P. Wellman. Search-improved game-theoretic multiagent reinforcement learning in general and negotiation games (extended abstract). In 32nd International Conference on Autonomous Agents and Multiagent Systems , AAMAS, 2023.

Michael L. Littman. Markov games as a framework for multi-agent reinforcement learning. In 11th International Conference on Machine Learning , pages 157-163, 1994.

Stephen McAleer, John Lanier, Kevin Wang, Pierre Baldi, and Roy Fox. XDO: A double oracle algorithm for extensive-form games. In 35th Conference on Neural Information Processing Systems , 2021.

Richard D. McKelvey, Andrew M. McLennan, and Theodore L. Turocy. Gambit: Software tools for game theory. http://www.gambit-project.org/ , 2016.

H. Brendan McMahan, Geoffrey J Gordon, and Avrim Blum. Planning in the presence of cost functions controlled by an adversary. In 20th International Conference on Machine Learning , pages 536-543, 2003.

Richard Mealing and Jonathan L Shapiro. Opponent modeling by expectation-maximization and sequence prediction in simplified poker. In IEEE Transactions on Computational Intelligence and AI in Games , volume 9, pages 11-24, 2015.

Vicent Michalski, Roland Memisevic, and Kishore Konda. Modeling deep temporal dependencies with recurrent grammar cells. In 27th Conference on Neural Information Processing Systems , 2014.

Johan S. Obando-Ceron and Pablo Samuel Castro. Revisiting rainbow: Promoting more insightful and inclusive deep reinforcement learning research. In 38th International Conference on Machine Learning , 2021.

Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard Lewis, and Satinder Singh. Actionconditional video prediction using deep networks in atari games. In 28th Conference on Neural Information Processing Systems , 2015.

Junhyuk Oh, Satinderg Singh, and Honglak Lee. Value prediction network. In 30th Conference on Neural Information Processing Systems , pages 6118-6128, 2017.

522

523

524

525

526

527

528

529

530

531

532

533

534

535

536

537

538

539

540

541

542

543

544

545

546

547

548

549

550

551

552

553

554

555

556

557

558

559

560

561

562

563

564

565

566

567

S. Phelps, M. Marcinkiewicz, and S. Parsons. A novel method for automatic strategy acquisition in N -player non-zero-sum games. In Fifth International Joint Conference on Autonomous Agents and Multiagent Systems , page 705-712, 2006.

Martin L Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming . John Wiley & Sons, Inc., 1994.

Julien Pérolat, Joel Z. Leibo, Vinicius Zambaldi, Charles Beattie, Karl Tuyls, and Thore Graepel. A multi-agent reinforcement learning model of common-pool resource appropriation. In 31st Conference on Neural Information Processing Systems , 2017.

Stéphane Ross and J. Andrew Bagnell. Reinforcement and imitation learning via interactive no-regret learning. CoRR , abs/1406.5979, 2014.

Stéphane Ross, Goeffrey J. Gordon, and J. Andrew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In 14th International Conference on Artificial Intelligence and Statistics , 2011.

Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy Lillicrap, and David Silver. Mastering atari, go, chess and shogi by planning with a learned model. Nature , 588:604-609, 2020.

L. Julian Schvartzman and Michael P. Wellman. Exploring large strategy spaces in empirical game modeling. In AAMAS-09 Workshop on Agent-Mediated Electronic Commerce , 2009.

L. Julian Schvartzman and Michael P. Wellman. Stronger CDA strategies through empirical game-theoretic analysis and reinforcement learning. In 8th International Conference on Autonomous Agents and Multi-Agent Systems , pages 249-256, 2009.

Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via self-supervised world models. In 37th International Conference of Machine Learning , pages 8583-8592, 2020.

David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of Go with deep neural networks and tree search. Nature , 529:484-489, 2016.

David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature , 529(7587):484489, 2016.

David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature , 550(7676):354-359, 2017.

David Silver, Hado van Hasselt, Matteo Hessel, Tom Schaul, Arthur Guez, Tim Harley, Gabriel Dulac-Arnold, David Reichert, Neil Rabinowitz, Andre Barreto, and Thomas Degris. The predictron: End-to-end learning and planning. In 34th International Conference on Machine Learning , volume 70, pages 3191-3199, 2017.

Max Olan Smith, Thomas Anthony, Yongzhao Wang, and Michael P. Wellman. Learning to play against any mixture of opponents. CoRR , 2020.

Max Olan Smith, Thomas Anthony, and Michael P. Wellman. Iterative empirical game solving via single policy best response. In 9th International Conference on Learning Representations , 2021.

Chen Sun, Per Karlsson, Jiajun Wu, Joshua B Tenenbaum, and Kevin Murphy. Stochastic 568 prediction of multi-agent interactions from partial observations. In 7th International Conference 569 on Learning Representations , 2019. 570

Richard S. Sutton. Integrated architectures for learning, planning, and reacting based on 571 approximating dynamic programming. In 7th International Workshop on Machine Learning , 572 pages 216-224. Morgan Kaufmann, 1990. 573

574

575

Richard S. Sutton. Dyna, an integrated architecture for learning, planning, and reacting. In SIGART Bulletin , volume 2, pages 160-163. ACM, 1991.

576

577

578

579

580

581

582

583

584

585

586

587

Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction . The MIT Press, 2018.

Richard S Sutton, Csaba Szepesvári, Alborz Geramifard, and Michael P. Bowling. Dyna-style planning with linear function approximation and prioritized sweeping. In 28th Conference on Uncertainty in Artificial Intelligence , 2012.

Erin Talvitie. Model regularization for stable sample rollouts. In 30th Conference on Uncertainty in Artificial Intelligence , 2014.

Gerald Tesauro. Temporal difference learning and td-gammon. Communications of the ACM , 38(3):58-68, 1995.

Zheng Tian, Ying Wen, Zhichen Gong, Faiz Punakkath, Shihao Zou, and Jun Wang. A regularized opponent model with maximum entropy objective. In International Joint Conference on Artificial Intelligence , 2019.

Karl Tuyls, Julien Pérolat, Marc Lanctot, Edward Hughes, Richard Everett, Joel Z. Leibo, Csaba 588 Szepesvári, and Thore Graepel. Bounds and dynamics for empirical game theoretic analysis. 589 Autonomous Agents and Multi-Agent Systems , 34(7), 2020. 590

Yevgeniy Vorobeychik. Probabilistic analysis of simulation-based games. ACM Transactions 591 on Modeling and Computer Simulation , 20(3), 2010. 592

Niklas Wahlström, Thomas B. Schön, and Marc Peter Deisenroth. From pixels to torques: 593 Policy learning with deep dynamical models. arXiv preprint arXiv:1502.02251 , 2015. 594

William Walsh, Rajarshi Das, Gerald Tesauro, and Jeffrey Kephart. Analyzing complex strategic 595 interactions in multi-agent systems. In AAAI-02 Workshop on Game Theoretic and Decision 596 Theoretic Agents , 2002. 597

598

599

600

Rose E Wang, Chase Kew, Dennis Lee, Edward Lee, Brian Andrew Ichter, Tingnan Zhang, Jie Tan, and Aleksandra Faust. Model-based reinforcement learning for decentralized multiagent rendezvous. In Conference on Robot Learning , 2020.

601

602

603

Manuel Watter, Jost Tobias Springenberg, Joschka Boedecker, and Martin Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. In 28th Conference on Neural Information Processing Systems , pages 2746-2754, 2015.

Michael P. Wellman. Methods for empirical game-theoretic analysis. In 21st National Confer604 ence on Artificial Intelligence , page 1552-1555, 2006. 605

Daniël Willemsen, Mario Coppola, and Guido CHE de Croon. MAMBPO: Sample-efficient 606 multi-robot reinforcement learning using learned world models. In IEEE/RSJ International 607 Conference on Intelligent Robots and Systems , 2021. 608

609

610

Ronald J. Williams and David Zipser. A learning algorithm for continually running fully recurrent neural networks. Neural Computation , 1(2), 1989.

611

612

613

614

615

616

Fan Yang, Gabriel Barth-Maron, Piotr Sta´ nczyk, Matthew Hoffman, Siqi Liu, Manuel Kroiss, Aedan Pope, and Alban Rrustemi. Launchpad: A programming model for distributed machine learning research. arXiv preprint arXiv:2106.04516 , 2021.

Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: Model-based offline policy optimization. In 33rd Conference on Neural Information Processing Systems , 2020.

617

618

619

620

621

622

Kaiqing Zhang, Sham Kakade, Tamer Basar, and Lin Yang. Model-based multi-agent rl in zero-sum markov games with near-optimal sample complexity. In 33rd Conference on Neural Information Processing Systems , 2020.

Qizhen Zhang, Chris Lu, Animesh Garg, and Jakob Foerster. Centralized model and exploration policy for multi-agent RL. In 21st International Conference on Autonomous Agents and Multiagent Systems , 2022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the problem of multi-agent RL by using learned world models over multiple potential opponent policies. The technique uses a Dyna-style algorithm to train the core policy with a combination of experiences generated through a world model and experiences playing against opponents. One evaluation demonstrates the world models benefits from training on data from multiple distinct policies. A second evaluation compares ways to use experiences generated from a world model to train a policy, showing pretraining on purely generated experiences is effective to warm-start a policy. An ablation study compares the proposed Dyna-PSRO model to vanilla PSRO on three MARL games, showing improvements over the PSRO model.\n\nSTRENGTHS:\n## originality\n\nModest. The paper extends existing lines of work on MARL and world models, specifically studying the question of the policy diversity for training the world model.\n\n## quality\n\nModest. The core results (figure 5) show clear improvements over PSRO. This is limited to a small number of games and the games themselves are relatively simple game domains.\n\nThe paper does a good job of breaking down specific claims to isolated experiments.\n\n## clarity\n\nLow. It was difficult to interpret many of the figures (questions and suggestions below). Generally the results of each experiment were hard to understand and would benefit from a single clear statement of the core outcomes in each section.\n\n## significance\n\nLow. The core audience of this work is researchers in MARL and particularly those considering world models as a solution.\n\nWEAKNESSES:\nThe experimental results are promising, but would benefit from expansion. There are a few experiments that would help:\n1. More games from MeltingPot. I hate asking simply for \"more\", but in this case it would help to show how well the agents perform on a wide variety of tasks. The results would help clarify where DynaPSRO benefits and may reveal limitations or areas for improvement. The wider set of results would give others confidence in the generality of the improvements gained by planning against diverse other agents.\n1. More complex games. Consider more complex environments from PettingZoo (https://pettingzoo.farama.org/) or SMAC (https://github.com/oxwhirl/smac) that would highlight the potential of these algorithms in more compelx scenarios. This would help address the point that world models can become unstable and the value of strategic diversity in scenarios that support a much wider array of behaviors.\n1. Scaling experiments. For example, when do prediction improvements level off when adding more policies? The experiments only examine adding 2 policies, which is a sparse sample of the space of strategies for most games.\n\nThe evaluations would benefit from other baselines to compare. What other algorithms could be used aside PSRO?\n\nThe full evaluation (last experiment) would benefit from a set of ablation studies. This could easily replace some of the planning experiments as the ablations would examine similar capabilities. I ask for ablations as these will be more convincing that the parts of Dyna being used add benefit over PSRO.\n\nQUESTIONS:\n- [Q1] How do the results here scale?\n\t- This is a broad question, but for any RL algorithm it's important to understand the scaling and sample efficiency of the method. For example, how does increasing the amount of warm-start background training impact asymptotic performance or convergence rate? How does increasing the number of policies used to train the world model alter prediction performance? Reward obtained during training?\n- [Q2] Figure 2\n\t- This figure is hard to understand and summarize. It may help to instead plot the average performances for each model combination and omit the heat maps.\n- [Q3] Figure 3\n\t- What is \"Plan: Model\"?\n\t- Why does \"Plan: Model\" start strong then decay? (Opponents get better?)\n\t- Please define the legend (labels like \"Plan: Model\") as the names are not immediately obvious.\n- [Q4] Figure 4\n\t- Is it correct to conclude that concurrent background planning has no benefit?\n- [Q5] What ablations can add more detail to Dyna-PSRO model?\n\t- What evidence is there that DynaPSRO was learning and responding to strategic diversity during these experiments?\n- [Q6] Section 3.2.2 DT planning\n\t- It seems like warm-start planning is sufficient to capture world model benefits. Why is that?\n\t- Consider adding an ablation: remove DT planning but retain BG:W and BG:C. This could clarify how much help the world model is for training the policy, independent of evaluating future states during execution.\n- [Q7] Figure 5\n\t- Why do the DynaPSRO traces stop partway through the real experiences?\n\nLIMITATIONS:\nYes. The work is focused on integrating world models into game playing agents and recognizes the preliminary nature of this work along with the potential risks introduced by using a simulation (world model) for decision making in real world applications.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new approach to PSRO algorithms, where a world model of the environment is learned concurrently to the iterative PSRO strategy expansion.\n\nSTRENGTHS:\n- The authors are right that the problem of having to re-learn policies from scratch is a large problem in the PSRO literature. Therefore, the idea to co-learn a world model alongside the expansion of the empirical game, in so taking advantage of the diversity of experience created by agents with slightly different best-response targets is an interesting approach to this problem. To the best of my knowledge this is also a novel solution to this problem.  \n- I really like the presentation of the paper, and in general I think the authors do a very good job in terms of analysing the different moving parts of the framework in a reasonable manner.\n\nWEAKNESSES:\nI have a few concerns with the paper, however none of these necessarily game-changing in my evaluation of the work.\n\n- I think the greatest misgiving I have with this work is that the related work seems to miss quite a large collection of PSRO papers that probably deserve mentioning. PSRO-style algorithms is a fairly small research area and I am surprised that the authors fail to make mention to many variants. In particular, as there is a section on strategic diversity itself in the paper, it seems odd that the authors have failed to comment on the line of works on diversity-based PSRO frameworks. For example, [1], [2], [3], [4], [5] are all diversity PSRO approaches. It also fails to place itself in the literature involving PSRO algorithms that attempt to speed up convergence times such as [6], [7].\n\n- Furthermore, I was additionally surprised at the lack of comparison to NeuPL [8] which is another population-based framework attempting to similarly deal with the best ways to transfer information between agents in the population. \n\n- I do not necessarily believe that the authors need to benchmark against all of the approaches that I have listed. I do however believe the paper still needs work in terms of placing itself within the current literature on PSRO and other population-based frameworks.\n\n- Based on the above, my score is set at a borderline accept. However, I am willing to revise this upwards upon seeing a better framing of this work in the current literature.\n\nREFERENCES  \n[1] Policy Space Diversity for Non-Transitive Games - Yao et al. 2023  \n[2] Open-ended learning in symmetric zero-sum games - Balduzzi et al. 2019  \n[3] Modelling behavioural diversity for learning in open-ended games - Perez-Nieves et al. 2021  \n[4] Towards unifying behavioural and response diversity for open-ended learning in zero-sum games - Liu et al. 2021  \n[5] A unified diversity measure for multi agent reinforcement learning - Liu et al. 2022  \n[6] Pipeline PSRO: A scalable approach for finding approximate Nash equilibria in large games - McAleer et al. 2020  \n[7] Neural auto-curricula in two-player zero-sum games - Feng et al. 2021  \n[8] NeuPL: Neural Population Learning - Liu et al. 2022\n\nQUESTIONS:\n- Discussion starting line 208: I think this whole section on strategic diversity is very interesting. However, I am not entirely convinced as of yet that there does exist strategic diversity between the PSRO policies, especially as the changes in performance of Fig. 2 when the PSRO policies is only minor. My question to the authors is: was the strategic diversity of the PSRO policies actively checked? If so, am I missing some proof of that here? If not, why was it not checked? In my experience with PSRO policies, it is quite easy to end up with policies that act similarly even under the presence of different training data.\n\n- Discussion starting line 264: Could the authors comment a little further on the gap between the world model and the real game - do they foresee any scenarios where this gap may be far more damaging to performance than in the proposed experiments? Do they have any intuition for why it does not see to matter as much as one would expect in these experiments?\n\n- Why is this a better transfer learning method than NeuPL? I would be interested to hear the authors thoughts on this.\n\nLIMITATIONS:\nI think the authors actively engage with the limitations of the work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors consider learning world models for deep reinforcement learning in combination with the construction of empirical games through PSRO. They first show that world models benefit from training on a diverse set of strategy profiles as can be generated through PSRO meta-game solvers. They then empirically show that PSRO best responses can enjoy sample efficiency benefits by training with simulated world model experience. Finally, they present Dyna-PSRO, in which PSRO best responses make use of a world model trained on all available experiences collected thus far in a run of the PSRO algorithm. Dyna-PSRO provides lower-regret solutions with higher sample efficiency than PSRO without a world model.\n\nSTRENGTHS:\n- The paper is very well written and presented, and the experiments are well designed. \n- World models are seeing increased use in the RL community, and PSRO is one of the more practical and general methods currently available for finding approximate game solutions. This paper provides insights on how to properly combine the two and make improvements to PSRO's sample efficiency, which is one of its largest issues.\n- The proposed Dyna-PSRO method is sound.\n- While many implementation details are not present in the main paper, the appendix describes these details thoroughly.\n\nWEAKNESSES:\nIt would have been nice to see how current high-performing world model methods such as Dreamer, which employs latent state spaces [1,2] might perform with the same approach. It's not immediately clear if experiments like in section 3.3.2 would have had the same outcome.\n\n[1] Hafner, Danijar, et al. \"Dream to Control: Learning Behaviors by Latent Imagination.\" International Conference on Learning Representations. 2019.\n\n[2] Hafner, Danijar, et al. \"Mastering diverse domains through world models.\" arXiv preprint arXiv:2301.04104 (2023).\n\nQUESTIONS:\nWas there a strong reason for adapting a Dyna-based method over other existing world model approaches?\n\nLIMITATIONS:\nAll limitations have been adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper describes combining two things: training a world model of a game, and doing Policy Space Response Oracles (PSRO) on the game.\n\nDoing PSRO involves getting a lot of episodes from the game (episodes are used to train the RL best-responses, and also to estimate the payoffs of the empirical game). The novel algorithm in this work (Dyna-PSRO) can be thought of as a modification of PSRO where those episodes are **also** used to train a world model (which is essentially a learned simulator of the game engine). Then, the world model is used to improve the training process of the best-response policies. \n\nThrough experimental results, the authors show that this improved training process (based on Dyna) can cause the best-response learners to learn a stronger policy than the normal method when using the same amount of interactions with the real game environment. It does this by training the policy using trajectories from the world model (in addition to the usual trajectories from the real game environment), and by equipping the agents with one-step lookahead planning during training.\n\nThis paper showcases experiments on the Dyna-PSRO algorithm in three games, and Dyna-PSRO outperforms PSRO in all three, as measured by an approximation of NashConv.\n\nThe paper also shows experiments to measure the quality of the learned world model, to test the hypothesis that Dyna-PSRO results in a good world model.\n\nI think the paper has some flaws* in its current form, but the core work of the paper is good, the charts are beautiful, and the results are strong.\n\n---\n\n*Edit: many of the abovementioned flaws were addressed during the rebuttal/discussion period.\n\nSTRENGTHS:\n- Overall, the paper is well-executed.\n    - It is well-written and polished.\n    - There has clearly been a lot of time and effort put into the engineering and writing of this paper. There are 4 sets of thorough experiments in the main paper, and more in the appendix.\n    - The figures are extremely readable.\n- The results concerning the performance of the best-response policies are impressive.\n- The effort will surely be helpful to future researchers: the research directions of (1) improving the efficiency of PSRO response calculations and (2) training better world models should continue to flourish, and the work presented in this paper contributes to both.\n    - The research direction seems natural, especially in the direction of using world models to improve the performance of PSRO.\n\nWEAKNESSES:\nI think the paper could be better in explaining or hypothesizing the \"why\" for a lot of things, even if just qualitatively.\n\nI think the paper states some conclusions too strongly.\n\nI think some things are not explained well enough and are confusing to the reader (at least, to me):\n\n- SumRegret metric:\n    - It's really not clear from the main paper how SumRegret works (even though it is explained in the appendix). This could be clarified by defining the terms \"method\" and \"combined game\".\n    - Also, I would feel a lot better if I saw results measured by an alternative metric, where the deviation set is the set of **all** policies. The $max_{\\pi_i \\in \\bar{\\Pi}_i}$ could then be approximated by just training one more response policy (as if doing one last epoch of PSRO). This seems like it would be a more accurate approximation of the Nash Conv. Is there any reason to use the metric in the paper instead of this?\n- Empirical Game Solution not described\n    - Since the settings here are general-sum, it's probably important to specify what solution concept is used for the meta-strategies in the main paper (even though it is included in the appendix).\n- Experiments in Section 3.1 Strategic Diversity\n    - Looking purely at Figure 2, the conclusion \"Overall, these results support the claim that strategic diversity enhances the training of world models\" does not ring true to me. For example, there are three world models which perform better on the metric (accuracy) used in Figure 2 than the most diverse one, for Observations.\n    - Even if I look in the appendix at E.1, there doesn't seem to be significant evidence to support the conclusion: multiple world models have similar recall scores than the most diverse one, and the one trained without the random policy seems to have better scores.\n    - I would be interested in seeing the cross-entropy loss instead of (or in addition to) the accuracy.\n- The discussion of the Decision-Time Planning results (3.2.2) seems incomplete:\n    - \"**The main outcome of these experiments is the observation that multi-faceted planning is unlikely to harm a response calculation,** and has a potentially large benefit when applied effectively. These results support the claim that world models offer the potential to improve response calculation through decision-time planning.\" (emphasis mine)\n    - However, Figure 4 does show that decision-time planning causes the response to be *worse*: The solid blue line (top) has no decision time planning, and the dashed gray line (second from the top) has decision-time planning, and performs worse.\n    - It would be nice if there was some discussion about this, perhaps an intuitive/qualitative reason why this is.\n- Dyna-PSRO results need more details (Figure 5):\n    - For each experiment, how many policies (iterations of PSRO) were there?\n    - Does each policy train for a fixed number of steps, or until some measure of convergence is reached?\n- Was \"policy\" vs. \"strategy\" ever defined like this before? In my opinion, we shouldn't define these terms like this, because they are usually considered synonymous. The terms I'm familiar with are \"policy\" or \"strategy\" for the former, and \"meta-policy\" or \"meta-strategy\" or \"meta-strategy distribution\" for the latter. Just my opinion!\n- I was very confused by the definition of World Model while reading the paper.\n    - Even after reading it through entirely, I was under the impression that each player had their own world model, and that it implicitly modeled the actions of the opponent.\n    - If one misses the bold notation of the definition of agent world model from line 137 to 141, it's easy to think that this is the case, especially since the phrasing is that \"the agent learns and uses a ... world model\" (instead of, say, \"the agents learn and use a ... world model).\n    - On one hand, the formal definition given for an \"agent world model\" is technically accurate, and I am just dumb. On the other hand, I suspect many of us are dumb, and will be similarly confused upon reading the paper. (Also, I'm not **that** dumb: it's really hard to tell that the O and A are bolded!) (Also, even if some of us are not dumb, we are likely lazy and will gloss over the explanation that boldface means joint.) This is all to say that I would suggest explicitly stating that the world model takes as input an observation and action from **each** player, and returns an observation and reward to each player. And that the world model does NOT model the actions of any player.\n- The bolding is nice, but it would be less confusing to **also** say \"strategy profile\" or \"joint strategy\" anytime this is meant instead of just \"strategy\" and something bold, as it's very easy to miss or forget what something bold means (plus, it seems incorrect to call a strategy profile a strategy). Also, maybe emphasize that sentence that explains what boldface means, so that readers don't miss it?\n    - For example in line 774 and 775 of the appendix:\n        - \"This is typically not tractable, but instead draws are taken from a dataset generated from play of a behavioral strategy **σ**. And the performance of the world model is measured under a target strategy **σ∗**\"\n        - should be \"strategy profile\" and \"target strategy profile\"\n    - and same in Line 159 and 160 of the main paper, and throughout section 3.1\n\nQUESTIONS:\n- The world model is deterministic. Can it easily be made stochastic?\n    - The Harvest game has stochasticity in the apple regeneration, right? Do you think the deterministic world model affected impacted anything in the experiments?\n- Section 3.1 Strategic Diversity\n    - I'm guessing the world model implemented for the categorical harvest game outputs some logits that can be interpreted for each cell as a probability distribution over the possible entities in that cell. Then the world model takes the argmax of each cell and returns the observation? Or does it sample from the probability distribution? If not, why not?\n    - What are the details of the weighted-average cross-entropy objective?\n- Did the empirical games often have multiple Nash equilibria with different values? If so, do the authors think it would change anything if specific Nash equilibria were sought -- e.g. welfare-maximizing?\n- Section 3.2.1 and 3.2.2\n    - Section 3.2.1: \"The planner employs the x world model and the opponent plays the previously held-out policy.\" The opponent plays the previously held-out policy in all 3 cases, right? (Baseline, Plan: Real, Plan: Model)\n    - It's stated that the planner switches from planned to real experience after 10,000 updates. Does this correspond to 1,280,000 experiences?\n    - In Figure 3 and 4, I would be interested in knowing how the baseline experiments do when they are run with as much experience as the amount of real+planned experience that the planning models get.\n        - On Figure 3, if the planning learners continue to outperform the baseline ones even on the real+planned exp x-axis, then what is the reason for that? Does the baseline policy settle into some local maximum that it can't escape, while the pretraining on the world model experience makes the planner walk into some better part of the policy space?\n    - Is there any explanation for why the baseline returns are so variable and spiky compared to the planner's? e.g. in Figure 3.\n- Section 4: Is the decision-time planning only used to train? Do the final learned agents use any decision-time planning?\n- Seems like a lot of problems were the result of a subpar World Model\n    - Could the performance of the world model be improved?\n        - In the experiments of 3.1\n            - Are the world models trained to convergence? Do they achieve near-0 training loss? What do the training curves look like?\n            - Did you try training the world model where one or more of the policies was a PSRO policy that played epsilon-random?\n- Out of curiosity, it would be nice to know what the results look like with wall-time on the x-axis.\n    - Or, how long does it take to query the world model? How long does it take to query the real game environment?\n- Will the code be open-sourced?\n\nLIMITATIONS:\nLimitations addressed", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}