Title: Generative Modeling of Individual Behavior at Scale
Venue: NeurIPS 2024
Accepted: no
Keywords: style

parameter efficient fine-tuning

peft

chess

stylometry

playstyle

representation learning

steerability
Abstract: Recent years have seen a growing interest in using AI to model human behavior, particularly in domains where humans learn from or collaborate with this technology. While most existing work attempts to model human behavior at an aggregate level, our goal is to model behavior at the individual level. Recent work in the domain of chess has shown that behavioral stylometry, or the task of identifying a person from their actions alone, can be achieved with high accuracy among a pool of a few thousand players. However, this approach cannot generate actions in the style of each player, and hence cannot reason about or influence player behavior in practice. We provide a new perspective on behavioral stylometry that addresses these limitations, by drawing a connection to the vast literature of transfer learning in NLP. Specifically, by casting the stylometry problem as a multi-task learning problem---where each task represents a distinct---we show that parameter-efficient fine-tuning (PEFT) methods can be adapted to model individual behavior in an explicit and generative manner, at unprecedented scale. We apply our approach at scale to two very different games: chess (47,864 players) and Rocket League (2,000 players).

Our approach leverages recent modular PEFT methods to learn a shared set of skill parameters that can be combined in different ways via style vectors. Style vectors enable two important capabilities. First, they are generative: we can generate actions in the style of a player simply by conditioning on the player's style vector. Second, they induce a latent style space that we can interpret and manipulate algorithmically. This allows us to compare different player styles, as well as synthesize new (human-like) styles, e.g. by interpolating between the style vectors of two players.

Abstract

There has been a growing interest in using AI to model human behavior, particularly in domains where humans interact with this technology. While most existing work models human behavior at an aggregate level, our goal is to model behavior at the individual level. Recent approaches to behavioral stylometry -or the task of identifying a person from their actions alone-have shown promise in domains like chess, but these approaches are either not scalable (e.g., fine-tune a model for each person) or not generative, in that they cannot generate actions in the style of each person. We address these limitations by casting behavioral stylometry as a multi-task learning problem-where each task represents a distinct person -and using parameter-efficient fine-tuning (PEFT) methods to learn an explicit style vector for each person. Style vectors are generative: they selectively activate shared "skill" parameters to generate actions in the style of each person. They also induce a latent style space that we can interpret and manipulate algorithmically. In particular, we develop a general technique for style steering that identifies a subset of players with a desired style property, and steers a new player towards that property. We apply our approach to two very different games, at unprecedented scale: chess (47,864 players) and Rocket League (2,000 players).

1 Introduction

The rapid advances in machine learning in recent years has made it increasingly important to find constructive ways for humans to interact with this technology. Even in domains where AI has achieved proficiency, it is often important to understand how humans approach these tasks. Such an understanding can help identify areas for improvement in humans, develop better AI collaborators or teachers, create more human-like experiences, and more.

A common method for capturing human behavior is behavioral cloning (BC), a form of imitation learning [Schaal, 1996] that applies supervised learning to fixed demonstrations collected for a given task. While traditionally used in domains such as robotics [Florence et al., 2022] and self-driving vehicles [Pomerleau, 1988], BC has seen increasing use in gaming, such as in Counter-Strike [Pearce and Zhu, 2022], Overcooked [Carroll et al., 2019], Minecraft [Schäfer et al., 2023], Bleeding Edge [Jelley et al., 2024], and chess McIlroy-Young et al. [2020].

The above work focuses on modeling human behavior in aggregate, with the goal of developing better 30

AI partners, opponents, and training tools. However, we believe that the most value for such goals can 31

be derived by modeling human behavior at the individual level. To that end, recent results in chess 32

have shown the most promise. McIlroy-Young et al. [2020] used behavior cloning to create a set of 33

models called Maia that match human play at 9 aggregate skill levels. By fine-tuning these models on 34

the data of 400 individual players, they created 400 personalized models that achieve 4-5% higher 35

move-matching accuracy on average [McIlroy-Young et al., 2022]. The authors use these models to 36

perform behavioral stylometry with high accuracy, where the goal is to identify which person played 37

a given query set of games; in this case, they simply apply each of the 400 models to the query set 38

and output the one with the highest accuracy. McIlroy-Young et al. [2021] propose a more scalable 39 approach of training a Transformer-based embedding on the games of each player, and use this to 40 perform accurate stylometry across 2,844 players; in this case, they compute the embedding of the 41 query set of games and match it to the closest player's embedding. 42

These approaches have different merits. The individual model approach creates a generative model 43 for each player, but it is not scalable and shares only initial (base model) knowledge across the 44 players; adding a new player requires fine-tuning a separate model. The embedding approach is 45 much more scalable: it learns a compact (single-vector) representation of each player in a shared 46 style space, and supports few-shot learning to embed a new player in this space. It cannot be used to 47 generate moves, however, and hence cannot reason about player behavior in practice. 48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

An ideal solution would combine these properties: generative, scalable, shared knowledge, compact representation. Our key insight for achieving this is to view behavioral stylometry as a multi-task learning problem, where each task represents an individual person . The goal here is to generalize across an initial set of players (tasks) while supporting few-shot learning of new players (tasks). To do this efficiently, we leverage recent advances in parameter-efficient fine-tuning (PEFT) [Ponti et al., 2023, Caccia et al., 2022]. Specifically, we augment an existing BC model with a set of Low Rank Adapters (LoRAs) as well as a routing matrix that specifies a distribution over these adapters for each player. Unlike approaches that train a separate LoRA for each task, this modular design allows players to softly share parameters in a fine-grained manner. We apply this adapter framework to two very different game models (which we create): a modified version of the Maia model for chess, and a Transformer-based BC model for Rocket League, a 3D soccer video game played by cars in a caged arena. (Our models scale beyond the prior art and may be of independent interest.) Our methodology first trains the BC models to convergence across all player data, and then fine-tunes the adapters and routing matrix on per-player data. This encourages the adapters to learn different latent skills that explain the variance between players, while each row of the routing matrix induces a weight distribution over these skills. We call each row the style vector for the corresponding player.

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

Style vectors are versatile and powerful. They support few-shot learning which enables stylometry at scale. They induce a generative model for each player that we can run and observe. They induce a shared style space that we can interpret and manipulate algorithmically. Leveraging these properties, we develop a general technique for style steering that identifies a subset of players who exhibit a desired style property, and steers a new player towards that property. Our main results include:

We perform behavioral stylometry at an unprecedented scale for chess (47,864 players, 94.4% accuracy) and Rocket League (2,000 players, 86.7% accuracy), using a query set of 100 games.

Our per-player generative models achieve move-matching accuracy in the range 45-69% for chess and 44-72% for Rocket League, even for players with very few (e.g., 50) games.

Style vectors capture a wide diversity of playing styles and strengths. They can be combined, interpolated, and steered, while reflecting consistent changes to play style and strength.

2 Background and Framing

We frame behavioral stylometry and per-player generative modeling as a multitask learning problem, to which we apply PEFT methods. In multitask learning [Caruana, 1997, Ruder et al., 2019], we are given a collection of tasks T = ( T 1 , . . . , T |T | ) , each task T i associated with a dataset D i = { ( x 1 , y 1 ) , ..., ( x n i , y n i ) } . Multitask learning exploits the similarities among related training tasks by transferring knowledge among them; ideally, this builds representations that are easily adaptable to new tasks using potentially few target examples. The premise of this paper is that modeling individual human behavior from a pool of players can be interpreted as a multitask learning problem. In other words, each task T i consists of modeling the behavior of a specific player i ; and dataset D i corresponds to the sequence of game actions taken by player i . Specifically, an ( x, y ) tuple denotes a game state x at a specific point in time during game, along with the action y that player i took in this state. For the rest of the paper, we use the notion of tasks and players interchangeably.

2.1 Parameter-efficient fine-tuning

Popularized in NLP, parameter-efficient fine-tuning (PEFT) [Houlsby et al., 2019, Hu et al., 2022, Liu et al., 2022] approaches have emerged as a scalable solution for adapting Large Language Models to several downstream tasks. Indeed, standard finetuning of pretrained LLMs requires updating (and

storing) possibly billions of parameters for each task. PEFT methods instead freeze the pretrained 92 model and inject a small set of trainable task-specific weights, or 'adapters". 93

One such approach is the use of Low Rank Adapters (LoRA) [Hu et al., 2022], which modify linear 94 transformations in the network by adding a learnable low rank shift 95

Here, W 0 ∈ R d × d are the (frozen) weights of the pre-trained model, and A , B ∈ R d × r the learnable 96 low-rank parameters of rank r ≪ d . With this approach, practitioners can trade off parameter 97 efficiency with expressivity by increasing the rank r of the transformation. 98

2.2 Polytropon and Multi-Head Adapter Routing 99

Standard PEFT methods such as LoRA can adapt a pretrained model for a given task. In multitask settings, training a separate set of adapters for each task is suboptimal, as it does not enable any sharing of information, or transfer , across similar tasks. On the other hand, using the same set of adapters for all tasks risks negative interference [Wang et al., 2021] across dissimilar tasks. Polytropon [Ponti et al., 2019] ( Poly ) addresses this transfer/interference tradeoff by softly sharing parameters across tasks. That is, each Poly layer contains 1) an inventory of LoRA adapters

with m ≪ |T | , and 2) a task-routing matrix Z ∈ R |T |× m , where Z τ ∈ R m specifies task τ 's distribution over the shared modules. This formulation allows similar tasks to share adapters, while allowing dissimilar tasks to have non-overlapping parameters. The collection of adapters M can be interpreted as capturing different facets of knowledge, or latent skills , of the full multitask distribution.

At each forward pass, Poly LoRA adapters for task τ are constructed as follows:

where α i = softmax ( Z [ τ ]) i denotes the mixing weight of the i -th adapter in the inventory, and A ( i ) , B ( i ) , A τ , B τ ∈ R d × r . Here, the τ -th row of the routing matrix Z is effectively selecting which adapter modules to include in the linear combination. In our setting, where each task consists of modeling an individual, Z [ τ ] specifies which latent skills are activated for user τ ; we call this their style vector . As per Eqn 1, the final output of the linear mapping becomes h = ( W 0 + A τ ( B τ ) T ) x .

In Poly , the module combination step remains coarse , as only linear combinations of the existing modules can be generated. Caccia et al. [2022] propose a more fine-grained approach, called MultiHead Routing ( MHR ), which is what we use in our work. Similar to Multi-Head Attention [Vaswani et al., 2017], the input dimension of A (and output dimensions of B ) are partitioned into h heads, where a Poly -style procedure occurs for each head. The resulting parameters from each head are then concatenated, recovering the full input (and output) dimensions. See A.1 for more details.

Routing-only fine-tuning. While LoRA adapters can reduce the parameter cost from billions to millions [Liu et al., 2022], training the adapters for each new task can still be prohibitive when dealing with thousands of tasks. To this end, Caccia et al. [2022] proposed routing-only finetuning, where after an initial phase of pretraining, the adapter modules are fixed, and only the routing parameters Z are learned for a new task. This reduces the parameter cost for each additional task by several orders of magnitude, while maintaining similar performance. We use this method for few-shot learning.

3 MLMethodology

In this section, we detail our methodology for creating a generative model of individual behavior that enables our style analyses. Our methodology applies to any behavior cloning scenario with access to human demonstrations from multiple individuals. To demonstrate this generality, we apply it to two very different games: chess and Rocket League. We start with a base model for each and apply the MHR adapter framework to it, and then discuss model training and evaluation.

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

game state

p(action|x)

possible actions

player style vector

s

kill

inventory

+

Wpretrained

LoRA

A

B T

active player's style vector

players

skills

or

…

Layer N

s

kill

inventory

Layer 1

…

3.1 Model architecture

For chess, we follow McIlroy-Young et al. [2022] and use the Squeeze-and-Excitation (S&E) Residual 129 Network [Hu et al., 2018] as a base model, but with a deeper and wider configuration (see A.2). 130 At every residual block, an additional 2-layer MLP rescales the residual output along the channel 131 dimension to explicitly model channel interdependencies. The input is a 112-channel 8 × 8 image 132 representation of the chess board; the output is the predicted move represented as a 1858-dimensional 133 vector. The total parameters is 15.7M. For Rocket League, we use the GPT-2 architecture from 134 Radford et al. [2019] with a dimensionality of 768, 12 attention heads, and 12 layers. The input is a 135 49-dimensional vector with game physics information; the output is 8 heads: 5 with 3 bins of [-1, 0, 136 1] and 3 binary. The model has no embedding layer, as the game data points are passed directly as 137 tokens after processing. The total parameters is 87.7M. 138

139

140

141

142

143

144

145

146

147

To enable user-based adaptation, we incorporate the MHR adapters described in §2.2 into our base models, as illustrated in Fig. 1. In chess, for every linear transformation in the MLP used for channelwise rescaling, we add an MHR layer built of LoRA adapters with rank 16, for a total of 12 × 2=24 MHR layers. We use an adapter inventory of size 32 and a multi-head routing strategy with 8 heads. Therefore, for each user we must learn 32 × 8=256 routing parameters as their style vector. This yields 5M additional parameters. For Rocket League, we attach the adapters to the fully connected layer of each transformer block, resulting in 12 MHR layers of LoRAs with rank 16. We use an inventory size of 16 and 64 heads. This yields 13.8M additional parameters. To facilitate interpretability and style analysis, we use the same routing (style vector) across all MHR layers.

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

3.2 Data collection and partitioning

We use data from the largest open-source online chess platform, Lichess.org [Duplessis, 2021], which boasts a database of over 4.8 billion games. We collected Blitz games played between 2013 and 2020 inclusive-these are games with 3 or 5 minutes per side, optionally with a few seconds of time increment per move-and applied the same player filtering criteria as McIlroy-Young et al. [2022]. The resulting dataset comprises 47,864 unique players and over 244 million games. (See A.2 for a discussion on data imbalance.) For Rocket League, we collect data from a large open-source replay database, Ballchasing.com [CantFlyRL, 2024]. We use 2.2 million 1v1 replays from 2015 to mid-2022, totalling several decades of human game play hours at 5 minutes per game. After parsing, each Rocket League game state is a vector holding the player's 3D position, linear and angular velocity, boost remaining, rotation, and team; we also include the opponent's state and the position, linear and angular velocity of the ball. Given a game state, we have to predict the user's throttle, steer (while grounded), pitch, yaw, roll (while aerial), jump, boost, and handbrake. Additional processing was needed to correct for missing aerial controls and inconsistent sampling rates (24-27hz). Our full data processing procedure, including the challenges we faced, are detailed in A.3.

We divide the set of players into a few subsets to support our training methodology. The base player 163 set comprises all data and is used to train the base models. The fine-tuning player set is used to 164 fine-tune the MHR architecture shown in Fig. 1. (For both, we split each player's data into 80/10/10 for 165 train/test/validation.) The few-shot player set is used for few-shot learning based on a reference set of 166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

100 games per player. For our chess experiments, to enable a direct comparison with prior work, we create an additional fine-tuning player set consisting of the same 400 players used in those studies. Currently, we treat each player's data holistically, but in principle one could partition a player's data in different ways to perform a finer analysis of their playing style. We explore this in A.4.

3.3 Model training and evaluation

Base model. We train our base Maia model for chess using data from a base player set of all 47,864 players, treating this as a classification task of predicting human move y made in chess position x , given a datapoint ( x, y ) . We use the same loss functions and evaluation criteria as the original Maia work: Maia's policy head uses a cross entropy loss while the value head uses MSE; the output of the policy head is used to evaluate the model's move-matching accuracy.

We train our Rocket League model using a base player set of over 800,000 players, though the vast majority of players have 5 games or fewer. We discretize the actions into 3 bins for throttle, steer, pitch, yaw, and roll, as most of this data is close to 0, -1, or 1. We use binary outputs for jump, boost, and handbrake. A next-move prediction is labelled correct if and only if all of the outputs are correct.

MHR fine-tuning. To train the MHR LoRA adapters, we adopt the methodology used in Caccia et al. [2022]: namely, we freeze the base model and fine-tune the MHR layers and routing matrix using data from a fine-tuning player set. Recall that the routing matrix Z has a row (style vector) for each player in the fine-tuning set. Following Ponti et al. [2019], we use a two-speed learning rate, where the style vectors' learning rate is higher than the adapters', to enable better specialization.

For chess, we use two fine-tuning player sets in our experiments, creating two separate MHR -Maia models. The first set comprises all 47,864 players and is used to evaluate behavioral cloning and stylometry at very large scale. The second set is comprised of the same 400 players used by McIlroyYoung et al. [2022], which we use to compare few-shot learning and stylometry results. For Rocket League, we train an MHR -Rocket model on a fine-tuning set of 2,000 players with 100 games each.

Few-shot learning. To perform few-shot learning on our MHR models, we perform the 'routing-only fine-tuning" described in section 2.2 that additionally freezes all MHR LoRA adapters. Given a fewshot player, we add a (randomly-initialized) new row to Z and fine-tune it on the player's reference set of games, eventually learning a style vector for the player. Using this style vector, we can invoke a generative model of the player and use it to evaluate move-matching accuracy, as described above. To perform stylometry, if the player is a seen player (i.e., part of the fine-tuning set), then a matching style vector already exists in Z , and we can find it using cosine similarity. Otherwise, if the player is unseen , then we simply repeat the few-shot learning process on a query set of games (from the same player), and compare this new style vector to the entries in Z .

For chess, (unless stated otherwise), all of our few-shot experiments use the MHR -Maia model finetuned on the 400-player set from McIlroy-Young et al. [2022]. For Rocket League, the few-shot player set consists of 1,000 of the 2,000-player set used to fine-tune MHR -Rocket.

Evaluation. We evaluate a fine-tuned MHR model in two ways. First, we measure its move-matching accuracy, similar to how we evaluate the base models. However, since our MHR models provide a generative model for each player (invoked through their style vector), we can separately evaluate each player's model by applying it to their test set and measuring move-matching accuracy. The overall move-matching accuracy for the model is simply the average of these per-player accuracies.

Our second evaluation method uses the model to perform behavioral stylometry among all players in the fine-tuning set. This is done by leveraging our few-shot learning methodology (above). That is, given a query set of games from some player, we learn a new style vector in Z for those games via few-shot learning, and compare this vector to every other vector in Z . Using cosine similarity as our distance metric, we simply output the player with the highest cosine similarity to the query set vector.

4 Style methodology

The style vectors in Z represent distinct distributions over latent skills that give us a starting point for comparing player styles. For example, our stylometry method above uses the cosine similarity of

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

Table 1: Stylometry accuracy results. Seen few-shot players are a subset of the fine-tuning player set, unlike unseen players. Numbers for McIlroy-Young et al. [2022] and McIlroy-Young et al. [2021] are borrowed from their respective papers.

these vectors to determine how similar or different players are. However, style vectors also enable much more powerful capabilities, such as the ability to synthesize new (human-like) styles.

To begin, we measure the intra-player consistency of style vectors by splitting a player's dataset into disjoint subsets of varying size, and few-shot learning a style vector for each subset. We then investigate inter-player consistency by merging the datasets of two players and seeing if the style vector trained on the merged dataset is similar to the average of the two player's style vectors.

The latter method actually creates a new playing style that is human-like and yet previously unseen in the world. This suggests a more general approach to style synthesis: interpolate between players using a convex combination of their style vectors. To determine the playing strength of these new players, we can simulate games between them and the players they are derived from. The results of these games can be used to calculate a win rate, which can then be converted to a strength rating.

Currently, our advanced style synthesis techniques focus on chess, where there is a robust mapping between win rates and playing strength (the Elo rating system), and simulating games is cheap. Rocket League simulations are quite costly at present, but in principle the same methodology should apply and we plan to reduce these costs in future work.

In order to make style comparisons more human-understandable, we again exploit the generative nature of our MHR models. Inspired by the concept probing technique used to analyze AlphaZero (a deep RL chess engine) [McGrath et al., 2022], we use a set of human-coded heuristic functions found in Stockfish (a traditional chess engine) to evaluate a player's model. These functions capture concepts such as: king safety, material imbalance, piece mobility, and so on. By invoking a player's model on a fixed set of chess positions and seeing which move they select, we can use this to summarize how much emphasis the player places on the corresponding concepts.

Finally, we combine the above methods to design a simple but general method for steering a player's game style towards a specific attribute a , such as increasing their king safety, while limiting the changes on other attributes (so as to preserve their style). To achieve this, we first collect a set players X who exhibit high values for attribute a -determined, for example, by running their generative models on a fixed set of game states. We then extract the common direction among these players, by averaging their style vectors and subtracting the population average. This yields a style delta vector that can be added to any player's style vector to elicit the desired change.

5 Experiments

In this section, we demonstrate two main findings. First, MHR -Maia performs competitively with 246 prior methods for behavior cloning and stylometry in chess, while achieving unprecedented scale. 247 We also show that our approach can be applied to Rocket League, for both stylometry and move 248 prediction. Second, we show that explicitly capturing style vectors allows us to reason about and 249 perform arithmetic operations on generated behaviors. 250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

5.1 Behavioral Stylometry

In this section, we show that our models perform competitively with previous behavioral stylometry methods for both seen and unseen players. Here, the goal is to predict the player who produced a given set of games. We compare to individual model fine-tuning [McIlroy-Young et al., 2022], fitting a pre-trained Maia to the data from a single player, and to a Transformer-based method [McIlroy-Young et al., 2021], which embeds players in a 512-dimensional style space based on their gameplay. All reported accuracies are top-1 unless stated otherwise.

To perform stylometry on a query set of games, McIlroy-Young et al. [2022] suggest measuring the move-matching accuracy of each available fine-tuned model and selecting the best performing model. As seen in Table 1, this procedure works well, but is tremendously expensive-requiring computationally intensive inference calls on the entire query set for every candidate player.

In contrast, both the Transformer-based method and MHR -Maia scale much better to large numbers of players. The Transformer-based method needs only to condition on these games to produce a vector, while MHR -Maia needs only to fit a new vector. In either case, the produced vectors need only be matched to those in the player set, e.g., using cosine similarity. Table 1 compares both approaches, showing that MHR -Maia performs competitively or better, on a much larger universe. To do this, we use few-shot learning to compute style vectors for 10,000 players based on their 100 game reference sets, then fit new style vectors for each player based on their respective query sets. Note that the individual model fine-tuning method is omitted from the larger few-shot study due to scalability reasons. The Transformer-based method can scale, but it is not a generative model.

For Rocket League, to the best of our knowledge, we are the first to attempt stylometry. We report player identification results averaged over the few-shot player set. For each prediction, our MHR -Rocket approach must correctly identify each of the 1,000 players among a pool of 2,000 players. Yet, it reaches an accuracy of 86.7% (random: 0.05%), showcasing the validity of our approach.

5.2 Move generation

Here we compare the efficacy of our method to using individually fine-tuned models for each player. Finetuning individual models generally results in superior results compared to PEFT methods, as the increased parameter count produces more expressive models. However, they are also more computationally intensive to train and store. That said, in the domain of modeling individual behavior in chess, MHR -Maia is able to perform comparatively well despite using a much smaller parameter budget. Figure 2 shows that MHR -Maia matches individual model fine-tuning over a wide range of game counts. The base model is frozen for all game counts in MHR -Maia. The model has already learned the set of skills required to differentiate the players, all that is

25

50

100

500

1000

5000 10000

Game Count

57

58

59

60

61

62

Accuracy (%)

MHR-Maia Accuracy

needed with very few-shot learning is to find a proper recombination of the learned skills within the new style vectors. The Transformer-based method is omitted, as it is incapable of generating moves.

For Rocket League, we compare the next move prediction of our base model, with MHR -Rocket, to validate that our user-based conditioning generates better predictions. We find that, on average, MHR -Rocket increases the next move prediction from 53.1% to 56.1% .

5.3 Analysis of style vectors

In this section, we explore the consistency of our style vectors across different players and datasets.

Consistency across a single player. To showcase intra-player consistency, we first partition 50 players' datasets into disjoint subsets. We use 50 splits for chess and 20 for Rocket League. The subsets are sampled across a wide range of dates, opposing players, and playing sessions. Next, we train a style vector for every split across all players. We find that vectors corresponding to the same player will be similar to each other, and have low similarity with the other players and general population. This is visualized in Figure 3. This suggests that our neural network is able to find

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

Rocket League

Chess

bishop pairs

threats

mid game

mobility

mid game

king danger

passed pawns

imbalance

end game

eval

0.2

0.4

0.6

0.8

Player 1

Player 2

Player 3

Player 4

Player 5

mid game

Rocket League

Chess

bishop pairs

threats

mid game

mobility

mid game

king danger

passed pawns

mid game

imbalance

end game

eval

0.2

0.4

0.6

0.8

Player A

Player B

Interpolated

distinct tendencies for each player. To confirm, we sampled 5 random chess players, predicted their preferred move across 2 17 positions, and measured a series of Stockfish evaluation metrics per player. Figure 4 shows the distribution of these metrics for each player, demonstrating that these vectors store a wide diversity of styles.

Consistency across merged players. To parse out whether we can generate new styles using this information, we merged two players' datasets together to generate a new set with the tendencies of both players, measuring inter-player consistency. We then compared this new set of vectors to a different set of vectors generated by simply averaging the style vectors of the player pair. As seen in Figure 5 (left and center), vectors with the same two source players have very high similarity in both chess and Rocket League. We then sampled a random pair in the merged dataset, created a new player by averaging the two players' vectors, and recorded their gameplay according to the previous section. The results are visualized in Figure 5 (right), showing that the new player (green) has an intermediate playing style to the source players (red, blue).

5.4 Synthesis of new styles

Convex combinations. We show that interpolating between skill vectors results in a player whose level is a weighted average of the interpolated players. Here, we take 100 pairs of learned player vectors, such that one item in the pair corresponds to a strong player and the other to a weaker player. We then gradually interpolate between the weak and strong player as (1 -λ ) u w + λu s , 0 ≤ λ ≤ 1 , where u w and u s are respectively vectors representing the weak and strong player. For each value of λ we simulate 1,000 games between the interpolated vector and u s , the stronger player.

Figure 6 plots the win rate of the interpolated player as a function of λ for each player pair we considered. This plot demonstrates that win rate progresses in a roughly linear fashion, starting off winning infrequently against the stronger player and eventually winning roughly half the time as the interpolated player converges to the stronger player.

Directly steering player style. Finally, we directly control the playing style of a player by creating 327 skill vectors according to the procedure described in 4. We choose players in our chess dataset with 328 high (>2 std) bishop pair utilization, and separately players with high king danger. Figure 7 shows 329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

0.0

0.2

0.4

0.6

0.8

1.0

0.1

0.2

0.3

0.4

0.5

attack

bishop pair

mobility

king danger pawns eg

Style attributes

0.0

0.5

1.0

1.5

2.0

Difference in std.

Steerability of game-playing style via delta style vectors

Manipulating Bishop pair

the change in 2,000 randomly sampled player's stockfish evaluations after adding the skill vector corresponding to each heuristic to their style vectors. Indeed, we see that the player's style is steered towards the attribute in question, with model impact on other attributes.

6 Related Work

Stylometry and player style modeling. Originally referring to performing author attribution via statistical analysis of text [Tweedie et al., 1996, Neal et al., 2017], stylometry has since come to refer to the general task of identifying individuals given a set of samples or actions, and has found broad application for tasks such as handwriting recognition [Bromley et al., 1993], speaker verification [Wan et al., 2018], identifying programmers from code [Caliskan-Islam et al., 2015], determining user age and gender from blog posts [Goswami et al., 2009], and identifying characteristics of authors of scientific articles [Bergsma et al., 2012]. In the context of gaming (covered in the introduction), stylometry is closely related to playstyle modeling, where the goal is to associate a player with a reference style, such as by building agents representative of different playstyles and find the closest behavioral match [Holmgård et al., 2014], or gathering gameplay data and applying methods such as clustering [Ingram et al., 2022], LDA [Gow et al., 2012], Bayesian approaches [Normoyle and Jensen, 2015] and sequential models [Valls-Vargas et al., 2015] to identify groups of players with similar styles. Unlike our work, these approaches focus on aggregate playstyles, and do not learn generative models that can be conditioned on an individual's style.

Our method for style synthesis is inspired by earlier work on vector arithmetic with embeddings [Church, 2017], as well as recent work on steering multiask models with task vectors [Ilharco et al., 2023]. Finally, our steering method is reminiscent of Radford et al. [2016], which manipulates the model's latent space to generate images containing specific attributes.

Parameter-efficient adaptation Approaches for efficient adaption of a pretrained model can be broadly grouped in two categories. Adapter based methods inject new parameters within a pretrained model, and only updates the newly inserted parameters while keeping the backbone fixed. Houlsby et al. [2019] defines an adapter as a two-layer feed-forward neural network with a bottleneck representation, and are inserted before the multi-head attention layer in Transformers. Similar approaches have been used for cross-lingual transfer [Pfeiffer et al., 2020]. Adapters have also been used in vision based multitask settings [Rebuffi et al., 2017]. More recently, Ansell et al. [2022] propose to learn sparse masks, and show that these marks are composable, enabling zero-shot transfer. Lastly, Hu et al. [2022] learn low-rank shifts on the original weights, and [Liu et al., 2022] learns an elementwise multiplier of the pretrained model's activations. Adapters have also been used in multitask settings. Chronopoulou et al. [2023] independently trains adapters for each task. In order to transfer to new tasks, the authors merge the parameters of the adapters of relevant training tasks.

7 Conclusion

We show that individual player behavior can be modeled at very large scale in games as different as chess and Rocket League. We cast this problem in the framework of multi-task learning and employ modular PEFT methods to learn a shared set of skills across players, modulated by a distinct style vector for each player. We use these style vectors to perform behavioral stylometry, analyze player styles, and synthesize and steer new styles.

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

References

Ansell, E. Ponti, A. Korhonen, and I. Vuli´ c. Composable sparse fine-tuning for cross-lingual transfer. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 1778-1796, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long.125. URL https: //aclanthology.org/2022.acl-long.125 .

Bergsma, M. Post, and D. Yarowsky. Stylometric analysis of scientific articles. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 327-337, 2012.

R.-A. Braaten. Rl-rpt - rocket league replay pre-training. https://github.com/Rolv-Arild/replaypretraining, 2022.

Bromley, I. Guyon, Y. LeCun, E. Säckinger, and R. Shah. Signature verification using a" siamese" time delay neural network. Advances in neural information processing systems , 6, 1993.

Caccia, E. Ponti, L. Liu, M. Pereira, N. L. Roux, and A. Sordoni. Multi-head adapter routing for data-efficient fine-tuning. arXiv preprint arXiv:2211.03831 , 2022.

Caliskan-Islam, R. Harang, A. Liu, A. Narayanan, C. Voss, F. Yamaguchi, and R. Greenstadt. De-anonymizing programmers via code stylometry. In 24th USENIX security symposium (USENIX Security 15) , pages 255-270, 2015.

CantFlyRL. Ballchasing.com. https://ballchasing.com/, 2024.

Carroll, R. Shah, M. K. Ho, T. Griffiths, S. Seshia, P. Abbeel, and A. Dragan. On the utility of learning about humans for human-ai coordination. Advances in neural information processing systems , 32, 2019.

Caruana. Multitask learning. Machine learning , 28:41-75, 1997.

Chronopoulou, M. Peters, A. Fraser, and J. Dodge. AdapterSoup: Weight averaging to improve generalization of pretrained language models. In Findings of the Association for Computational Linguistics: EACL 2023 , pages 2054-2063, Dubrovnik, Croatia, May 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-eacl.153. URL https://aclanthology.org/2023.findings-eacl.153 .

W. Church. Word2vec. Natural Language Engineering , 23(1):155-162, 2017.

Duplessis. Lichess. http://lichess.org, 2021. Accessed: 2021-01-01.

Emery. Rlgym - the rocket league gym. https://rlgym.org/, 2021.

Florence, C. Lynch, A. Zeng, O. A. Ramirez, A. Wahid, L. Downs, A. Wong, J. Lee, I. Mordatch, and J. Tompson. Implicit behavioral cloning. In Conference on Robot Learning , pages 158-168. PMLR, 2022.

Goswami, S. Sarkar, and M. Rustagi. Stylometric analysis of bloggers' age and gender. In Proceedings of the International AAAI Conference on Web and Social Media , volume 3, pages 214-217, 2009.

Gow, R. Baumgarten, P. Cairns, S. Colton, and P. Miller. Unsupervised modeling of player style with lda. IEEE Transactions on Computational Intelligence and AI in Games , 4(3):152-166, 2012.

Holmgård, A. Liapis, J. Togelius, and G. N. Yannakakis. Evolving personas for player decision modeling. In 2014 IEEE Conference on Computational Intelligence and Games , pages 1-8. IEEE, 2014.

Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly. Parameter-efficient transfer learning for NLP. In International Conference on Machine Learning , pages 2790-2799, 2019. URL http://proceedings.mlr.press/v97/ houlsby19a/houlsby19a.pdf .

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9 .

Hu, L. Shen, and G. Sun. Squeeze-and-excitation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 7132-7141, 2018.

Ilharco, M. T. Ribeiro, M. Wortsman, L. Schmidt, H. Hajishirzi, and A. Farhadi. Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=6t0Kwf8-jrj .

Ingram, B. Rosman, C. van Alten, and R. Klein. Play-style identification through deep unsupervised clustering of trajectories. In 2022 IEEE Conference on Games (CoG) , pages 393-400. IEEE, 2022.

Jelley, Y. Cao, D. Bignell, S. Devlin, and T. Rashid. Aligning agents like large language models, 2024. URL https://openreview.net/forum?id=kQqZVayz07 .

Liu, D. Tam, M. Muqeeth, J. Mohta, T. Huang, M. Bansal, and C. Raffel. Few-shot parameterefficient fine-tuning is better and cheaper than in-context learning, 2022. URL https://arxiv. org/abs/2205.05638 .

McGrath, A. Kapishnikov, N. Tomašev, A. Pearce, M. Wattenberg, D. Hassabis, B. Kim, U. Paquet, and V. Kramnik. Acquisition of chess knowledge in alphazero. Proceedings of the National Academy of Sciences , 119(47):e2206625119, 2022.

McIlroy-Young, S. Sen, J. Kleinberg, and A. Anderson. Aligning superhuman ai with human behavior: Chess as a model system. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , page 1677-1687, 2020.

McIlroy-Young, Y. Wang, S. Sen, J. Kleinberg, and A. Anderson. Detecting individual decisionmaking style: Exploring behavioral stylometry in chess. Advances in Neural Information Processing Systems , 34:24482-24497, 2021.

McIlroy-Young, R. Wang, S. Sen, J. Kleinberg, and A. Anderson. Learning models of individual behavior in chess. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , page 1253-1263, 2022.

Neal, K. Sundararajan, A. Fatima, Y. Yan, Y. Xiang, and D. Woodard. Surveying stylometry techniques and applications. ACM Computing Surveys (CSuR) , 50(6):1-36, 2017.

Normoyle and S. Jensen. Bayesian clustering of player styles for multiplayer games. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment , volume 11, pages 163-169, 2015.

Pearce and J. Zhu. Counter-strike deathmatch with large-scale behavioural cloning. In 2022 IEEE Conference on Games (CoG) , pages 104-111. IEEE, 2022.

Pfeiffer, I. Vuli´ c, I. Gurevych, and S. Ruder. MAD-X: An Adapter-based framework for multi-task cross-lingual transfer. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 7654-7673, Nov. 2020. URL https://aclanthology. org/2020.emnlp-main.617 .

A. Pomerleau. Alvinn: An autonomous land vehicle in a neural network. Advances in neural information processing systems , 1, 1988.

M. Ponti, H. O'Horan, Y. Berzak, I. Vuli´ c, R. Reichart, T. Poibeau, E. Shutova, and A. Korhonen. Modeling language variation and universals: A survey on typological linguistics for natural language processing. Computational Linguistics , 45(3):559-601, 2019. URL https://watermark.silverchair.com/coli_a_00357.pdf .

M. Ponti, A. Sordoni, Y. Bengio, and S. Reddy. Combining parameter-efficient modules for tasklevel generalisation. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics , pages 687-702, Dubrovnik, Croatia, May 2023. Association for Computational Linguistics. URL https://aclanthology.org/2023.eacl-main.49 .

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

Radford, L. Metz, and S. Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In International Conference on Learning Representations , 2016.

Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language models are unsupervised multitask learners. 2019.

S.-A. Rebuffi, H. Bilen, and A. Vedaldi. Learning multiple visual domains with residual adapters. Advances in neural information processing systems , 30, 2017.

RLBot. Rlbot. https://github.com/RLBot/RLBot, 2017.

Ruder, M. E. Peters, S. Swayamdipta, and T. Wolf. Transfer learning in natural language processing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Tutorials , pages 15-18, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-5004. URL https://aclanthology.org/N19-5004 .

SaltieRL. Carball. https://github.com/SaltieRL/carball, 2024.

Schaal. Learning from demonstration. Advances in neural information processing systems , 9, 1996.

Schäfer, L. Jones, A. Kanervisto, Y. Cao, T. Rashid, R. Georgescu, D. Bignell, S. Sen, A. T. Gavito, and S. Devlin. Visual encoders for data-efficient imitation learning in modern video games, 2023.

J. Tweedie, S. Singh, and D. I. Holmes. Neural network applications in stylometry: The federalist papers. Computers and the Humanities , 30:1-10, 1996.

Valls-Vargas, S. Ontanón, and J. Zhu. Exploring player trace segmentation for dynamic play style prediction. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment , volume 11, pages 93-99, 2015.

Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. CoRR , abs/1706.03762, 2017. URL http://arxiv.org/abs/1706. 03762 .

Wan, Q. Wang, A. Papir, and I. L. Moreno. Generalized end-to-end loss for speaker verification. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 4879-4883. IEEE, 2018.

Wang, Y. Tsvetkov, O. Firat, and Y. Cao. Gradient vaccine: Investigating and improving multitask optimization in massively multilingual models. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum?id=F1vEjWK-lH_ .

Zhou, C. Barnes, J. Lu, J. Yang, and H. Li. On the continuity of rotation representations in neural networks, 2020.

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

516

517

518

519

520

521

522

523

524

525

526

527

528

529

530

531

532

533

534

535

536

537

538

539

540

541

542

A Appendix

A.1 Multi-Head Adapter Routing

In Poly , the module combination step remains coarse , as only linear combinations of the existing modules can be generated. Caccia et al. [2022] propose a more fine-grained module combination approach, referred to as Multi-Head Routing ( MHR ). Similar to Multi-Head Attention [Vaswani et al., 2017], the input dimension of A (and output dimensions of B ) are partitioned into h heads, where a Poly -style procedure occurs for each head. The resulting parameters from each head are then concatenated, recovering the full input (and output) dimensions. This makes the module combination step piecewise linear , with a separate task-routing matrix Z learned for each head.

Formally, a MHR layer learns a 3-dimensional task-routing tensor Z ∈ R |T |×|M|× h . The 2D slice Z : , : ,k ∈ R |T |×|M| of the tensor Z denotes the distribution over modules for the k -th head, and W [ k ] ∈ R d h × r the k -th partition along the rows of the matrix W ∈ R d × r . The adapter parameters A τ ∈ R d × r for task τ , and for each adapter layer, are computed as (similarly for B τ ):

where α i,k = softmax ( Z [ τ ,:,k] ) i . Importantly, the number of LoRA adapter parameters does not increase with the number of heads. Only the task-routing parameters linearly increase with h for MHR vs. Poly . However, this cost is negligible as the parameter count of the routing matrices is much smaller than for the LoRA modules themselves.

A.2 Maia Architecture/Data

Our base Maia architecture follows McIlroy-Young et al. [2022] and uses the Squeeze-and-Excitation (S&E) Residual Network of [Hu et al., 2018]. At every residual block, channel information is aggregated across spatial dimensions via a global pooling operation. The resulting vector is then processed by a 2-layer MLP, with a bottleneck representation compressing the number of channels by r . The output of this MLP is a one-dimensional vector used to scale the output of the residual block along the channel dimension. We use 12 residual blocks containing 256 filters, and a bottleneck compression factor of r = 8 . We note that this differs from the base Maia model in McIlroy-Young et al. [2022], which uses 64 filters and 6 residual blocks.

While our dataset has a median game count of 3,479 games, many players may have as few as 10-50 games, implying some degree of data imbalance. Our evaluation of few-shot learning shows that 100 games is sufficient to learn the style vector of an unseen player. However, one might still ask how accurately such a style vector is given a very small number of games. To explore this, we first split a player into disjoint sets of 10, 25, 50, 100, 500, and 1,000 games. We then train a style vector on each set. As a baseline, we train a style vector on 10,000 games and track the cosine similarity of the smaller-set style vectors relative to this baseline vector. We show the results in Figure 8.

A.3 Rocket League Architecture/Data

The 1v1 replays dataset was scraped over the course of several weeks from the Ballchasing.com API using the Grand Champion subscription tier, though the API does have a slower free tier. This API yields raw game replays, which are uploaded by users either manually or using a community-made plugin for the game. The replays are in a binary format which must be parsed using community-made projects such as Carball [SaltieRL, 2024].

The Carball library allows us to convert the binary replay format to a more standard CSV format, which we save to a Cloud binary blob storage. The data present in both is a lossy reconstruction of game states, and requires some processing to be usable. In particular, the data is sampled at an inconsistent rate (varying between 24hz and 27hz), contains repeated physics ticks, and is missing action data for aerial controls (pitch, yaw, roll).

We resolve the issue of sampling rate and repeated ticks by removing repeated ticks, and doing a time-weighted resampling and interpolation to a standard 10hz for model training, though we found

543

544

545

546

547

548

549

550

551

552

553

554

555

556

557

558

559

560

561

562

563

564

565

566

567

568

569

570

571

572

573

574

10

25

50

100

500

1000

Games

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Cosine Similarity

that 30hz also works well. Note that the actual game physics ticks occur at 120hz, so any value aligned with this should work. Without these changes, the model performs extremely poorly and is unable to navigate the arena.

We resolve the issue of missing aerial controls through the physics-based solver present in the Carball library. The estimation of these controls is not perfect, but it is sufficient for our purposes. Some previous community work has used inverse dynamics [Braaten, 2022] trained from rollouts of in-game bots to solve for these actions, though we opted to not use this due to the inconsistency in replay data sampling.

The data returned by the CSVs are fairly large, messy, and inconsistent. We apply the following transformations to the dataframe to bring the values closer to 0:

Divide position by 2300

Divide linear velocity by 23000

Divide angular velocity by 5500

Divide boost by 255

Encode rotation Euler angles according to Zhou et al. [2020]

Additionally, when turning the data into tokens for use in our model, we add in an extra dimension to represent the team, and concatenate the opponent's data points along with the position, linear and angular velocity of the ball. We complete all of these transformations at runtime.

We also have to align the data returned by the simulators for Rocket League with the data used to train the model, RLBot [RLBot, 2017] and RLGym [Emery, 2021]. Along with including an extra dimension to represent the team, we apply the following transformations to all samples obtained from the game:

Divide position by 2300

Divide linear velocity by 2300

Divide angular velocity by 5.5

Divide boost by 100

The skill distribution of the players in our dataset can be found in Figure 9.

A.4 Implicit Stationarity Assumptions

Most of the existing work in chess assumes that a player remains stationary over time and across gameplay situations. However, in reality, a player's style may depend on the type of opponent they are facing, which opening is used, which stage of the game they are in (opening, middle, endgame), and so on. For instance, McIlroy-Young et al. [2021] observe that stylometry accuracy drops when

bronze-1

bronze-3

silver-1

silver-2

gold-2

gold-3

platinum-1

platinum-2

platinum-3

diamond-1

diamond-2

diamond-3

champion-1

champion-2

champion-3

grand-champion

grand-champion-1

grand-champion-2

grand-champion-3

supersonic-legend

Rank

0.00

0.05

0.10

%

Rocket League Rank Distribution

removing the opening (e.g., the first 15 moves) moves, suggesting that the opening has an outsized 575 effect on style identification. Our approach does not rely on these assumptions and can in principle 576 be applied to arbitrary subsets of a player's data. For instance, one could split a player's data into 577 opening, middlegame, and endgame moves and train a separate style vector for each. One could 578 further split the data based on which defense the opponent uses, what time of the day it is, etc.. 579 Despite treating players holistically and avoiding any splits of their data, we are still able to capture 580 the peculiarities of each individual's playing style and perform stylometry with high accuracy. This 581 also enables us to compare our results to those of prior work, which also treats player data holistically. 582

583

A.5 Delta Style Vector Computation

Algorithm 1 Style Delta Vector computation

Input:

X : Style vectors of top-k players for attrib. a ; P : Style vectors of all players in population Output ∆ a : Style delta vector for attr. a V a = mean ( X, axis = 'players' ) V P = mean ( P, axis = 'players' ) ∆ a = V a -V P Returns ∆ a

584

585

586

587

588

589

590

591

592

593

594

595

596

597

598

599

600

601

602

603

604

605

606

607

608

609

610

611

612

613

614

615

616

617

618

619

620

621

622

623

624

625

626

627

628

629

630

631

632

633

634

635

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract directly summarizes the key results of the paper, which focus on performing behavioral stylometry at scale in games (chess and Rocket League)

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Please see Related work and explanation of our limited style synthesis/steering results for Rocket League.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: we dont use proofs as a contribution

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide thorough implementation details, some of which appear in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We will open source our data and models upon publication.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See Appendix and main paper for full experimental details.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: While we do not use error bars, our methodology is properly described and clarifies the significance or our results.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

740

741

742

743

744

745

746

747

748

749

750

751

752

753

754

755

756

757

758

759

760

761

762

763

764

765

766

767

768

769

770

771

772

773

774

775

776

777

778

779

780

781

782

783

784

785

786

787

788

789

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We talk about the exact model parameter sizes, and use standard models that are very small. Due to the use of standard base models, information on computational resources required to train them based on token count is readily available.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Upon reading the code of Ethics, the paper conforms to the code of ethics. Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: This paper extends prior work on behavior cloning of individual behavior, but it is not the first to perform such fine-grained behavior cloning or observe their societal implications. Prior work by McIlroy-Young et al. discusses the implications of mimicking individual behavior with high fidelity (see "Mimetic Models: Ethical Implications of AI that Acts Like You" in AIES '2022).

Guidelines: 790

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Papers and codebases are properly cited.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper adapts Maia, a group-level model and variant of the AlphaZero model proposed by McIlroy-Young et al. (2020), to function at an individual level. The authors focused on scaling the fit to individual behavior using techniques like LoRA for fine-tuning LLMs. Moreover, the learned embeddings were shown to encode decision-making styles that correspond to individual behaviors. Linear combinations of these embeddings can result in desirable decision-making styles, creating synthetic styles.\n\nSTRENGTHS:\nThis is an interesting paper that treats fitting individual data as a task in a multi-task learning framework. Recent fine-tuning techniques for LLMs have shown promise in adapting a base model, which captures group-level behavior, to individual-level data on a large scale, accommodating hundreds of thousands of individual datasets.\n\nWEAKNESSES:\nThe paper lacks a proper benchmark for comparing the accuracy gains from their proposed method. Hypothetically, if the datasets they considered (Chess and Rocket League) exhibit no individual variability in behaviors, meaning all individuals behave identically to the group-level agent, we should expect no accuracy gain whatsoever from adapting a group-level base model to individual-level data. Conversely, if significant variability stems from individual differences, we should expect a substantial gain. Currently, the 0.4% and 4.8% accuracy gains reported in Table 1 are not compared to any benchmark that considers the maximal possible variability that could be extracted from individual-level data. This leaves readers uncertain about the actual gains achieved by the method. It would be beneficial to establish a theoretically optimal benchmark for maximal possible accuracy gains across all datasets.\n\nQUESTIONS:\nIs it possible to use steering or synthetically created new styles to predict human decisions? This approach might provide a stronger test for the learned style vectors that capture human-like behaviors.\n\nLIMITATIONS:\nThis paper did not explicitly include a section for Limitation and Future Research.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper explores modelling user behaviour for chess and rocket league using a PEFT-based method, wherein users are modelled using a composition of MHR adapters. The authors evaluate their approach both for predicting which player played a particular game and for predicting the next move of a given player, and find that their MHR-based approach outperforms prior work in both cases. Further analysis shows that the MHR-based approach further allows steering and combining user vectors to produce new styles, and that the vectors are able to capture a wide diversity of strategies.\n\nSTRENGTHS:\nThe approach is simple but intuitive, applying MHR-style adapters to stylometry and exploring the benefits of this approach. The approach does seem to result in improved performance over prior work, and the analysis of the adapters showing that they can be combined to create new styles, or ‘steer’ a user adapter towards a particular style, is interesting, and highlights that using MHR for learning style vectors allows for interesting analyses/applications. The experimental setup seems reasonable, and the authors show their approach works across more than just chess (although chess is the main focus of the work).\n\nWEAKNESSES:\n- The MHR approach does seem to underperform full-finetuning when there is a large number of games available, as shown in Figure 2, and as expected for a PEFT-based method.\n- While older stylometry work [1] was able to produce user vectors by just performing inference (albeit not being generative), this approach requires training, and so might be more computationally expensive as a result (PEFT training is cheap, but still requires computing the full backward pass to compute gradients for the adapters). Grounding some of the discussion in section 5.1 with compute estimates (e.g. estimated FLOPs) would be useful.\n- The novelty is somewhat limited, as the primary method (MHR adapters, linearly interpolating between adapters) has been explored in prior work. However, I think that the application of this idea to a new domain (user identification/move generation) is still novel and interesting.\n\n[1] R. McIlroy-Young, Y. Wang, S. Sen, J. Kleinberg, and A. Anderson. Detecting individual decision making style: Exploring behavioral stylometry in chess. Advances in Neural Information Process439 ing Systems, 34:24482–24497, 2021.\n\nQUESTIONS:\n1. I’m curious about the compute cost of your approach against the other methods. Taking into account training and inference, what are the differing FLOPs costs? Using adapters is typically useful for reducing memory costs (since you don’t need to store optimizer states for all parameters), but still requires a similar number of FLOPs to train.\n2. In the Polytropon paper, they attempt to analyse how tasks are distributed among the skill vectors. While this might be a bit trickier with MHR, I’m curious if you observe that if the skill vectors learnt through MHR training correspond to any human-interpretable features? This would be an interesting analysis!\n\nLIMITATIONS:\nThe authors address the limitations of their approach reasonably.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors found a way to obtain player stylometry using BC on a massive amounts of data. Their idea uses multiple LoRAs and routing matrix to obtain a style vector for each player.\n\nSTRENGTHS:\n- Very interesting technical solution and innovative way to use LoRA and routing matrix to obtain player style vectrors using behavioural cloning. \n- I think there is a large potential in using this technique in analyzing large game play databases. Possibly to glean new insights. \n- Potential also exists to combine these ideas to IRL / adversarial imitation learning schemes. It is not clear that which expert demos we should use to show to the IRL algorithm. Should it be just one pleayer, or maybe k-most similar players to some seed player?\n\nWEAKNESSES:\n- Applications as of now are not very compelling. One option would be to take lichess.org data and ask some interesting research question about the player data that has not seen a sufficient answer. Then use this technique to answer it. \n- Another option would be to integrate this technique to some IRL scheme. \n- Authors cite speaker recognition in the related work, but did not study it more closely. In biometric recognition we have essentially similar task as in the present paper. Idea is to turn observed data (audio in speaker recognition) into fixed length vector and then by comparing these vectors obtain downstream recognition. Where the present paper goes wrong is that evaluation metric is identification accuracy, whereas in biometrics it is known that task is verification task and metric is ROC or DET curve and equal error rate (EER) summary statistic. Authors should use it, in addition to identification. \n- Speaker verification also gives nice possibility for extensions to the current work. One idea back in the day was to do Baysian adaptation from the general speaker model to a specific speaker model. These ideas were also presented in the following paper https://arxiv.org/abs/2012.01244\n\nQUESTIONS:\n-\n\nLIMITATIONS:\n-", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose to solve the problem of behavior stylometry, which is to identify the style of a player’s policy in the game, by regarding it as a multi-task learning problem. Each player’s style is a distinct task. Previous methods are either not scalable or not generative, in that they cannot predict the moves of a player given a query set of games played by that player. The authors aim to design a model capable of generating the moves of different players in a scalable manner. To do that, the authors trained a set of Low Rank Adapters (LoRAs) over a base model. The base model is trained with behavior cloning on the whole dataset, and the LoRAs are trained on a specific set of training player dataset separately. Additionally, a routing matrix is trained with LoRAs to specify the distribution over the LoRAs for each player. In this case, the authors claim that the routing matrix is a compact representation of the players’ skills, and encourage the adapters to learn different latent skills while preserved the shared knowledge within the base model. The authors also claim that the routing matrix supports few-shot learning, and induces a series of applicable benefits on stylometry, such as interpreting and manipulating the style of a player.\n\nSTRENGTHS:\n1. The paper is well written and clearly conveyed.\n2. The proposed method achieves convincing results. \n3. Their dataset is partitioned and rotated properly so that they can analyze the inter-player and intra-player consistency of the method, which provides strong convincing for the paper.\n\nWEAKNESSES:\n1. The main concern I have for this paper is the novelty of the proposed method. The base model for Chess is from a previous work called Maia, and trained with existing techniques such as LoRA and Polytropon, either of which is novel to me.\n2. The significance of the problem setting, behavior stylometry, is not clear. The authors only have a very short introduction of what it is and its usage in the first section. However, the benefit of modeling accurate behavior stylometry is still vague. For example,  we have very high performance models in chess, even outperforming human players, then why do we need to model the style of human players via behavior stylometry? To me, accurately predict the moves of a player is not the ultimate goal of a game, the ultimate goal is to win the game. How can behavior stylometry help in winning the game? This should be better explained with more introduction and related experiments.\n\nQUESTIONS:\n1. How can behavior stylometry enhance human players? Why it is important to predict the moves of different human players, especially in the case of chess where super human models already exist? Can we analyze the style of AI methods with your method?\n2. Why do you choose chess and Rocket League to test your approach? Are there any other potential games or environments applicable for your method?\n\nLIMITATIONS:\nAs stated in the Weakness part, the analyze on the significance of behavior stylometry is limited. The authors work on a sub-problem of the game, and didn’t connect it back to the game itself.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}