Generative Modeling of Individual Behavior at Scale

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 17:04:51

Abstract: Recent years have seen a growing interest in using AI to model human behavior, particularly in domains where humans learn from or collaborate with this technology. While most existing work attempts to model human behavior at an aggregate level, our goal is to model behavior at the individual level. Recent work in the domain of chess has shown that behavioral stylometry, or the task of identifying a person from their actions alone, can be achieved with high accuracy among a pool of a few thousand players. However, this approach cannot generate actions in the style of each player, and hence cannot reason about or influence player behavior in practice. We provide a new perspective on behavioral stylometry that addresses these limitations, by drawing a connection to the vast literature of transfer learning in NLP. Specifically, by casting the stylometry problem as a multi-task learning problem---where each task represents a distinct---we show that parameter-efficient fine-tuning (PEFT) methods can be adapted to model individual behavior in an explicit and generative manner, at unprecedented scale. We apply our approach at scale to two very different games: chess (47,864 players) and Rocket League (2,000 players).

Our approach leverages recent modular PEFT methods to learn a shared set of skill parameters that can be combined in different ways via style vectors. Style vectors enable two important capabilities. First, they are generative: we can generate actions in the style of a player simply by conditioning on the player's style vector. Second, they induce a latent style space that we can interpret and manipulate algorithmically. This allows us to compare different player styles, as well as synthesize new (human-like) styles, e.g. by interpolating between the style vectors of two players.

Main paper: 
 introduction: 
 With the rapidly advancing capabilities of machine learning in recent years, it has become increasingly important to find constructive ways for humans to interact with this technology. Even in domains where AI has achieved proficiency or superhuman ability, there is still considerable value in using machine learning to understand how humans approach these tasks. Such an understanding can help identify areas for improvement in humans, find compatible human partners, develop effective AI collaborators or teachers, and create more human-like experiences, for example.A common method for capturing human behavior is behavioral cloning (BC), a form of imitation learning (Schaal, 1996) which applies supervised learning on fixed demonstrations collected for a given task. While traditionally used in domains such as robotics (Florence et al., 2022) and selfdriving vehicles (Pomerleau, 1988) , BC has seen increasing use in gaming, where the above use cases can be explored in a safe, virtual environment. Indeed, recent work in games like Counter-Strike (Pearce & Zhu, 2022; Pearce et al., 2023) and Overcooked (Carroll et al., 2019; Strouse et al., 2021; Yang et al., 2022) have developed human-like models that match human demonstrations closely; in the case of Overcooked, these models were used to train AI partners that collaborate effectively with human players. While the majority of this work has focused on modeling human behavior in aggregate, we believe the most value can be derived from modeling human behavior at the individual level. To that end, recent results in the game of chess have shown the most promise. McIlroy-Young et al. (2020) used behavior cloning to create a set of models called Maia that match human play at 9 aggregate skill levels. By fine-tuning these models on the data of 400 individual players, they created 400 personalized models that achieve a 4-5% higher move-matching accuracy on average compared to the best performing aggregate Maia model (McIlroy-Young et al., 2022) . The authors show that this is sufficient to perform behavioral stylometry with high accuracy, where the goal is to identify which person played a given query set of games. In this case, the authors simply apply the 400 personalized models to the query set and output the model with the highest move-matching accuracy. As a more scalable approach, the authors designed a Transformer-based embedding for games and players (a player's embedding is the average of their game embeddings), and showed that this can be used to perform stylometry among 2,844 players with high accuracy, by embedding the query set of games and matching it to the closest player in the embedding (McIlroy-Young et al., 2021) . This method supported few-shot learning of unseen players given a reference set of 100 games. Unlike the personalized models, however, the embedding was not generative: it could be used to identify players, but could not generate moves in their playing style. This paper advances the above work in three important ways, focusing on the game of chess:1. We perform behavioral stylometry at an unprecedented scale, across 47,864 players. Using query sets of 100 games, we achieve an accuracy of 94.4%. 2. Our model is generative: it learns an explicit style vector for each player that can be used to generate actions in their playing style. Our per-player models have move-matching accuracy in the range 45-69%, with a mean of 59%, even for players with very few (e.g., 50) games. 3. Our style vectors induce a latent space that can be interpreted and manipulated algorithmically. This allows us to analyze a player's style using human-understandable concepts, and synthesize new, human-like styles by interpolating between two existing player styles.Our key insight for achieving these results is that behavioral stylometry can be viewed as a multi-task learning problem, where each task represents an individual person. The goal here is to generalize across an initial set of players (tasks) while supporting few-shot learning of new players (tasks).To do this efficiently, we leverage recent advances in parameter-efficient fine-tuning (PEFT) (Ponti et al., 2023; Caccia et al., 2022) . Specifically, we augment an existing model (such as Maia) with a set of Low Rank Adapters (LoRAs) and a routing matrix that specifies a distribution over these adapters for each player. Unlike approaches that train a separate LoRA for each task, this modular design allows players to softly share parameters in a fine-grained manner. We apply this adapter framework to a modified version of the Maia model. By training Maia to convergence across all player data before fine-tuning the adapters and routing matrix on per-player data, our methodology encourages the adapters to learn different facets of knowledge, or latent skills, that explain the variance between players. Since each row of the routing matrix implies a distribution over these latent skills, we can interpret each row as a style vector for the corresponding player. Our experiments show that this vector is a good characterization of the player's style, enabling accurate stylometry at scale, and facilitating interpretation and synthesis of styles. 
background: 
 In multitask learning (Caruana, 1997; Ruder et al., 2019) , we are given a collection of tasks T = T 1 , . . . , T |T | , each task T i associated with a dataset D i = (x 1 , y 1 ), ..., (x ni , y ni ) . Multitask learning exploits the similarities among related training tasks by transferring knowledge among them, which in turn increases sample efficiency. Ideally, it builds representations easily adaptable to new tasks from potentially few target examples. The premise of this paper is that modeling individual human behavior from a pool of players can be interpreted as a multitask learning problem. In other words, each task T i consists of modeling the behavior of a specific player; and the dataset D i corresponds to the set of game moves made by player i. Specifically, an (x, y) tuple denotes a chess board position at a specific point in time during a game, along with player i's move in this game state. For the rest of the paper, we will thus use the notion of tasks and players interchangeably.Popularized in NLP, parameter-efficient fine-tuning (PEFT) (Houlsby et al., 2019; Hu et al., 2022; Liu et al., 2022) approaches have emerged as a scalable solution for adapting Large Language Models to several downstream tasks. Indeed, standard finetuning of pretrained LLMs requires updating possibly billions of parameters for each task at hand, which makes storing and serving each taskspecific model expensive. PEFT methods instead freeze the pretrained model and inject a small set of trainable task-specific weights, or "adapters".One such approach is the use of Low Rank Adapters (LoRA) (Hu et al., 2022) , which modify linear transformations in the network by adding a learnable low rank shifth = W 0 + ∆W x = W 0 + AB T x.(1)Here, W 0 ∈ R d×d are the (frozen) weights of the pre-trained model, and A, B ∈ R d×r the learnable low-rank parameters of rank r ≪ d. With this approach, practitioners can trade off parameter efficiency with expressivity by increasing the rank r of the transformation.Standard PEFT methods such as LoRA can adapt a pretrained model for a given task. In multitask settings, a common approach is to train individual sets of adapters for each task. This approach however is suboptimal, as it does not enable any sharing of information, or transfer, across similar tasks. On the other hand, if one uses the same set of adapters across all tasks, they risk negative interference (Wang et al., 2021) That is, each Poly layer contains 1) an inventory of LoRA adaptersM = {A (1) B (1) , . . . , A (m) B (m) },with m ≪ |T | and 2) a task-routing matrix Z ∈ R |T |×m , where Z τ ∈ R m specifies the task τ 's distribution over the shared modules. This formulation allows similar tasks to share the same adapters and yield transfer, while still being flexible so that dissimilar tasks may have a non-overlapping set of parameters. The collection of adapters M can be interepreted as capturing different facets of knowledge, or latent skills, of the full multitask distribution.At each forward pass, Poly LoRA adapters for task τ are constructed as follows:EQUATIONwhere α i = softmax(Z [τ ]) i denotes the mixing weight of the i-th adapter in the inventory, andA (i) , B (i) , A τ , B τ ∈ R d×r .Here, the τ -th row of the routing matrix Z is effectively selecting which adapter modules to include in the linear combination of the adapter for task τ . In the stylometry setting, where each task consists of modeling an individual, Z [τ ] specifies which latent skills are activated for user τ . As per Eqn 1, the final output of the linear mapping modified with a Poly LoRA adapter becomes h = W 0 + A τ (B τ ) T x.In Poly, the module combination step remains coarse, as only linear combinations of the existing modules can be generated. Caccia et al. (2022) propose a more fine-grained module combination approach, referred to as Multi-Head Routing (MHR). Similar to Multi-Head Attention (Vaswani et al., 2017) , the input dimension of A (and output dimensions of B) are partitioned into h heads, where a Poly-style procedure occurs for each head. The resulting parameters from each head are then concatenated, recovering the full input (and output) dimensions. See Appendix A.1 for a detailed description of MHR.Routing-only fine-tuning While LoRA adapters can reduce the parameter cost from billions to millions (Liu et al., 2022) , deploying a set of adapters for each new task can still be prohibitive when dealing with thousands of tasks. To this end, Caccia et al. (2022) proposed routing-only finetuning (MHR-z), where after an initial phase of pretraining, the adapter modules are fixed, and only the routing parameters Z are learned for a new task. This reduces the parameter cost for each additional task by several orders of magnitude, while performing similarly as standard adapter training. 
methodology: 
 In this section, we detail our methodology for creating a generative model of individual player behavior that supports stylometry, style analysis, and style synthesis. We start by applying the MHR adapter framework to a modified version of Maia's architecture, and then discuss our model training and evaluation methodology, as well as our approach for analyzing and synthesizing style.The base model used for our experiments is the Squeeze-and-Excitation Residual Network proposed in Hu et al. (2018) . This approach explicitly models the interdependencies across channels. At every residual block, channel information is aggregated across spatial dimensions via a global pooling We modify the base Maia model to predict a chess move in the style of an individual player by inserting adapters generated from the player's style vector. (right) Inside each residual block, we inject a low-rank adapter at every fully-connected layer. This adapter is generated from a skill inventory that is shared across players. The player's style vector specifies which skills are active to generate the player's adapter. In this example, following the active player's style vector, the first and third skill in the inventory are selected and averaged, and the resulting low rank weight shift is applied on the (frozen) fully connected layer. operation. The resulting vector is then processed by a two layer MLP, with a bottleneck representation compressing the number of channels by r. The output of this MLP is a one-dimensional vector used to scale the output of the residual block along the channel dimension. The workflow of an SE ResNet layer is shown in Fig. 1 (left). In practice, we use 12 residual blocks containing 256 filters, and a bottleneck compression factor of r = 8. We note that this differs from the base Maia model in McIlroy-Young et al. (2022) , which uses 64 filters and 6 residual blocks.To enable user-based adaptation, we incorporate the MHR adapters described in Sec. 2.2 into our base model. For every linear transformation in the two-layer MLP used for the channelwise rescaling operation described above, we add an MHR layer built of LoRA adapters. In our experiments, the size of our inventory adapter is 32, and we use a multi-head routing strategy with 8 heads. Therefore, for each user we must learn 32×8=256 routing parameters, and these parameters make up our player style vector. To facilitate interpretability and style analysis, we use the same routing (or style vector) across all MHR layers. A depiction of this framework is shown in Fig. 1 (right). We used a rank of 16 for each LoRA, across all 12×2=24 MHR layers, totaling 5M additional parameters.We use data from the largest open-source online chess platform, Lichess.org (Duplessis, 2021), which boasts over 100 million games played per month, a globally diverse player community, and a database of over 4.8 billion games. We collected Blitz games played between 2013 and 2020 inclusive-these are games with 3 or 5 minutes per side, optionally with a few seconds of time increment per move-and applied the same player filtering criteria as McIlroy-Young et al. (2022) . The resulting dataset comprises 47,864 unique players and over 244 million games. We enforce a strict lower bound of 10 games per player to preserve validation and testing sets, though most players tend to have many more games (e.g. the median player has 3,479 games). See Appendix A.2 for a discussion on data imbalance. We divide this set of players into a few subsets to support our training methodology below. The base player set is used to train the base Maia model. The fine-tuning player set is used to fine-tune the MHR-Maia architecture shown in Fig. 1 . The few-shot player set is used for few-shot learning on the MHR-Maia. For players used during base model training or MHR fine-tuning, we split their games into a training set (80%), validation set (10%), and testing set (10%). For players used during few-shot learning, we use a reference set of 100 games to learn their style vectors, and a query set of 100 games to evaluate them for stylometry. If a few-shot player was previously trained on during fine-tuning, we refer to them as seen; otherwise, they are unseen. Currently, we treat each player's data holistically, but in principle one could partition a player's data in different ways to perform a finer analysis of their playing style. We explore this in Appendix A.4.Base model. We train our (modified) base Maia model using data from a base player set of all 47,864 players, treating this as a classification task of predicting human move y made in chess position x, given a datapoint (x, y). We use the Adam optimizer (Kingma & Ba, 2015) with a learning rate of 0.0003 (and no learning rate schedule). We use the same loss functions and evaluation criteria as the original Maia work: Maia's policy head uses a cross entropy loss while the value head uses MSE; the output of the policy head is used to evaluate the model's move-matching accuracy. Specifically, given data from the testing sets of all players, we compare the move with highest probability from the policy head with the ground truth label (the human move).MHR fine-tuning. To fine-tune the MHR-Maia architecture shown in Fig. 1 , we adopt the methodology used in Caccia et al. (2022) , namely, we freeze the Maia model and fine-tune the MHR layers and routing matrix using data from a fine-tuning player set. Recall that the routing matrix Z has a row (style vector) for each player in the fine-tuning set. When fine-tuning on data from player i, Maia's output is influenced during the forward pass through the adapter modules activated by player i's style vector. During the backward pass, the gradient updates flow through the shared adapters but only update player i's style vector. Our fine-tuning process uses the Adam optimizer with a higher learning rate of 0.005 for the LoRAs, and an even higher learning rate of 0.02 for the style vectors.We use two fine-tuning player sets in our experiments, creating two separate MHR-Maia models. The first set comprises all 47,864 players and is used to evaluate behavioral cloning and stylometry at very large scale. The second set is comprised of the same 400 players used by McIlroy-Young et al. (2022) ; we use this to evaluate few-shot learning and stylometry on unseen players, and compare these results to the original Maia work.We evaluate a fine-tuned MHR-Maia model in two ways. First, we measure its move-matching accuracy, similar to how we evaluate the base Maia model. However, since MHR-Maia provides a generative model for each player (which can be invoked by indexing into the player's style vector), we can separately evaluate each player's move-matching accuracy by invoking their generative model and evaluating it on the player's testing set. The overall move-matching accuracy for the model is simply the average of these per-player accuracies.Our second evaluation method uses the model to perform behavioral stylometry among all players in the fine-tuning set. In theory, we could adopt the methodology of McIlroy-Young et al. (2022) and compute the move-matching accuracy of every player applied to every other player's query set, but such a quadratic computation is infeasible beyond a few thousand players. A much more efficient approach is to simply compare the style vectors. That is, given a query set of games, we learn a new style vector in Z for those games via few-shot learning (discussed below), and compare this vector to every other vectors in Z. Using cosine similarity as our distance measure, we then output the the player with the highest cosine similarity to the query set vector.Few-shot learning. To perform few-shot learning on MHR-Maia, we perform "routing-only finetuning" as described in section 2.2 and additionally freeze all MHR adapters, allowing only the entries of Z to change. Specifically, given a reference set of games for a few-shot player, we add a (randomly-initialized) new row to Z and fine-tune it on data from the reference set, eventually learning a style vector for the player. Using this style vector, we can invoke a generative model of the player and use it to evaluate move-matching accuracy, as described above. To perform stylometry, if the player is a seen player (i.e., part of the fine-tuning set), then a matching style vector already exists in Z and we can find it using cosine similarity, as described above. Otherwise, if the player is unseen, then we simply repeat the few-shot learning process on a query set of games (from the same player), and compare this new style vector to the entries in Z. All of our few-shot experiments use the MHR-Maia model fine-tuned on the 400-player set from McIlroy-Young et al. (2022) .The style vectors in Z represent distinct distributions over latent skills that give us a starting point for comparing player styles. For example, we can use the cosine similarity of these vectors to determine how similar or different players are, as employed by our stylometry methodology above.In order to make such comparisons more human-understandable, we exploit the generative nature of MHR-Maia to connect style vectors to human concepts in chess. Specifically, inspired by the concept probing techniques used to analyze AlphaZero (a deep RL chess engine) (McGrath et al., 2022) , we use a set of human-coded heuristic functions found in Stockfish (a traditional chess engine) that are used to evaluate a given chess position. These functions capture concepts such as: king safety, material imbalance, piece mobility, and so on. By invoking each player's generative model on a fixed set of chess positions and seeing which move they select, we can measure the deltas in the heuristic functions before and after the move, and use this to summarize how much emphasis each player places on the corresponding concepts.In addition to interpreting existing style vectors, we can also synthesize new ones. We do this by performing simple vector arithmetic, for example by interpolating between two players using a convex combination of their style vectors. This allows us to create new player styles that are humanlike and yet previously unseen in the world. To determine the playing strength of these new players, we run a simulated tournament between them and the players they are derived from. The results of this tournament can be used to calculate a win probability (based on a standard scoring system of win=1 point, loss=0 points, draw=0.5), which can then be converted to an estimated Elo rating. 
experiments: 
 In this section, we demonstrate two main findings. First, MHR-Maia performs competitively with previous methods in their respective tasks at unprecedented scale. We are able to closely match the accuracy of move prediction with individual models (McIlroy-Young et al., 2022) , while also retaining the computational efficiencies of stylometry with a Transformer-based method (McIlroy-Young et al., 2021) . Second, we show that explicitly capturing style vectors allows us to reason about and perform arithmetic operations on generated behaviors. Specifically, we generate new players by combining learned style vectors in a way similar to how early vector-based image generation showed the ability to generate image analogies (Radford et al., 2016) .Here we compare the efficacy of our method to using individually fine-tuned models for each player. Fine-tuning individual models generally results in superior results compared to PEFT methods, as the increased parameter count produces more expressive models. However, they are also more computationally intensive to train and store. That said, in the domain of modeling individual behavior in chess, MHR-Maia is able to perform comparatively well despite using a much smaller parameter budget. Figure 2 shows that MHR-Maia matches individual model fine-tuning over a wide range of game counts. For 25, 50, and 100 games, the LoRAs are frozen to combat overfitting (Table 1 ), and only player style vectors are fit (256 parameters). The base model is frozen for all game counts in MHR-Maia. Table 1 details move-matching performance with various parameters. The model has already learned the set of skills required to differentiate the players, all that is needed with very few-shot learning is to find a proper recombination of the learned skills within the new style vectors. The Transformer-based method is omitted, as it is incapable of generating moves.In this section, we show that our models perform competitively with previous behavioral stylometry methods for both seen and unseen players. Here, the goal is to predict the player who produced a given set of games. We compare to individual model fine-tuning (McIlroy-Young et al., 2022) , fitting a pre-trained Maia to the data from a single player, and to a Transformer-based method (McIlroy-Young et al., 2021) , which embeds players in a 512-dimensional style space based on their gameplay. All reported accuracies are top-1 unless stated otherwise.To perform stylometry on a query set of games, McIlroy-Young et al. (2022) suggest measuring the move-matching accuracy of each available fine-tuned model and selecting the one that seems most performant. As seen in Table 2 , this procedure works well, but is tremendously expensive-requiring computationally intensive inference calls on the entire query set for every candidate player.In contrast, both the Transformer-based method and MHR-Maia scale much better to large numbers of players. The Transformer-based method needs only to condition on these games to produce a vector, while MHR-Maia needs only to fit a new vector. In either case, the produced vectors need 2021) are borrowed from their respective papers.only be matched to those in the player set, e.g., using cosine similarity. Table 2 compares both approaches, showing that MHR-Maia performs competitively or better, on a much larger universe.To do this, we use few-shot learning to compute style vectors for 10,000 players based on their 100 game reference sets, then fit new style vectors for each player based on their respective query sets.Note that the individual model fine-tuning method is omitted from the larger few-shot study due to scalability reasons. The Transformer-based method can scale, but it is not a generative model.This section provides evidence for the consistency of style vectors in MHR-Maia and probe the information learned within them. We additionally create novel intermediary styles of players. Finally, we visualize the individual style of our vectors using traditional chess evaluation methods.Style vector consistency. The result shown in Figure 3 suggests that MHR-Maia, used to fit player vectors, will find similar vectors for two non-overlapping datasets if they are generated by the same player. Here we examine that phenomenon in more detail. Figure 3 is created by splitting 50 players data into an average of 50 separate game sets each. We then train each game set as a separate player, and measure to cosine similarity between vectors that are learned from the same player to those learned by different players. We find that the cosine similarity between same-player vectors is far larger than those measured between the 47,864 different players, demonstrating a self-consistency in what is captured by these vectors. To further probe the consistency of this style space, we create intermediary players by merging two players' game sets. Interestingly, we find that the vectors fitted to these joint datasets are quite similar to vectors obtained by simply averaging the vectors of each of the distinct player datasets. This Figure 4 shows how these cosine similarities are distributed.Relative player quality. Learned style vectors appear to capture information related to the quality of the corresponding gameplay. For example, these vectors contain information related to Elo score, the standard system for ranking the relative quality of players. A linear regressor trained to predict player Elo from their style vector is able to do so with a mean absolute error (MAE) of 54-a naive approach that simply picks the mean Elo obtains an MAE of roughly 204. This experiment was carried out over a test set of 7,864 randomly held-out players with an average Elo rating of 1600.Differences in player strategy. To confirm the distinctness of playstyle across style vectors, we sampled five learned player vectors uniformly at random from a MHR-Maia model trained on the 400 players from McIlroy-Young et al. (2022) . For each, we predicted their preferred move across 2 17 roughly neutral board positions (within 1 pawn of balanced). For each move, we measured a series of Stockfish evaluation metrics, which are human-interpretable scores for different qualities of chess gameplay, including piece mobility, king safety, etc. Figure 5 shows the distribution of these metrics for each of the five players, demonstrating that these vectors store a wide diversity of styles.Convex combinations. Finally, we show that interpolating between skill vectors results in a player whose level is a weighted average of the interpolated players. Here, we take 100 pairs of learned player vectors, such that one item in the pair corresponds to a strong player and the other to a weaker player. We then gradually interpolate between the weak and strong player as (1 -λ)u w + λu s , 0 ≤ λ ≤ 1, where u w and u s are respectively vectors representing the weak and strong player.For each value of λ we simulate 1,000 games between the interpolated vector and u s , the stronger player. Figure 6 plots the win rate of the interpolated player as a function of λ for each player pair we considered. This plot demonstrates that win rate progresses in a roughly linear fashion, starting off winning infrequently against the stronger player and eventually winning roughly half the time as the interpolated player converges to the stronger player. In Appendix 6, we further investigate the effect of interpolation on style using the the human-interpretable Stockfish evaluation metrics. These results suggest that simple arithmetic operations in this vector space seem to respect humanidentifiable notions of style. 
related work: 
 We discuss related work on stylometry, player style modeling (outside of chess), as well as other PEFT methods for transfer and multi-task learning.Originally referring to performing author attribution via statistical analysis of text (Tweedie et al., 1996; Neal et al., 2017) , stylometry has since come to refer to the general task of identifying individuals given a set of samples or actions and has found broad application for tasks such as handwriting recognition (Bromley et al., 1993) , speaker verification (Wan et al., 2018) , identifying programmers from code (Caliskan-Islam et al., 2015) , determining user age and gender from blog posts (Goswami et al., 2009) , and identifying characteristics of authors of scientific articles (Bergsma et al., 2012) . Along these lines, McIlroy-Young et al. (2021) introduced behavioral stylometry, the task of identifying individuals based on their decisions or actions in a given situation. In the context of games (such as chess), this takes the form of identifying players based on their in-game actions or moves. This characterization can be viewed as a form of playstyle modeling, where an individual's playing style is defined by the actions they choose to perform in a certain game state (Holmgård et al., 2013) . This has been used to determine a player's style by building agents representative of different playstyles and then assigning a player the style of the agent whose behavior most closely matches the player (Holmgård et al., 2014) . An alternate approach for identifying playstyles involves gathering gameplay data (e.g., event logs, player trajectories) and applying methods such as clustering (Ingram et al., 2022) , LDA (Gow et al., 2012) , bayesian approaches (Normoyle & Jensen, 2015) and sequential models (Valls-Vargas et al., 2015) to identify groups of players with similar styles. Note that, unlike our work, these approaches do not learn a behavior model conditioned on the learned styles to generate actions suited to a specific style. Also worth noting is that both prior sets of approaches focus on identifying aggregate playstyles. In a similar vein, behavior cloning methods for games also typically learn models of behavior aggregated across a number of individuals as seen in such approaches for games like Counter-Strike (Pearce & Zhu, 2022) and Overcooked (Carroll et al., 2019) . Our work differs in that we model behavior separately for each individual player.Approaches for efficient adaption of a pretrained model can be broadly grouped in two categories.Adapter based methods inject new parameters within a pretrained model, and only updates the newly inserted parameters while keeping the backbone fixed. Houlsby et al. (2019) defines an adapter as a two-layer feed-forward neural network with a bottleneck representation, and are inserted before the multi-head attention layer in Transformers. Similar approaches have been used for cross-lingual transfer (Pfeiffer et al., 2020) . Adapters have also been used in vision based multitask settings (Rebuffi et al., 2017) . More recently, Ansell et al. (2022) propose to learn sparse masks, and show that these marks are composable, enabling zero-shot transfer. Lastly, Hu et al. (2022) learn lowrank shifts on the original weights, and (Liu et al., 2022) learns an elementwise multiplier of the pretrained model's activations. Adapters have also been used in multitask settings. Chronopoulou et al. (2023) independently trains adapters for each task. In order to transfer to new tasks, the authors merge the parameters of the adapters of relevant training tasks.Another popular approach is the use of soft prompts (Lester et al., 2021) . In the context of natural language, where the input is a sequence of work tokens, soft prompts append new, learnable tokens to the sequence. Closer to the multitask setting presented in this work, Vu et al. (2021) learns a collection of soft-prompts amassed from a multitask training set, and given a novel task, retrieves relevant prompts for efficient transfer. 
conclusion: 
 We show that behavioral cloning of individuals is possible at very large scale in the game of chess.We cast this problem in the framework of multi-task learning and employ modular parameterefficient fine-tuning methods to learn a shared set of skills across players, modulated by a distinct style vector for each player. We use these style vectors to perform stylometry, predict Elo ratings, probe a player's playing style, and create novel playing styles.  
 
 Appendix: A.1 MULTI-HEAD ADAPTER ROUTINGIn Poly, the module combination step remains coarse, as only linear combinations of the existing modules can be generated. Caccia et al. (2022) propose a more fine-grained module combination approach, referred to as Multi-Head Routing (MHR). Similar to Multi-Head Attention (Vaswani et al., 2017) , the input dimension of A (and output dimensions of B) are partitioned into h heads, where a Poly-style procedure occurs for each head. The resulting parameters from each head are then concatenated, recovering the full input (and output) dimensions. This makes the module combination step piecewise linear, with a separate task-routing matrix Z learned for each head. EQUATIONA τ = concat(A τ 1 , . . . , A τ h ), where α i,k = softmax(Z[τ ,:,k]) i . Importantly, the number of LoRA adapter parameters does not increase with the number of heads. Only the task-routing parameters linearly increase with h for MHR vs. Poly. However, this cost is negligible as the parameter count of the routing matrices is much smaller than for the LoRA modules themselves.A.2 FEW-SHOT PLAYERS AND DATA IMBALANCE While our dataset has a median game count of 3,479 games, many players may have as few as 10-50 games. Our evaluation of few-shot learning shows that 100 games is sufficient to learn the style vector of an unseen player. However, one might still ask how accurately such a style vector is given a very small number of games. To explore this, we first split a player into disjoint sets of 10, 25, 50, 100, 500, and 1,000 games. We then train a style vector on each set. As a baseline, we train a style vector on 10,000 games and track the cosine similarity of the smaller-set style vectors relative to this baseline vector. We show the results in Figure 7 .To further investigate the effect of interpolation on style, we compare the playing style of two random players to a third player produced by averaging their style vectors. The result is shown in Figure 8 , where as before, we use a set of human-interpretable evaluation metrics from Stockfish to characterize each player's style. We observe that the green player (labeled Interpolated) retains an intermediate set of skills relative to the two source players. This corroborates the results of Figure 4 , where we show that a style vector trained by merging two players' game sets has high cosine similarity to a style vector generated by averaging the style vectors trained on each game set individually.Most of the existing work in chess assumes that a player remains stationary over time and across gameplay situations. However, in reality, a player's style may depend on the type of opponent they are facing, which opening is used, which stage of the game they are in (opening, middle, endgame), and so on. For instance, McIlroy-Young et al. (2021) observe that stylometry accuracy drops when removing the opening (e.g., the first 15 moves) moves, suggesting that the opening has an outsized effect on style identification. Our approach does not rely on these assumptions and can in principle be applied to arbitrary subsets of a player's data. For instance, one could split a player's data into opening, middlegame, and endgame moves and train a separate style vector for each. One could further split the data based on which defense the opponent uses, what time of the day it is, etc.. Despite treating players holistically and avoiding any splits of their data, we are still able to capture the peculiarities of each individual's playing style and perform stylometry with high accuracy. This also enables us to compare our results to those of prior work, which also treats player data holistically. 


Decision: This paper considers the problem of behavior stylometry by regarding it as a multi-task learning problem. In particular, it seeks to develop a generative approach that is scalable, which is a major challenge in existing methods. It develops a method by training multiple LoRA adaptors over a common base model to represent different styles. And then these adaptors / vectors are combined in different ways for action generation in the given style of a player. And they can also be used for interpreting and manipulating in the latent style space. The reviewers agree that this is an interesting but intuitive method. On the other hand, there are also concerns raised by the reviewers about the paper, such as slight lack of novelty and the significance of the problem being unclear. A better way to make this paper stronger is to extend the approach to more applications domains (beyond just chess games) and how the model can be used beyond stylometry.

Reviews: review: paper_summary: paper_summary: In this paper, the authors propose to solve the problem of behavior stylometry, which is to identify the style of a player’s policy in the game, by regarding it as a multi-task learning problem. Each player’s style is a distinct task. Previous methods are either not scalable or not generative, in that they cannot predict the moves of a player given a query set of games played by that player. The authors aim to design a model capable of generating the moves of different players in a scalable manner. To do that, the authors trained a set of Low Rank Adapters (LoRAs) over a base model. The base model is trained with behavior cloning on the whole dataset, and the LoRAs are trained on a specific set of training player dataset separately. Additionally, a routing matrix is trained with LoRAs to specify the distribution over the LoRAs for each player. In this case, the authors claim that the routing matrix is a compact representation of the players’ skills, and encourage the adapters to learn different latent skills while preserved the shared knowledge within the base model. The authors also claim that the routing matrix supports few-shot learning, and induces a series of applicable benefits on stylometry, such as interpreting and manipulating the style of a player.
strength_weakness: strength_weakness: 1. The main concern I have for this paper is the novelty of the proposed method. The base model for Chess is from a previous work called Maia, and trained with existing techniques such as LoRA and Polytropon, either of which is novel to me.
2. The significance of the problem setting, behavior stylometry, is not clear. The authors only have a very short introduction of what it is and its usage in the first section. However, the benefit of modeling accurate behavior stylometry is still vague. For example,  we have very high performance models in chess, even outperforming human players, then why do we need to model the style of human players via behavior stylometry? To me, accurately predict the moves of a player is not the ultimate goal of a game, the ultimate goal is to win the game. How can behavior stylometry help in winning the game? This should be better explained with more introduction and related experiments.
questions: questions: 1. How can behavior stylometry enhance human players? Why it is important to predict the moves of different human players, especially in the case of chess where super human models already exist? Can we analyze the style of AI methods with your method?
2. Why do you choose chess and Rocket League to test your approach? Are there any other potential games or environments applicable for your method?
limitations: limitations: As stated in the Weakness part, the analyze on the significance of behavior stylometry is limited. The authors work on a sub-problem of the game, and didn’t connect it back to the game itself.
paper_summary: paper_summary: Authors found a way to obtain player stylometry using BC on a massive amounts of data. Their idea uses multiple LoRAs and routing matrix to obtain a style vector for each player.
strength_weakness: strength_weakness: - Applications as of now are not very compelling. One option would be to take lichess.org data and ask some interesting research question about the player data that has not seen a sufficient answer. Then use this technique to answer it. 
- Another option would be to integrate this technique to some IRL scheme. 
- Authors cite speaker recognition in the related work, but did not study it more closely. In biometric recognition we have essentially similar task as in the present paper. Idea is to turn observed data (audio in speaker recognition) into fixed length vector and then by comparing these vectors obtain downstream recognition. Where the present paper goes wrong is that evaluation metric is identification accuracy, whereas in biometrics it is known that task is verification task and metric is ROC or DET curve and equal error rate (EER) summary statistic. Authors should use it, in addition to identification. 
- Speaker verification also gives nice possibility for extensions to the current work. One idea back in the day was to do Baysian adaptation from the general speaker model to a specific speaker model. These ideas were also presented in the following paper https://arxiv.org/abs/2012.01244
questions: questions: -
limitations: limitations: -
paper_summary: paper_summary: The paper explores modelling user behaviour for chess and rocket league using a PEFT-based method, wherein users are modelled using a composition of MHR adapters. The authors evaluate their approach both for predicting which player played a particular game and for predicting the next move of a given player, and find that their MHR-based approach outperforms prior work in both cases. Further analysis shows that the MHR-based approach further allows steering and combining user vectors to produce new styles, and that the vectors are able to capture a wide diversity of strategies.
strength_weakness: strength_weakness: - The MHR approach does seem to underperform full-finetuning when there is a large number of games available, as shown in Figure 2, and as expected for a PEFT-based method.
- While older stylometry work [1] was able to produce user vectors by just performing inference (albeit not being generative), this approach requires training, and so might be more computationally expensive as a result (PEFT training is cheap, but still requires computing the full backward pass to compute gradients for the adapters). Grounding some of the discussion in section 5.1 with compute estimates (e.g. estimated FLOPs) would be useful.
- The novelty is somewhat limited, as the primary method (MHR adapters, linearly interpolating between adapters) has been explored in prior work. However, I think that the application of this idea to a new domain (user identification/move generation) is still novel and interesting.

[1] R. McIlroy-Young, Y. Wang, S. Sen, J. Kleinberg, and A. Anderson. Detecting individual decision making style: Exploring behavioral stylometry in chess. Advances in Neural Information Process439 ing Systems, 34:24482–24497, 2021.
questions: questions: 1. I’m curious about the compute cost of your approach against the other methods. Taking into account training and inference, what are the differing FLOPs costs? Using adapters is typically useful for reducing memory costs (since you don’t need to store optimizer states for all parameters), but still requires a similar number of FLOPs to train.
2. In the Polytropon paper, they attempt to analyse how tasks are distributed among the skill vectors. While this might be a bit trickier with MHR, I’m curious if you observe that if the skill vectors learnt through MHR training correspond to any human-interpretable features? This would be an interesting analysis!
limitations: limitations: The authors address the limitations of their approach reasonably.
paper_summary: paper_summary: This paper adapts Maia, a group-level model and variant of the AlphaZero model proposed by McIlroy-Young et al. (2020), to function at an individual level. The authors focused on scaling the fit to individual behavior using techniques like LoRA for fine-tuning LLMs. Moreover, the learned embeddings were shown to encode decision-making styles that correspond to individual behaviors. Linear combinations of these embeddings can result in desirable decision-making styles, creating synthetic styles.
strength_weakness: strength_weakness: The paper lacks a proper benchmark for comparing the accuracy gains from their proposed method. Hypothetically, if the datasets they considered (Chess and Rocket League) exhibit no individual variability in behaviors, meaning all individuals behave identically to the group-level agent, we should expect no accuracy gain whatsoever from adapting a group-level base model to individual-level data. Conversely, if significant variability stems from individual differences, we should expect a substantial gain. Currently, the 0.4% and 4.8% accuracy gains reported in Table 1 are not compared to any benchmark that considers the maximal possible variability that could be extracted from individual-level data. This leaves readers uncertain about the actual gains achieved by the method. It would be beneficial to establish a theoretically optimal benchmark for maximal possible accuracy gains across all datasets.
questions: questions: Is it possible to use steering or synthetically created new styles to predict human decisions? This approach might provide a stronger test for the learned style vectors that capture human-like behaviors.
limitations: limitations: This paper did not explicitly include a section for Limitation and Future Research.

Comments: title: Response to reviewer tLZq
re: rebuttal
comment: Thank you for your detailed response. I am pretty much satisfied with it. I am inclined to raise my score by one.
Thank you for your response! If you feel we have adequately addressed your concerns and are inclined to increase your score to a 5, we would very much appreciate it. Please let us know if anything else is needing of clarification.
Thank you for the response. Most of my concerns have been adequately addressed, and I am inclined to increase my score to 5.
Thank you for the detailed rebuttal! I'm keeping my positive score as-is, and my concerns have mostly been addressed.