Detecting Individual Decision-Making Style: Exploring Behavioral Stylometry in Chess
Authors: Reid McIlroy-Young
Russell Wang
Siddhartha Sen
Jon Kleinberg
Ashton Anderson
Venue: NeurIPS 2021 Poster (2021)
Keywords: behavioral stylometry
player identification
chess
few-shot classification
transformer

Abstract: The advent of machine learning models that surpass human decision-making ability in complex domains has initiated a movement towards building AI systems that interact with humans. Many building blocks are essential for this activity, with a central one being the algorithmic characterization of human behavior. While much of the existing work focuses on aggregate human behavior, an important long-range goal is to develop behavioral models that specialize to individual people and can differentiate among them.

To formalize this process, we study the problem of behavioral stylometry, in which the task is to identify a decision-maker from their decisions alone. We present a transformer-based approach to behavioral stylometry in the context of chess, where one attempts to identify the player who played a set of games. Our method operates in a few-shot classification framework, and can correctly identify a player from among thousands of candidate players with 98% accuracy given only 100 labeled games. Even when trained on amateur play, our method generalises to out-of-distribution samples of Grandmaster players, despite the dramatic differences between amateur and world-class players. Finally, we consider more broadly what our resulting embeddings reveal about human style in chess, as well as the potential ethical implications of powerful methods for identifying individuals from behavioral data. 

Detecting Individual Decision-Making Style: Exploring Behavioral Stylometry in Chess
Reid Mcllroy-Yourr University of Toronto reidncydcs. toronto. edu
Russell Wang University of Toronto russellêcs. toronto.
Siddhartha Sen
Microsoft Research
didsen@microsofe
Jon Kleinberg
Ashton Anderson University of Toronto ashton&cs. toronto. edu
Cornell University
kleinbergScornell.cco
Abstract
The advent of machine learning models that surpass human decision-make ability in complex domains has initiated a movement towards building Al systems that interact with humans. Many building blocks essential for this activity, with central one being the algorithmic characterization of human behavior. While much of the existing work focuses aggregate human behavior, important long-range goal is to develop behavioral models that specialize to individual people and can differentiate among them.
To formalize this process, we study the problem of behavioral stylometry, in which the task is to identify a decision-make makkir from their decisions alone. We present a transformer-Be based approach to behavioral stylometry
the context of chess, where one attempts to identify the player who played a set of games. Our method operates a few-shot classification framework, and can correctly identify player from among thousands of candidate players with 98% accuracy given only 100 labeled games. Even when trained on amateur play, our method generalises to out- of-distribution samples of Grandmaster players, despite the dramatic differences between amateur and world-class players. Finally, we consider more broadly what our resulting embeddings reveal about human style in chess, well the potential ethical implications of powerful methods for identifying individuals from behavioral data.
Introduction
The advent of machine learning models that surpass human decision-making ability in complex domains raises the prospect of building AI systems that can collaborate and interact productively with humans. While many advances will be essential in order to achieve this far-reaching goal, algorithmically characterizing aspects of human behavior will be significant step. Whereas existing work focuses on characterizing human behavior in an aggregate or generic sense, genuine human-Al interaction will require specializing these characterizations to individual people.
Developing algorithms that can detect signatures of individual decision-making style would constitute an important advance for many reasons. First, being able recognize an individual from their behavioural characteristics would greatly benefit an AI system's 's ability to coordinate with that individual. In general, preision-making styles can range from complementary to clashing, and predictive characterization of an individual's 's decisions would be useful for an AI system trying to adapt their style. Second, such ame-grained, individual-level characterizations would help isolate the distinguishing features of individual styles. In domain where ML systems are able to match
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

or surpass human performance, it becomes natural to try extending these systems to pinpoint the types of decisions a person is particularly good at, and types of decisions they could potentially improve. Finally, applying these algorithms to many different individuals would shed light on the structure of decision- -aaiinnnnmmiingggge the domain itself. For example, certain instances could prove particularly discriminative in detecting individual style. In addition to these benefits, the ability to identify individuals from behavioral traces also raises ethical concerns centered around privacy and generalization to high-stakes domains. We discuss the ethical consequences of our work in Section S
To formalize the above process, we study the problem of behavioral stylometry, which the goal is to identify decision-make from their decisions alone. We pursue this task in the domain of chess, which for many reasons is an ideal model system for developing numan-compation ability in chess at least 15 years ago. Despite this, millions of
algorithms. Al
systems surpassed all human-leye people at all skill levels still actively play. Through their activity, people generate detailed traces of their decisions, which are digitally recorded and publicly available. In chess, behavioral stylometry reduces to predicting the identity of a player from the moves they played in a given set of games.
We present a transformer-basee approach to behavioral stylometry in chess. We show how to adapt recent work in speech verification to the problem of detecting individual decision-making style. Working in the domain of chess requires us to extend these methods to the complex types of data found in the game-playing structure of chess. We apply our model a few-shot classification setup, where only a small number of games are provided for the model to learn a representation of a player's s decisions, and the model must then identify the player from thousands of possible candidates, based on another small sample of their games. Compared to recent approach, our method achieves high accuracy on this task, while being much more scalable and data-efficient. Our resulting embeddings of players and their games reveals structure of human style in chess, with players who are stylistically similar clustering together in the space. We consider the implications of these findings, including the potential ethical implications of identifying individuals from their behavioral data.
2 Related Work
Speaker verification. Our work adapts recent work in speaker verification 46 59
where given a
speaker's known ufterances, the task is verify whether an utterance belongs to that speaker. Recent approaches embed utterances into a vector space such that different speakers are distinguishable [28]
42].
These embedding approaches
also be used for downstream tasks such voice cloning and
multi-speaker speech synthesis 21 27 34].
A key decision for our task is the choice of loss function, of which two families are most relevant K
Classification- objectives are designed to identify (classify) an unknown target speaker from a pool of possible candidates. Cross-entropp loss is simple approach, with many variations including angular softmax loss [29]. cosine loss [50), and additive angular margin loss [4] Contrast or
metric-learning based objectives instead directly optimize the contrastiveness for samples in different
classes, and similarity for those within the same class. Contrastive loss [Л.. triplet loss 38, and generalized end-to-end loss (GE2E) 40 used in face verification, face clustering, and speaker verification respectively. In this work, we adapt GE2E to the problem of behavioral stylometry.
Stylometry. Both speaker verification and handwriting recognition are stylometry tasks where the goal is detect and analyze authorial style. Stylometry has historically been associated with attribution studies [45] and has been applied to de-anonymizing programmers from their code [ identifying age and gender from blog posts [2)) and distinguishing machine-generated text from human- generated text [5]. It is often framed as task to determine the similarity between samples or features extracted from the samples 32 Handwriting recognition has employed neural methods to
identify individuals for decades R. and development is ongoing (23) 24]. We introduce the task of
behavioral stylometry, in which the goal is to identify decision-makers makers from their decisions.
Chess. Chess has a long history in academic research as both a test bed for artificial intelligence and for understanding human behaviour 11 g. Recently, these two goals have begun to merge, as researchers have begun to create chess engines that behave more human-like manner 30 [7 Recent work presents personalized chess engines based on AlphaZero 40 that are trained to predict
39

---

Residual
1st Move
2nd Move
Residkal
CNN
MLP
MLP Tanh
Normalzze
Game Voctor
Blocks
Transformer Encoder
Residual
N-th Move
CMN
MLP
Blocks
Figure I: Overview of our model's inference procedure. It first takes in sequence of chess moves and extracts move-level features with series residual blocks, then produces game vector with a transformer encoder followed by projection layers and normalization.
moves made by specific players 611 As side effect, these models can perform behavioral stylometry in chess. We develop methodology that is both more scalable and more accurate than this approach.
Few-shot learning. We approach behavioral stylometry few-shot learning task with tens to hundreds of samples per class, which is a well-studied problem formulation [1
19 [5]] Transfer
learning is main approach to few-shot learning, where models are first trained on large dataset
and then specialized to several smaller datasets 16 33
4 53
Embedding learning [26] is another
approach. where an embedding space is constructed so that similar training samples are close together and different samples are further apart. Test samples can then be mapped to this space and compared with the closest training samples and their associated classes. Several architectures have been proposed, such as matching networks 4 I0 48 and prototypical networks [36 41] Finally, meta- learning [ 44 is another popular approach, which uses a two-level learning process train a model across similar tasks and adapts it to solve new tasks with small number of training samples [20 35
3 Methodology
We begin by describing model design for performing behavioral stylometry in chess. The model has three main components: the first extracts features from individual moves, the second aggregates information from the move level to the game level, and the third aggregates information from the game level to the player level. Given a chess position as input, the first step uses residual blocks to extract move-level features. Then in the second step, aggregate move-level features into game-level vector using a transformer based method. Finally, the third step aggregates a player's game vectors by calculating their centroid, producing a player vector that represents the player's individual playing style.
Move-level feature extraction. To extract features from chess boards and moves, we draw upon previous work (AlphaZero and Maia) and encoded as 34-channel 8 x 8 chess move representation; we represent a move with two board positions, corresponding to the boards before and after the player's move. The first 24 channels each encode a specific type of chess piece (6 for each side in one position, and we encode two positions), and the next 10 channels encode position metadata relevant to chess such repetition, castling rights, the active player's 's side, 50-move-rule count, and border information. The output is 320-dimensional feature vector that characterizes each move. Each residual block consists of convolutional layers, batch normalization, ReLU activations, and squeeze and-exxitation (SE) blocks (25). On top of the residual blocks we add global average pooling, which summarizes the output from residual blocks into 1 x x 320 feature map, which we flatten into 320-dimensional vector. The hyperparameters of the model were chosen to match the Maia architecture: residual blocks with channel size of 64 and reduction ratio for SE blocks of 8.
series of residual blocks. The input to the model
3

---

Aggregating information to the game level. The next component of the model takes in a sequence of move features extracted from the first component and constructs a game representation. The imput features correspond to the moves played by the target player in a particular game. The output is game vector that aggregates information about the move features. Although previous work used an LSTM to perform the analogous task in speaker verification, our experimentation revealed that transformer 4 worked better for our task. Hence, in our approach we use a modified version of the Vision Transformer (ViT) [7.. which was originally proposed as pure transformer replacement for CNNs in vision tasks, achieving comparable results to state-of-the-arr CNN architectures with fewer computational resources. As we consider chess moves 8 8 images with 34 channels, we can adapt ViT to fit in our setting.
To do this, we first modified the input to VIT to take in a sequence of move features, instead of fixed-size patches extracted from an image. Both BERT [55 and ViT append classification token to the input sequence, and the final hidden state of this token is considered as a representation of the entire sequence. However, our task we achieve better performance if we directly average the output vectors of transformer blocks, so we remove the classification token in the input. As all move features are fed into the transformer simultaneously, we retain the ordering of the chess moves by adding positional encoding to each move feature. The original ViT architecture includes a learnable ID position embedding in the input sequence, but it cannot easily accommodate variable length inputs during training. Therefore, we use sine and cosine functions to encode positions, a method proposed in the standard transformer, and we include support for encoding up to 500 chess moves.
As an overview of our model, each 320-dimensional input move feature vector is first projected into 1024 dimensions, and a positional encoding is appended to the resulting vectors before they are fed into the transformer encoder. The encoder has 12 transformer blocks, each composed of an attention layer followed by a feed-forward layer, with layer normalization are 8 attention heads, each with 64 dimensions, in each attention layer. The feed-forward layer uses MLP with two layers to project the 1024-dimensional vector into 2048 dimensions, where GELU activation is applied, and then another layer projects it back to 1024 dimensions. Each output vector of the encoder corresponds to an imput move feature, and we average these vectors to get a single vector of dimension 1024, which is finally projected with MLP into our final embedding space. This 512-dimensional embedding vector encodes the decisions the player made in the game. At inference time, the player vector is obtained by simply averaging all the game vectors (i.e. the centroid) that belong to the player.
3 applied before each layer. There
Loss Function. In order to capture individual playing style, we adapt Generalized End-to-End Loss for Speaker Verification (GE2E) 40 our setting, which allows train the game vectors to maximize both intra-player compactness and inter-player discrepancy in the resulting embedding space. GE2E builds a similarity matrix on batch of N x . games, where N is the number of players and M is the number of games per player. For each game from each player, we extract the current 512-dimensional embedding vector Yıı (i-th game from the j-th player) output by the residual blocks and transformer network. We also find the centroid vector c of specific target player by averaging all of their game vectors in the batch. However, we remove yıı when computing the centroid of the true player as suggested in [4), to help avoid trivial solutions.
We then construct a similarity matrix Sji,k containing the cosine similarities between each game embedding vector and each player centroid (for all players in the batch), which we fur- ther scale by two learned parameters and b. For the target player, we calculate the centroid
(-1)) = N Emzz yim- and otherwise simply use the average of game vectors for all other players (A: + j).
w cos(y, cf + if k= w - C&) + b otherwise.
Sji
(1)
The final loss function is designed to maximize the similarity for the correct entries, while pe- nalizing all other entries in the similarity matrix. As a result, the loss function Lyy;;) -Sy. + log Σ exp(S)t.)) pushes each embedding vector closer towards the player's centroid and further from all other players' centroids simultaneously.

---

seen
unseen
Universe
Candidate pool
Evaluation pool
Individual player
Individual game
move k
Training
Reference
Query games
A schematic depiction of how players are organized in behavioral stylometry task; how each player's games are organized into training, query, and reference sets; and how each game's moves are in sequential order.
Figure 2:
Training procedure. We randomly sample N x M games as a batch, where N = 40 is number of players and M 20 is number games per player. In order to speed up training and perform batch computing, we randomly sample 32-move sequence from each game and pad Os to games with length less than 32 moves. The entire model was trained with 4 Tesla K80 GPÜs and SGD optimizer with an initial learning rate of 0.01 and momentum of 0.9. The learning rate was reduced by half every 40K steps. Initial values of w and b for the similarity matrix were chosen be (w, b) (10, -5), and we used a smaller gradient scale of 0.01 to match the original GE2E model.
4 Evaluation
We now train model using our methodology and evaluate against competitive baselines on several behavioral stylometry tasks.
4.1 Data
Lichess. We use chess games downloaded from Lichess] a popular open-source chess platform. Their game database contains over two billion games played by players ranging from beginners to the current world champion, Magnus Carlsen. From this database, we collect rated Blitz games (3-7 minutes per player per game) played on Lichess between January 2013 and December 2020 by players who match all of the following criteria: active as of December 2020, mean rating between 1000-2000, low variance in Blitz rating, and least 1.000 games played during the time interval. We also restrict our attention to games with at least 10 moves (20 ply). To investigate whether behavioral stylometry performance varies with number of games played, we bucket the players according to how many games they have played: 1K-5K, 5K-10K, 10K-20K, 20K-30K, 30K-40K, and 40K+ (buckets are hereafter referred to by the minimum of their respective range, e.g. "20K" for the 20K-30K bucket). The union of these sets constitutes our universe of players U.
Players are further divided into seen and unseen sets, using the same players as our previous work for those with more than 1OK games played [31. Players are seen if their games are used in model training and unseen if Finally, each player's games are randomly split into training games (80% of their games), reference games (10%). and query games (10%). Training games are only used for model training (and thus only seen players' training games are used); reference games are what is known about each candidate player; and query games are what is given about an unknown target player we are attempting to identify. all of our evaluations, we only use 100 games per player for both the reference and query sets, although in principle they could vary, and for each player their reference and query sets are non-overlapping: 2r (... 0. The numbers of players and games in each player set is shown in Table
5

---

Dataset
IK
5K
10K
20K
30K
40K+
Ranked
Seen Players
8,617
5,298
1,866
295
86
19
1,813
Seen Players' Games
14M
24M
17M
4.9M
2.0M
0.7M
9.5M
Unseen Players
23,101
1,324
466
72
20
20
451
Unseen Players' Games
4.6M
264.8K
93.2K
14.4K
4K
4K
90.2K
Table 1: The numbers of players and games used in training (seen) and evaluation (unseen).
High-ranking players. We also assembled a dataset of high-ranking online players from both Lichess and Chess.com the two most popular online chess platforms. To do so, we took the lists of the top 1, 1,500 players on both sites' leaderboards, downloaded all of their games, and filtered out games with fewer than 10 moves and players with fewer than 950 games. We split the remaining high-ranking players into seen (80%) and unseen (20%) sets, and their counts are shown in Table L (again, we use only 100 reference and 100 query games per player in our evaluations). Note that these games include all time controls and tend towards quicker games than the general population. These sets include many of the best players in the world, including the world champion.
4.2 Experimental Setup
Task formulation. Our central evaluation task is performing behavioral stylometry in chess, in which we are given a set of query games played by particular player, and the task is to identify them from amongst a pool of candidate players C, each of whom is associated with a set of reference games.
Formally, we set this up as a few-shot classification task. Within our universe of players U, we have an evaluation pool E U of players we are trying to identify, and a candidate pool C' U of players which they are guaranteed to belong to (E C C). Every target player in E is associated with small set of query games Vig. and every candidate player in C' is associated with a small set of reference games Tr (the conceptual organization of players, games, and moves is shown in Figure D In general, the candidate pool C' can be much larger than the evaluation pool E-for example, one possible task would be to try to identify some set of E = 10 unknown players, but we are only promised that they are members of some much larger pool of |C| = 10, 000 players. Finally, we also specify task parameter & where all games, both query and reference games, only contain moves starting at move k. As we will see, the opening moves in chess can be highly discriminative because many players repeatedly play the same openings. For k = 0 we are given entire games, but for, e.g.. 15, we are only given moves 15 and beyond for each game. The behavioral stylometry task is thus fully specified by C, E, |xr,], [zgll, and k: for each player in E we are given |zrg] query games and for each player in C we are given |x,,1 reference games, all starting at move k. For each set of query games Zg seek to identify which player in C' played them.
Openings baseline. Chess games always start from the same initial position, and the opening moves are heavily studied and well-understood. Opening choice shapes the resulting game, and players often study small set of openings in depth and stick to them. As result, the opening moves are likely to contain a lot of information about who is playing the game, albeit for the somewhat unsatisfying reason that they memorized patterns of play rather than new decisions being made. Indeed, when strong anonymous accounts play on online platforms, informed observers often rely on their opening moves to guess who they are. As strong baseline for comparison, we create an embedding model that uses the opening moves. The model creates game vectors by sampling 5 sequential moves by the target player, converts each into a one-hot 4096-dimensional encoding vector (moves that are never used will be 0 for all vectors and do not affect our distance measurements). and adds them together to yield a 5-hot vector encoding of the 5 opening moves. Doing this process for each of the xg games in the target player's 's set of unlabeled games yields xy 5-hot vectors in a 11996--dmmsiional space, which we then convert into player vector by simply taking their centroid. These player vectors can then be used in the same manner as those in our main model to perform behavioral stylometry.
Our model. We apply our methodology to the behavioral stylometry task. To construct our main model, we trained on all training games by all players in the seen universe-63.1 -..e-e63.MMM games by 16,181 181 players in total. Our model learns player vectors in an embedding space that captures chess style,

---

into which new, unseen players can be encoded. To perform few-shot stylometry on an unseen player p for some candidate pool C, we first infer player representations from the reference set of each player in C, then infer p's representation by running the model on p's query set. We then compute the cosine distance between y's I's query representation and all candidate players' reference representations and return the closest candidate player as p's identity. In all of our evaluations, the reference sets are drawn from players' 10% validation sets and the query sets are drawn from players' 10% testing sets. We report Precision at I (P@ 1) unless otherwise noted. We also trained two additional models, one on high-ranking players and one on the set of players used in our prior work [3D, which are discussed in the Supplement.
4.3 Behavioral Stylometry
Stylometry performance. Our main evaluation is the behavioral stylometry task where C = 10K 20O 30K 440K+ (all 2,844 Lichess players with at least 10,000 games played, both seen and unseen, are possible identities); E = 10K+ (the 578 unseen Lichess players with 10K-40K games the target players to be identified); |sig|1 | = 100 (reference and query sets are 100 games each); and k = 15 (we only consider games starting from move 15, so that memorized openings do not factor into our evaluation of capturing human decision-making style). On this task, our model achieves a P@ I of 0.86, exactly identifying player from 100 of their games out of 2,844 possible candidates 86% of the time (see Table 2 (top)). We emphasize that this is without access to the opening, which is the most regimented, repeated, and identifiable part of chess game. In comparison, the baseline with access the first 5 given moves of each game (i.e. moves 16 16-20) only achieves a P@ 1 score of 0.234-far better than random guessing, indicating that specific moves indeed capture a significant amount of information, but far worse than our method. We also tested an alternative baseline that uses all moves of the game, which performs slightly better than the opening baseline, but still much worse than our method (P@ 1 = 0.277, see Supplement for details). In the Supplement, we also show how performance varies for different choices of the evaluation pool. Both our model and the baseline are robust to the choice of minimum total number of games per player, indicating that strong performance on this task is not contingent on players being very active on the platform.
powerful property of our method is that it adapts to players it has never seen before. We compare performance on unseen players versus seen players (recall that seen players are those whose training games were used during model training). The difference in performance between players the model explicitly trained on and players it has never seen before is surprisingly small. For example, the model achieves P@ 0.86 on unseen players and P@ = 0.853 on seen players. This suggests that the model has learned comprehensive space of decision-making style from the seen players, and can leverage this to characterize unseen players.
In Table 2 (bottom), we also show the version of the task where the opening moves are given to the models (k = 0). Our model performance increases to near- near-perfect accuracy, but the openings baseline also increases substantially. This indicates that using the opening moves to perform stylometry- identical moves that players repeat out of memorization is less interesting than using moves later on during games, when players are making decisions in instances they (and our models) have not seen before. Although our model is still significantly better than the openings haseline in this formulation of the task, we use k- = 15 for the rest of our analyses.
To gauge whether our performance relies on discriminating between player skill levels, we show a variation of behavioral stylometry limited to subsets of players with similar ratings. Restricting our attention to players with ratings within a narrow range around 1900 yields P@ = 92.6% (see Supplement Table 2 for full details).
Comparison with personalized move-matching models. The previous state-of-the-an performance for chess player stylometry was achieved by training individual move-matching models for every player in C' [3 This requires thousands of labeled games per player, and is only computationally feasible for small |C], hence we are unable to include this method our main analyses due to its lack of scalability. However, we directly compare our method on the small, 400-player candidate pool tested in 31 For this comparison, the evaluation and candidate pools are both 400 players, | 100. The results are shown in Table top), where personalized models
k 15, and |z,]|
7

---

Evaluation Pool (E)
[
[C]
k
Random
Bascline
Our Model
10k+ Lichess (unseen)
578
2844
15
0.0004
0.244
0.860
10k+ Lichess (seen)
2266
2844
15
0.0004
0.274
0.853
10k+ Lichess (all)
2844
2844
0.0004
0.268
0.854
10k+ Lichess (unseen)
578
2844
0
0.0004
0.929
0.979
10k+ Lichess (seen)
2266
2844
0
0.0004
0.930
0.982
10k+ Lichess (all)
2844
2844
0
0.0004
0.929
0.982
Table 2: Behavioral stylometry performance Lichess players with all games starting at move
k = 15 (top) or k = 0 (bottom). The candidate pool is the entire set of Lichess players with over 10K games, and the evaluation pool consists of all players that were seen, unseen, both. Each players' reference and query sets consisted of 100 games each.
Task
[E]
|C|
Baseline
Personalized
Our Model
Sellroy-Young et al.
400
400
0.478
0.552
0.953
High-ranked players
408
2157
0.039
0.308
U 10K-40K+ in C
408
5001
0.027
0.301
Table 3: (Top) Comparison with personalized move- move--rediction models [31] with k 15. (Bottom)
Performance on high-ranked players, either with high-ranked players only in the candidate pool or high-ranked players and Lichess amateurs together, with k = 15 and [x:g] = [r = 100.
achieve P@] I of 0.552. However, our model achieves P@ 1 of 0.953, far outstripping the previous method's performance.
High-ranking players. To examine how our model performs on out-of-distribution examples, we evaluated its performance on our set of high-ranking players (Section 4.1). These players are not only unseen by the model, but they are of much higher skill than any of the players it was trained on, and half of them drawn from separate online platform. We begin with the task of identifying the 408 unseen high-ranked players out of the complete set of both seen and unseen high-ranked players (Table 3 (bottom)). This task is significantly more difficult: both the baseline and model have lower P@ scores. However, our model still outperforms the baseline by almost factor of 10. Interestingly, when the set of Lichess amateurs is added to the candidate pool C, the model performance hardly changes at all, indicating that high-ranked players occupy a different part of the embedding space.
Reference and query set sizes. We studied how model performance varies with |z,,| and |=,/]- As Figure 3a shows, model performance rises sharply for earlier values and starts to saturate around x
= 50 games. Even when given a small set of labeled examples, i.e., [a>, 10. our model performs reasonably well. In comparison, personalized move- preeeiredinioon models require thousands of games begin seeing improvements over unpersonalized models [II]
Expanded dataset. We examined the model's performance on our complete dataset (1K 40K+), where all 41,184 players are used in both the candidate pool and evaluation pool. Even in this extensive regime, our model achieves a P@ of 0.54, while the baseline model achieves P@ of 0.0849 (see Supplement for details).
4.4 Qualitative Analysis
Attention. Our use of attention allows us to examine which sections of the game are most useful for identifying individual style. Figure 3 shows average per-move attention for the main model when applied to 100 games for each player in the 40K+ set. Four different starting moves were examined (k: 0, 5, 10, 15) show how the models handle being given incomplete games. The earlier parts of the game are most informative, with monotonic decay in attention games progress.
8

---

1.00
100
Attention
0.06
15
75
Accura
0.04
25
10
0.25
0.02
60
80
20
40
60
80
Query Set Size (|x,))
Move
(a)
(b)
(c)
Figure 3: (a) Stylometry performance (P@) 1) function of query size (x₂)) and reference set size (z,). (b) Distribution of attention across 1,900 games (40K+ seen players) from 4 different starting moves (k). Confidence intervals represent standard deviation. (c) -wo-dimensional t-SNE projection of high-ranked game vectors generated from games starting at move & = 15, colored by White's first move.
Playing style. Figure shows three different colorings of 2D t-SNE projection of the high-skill players. The colorings are based on the preferred opening moves of the players, when playing as White (see Supplement, Figure 7 for the corresponding projection when playing black). Despite these player vectors being learned from games starting on move k = 15, where the opening moves discarded, the resulting player embeddings tightly related to their opening preferences. This indicates that the space is aligned with the stylistic preferences of players.
Ethical Considerations
As this work pursues the identification of individuals from behavioral traces, it raises important ethical questions. In principle, although our main goal is to specialize behavioral models in the service of more human-compatinn actors. The most salient concerns revolve around user privacy, as sufficiently powerful stylometry methods could be used to deanonymize individuals. We now discuss these issues in detail, well as comment on our choice of domain and possible generalization to other domains.
AI, the methodology present and others like it could be abused by bad
Privacy concerns. Our methods can be used to accurately identify people from limited behavioral traces. This raises significant privacy concerns, well as concerns about power differentials and inequality, since those who are marginalized in society often benefit most from anonymized communication and behavior. The relatively small amount of data required to identify an individual with high degree of accuracy, at least in the context of chess, would likely surprise those who wish remain anonymous. Engaged players often have several accounts (either on the same online server or across multiple platforms), and our embedding methodology could be used to link them together. This could be problematic if player intended to keep an anonymous account separate from main account, potentially concealing ideas they are experimenting with or details about when they are active online. Similar concerns are also present the contexts of speech recognition, the domain we adapted our methodology from, and author identification in text, subfield with long history.
The history of author identification also makes clear that people have interest in developing countermeasures to being identified (and again, especially those with limited power). Our work contains insights that can offer help to those secking such countermeasures in our chosen domain of chess. Consider again player with multiple accounts. We find that if their accounts were to differ non-trivially in their opening repertoire, this would reduce our model's accuracy in identifying them as the same individual, since we find that the opening phase of the game is the most revealing part. To illustrate, we divide the games played as White by a single player into two sets, depending on whether they started with the opening move e4 or d4, the resulting set centroids are on average as distant from each other (cosine distance 0.27) as they are from other players' centroids. Of course, any such solution is far from ideal since it requires people to put effort into avoiding identification. But the framework we develop may be able to continue providing further insight into the design of countermeasures to identification requiring limited effort or adaptation from individuals.

---

Our methods could possibly reveal other information about players through the embedding represen- tations we learn. We do not examine if characteristies such as gender or nationality are correlated with the embedding representations, since the public dataset we use contains no such personally identifiable information, but other machine learning representations have been found to unwittingly incorporate private characteristies [2]
Domain and generalization. We chose chess as first domain to study since it is relatively benign compared to other possible contexts such as human speech, driving behavior, and cellphone mobility data. Having onc's 's chess behavior connected with onc's identity, while not always harmless, is arguably less serious than in other domains. It is not a priori clear, in any domain, how uniquely identifying particular forms of behavioral data will be; but it is crucial to understand this since it is easy to build individual-level classifiers from public data of this form, and individuals may act with expectation that they will remain anonymous, without knowing how revealing their data is. Moreover, there is a quantitative dimension to this-it is not simply all-or-nothing, but a question of how accurately these determinations can be made as function of the data available. Only through careful analysis of stylometry can we determine the level of risk in any given domain, and as such it an important part of making people aware of privacy risks. Our work is following this principle in illuminating the power of stylometry from behavioral traces. By deliberately investigating and quantifying these risks in the domain of chess, we can begin to develop this understanding before something similar is attempted in higher-stakes domains.
The above is particularly important since our methodology can be applied to other potentially more concerning domains. The framework of mapping sequences of decisions into a vector space, and then using an appropriate stylometry loss function, is not unique chess. We hope that the research community continues to develop both the methodological and ethical frameworks necessary to ensure behavioral stylometry techniques enhance human-Al compatibility and collaboration.
6 Conclusion
Motivated by the goal of algorithmically characterizing human behavior, we propose a method for performing behavioral pyllometr-iddeniffyeee people down to the level of specific individuals from their patterns of decisions and demonstrate the effectiveness of these methods in the domain of chess. We adapt techniques developed for speaker verification tasks to this decision-making setting, and construct a neural network architecture that can reliably identify individual-level decision- making style. Our architecture uses residual blocks to extract position- and move-level features, then combines these with a position encoding and inputs them to modified Vision Transformer that learns how to accurately aggregate this information into a game-level representation. We use GE2E loss framework to maximize both intra- player compactness and inter- player discrepancy in the resulting embedding space. The final model is then capable of performing highly accurate few-shot classification of unseen players, outperforming both the previous state of the art and competitive opening baseline. The resulting embedding space generalizes in several ways- it can identify high-ranked players even though it only trained amateurs, it identifies unseen players equally well as players it was trained on, and maintains its strong performance when expanded to tens of thousands of players.
Our methodology has several limitations that raise natural questions for future work. First, while our methodology far outperforms the baseline on high-ranking players, the lower accuracy suggests there is room for improvement in capturing top-class, complex styles. Second, our methodology is geared towards identifying individuals from their behavior alone, which could lead to privacy concerns. While this critique pertains to any stylometry task, including hand-writing recognition and speaker verification, our work raises similar questions in behavioral domains. Future work could investigate whether the benefits of methodologies capable of stylometry could be achieved without the privacy risks of uniquely identifying people. Finally, our methodology could be extended to other behavioral domains where fruitful human-AI collaboration is on the horizon.
Acknowledgments. Supported in part by Microsoft, NSERC, CFI. a Simons Investigator Award, Vannevar Bush Faculty Fellowship. MURI grant W91 1E0INN-119900 19-0217, AFOSR grant FA9550- 19- 1-0183, ARO grant 1911N719-1-0007, a Simons Collaboration grant, and grant from the MacArthur Foundation.
10

---

Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], or [N/A] . You are strongly encouraged to include justification your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license the code and datasets? [Yes] See Section.
Did you include the license to the code and datasets? [No] The code and the data are proprictary.
Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] We discuss them in the Conclu- sion.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We discuss the privacy risks in the Conclusion.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material as URL)? [Yes] Our supplement contains the code, the model files are too large for the supplement so will be released when the paper goes public
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] We have data and methods sections discussing this along with more information in the supplement
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [N/A] We ran over thousands of players to get error estimates, but did not train or test multiple times
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Our methods and supplement discuss this
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] Our code release links to the code we build on
(b) Did you mention the license of the assets? [Yes] The code we are using is GPL or MIT licensed and is mentioned in the code release
(c) Did you include any new assets either in the supplemental material as URL? [Yes] We plan to release the code and models after publication
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [Yes] We discuss this in the Data section, all data used is public and not sensitive
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] We discuss this in the Data section, there no PIl in our data
11

---

5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
12

---

References
[1] Ashton Anderson, Jon Kleinberg, and Sendhil Mullainathan. Assessing human error against a benchmark of perfection. ACM Transactions on Knowledge Discovery from Data (TKDD), 14)):45 2017.
[2] Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. In Advances in neural information processing systems, pages 3981-3989, 2016.
[3] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization, 2016.
[4] Philip Bachman, Alessandro Sordoni, and Adam Trischler. Learning algorithms for active learning. Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 301-310. PMLR, 06- 11 Aug 2017.
[5] A. Bakhtin, S. Gross, Myle Ott, Y. Deng, Marc' Aurelio Ranzato, and Arthur D. Szlam. Real or fake? learning to discriminate machine from human generated text. ArXiv, fbss/190600331,, 2019.
[6] Tamal Biswas and Kenneth W Regan. Measuring level-k reasoning, satisficing, and human error in game-play data. In 2015 IEEE 14th International Conference Machine Learning and Applications, pages 941-947, Miami, FL, 2015. IEEE.
[7] Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. Signature verification using a" siamese" time delay neural network. Advances in neural information processing systems, 6:737-744, 1993.
[8] Aylin Caliskan Islam, Richard Harang, Andrew Liu, Arvind Narayanan, Clare Voss, Fabian Yamaguchi, and Rachel Greenstadt. De-anonymizing programmers via code stylometry. In 24th USENIX Security Symposium (USENIX Security 15), pages 255-270, Washington, D.C., August 2015. USENIX Association.
[9] Neil Charness. The impact of chess research on cognitive science. Psychological research, 54(1):4-9, 9, 1992.
[10] Jonghyun Choi, J. Krishnamurthy, Aniruddha Kembhavi, and Ali Farhadi. Structured set matching networks for one-shot part labeling. 2018 IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 3627-3636, 2018.
[11] S. Chopra, R. Hadsell, and Y. LeCun. Learning similarity metric discriminatively, with application to face verification. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) volume 1, pages 539-546 vol. 1, 2005.
[12] Alexandra Chouldechova and Aaron Roth. A snapshot of the frontiers of fairness in machine learning. Communications of the ACM, 63858822-8, 2020.
[13] Joon Son Chung, Jaesung Huh, Scongkyu Mun, Minjae Lee, Hee-Soo Heo, Soyeon Choe, Chiheon Ham. Sunghwan Jung, Bong-Jin Lee, and Icksang Han. In Defence of Metrie Learning for Speaker Recognition. In Proc. Interspeech 2020, pages 2977-293 2020.
[14] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4685-4694 2019.
[15] Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre- Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
13

---

[16] Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In International conference machine learning, pages 647-655, 2014.
[17] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[18] Li Fei-Fei, R. Fergus, and P. Perona. One-shot learning of object categories. IEEE Transactions on Puttern Analysis and Machine Intelligence, 28(4):594-611, 2006.
[19] Michael Fink. Object classification from a single example utilizing class relevance metrics, In L. Saul, Y. Weiss, and L.. Bottou, editors, Advances in Neural Information Processing Systems, volume 17. MIT Press, 2005.
[20]
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adap- tation of deep networks. In Proceedings of the 34th International Conference on Machine 70, pages 122--1135. JMLR. org, 2017.
[21] Andrew Gibiansky, Sercan Arik, Gregory Diamos, John Miller, Kainan Peng, Wei Ping, Jonathan Raiman, and Yanqi Zhou. Deep voice 2: Multi-speaker neural text-to-speech. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[22] Sumit Goswami, Sudeshna Sarkar, and Mayur Rustagi. Stylometric analysis of bloggers' age and gender. In ICWSM, 2009.
[23] Luiz G. Hafemann, Robert Sabourin, and Luiz S. Oliveira. Writer-independent feature learning for offline signature verification using deep convolutional neural networks. In 2016 International Joint Conference on Neural Networks (IJCNN), pages 2576-2582 2016.
[24] Luiz G Hafemann, Robert Sabourin, and Luiz S Oliveira. Learning features for offline hand- written sigmature verification using deep convolutional neural networks. Pattern Recognition, 70(C):163 176, 2017.
[25] Jie Hu, Li Shen, and Gang Sun. Squeeze-and setpree-anndxxiiiatinoon networks. In Proceedings the IEEE conference on computer vision and pattern recognition, pages 7132-7141, 2018
the
[26] Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. Caffe: Convolutional architecture for fast feature embedding. In Proceedings of the 22nd ACM International Conference Multimedia, MM '14, page 678, New York, NY, USA, 2014. Association for Computing Machinery.
[27] Ye Jia, Yu Zhang, Ron Weiss, Quan Wang, Jonathan Shen, Fei Ren, Zhifeng Chen, Patrick Nguyen, Ruoming Pang. Ignacio Lopez Moreno, et al. Transfer learning from speaker verifica- tion to multispeaker text-to soteehh-peecch synthesis. arXiv preprint 1806.08800.04555 2018.
(28) Chao Li, Xiaokong Ma, Bing Jiang, Xiangang Li, Xuewei Zhang, Xiao Liu, Ying Cao, Ajay Kannan, and Zhenyao Zhu. Deep speaker: an end-to-end neural speaker embedding system. arXiv preprint arXiv: 7777.02000344 2017.
[29] Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. Sphereface: Deep hypersphere embedding for face recognition. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6738-6746, 2017.
[30] Reid Mcllroy- Young, Siddhartha Sen, Jon Kleinberg, and Ashton Anderson. Aligning super- human ai with human behavior: Chess model system. In Proceedings of the 25th ACM SIGKDD international conference on Knowledge discovery and data mining, 2020.
[31] Reid Mellroy-Young Russell Wang, Siddhartha Sen, Jon Kleinberg, and Ashton Anderson. Learning personalized models of human behavior in chess. arXiv preprint arXiv:2008. 10086, 2020.
14

---

[32] Tempestt Neal, Kalaivani Sundararajan, Aneez Fatima, Yiming Yan, Yingfei Xiang, and Damon Woodard. Surveying stylometry techniques and applications. ACM Comput. Surv, 50(6), November 2017.
[33] Maxime Oquab, Leon Bottou, Ivan Laptev, and Josef Sivic. Learning and transferring mid- level image representations using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1717-1724, 2014.
[34] Wei Ping, Kainan Peng. Andrew Gibiansky, Sercan O. Arik, Ajay Kannan, Sharan Narang, Jonathan Raiman, and John Miller. Deep voice 3: 2000-speaker neural text-to-speech. In International Conference on Learning Representations, 2018.
[35] Sachin Ravi and Hugo Larochelle. Optimization model for few-shot learning, In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
[36] Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B. Tenen- baum, Hugo Larochelle, and Richard S. Zemel. Meta-learning for semi-supervised few-shot classification. In Proceedings of 6th International Conference on Learning Representations ICLR, 2018.
[37] Hanan Rosemarin and Ariel Rosenfeld. Playing chess at a human desired level and style. In Proceedings of the 7th International Conference on Human-Agent Interuction, pages 76-80, 2019.
[38] Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: unified embedding for face recognition and clustering. In 2015 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 815-823, 2015.
[39] David Silver, Thomas Hubert, Julian Schritwieserr, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 2018.
[40] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, al. Mastering the game of go without human knowledge. Nature, 2017.
[41] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[42] David Snyder, Pegah Ghahremani, Daniel Povey, Daniel
Yishay Carmiel, and Sanjeev Khudanpur. Deep neural network-based speaker embeddings for end-to-end speaker verification. In 2016 IEEE Spoken Language Technology Workshop (SLT), pages 165-170, 2016.
[43] Nima Tajbakhsh et al. Convolutional neural networks for medical image analysis: Full training or fine tuning? IEEE transactions on medical imaging, 15(5):12999110 1312, 2016.
[44] Sebastian Thrun and Lorien Pratt, editors. Lifelong Learning Algorithms, pages 181-209. Springer US Boston, MA, 1998.
[45] Fiona Tweedie, Sameer Singh, and David Holmes. Neural network applications in stylome- try: The Federalist Papers. Comput. Humanit., 30(1):1-10, 1996.
[46] Ehsan Variani, Xin Lei, Erik McDermott, Ignacio Lopez Moreno, and Javier Gonzalez- Dominguez. Deep neural networks for small footprint text-dependent speaker verification. In 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4052-4056 2014.
15

---

[47] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Guyon, U. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[48] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, koray kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
[49] Li Wan, Quan Wang, Alan Papir, and Ignacio Lopez Moreno. Generalized end-to-end loss for speaker verification. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4879-4883, 2018.
[50] Hao Wang. Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu. Cosface: Large margin cosine loss for deep face recognition. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5265-5274, 2018.
[51] Yaqing Wang, Quanming Yao, James T. Kwok, and Lionel M. Ni. Generalizing from a few examples: A survey few-shot learning, ACM Comput. Surv, 53(3), June 2020.
[52] Shi-Xiong Zhang, Zhuo Chen, Yong Zhao, Jinyu Li, and Yifan Gong. End-to-end attention based text-dependent speaker verification. In 2016 IEEE Spoken Language Technology Workshop (SLT), pages 171-178. 2016.
[53] Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. Transfer learning for low-resource neural machine translation. arXiv, 2016.
16


Reviews:
[{"review_id": "pv__8wyl3yf", "reviewer": "Reviewer_a4DH", "summary": "This paper describes a transformer-based approach to determining chess player identity from moves played.", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Review Criteria**\n\nOriginality: McIlroy-Young et al. (2020b) have previously studied behavioral stylometry in chess. This paper provides a more accurate and scalable solution to an existing task domain using transformer-based game embeddings. The architecture is adapted from work on speaker adaptation but is novel in its application to this domain. \n\nQuality: The results appear technically sound and the proposed method achieves high accuracy. The authors run appropriate ablations, such as not including opening moves, evaluating out-of-domain generalization, etc. [These ablations are especially important because the accuracy values are *shockingly* good; as an expert chess player (~ELO 2100), I am quite surprised that it is theoretically possible to identify chess players with such high accuracy, and I was quite skeptical of the results at first.]\n\nClarity: The task and model architecture are clearly described. Some details of the few-shot classification setup are difficult to understand. \n\nSignificance: Behavioral stylometry in chess is not a particularly significant task, but the methods described could easily apply to other domains involving decision-making style, and chess serves as an appropriate and convincing testbed. In particular, the comparison between the embedding method and personalized \"move-matching\" models is interesting and likely to generalize to other domains. However, as noted below, stylometry is wrought with potential ethical issues that should be discussed further.\n\n**Notes on Specific Sections**\n\nAbstract: First paragraph of the abstract can be removed, since it reads more like an introduction and is in fact repeated in the introduction. I would recommend starting the abstract by describing what you did, rather than motivating the problem domain.\n\nIntroduction: It might be worth defining \"style\" and what it means in the context of chess. For example, one common distinction is between aggressive/tactical vs. positional play, with certain players (Shirov, Tal) being known for aggressive play, and certain openings (e.g., gambits) being more or less tactical in nature. Presumably, this method also picks up on elements of \"style\" which do not readily correspond to human-interpretable concepts.\n\nRelated Work/Methodology: Overall, this is all very clear and well-written. Possibly out-of-scope, but I wonder if you could achieve better results by initializing with board embeddings from the policy network of AlphaZero or some other neural chess engine. [But the results are already quite good, so this isn't top priority.]\n\nData: I'm not sure why the paper groups by the number of games played when the evaluation uses only 200 games for each player. However, sampling from ELO 1000-2000 is quite a large range, and I'm curious if these results would still hold for a narrower range. It seems possible to me that a model could be evaluating player strength rather than \"style.\" Similarly, the top 1500 players from Chess.com/Lichess will also exhibit a lot variance in strength.\n\nEvaluation: \n - Task formulation: I found this section difficult to read, possibly because I had previously assumed D = R.\n - Can you confirm that k=15 means 15 moves for both players? Normally in chess a \"move\" consists of plays from both white and black, but I just wanted to confirm that's the case in your paper. There is technically opening theory that extends beyond the 15th move, but especially in Lichess 1000-2000 games, I wouldn't expect it to be very common. However, if k=15 actually meant 15 turns for both players combined (i.e., 7.5 moves), then this would definitely still be within the realm of opening theory, even for many amateur players.\n - Results for k=0/k=15 and varying sizes of support/query sets are useful, thanks!\n - Attention results seem straightforward and confirm my intuitions that openings are the most useful part of the game for stylometry, except for maybe the first move (where there are just less choices to be made). I'm less clear what takeaways I should have from the t-SNE plots, especially 3b and 3c. Does this just mean that choice of first move is an important factor in style but responses or not? Or how should I interpret 3b-3c?\n\nConclusion: Additional discussion of privacy concerns would be appreciated. See below.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper would benefit from further motivation of the problem domain and further discussion of ethical issues concerning behavioral stylometry. Is chess simply a testbed for general-purpose behavioral stylometry methods, or are there real-world applications of behavioral stylometry in chess? I see two main ethical issues with general-purpose behavioral stylometry, which merit further discussion:\n - Privacy Concerns -- this is briefly mentioned in the conclusion but deserves further discussion and citation of related ethics work. For example, the related work stylometry section mentions face verification, de-anonymization of programmers, and \"identification of age and gender from blog posts\" but does not mention the ethical issues raised by these application domains. Further, detecting user identity from online behavior carries many ethical risks and the tradeoffs are not well-described in this paper. For example, what are the \"benefits of methodologies capable of stylometry\" in this context? What are the risks? [Although I think this is mainly an issue in other stylometry domains, I don't particularly want my (anonymous) Lichess account to be connected to my (public) Chessbase data, either.]\n - Performance Ceilings -- high-accuracy stylometry is impossible in many domains, such as author identification. This can have ethical implications for e.g., forensic domains, where it is critical that policymakers and other non-technical individuals understand the limitations of these methods. Even in a domain like chess, methods for cheating detection (which became especially popular during virtual tournaments due to COVID-19) are limited in their accuracy, and accurately conveying that *it is probably impossible to identify players with perfect accuracy* is important for helping non-ML practitioners understand and make use of these results.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "NaB7iAHzq6", "reviewer": "Reviewer_iTpg", "summary": "This paper introduces a transformer-based neural network architecture that can be used to learn and identify unique playing styles of individual chess players. The neural network learns an embedding over players' actions and by using the distance between the learned embedding and a sequence of chess moves, the network can identify the player taking those moves. While this problem has been studied before, the main contribution of the paper is the ability of the architecture to learn the playing styles of unseen players using a small number of example moves.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main idea and motivation behind the paper is well defined and I found the problem of few-shot learning of player styles interesting. However, the comparison with a baseline used in the experiments from a previous paper is flawed -- see below.\n\nOriginality: While none of the techniques used in this paper are novel, the authors do employ previous techniques to a problem in which they are yet to be used. The use of learned embeddings to characterize player styles was clever.\n\nClarity: The paper leaves much to be desired in terms of clarity. I think the paper could have been written better. One subsection in which this is apparent is the description of the loss function. The explanation reads as if it structured in reverse --- that is, it would have made more sense if the similarity matrix was described before the embeddings. In fact, the description of the embeddings uses notations not defined until the similarity matrix was explained. I ended up having to refer the to the cited literature in order to understand this part. Furthermore, I think the paper focusses too much on the description of the architecture specifications rather than on the reason this architecture was chosen. Given that the authors claim that the strength of their proposed model lies in the use of transformers, it would have been prudent for the authors to describe why this is so --- why are transformers better? What characteristics of transformers make them more suitable to the task of behavioral stylometry?\n\nQuality: The experimental evaluation has a potentially major issue in regards to the comparisons performed against previous methods. Authors, please clarify this in your response, if I misunderstood the comparison. As far as I can tell, the paper misrepresents its comparative performance. The paper presents behavioral stylometry on settings of k = 0 and k = 15. While the authors state that they compare against previous work at a setting of k = 15, this is not what they do. The results from previous work on an identical problem were performed at settings of k = 0, 10 and 30. The performance of the previous work reported in the paper were for k = 30. The prediction problem is much more difficult with higher k values (as the authors themselves make clear.) This invalidates any comparison with the previous algorithm. \n\nSignificance: The paper does not present anything wholly unique, but the results reported are promising (if one discounts the potential major flaw described in the paragraph above). Another limiting factor of the paper's significance is that it uses a single problem domain in its evaluation.\n\nCOMMENT AFTER RESPONSE\n\nThanks for your response. In particular for clarifying the issue with the k-value. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors do not sufficiently address the societal impacts of their work. There is one line in the conclusion that addresses this but it is not enough. Especially, given the security and privacy concerns that could arise from misuse of the research. While the work is limited to a relatively harmless domain, there is potential of its use in more sensitive domains. Given that identifying people based incomplete information can be abused in a number of situations, the authors should have spent more time elaborating on the positive/negative implications of their work.", "review_text": "", "extra_scores": {"ethical_concerns": "The work pertains to explicitly de-anonymizing the identities of people based on limited data. The authors do not sufficiently address the negative implications of abuse of such technology.", "needs_ethics_review": "Yes", "ethics_review_area": ["Inappropriate Potential Applications & Impact  (e.g., human rights concerns)", "Privacy and Security (e.g., consent)"], "time_spent_reviewing": "12"}, "extra_text": {}}, {"review_id": "KYPi4IYkw1b", "reviewer": "Reviewer_RGQQ", "summary": "This paper introduces a task of behavioral stylometry in the domain of chess plays. It uses a few-shot learning framework and uses the optimization objective inspired by the speaker identification domain to solve this problem. It demonstrated clear gains over previous baselines and has performed a satisfactory analysis that can be used by future researchers to extend this work.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality**: this work opens up a new and meaningful task domain for few-shot learning and representation learning. Though the framework and optimization objective are heavily inspired by other domain, the application is very novel.\n\n**Quality**: the quality of this paper is very high. The authors have conducted satisfactory experiments to evaluate the model's performance. The original experiment shows the promise of using few-shot learning as a framework to solve this task. The ablation studies (by whether to include first k moves) and evaluation of top players point the research community to future research directions.\n\n**Clarity**: the structure and writing are very clear.\n\n**Significance**: it is very important to introduce new problems to the machine learning community. Online chess communities have been growing for quite a few years, and this domain has a strong connection to human behavior modeling, teaching, and human-AI collaboration. It is left to the ML community to decide what are interesting problems that ML should solve in the domain of chess play. The authors of this paper built upon prior work and defined player identification as one of the important tasks to solve -- embeddings that capture the behavioral style of play are effective embeddings for chess moves. This has a huge implication for unsupervised learning and embedding evaluations in the future. It might not be THE task or THE evaluation metric that everyone needs to care about, but this is a very solid first step.\n\nI don't have any significant issues with this paper, mainly because this paper introduces a new problem in a new domain. This is not to say that the paper is perfect. For example:\n1. The paper proposes to build \"embeddings\" of player style, but the visualization in Figure 3 (b) (c), I can't easily recognize clusters, making me wonder if the embeddings do contain valuable information beyond player identification.\n2. It seems that the data is public but the player IDs can only be obtained through private communication. I'm concerned that this level of privacy will impact future researchers using this as a valid task to test new algorithms. I hope the authors of this paper will spend more time thinking about how to build a research community around this domain and how to enable rapid model development in this domain in the future. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have addressed other limitations well. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "I_X82eB9pvI", "reviewer": "Reviewer_qhYv", "summary": "In this paper the authors propose a transformer-based approach for capturing and identifying individual styles in chess. This goal, which the authors name Behavior Stylometry, is framed as an essential step towards algorithmic personalization and person-agent interaction. The suggested framework borrows ideas from advances in speech processing and only utilizes a small sample in order to learn a representation of players' actions. Once trained, the model can accurately identify the users in a larger pool of unseen samples. Lastly, the authors suggest that their approach reveals structural properties for characterizing playing style in chess.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, this paper seems methodologically sound to me, and the results seem compelling enough. I do have certain qualms, however.\n\n* The paper seems to be very much tailored for chess applications. While that makes the paper coherent and easy to follow, it also diminishes its impact. It is hard to estimate how much of the proposed approach would generalize to other domains, for instance, or how easy it would be to adapt the proposed framework for other tasks. \n\n* The design of the loss function is meant to promote an embedding which keeps *individuals* as far apart in the embedding space as possible, but this approach seems to discount the fact that different people may have very similar playing styles, thus leading to artifacts in more general applications (I have more on that note, and how it relates to privacy concerns, see my last comment).\n\n* From a training perspective, it is unclear how much intrinsic endogeneity may exist in the training data. That is to say, while you are separating games, entire subsequences within the game may be close to identical. To mitigate or at least study this aspect, it would make sense to cluster state-action subsequences within game trajectories and group/cluster them. How well would the model fare when seeing familiar people take slightly novel sequences?\n\n* It would  have been interesting to see how well a naïve approach, such as comparing conditional action distributions, would have fared in this task.\n\n* Lastly I'd like to point out that any approach for identifying users by a small sample of their actions may pose dire privacy concerns and should not be explored lightly. While I can see its utility in very restricted game-playing scenarios (for instance, opponent modeling), I believe it is a lot more valuable in typical contexts such as game-playing, recommendation etc. to characterize the *type* (indeed, the *style*) of a person or agent, rather than the unique identity of that person. Going for the latter goal seems to assume unnecessary privacy risk without adding much real-world utility to me.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I believe the greatest limitation of this work is its specificity to the chess domain, making it harder to estimate how well the approach would generalize in other contexts, and therefore its broader impact. ", "review_text": "", "extra_scores": {"needs_ethics_review": "Yes", "ethics_review_area": ["Privacy and Security (e.g., consent)"], "time_spent_reviewing": "3"}, "extra_text": {"ethical_concerns": "As stated in the main review, one obvious ethical concern I have is that being able to identify users based on a small sample of their behavior, even in a context as seemingly innocuous as chess, may pose substantial privacy issues, which the authors themselves rightly note as well in the concluding section."}}]

Author rebuttal:

Meta-review: UPDATE:  The revision has been reviewed and this paper has been accepted.  We recommend that the authors integrate the final sentence of the introduction into the end of Paragraph 2, where they are talking about applications of their ideas.  Some applications raise ethical considerations and it's important to admit that alongside the other applications rather than it appearing as an afterthought at the end.

----

This paper is very difficult to make a decision.  The reviewers all agree the paper is excellent technically (even the low reviewer score is not basing their score on the technical contribution).  However, there is also general agreement that the paper does not sufficiently address what amounts to serious ethical considerations of this work, specifically the opportunity for it or derivatives to be abused.  The authors acknowledge that a more substantial discussion is needed in the paper, so I trust they understand this significance of this concern.

However, I (and other reviewers) also feel the authors' response lacks a recognition of appropriate mitigations of possible abuse.  They primarily appeal to the public nature of the chess data as a consideration in why this line of research is ethically appropriate.  This is not sufficient.  Abuses, for example where a bad actor uses even public data to identify/track an otherwise anonymous person in order to discriminate or restrict rights, are not limited to use only private data.  In fact, it seems to me public data abuses are more likely or more damaging than private data abuses (as the latter limits the abuse to owners of private data).

I do believe advances of stylometry in identifying chess players is a pretty benign application, however there's little discussion about how general this work might be in its ability to do stylometry in other domains.  The authors mention an additional social benefit of understanding what identifiability exists in our datasets, but this argument needs more attention and peer review than can be completed through the author response/discussion.

As a consequence this paper is being **conditionally accepted**.  The final determination will only be made upon review of the revision.   The authors should take advantage of the extra page to address concerns as reflected in their own response(s), although without the emphasis that use of public data lessens the ethical concerns.  We hope that the authors' added discussion could be used to send a strong signal commensurate with the gravity of the situation, so that appropriate community norms can be developed.
Decision: Accept (Poster)