Title: Maia-2: A Unified Model for Human-AI Alignment in Chess
Venue: NeurIPS 2024
Accepted: yes
Keywords: Human-AI Alignment

Action Prediction

Chess

Skill-aware Attention
Abstract: There are an increasing number of domains in which artificial intelligence (AI) systems both surpass human ability and accurately model human behavior. This introduces the possibility of algorithmically-informed teaching in these domains through more relatable AI partners and deeper insights into human decision-making. Critical to achieving this goal, however, is coherently modeling human behavior at various skill levels. Chess is an ideal model system for conducting research into this kind of human-AI alignment, with its rich history as a pivotal testbed for AI research, mature superhuman AI systems like AlphaZero, and precise measurements of skill via chess rating systems. Previous work in modeling human decision-making in chess uses completely independent models to capture human style at different skill levels, meaning they lack coherence in their ability to adapt to the full spectrum of human improvement and are ultimately limited in their effectiveness as AI partners and teaching tools. In this work, we propose a unified modeling approach for human-AI alignment in chess that coherently captures human style across different skill levels and directly captures how people improve. Recognizing the complex, non-linear nature of human learning, we introduce a skill-aware attention mechanism to dynamically integrate players’ strengths with encoded chess positions, enabling our model to be sensitive to evolving player skill. Our experimental results demonstrate that this unified framework significantly enhances the alignment between AI and human players across a diverse range of expertise levels, paving the way for deeper insights into human decision-making and AI-guided teaching tools.

Maia-2: A Unified Model for Human-AI Alignment in Chess

Abstract

There are an increasing number of domains in which artificial intelligence (AI) systems both surpass human ability and accurately model human behavior. This introduces the possibility of algorithmically-informed teaching in these domains through more relatable AI partners and deeper insights into human decision-making. Critical to achieving this goal, however, is coherently modeling human behavior at various skill levels. Chess is an ideal model system for conducting research into this kind of human-AI alignment, with its rich history as a pivotal testbed for AI research, mature superhuman AI systems like AlphaZero, and precise measurements of skill via chess rating systems. Previous work in modeling human decision-making in chess uses completely independent models to capture human style at different skill levels, meaning they lack coherence in their ability to adapt to the full spectrum of human improvement and are ultimately limited in their effectiveness as AI partners and teaching tools. In this work, we propose a unified modeling approach for human-AI alignment in chess that coherently captures human style across different skill levels and directly captures how people improve. Recognizing the complex, non-linear nature of human learning, we introduce a skill-aware attention mechanism to dynamically integrate players' strengths with encoded chess positions, enabling our model to be sensitive to evolving player skill. Our experimental results demonstrate that this unified framework significantly enhances the alignment between AI and human players across a diverse range of expertise levels, paving the way for deeper insights into human decision-making and AI-guided teaching tools. Our implementation is available here.

1 Introduction

There are an increasing number of domains in which artificial intelligence (AI) systems both surpass human ability and accurately model human behavior. This combination of machine mastery over a domain and computational understanding of human behavior in it introduces the possibility of algorithmically-informed teaching and learning. AI-powered aids could guide people along reliable and efficient improvement paths, synthesized from their knowledge of both human trajectories and

objective performance. Relatable AI partners, on the other hand, could learn to act alongside human counterparts in synergistic and complementary ways.

Researchers have begun to tackle this challenge in the model system of chess. Once held to be an ideal testbed for developing artificial intelligence, it is now the perfect domain to pursue human-AI alignment. The AI community finally surpassed all human ability in chess approximately 20 years ago, a milestone achievement and watershed cultural moment. Now, superhuman AI chess engines are ubiquitous and widely used. Despite this transformation, chess has never been more popular, becoming a mainstream activity in many countries during the last few years. There is now both unprecedented demand for chess education, as well as mature superhuman AI that could in principle help meet it.

However, existing models fall short of being effective learning tools and relatable partners. Traditional chess engines such as Stockfish and AlphaZero are unimaginably strong, but they don't play in ways that humans can easily understand or learn from. Comparing one's own decisions with those of traditional engines, it is easy to see how near-perfect AI would have improved upon your play but hard to see how you could realistically do the same. Recent work has resulted in the development of Maia, a suite of models that aim to mimic human behavior in chess at various skill levels by learning to predict actual human moves from a wealth of online gameplay data [1]. While substantially more human-like, these models still cannot power effective algorithmic teaching tools because of several limitations.

First and foremost, Maia models players at different skill levels completely independently; games by players at one skill level and those by an adjacent skill level are fed into separate instances of the same architecture and result in separate models. This has the downside that predictions from one model are independently made of predictions from any other. Viewed as a whole, they are volatile: the Maia models might predict that at one level players will approach a position correctly, then at the next level they will make a horrible mistake, then at the next level they will do fine again, and so on. In a word, they fail to cohere . People don't improve along volatile paths, they steadily get better. The unrealistically incoherent predictions made by separate models don't suggest realistic pathways that people can take in order to get better. In order to serve as algorithmic teachers or learning aids, our models of human behavior must be coherent.

Building a coherent model of human skill in chess is difficult, because the breadth of skill in chess is almost incomprehensibly large. Decisions made by beginners bear only the faintest of relations to those made by masters. A difference of 200 points in chess rating systems roughly equates to a 75% win rate for the higher-rated player-typically higher than the best record of any team in the entire National Basketball Association. On the online chess platform we study, there are players who are 2600 rating points apart-or 13 successive steps of 75%-vs.-25% dominance apart from each other. Capturing this breadth of skill in a single model, in a coherent, smooth fashion, is a challenge.

We contribute a unified modeling approach for human-AI alignment in chess that coherently captures human style across different skill levels and directly captures how people improve. Since our model builds directly on original Maia, we call it Maia-2 . Maia-2 consists of a standard residual network tower that processes chess positions into features, and our novel contribution of a skill-aware attention module with channel-wise patching. This innovation takes the position representation outputted by the residual network tower and simple player skill encodings and learns how player skill levels interact with chess positions to produce the moves humans make. Unlike previous models, Maia-2 only requires the current board position as input (as opposed to six), which dramatically reduces training time and increases flexibility (e.g. for applying the model in non-game contexts where there may be no 6-board history). In addition to policy and value heads like in previous work, we also add an additional auxiliary information head that helps the model learn a deeper understanding of human chess moves.

We evaluate Maia-2 along two key dimensions: move prediction accuracy and coherence . Testing it against the original Maia models, Stockfish, and AlphaZero, Maia-2 emerges as the most accurate human move predictor by far, surpassing original Maia by almost 2 full percentage points. Analyzing move prediction accuracy by skill level, Maia-2 matches and surpasses all other models on all skill levels. Furthermore, Maia-2's gains in perplexity are similarly striking, reducing average perplexity from a previous record of 4.67 bits down to 4.07 bits. Maia-2 achieves these accuracy gains while being substantially more coherent than the original Maia models. For example, call a model's treatment of a position monotonic if it assigns a monotonically increasing probability to the

correct move as we increase skill. While original Maia treats 1% of a random sample of positions monotonically, Maia-2 treats a remarkable 27% of the same positions monotonically. This is in keeping with our intuitive understanding of how chess players steadily and smoothly improve across the skill range. Finally, we conduct an investigation of the human chess concepts Maia-2 learns and varies with skill via linear probes, and find that skill-dependent concepts like overall board evaluation indeed vary with skill, but skill-independent concepts do not, which also accords with our understanding of how human players make decisions.

2 Related Work

Chess and AI. This paper draws on the long history of chess at the forefront of AI research [2, 3, 4, 5]. We engage with 3 distinct approaches to building chess AI: heuristic [6], learned [7], and textual [8]. Heuristic search: The original approach to computer chess was heuristics-based [4, 9]. This method was famously used by IBM's Deep Blue to defeat Garry Kasparov [2] and is currently used by Stockfish [6], one of the strongest chess engines in the world. Learned search: Alpha(Zero) Go [7, 10] is a set of neural networks that learn to play Go with methods that generalized to other games, including chess, with AlphaZero [10]. Chess AI with learned search is also extended to multi-agent systems [11], where diverse AI systems can outperform a single AI in challenging tasks such as chess. Chess as text: Large language models [12, 13, 14] have recently been found to perform well on tasks that the models were not explicitly trained on [15], including playing chess without fine-tuning [16, 17, 18]. This has lead to chess knowledge being one of the tested features in BIGBench [19], a popular LLM evaluation suite. Additionally, fine-tuning a language model can lead to systems that not only play chess, but can also generate comments, describe positions, and create other simple analyses of a game [20, 21, 8].

Human-AI Alignment in Chess. Building a chess engine that can defeat any human has been a solved problem for over 20 years. This has led to a new research agenda in extracting useful knowledge from these superhuman systems. A direct way of doing this is to probe an AI chess engine in a human representation space. Without any prior human knowledge or guidance, evidence of human chess concepts learned by AlphaZero is found and measured by linear probes [22]. Going further, AlphaZero also encodes knowledge that extends beyond existing human knowledge but is ultimately learnable by humans [23].Another direction was the creation of a 'behavioral stylometry' model that can identify chess players from the moves they play [24]. An alternative approach to creating systems that can act as guides to humans is demonstrated by Maia [1, 25], in which a model is trained to predict the next move a human will play, instead of optimizing for winning the game. In addition to predicting human actions the models have been fine-tuned to predict a given player's actions [25]. The prediction accuracy can be improved via a reinforcement learning-style search [26].

3 Methodology

We propose a unified model architecture to capture human decision-making in chess across a broad spectrum of skill levels. Since this model builds upon the previous Maia move-matching models, we call it Maia-2 . As shown in Figure 1, Maia-2 first encodes active and opponent skill levels and the chess positions, respectively. Then the encoded skill levels and positions are fused using our skill-aware attention with channel-wise patching architecture. The fused representations are then used for move prediction (policy head), auxiliary information prediction (auxiliary head), and game outcome prediction (value head). We now discuss each of these components in detail.

3.1 Skill Level Encoder

Instead of directly incorporating player ratings as numerical inputs, we use categorical skill level embeddings for two reasons. First, player behavior and decision-making in chess are not linearly related to their rating. Categorical embeddings allow for capturing complex, non-linear relationships between player strength and their moves. They can encode nuanced differences in play style and strategy that are not directly proportional to player ratings. Second, Generalization across similar skill levels: Players within a certain skill level may exhibit similar playing styles, strategies, and common mistakes. Categorical embeddings group players into these ranges, helping the model

/0 /1

/1

/2

/i255

/0 /4

/5

/5

/2

/8/9

/10

/11

/12

/13

/14

/1

/2

/i255

/15 /1

/10

/16

/13

/11

/11

/1

/17

/11

/12

/4

/17

/18

/19

/20

/21

/21

/22

/23

/24 /25

/26

/27

/i255

/23 /29

/29

/27

/30

/29

/20

/31

/30

/32 /9

/1

/5

/33

/34 /1

/33

/35 /7

/10

/9

/1

/18

/19

/20

/21

/21

/22

/23

/24 /25

/26

/27

/i255

/32 /9

/1

/5

/33

/36

/4 /17

/37

/38

/39 /40

/41

/i255

/42

/15

/14

/4

/5

/11

/43

/9

/11

/15

/44

/12

/10

/10

/i255

/45 /1

/37 /1

/10

/i255

/46

/17

/43

/4 /2 /1

/5

/36 /4

/17

/37

/38

/39 /40

/41

/i255

/42

/36 /4

/17

/37

/38

/39 /40

/41

/i255

/42

/47 /27

/48

/49 /27

/29

/i255

/50

/25

/51

/19

/52

/31

/30

/27

/54

/4 /4

/10

/12

/17

/55

/i255

/56

/0 /1

/1

/2

/i255

/0 /4

/5

/5

/2

/54

/4 /60

/12

/11

/12

/4

/17

/i255

/46

/17

/43

/4 /2 /1

/5

RATING1942

RATING1088

田

/53

/53

/53

/36

/14

/17

/17

/1

/10

/13

/6 /12

/60

/1

/i255

/54 /7

/11

/43

/14

/12

/17

/55

/0 /1

/1

/2

/i255

/0 /4

/5

/5

/2

/36

/4 /17

/37

/38

/39 /40

/41

/i255

/42

/57 /9

/58

/12

/10

/12

/5

/33

/8/4

/37 /1

/35 /7

/10

/9 /1

/53

/53

/53

to better generalize across players with similar strengths, as opposed to treating each rating as a numerical input.

/59 /1

/2

/59 /1

/2

/59 /1

/2

Let E ∈ R | E |× d s be the matrix of player rating embeddings, where each row corresponds to the embedding of a skill level with dimension d s : E = [ e (0 , 1000] , e (1000 , 1100] , ..., e (2000 , + ∞ ) ] ⊤ . Given the skill levels a and o of an active player (i.e. the player to move) and the opponent player, we look up the embedding matrix E by rows to map the skill levels to active and opponent skill embeddings: e a = E [ a ] , e o = E [ o ] .

Note that previous work [1, 26] uses completely independent models for human-AI alignment at different skill levels-e.g. decisions by 1100-rated chess players are encoded in one model and decisions by 1500-rated players are encoded in a separate model. Further, these models ignore opponent skill level, meaning that predictions cannot vary as a function of opponent strength. However, the active player's decisions may be significantly affected by the opponent's skill level in certain types of situations, or even in general. Players may adjust their strategy based on their perception of the opponent's skill, e.g. a higher-skill opponent might prompt more (or less) cautious play, while against a lower-skill opponent a player may pursue more aggressive tactics. Thus, the interaction between the skill levels of both players is an important component of matching human moves. Unlike existing models that ignore opponent skill level (and actually only consider games in which both players are at the same skill level), we explicitly model not only opponent skill but also the complex interplay between the two players' skill levels, and how it affects human decision-making.

3.2 Position Encoder

Position representation. We use a well-established method [10, 1] to represent each chess position as a multi-channel tensor P input ∈ R C board × 8 × 8 , which includes channels for each type of chess piece, which color is to move, and states of the position that are not derivable from the position alone (castling rights and en passant), where C board denotes the number of channels. One important departure from previous work is that we only use the current chess position, and not the last few chess positions that occurred in the game (models have typically incorporated the six most recent positions in the game). Many games with perfect information, including chess, can be modeled as alternating Markov games [27, 7], where future states are independent of past states given the current game state. Therefore, the current chess position theoretically encapsulates all the information necessary to make future decisions. Although human decision-making in chess may sometimes subtly depend on the historical lead-up to the current position, these effects are anecdotally small. In exchange, we gain two large practical benefits. First, modeling AI-human move matching in a Markovian way vastly improves training efficiency by reducing the computational load via significantly smaller data usage for each decision. Second, it also enhances flexibility , enabling our resulting model to make predictions even without historical data, which is particularly advantageous in situations where only the current position is available, like chess training puzzles or any position that didn't necessarily occur in a full game.

Position encoding. To process the position representation P input, we encode P input with the wellestablished ResNet-based [28] backbone architecture for chess position modeling with K Conv sequentially connected blocks [1]: P encoded = Backbone × K Conv ( P input ) ∈ R C patch × 8 × 8 , where P encoded denotes the encoded position representation of C patch channels. More details about position representation and the backbone architecture can be found in Appendix Section B.

3.3 Bridging Skill Levels and Positions

A central challenge we face is learning how players at different skill levels interact with chess positions differently. How does an expert player evaluate and process a chess position to come up with a move, and how does this differ from a novice? The relationship between positions and skill levels is complicated by the non-linearity in how players of various skill levels interpret and react to chess positions. This complexity presents a significant challenge in human move prediction using a unified model for diverse skill levels. To bridge skill levels and positions-decision-makers and decisions-we propose skill-aware attention with channel-wise patching .

Channel-wise patching. In contrast to the area-wise patching approach in Vision Transformers (ViTs) [29], we employ channel-wise patching. Each channel is flattened and linearly transformed, regarding the number of channels in P encoded , i.e., C patch , as the sequence length: P patched = Patching ( P encoded ) ∈ R C patch × 64 , P = P patched W + b ∈ R C patch × d att , where W ∈ R 64 × d att and b ∈ R d att denote the parameters of the linear projection from the patching dimension to the hidden dimension of the skill-aware attention blocks d att. This is particularly suitable for patching encoded chess positions as inputs to Transformer-like architectures, where channels are essentially feature maps that represent different learned latent concepts. These concepts in feature maps are then interactively selected and aggregated considering skill levels via skill-aware attention.

Skill-aware Attention. Given position representations P patched and skill level representations e a and e o , our proposed skill-aware multi-head self-attention is computed as follows. For each head k , we learn weight matrices W Q k ∈ R d att × d h , W K k ∈ R d att × d h , and W V k ∈ R d att × d h , where d h denote the dimension of each head. The queries Q k , keys K k , and values V k for each head are computed as: Q k = P patched W Q k , K k = P patched W K k , V k = P patched W V k . In order to fuse player skill levels and chess positions progressively and interactively, we inject skill level embeddings into queries within the multi-head self-attention: Q ∗ k = Q k +( e a ⊕ e o ) W ∗ , where W ∗ ∈ R 2 d s × d h denotes the weight matrix for feature transformation to the query space, and ⊕ is the concatenation operator. We choose to incorporate skill levels in queries because queries directly influence how attention is distributed across patched channels. Using skill-aware queries Q ∗ k , the attention mechanism can adjust its focus to reflect the strategic considerations and positional understanding of players at different skill levels. This adjustment allows Maia-2 to adaptively prioritize features of the positions that are more relevant to the skill levels involved, enhancing the model's contextual sensitivity. The skill-aware scaled dot-product attention for each head is thus defined as: h k = softmax ( Q ∗ k K T k √ d k ) V k . The outputs of all heads h 1 , h 2 , . . . , h h are concatenated and then linearly transformed: P att = σ (( h 1 ⊕ h 2 ⊕ . . . ⊕ h h ) W O ) , where W O ∈ R hd h × d att denote the weight matrix for multi-head attention and σ ( · ) denotes the activation function. We apply the vanilla ViT's feed-forward network and add & norm components upon P att to obtain the output of each skill-aware attention block P out . In Maia-2, we employ a sequence of skill-aware attention blocks to progressively fuse skill levels and positions. Specifically, the output P out for the previous block is fed into the next block as the input. We denote the final output after K Att blocks as P . This procedure enables the model to refine its understanding and interpretation of the positions with each successive block.

3.4 Model Training

Infusing auxiliary information. To enhance the model's understanding of the game state, we inject auxiliary information as labels, including legal moves represented by multi-hot vectors and human move information : one-hot vectors of which piece is moved, which piece is captured (if any), the move's originating square, the move's destination square, and whether or not the move will deliver a check. These segments are used as labels for classification, serving a dual purpose: 1) It offers a more granular understanding of human moves by providing detailed context beyond just the move indices produced by the policy head labels, enriching the model's insight of player decisions; and 2) It ensures the model also learns about objective (i.e. chess-specific as opposed to behavioral) knowledge in chess, which is essential for developing a comprehensive understanding of both human moves and the fundamental mechanics of the game.

Data balancing and filtering. Chess games between players of significantly different skill levels are relatively rare but help us understand how players of lower skill levels approach games against far stronger opponents and vice versa. While previous work has ignored these games completely, they play a central role in our approach. Since games between players of similar skill levels vastly outnumber more uneven matchups, we use a data balancing strategy to effectively train our unified model for aligning players across all skill levels, in which games between players of different skill levels are over-sampled. Online chess platforms feature a variety of game types, including blitz, rapid, and classical, each representing games played at different time controls (amount of time given to each player for the whole game). We focus on Rapid games, which are medium-length games that lie between the fast-paced decisions of 'Blitz' games and the slower, more strategic considerations of 'Classical' games. In addition, we follow the procedures in [1] to filter valid positions within each game. More details about data balancing and filtering are available in Appendix B.

Training objectives. With the fused skill level and position representation P as input, we construct the policy head on top to predict human moves, which is optimized using cross-entropy loss with one-hot labels representing the recorded human move. We also build the auxiliary information head to infuse additional knowledge into Maia-2 as introduced in Section 3.4. This head is trained using bit-wise binary cross-entropy loss with multi-hot labels. Finally, following previous work [1, 25] we include a value head to predict the game outcome as a regression task, where the labels 1, 0, -1 denote winning, drawing, and losing, respectively. The training objectives of these heads are balanced to contribute equally to Maia-2 model optimization. Hyperparameter settings used for Maia-2 training can be found in Appendix Table 5.

4 Results

We empirically evaluate Maia-2 along two key dimensions: move prediction accuracy , how well it can predict human moves at varying skill levels, and move prediction coherence , how aligned its predictions are across skill levels. We train Maia-2 on Lichess games played between Jan 2013 and Nov 2023, with the exception of December 2019, since that is the month used for testing in the original Maia paper (and we also test on this month for consistency) [1]. After game filtering and balancing, we end up with a training set of 169M games (9.1B positions). We also train Maia-2subset with identical model architecture and training configurations as Maia-2, except it only has access to the same training data that Maia had for fair comparisons. Dataset statistics are reported in Appendix Tables 7, 9, and 10. We compare Maia-2 with Stockfish [6], the strongest chess engine, Leela, an open-source counterpart to AlphaZero [10]. and Maia [1], the state-of-the-art model for human-like chess play. Maia is actually a set of 9 separate models, each trained on a different set of players at different skill levels from 1100 to 1900. We use the benchmarking Maia Testset [1] for performance comparisons where both players have identical skill levels. We report the results on Maia Testset by grouping players into three categories: Skilled (Rapid rating up to 1600, which slightly exceeds the initial rating of 1500), Advanced (Rapid rating between 1600 and 2000), and Master (Rapid rating over 2000). In addition, we aim to evaluate move prediction across diverse skill combinations with the Cross-skill Testset constructed from Dec 2023 games. Finally, we construct Grounded Testset with 450,000 positions that has recorded Stockfish evaluations, which can serve as grounded facts to measure move quality. Statistics of datasets are summarized in Appendix Table 8.

1200

1600

2000

Active Player's Skill Level

1200

1600

2000

Opponent Player's Skill Level

48.0

47.5

47.3

46.1

49.2

48.4

48.8

48.1

47.5

49.5

49.3

49.0

49.1

48.4

47.8

50.1

50.0

49.9

49.8

49.2

48.7

48.6

50.4

50.8

50.6

50.4

50.1

50.1

49.6

51.0

50.9

51.5

50.9

51.0

50.7

50.8

50.2

50.5

51.5

51.6

51.5

51.5

51.4

50.5

51.9

51.6

52.0

50.8

51.8

51.5

51.9

52.2

52.2

50.6

51.6

51.8

52.0

52.8

Maia 1100

1200

1600

2000

Active Player's Skill Level

51.1

52.0

52.5

52.9

51.1

51.5

52.8

52.9

53.3

49.4

51.2

51.6

52.9

53.0

54.0

49.0

49.9

51.4

52.0

52.7

53.5

54.4

47.8

49.5

50.9

51.6

52.2

53.3

54.2

47.5

48.3

50.4

50.5

51.8

52.9

53.9

46.2

47.3

49.6

50.3

50.7

52.0

53.0

46.3

48.7

49.2

50.3

49.8

52.3

47.7

48.4

49.4

50.2

49.0

47.3

48.3

48.8

50.2

Maia 1900

1200

1600

2000

Active Player's Skill Level

52.4

53.1

53.8

53.6

52.7

53.3

54.2

54.2

54.3

51.5

53.0

53.2

54.3

54.5

55.0

51.6

52.2

53.4

53.8

54.4

54.9

55.7

51.1

52.1

53.1

53.7

54.2

55.2

55.6

51.1

51.5

53.4

53.1

54.0

54.8

55.6

50.0

50.6

52.8

53.4

53.3

54.5

55.1

50.2

52.2

52.6

53.4

52.3

54.8

51.4

52.0

52.9

53.3

52.2

51.2

52.1

52.4

53.9

Maia-2

47

48

49

50

51

52

53

54

55

Move Prediction Accuracy

4.1 Move Prediction Accuracy

Maia-2. In Table 1, we show the top-1 move prediction accuracy of all models across all groups of players on the Maia-1 Testset . Maia-2 demonstrates strong and consistent performance across all skill levels, surpassing all baselines. Specifically, despite Maia-1 models being specifically trained to mimic chess moves by players at specific skill levels, Maia-2 emerges as a unified one-for-all model that is consistently effective across the entire spectrum of chess skills. The largest improvement is on Advanced players, where Maia-2 gains 1.5 percentage points over the nearest competitor (Maia 1500). When averaging across skill levels, Maia-2 outperforms all other models by almost 2 full percentage points in overall accuracy. Note that the ceiling accuracy of human move prediction is far below 100% given the randomness and diversity of human decisions-even the same player won't always make the same decision when faced with the same position. Our 2 percentage point gain is substantial considering that the difference between Maia-1 and Leela, the previous state-of-the-art model for this task and a traditional chess engine not trained for this task at all, is only 6 percentage points. Furthermore, Maia-1 is essentially a mixture of 9 experts targeting the specific players' skill level, where each expert has 10.3M parameters. Regarding the routing function to select the best-performing expert as a nonparameterized function, Maia-1 has 92M parameters in total. Maia-2, on the other hand, is a one-for-all model with 23.3M parameters under our default settings. Therefore, Maia-2 achieves better human move prediction accuracy with even much fewer trainable parameters.

Baseline models. Both Maia-2 and Maia-1 significantly outperform Stockfish and Leela, typically by 5-15 percentage points. Note that Stockfish and Leela aim to play optimal chess (as most humans do too), and only 'predict' human moves when their approximations to optimality happen to overlap with those of human players. However, we compare to these traditional chess engines because besides Maia-1, there are still the default method of creating 'human-like' AI agents. The accuracy gap between Maia-1 architectures and traditional chess engines demonstrates the necessity of developing specialized models to mimic human chess moves.

Maia-2subset. Maia-2 differs from Maia-1 in two main ways: it has a different architecture and it has access to more training data. To control for the difference in training data and isolate the effects of our architecture, we create Maia-2subset which has access to the exact same training data that Maia-1 was developed with. Comparing the two, we see that Maia-2subset matches or outperforms all baselines and alternate models. Recall that Maia-2 and Maia-2subset don't have the recent history passed as input to them, yet still achieve state-of-the-art results. It is important to note that each Maia-1 model is specifically trained for its respective skill level, relying solely on games where the active and opponent skill levels match for its training data. On the contrary, the unified modeling approach with skill-aware attention of Maia-2subset allows it to utilize a broader spectrum of games, featuring a variety of skill-level pairings, for training purposes. Consequently, while both Maia-1 and Maia-2subset draw from the same source dataset, Maia-2subset can leverage a significantly larger portion of this data for its training, improving its learning and predictive capabilities. The improvement from Maia-2subset to Maia-2 underscores the importance of extensive training with vast datasets. A broader range of games provides Maia-2 with access to more comprehensive and nuanced patterns in human chess moves. Using Maia-2subset as a comparison, we can determine the relative contributions of model architecture and training data to Maia-2's 1.9 percentage point gap over its nearest rival (Maia 1500). This calculation suggests that 73% of the increase in performance is due to the architecture improvements and 27% is due to increased training data.

Table 3: Ablation study results, where Maia2subset is compared with versions without skillaware attention ('w/o Att') and without infusing auxiliary information ('w/o Aux').

Move prediction perplexity. While top-1 accuracy gains are important, they may overshadow larger improvements in prediction quality. To account for this, we also measure the perplexity of move predictions , which reflects the model's confidence in its predictions. A lower perplexity indicates the model is more confident and accurate in human move prediction, as it corresponds to a higher likelihood of the correct human move. As shown in Table 2, Maia-2 consistently yields substantially lower perplexity in all groups of skill levels compared to Maia-1. In particular, Maia-2 significantly outperforms Maia-1 in Advanced and Master moves with relatively large margins, demonstrating the effectiveness of our unified modeling approach across diverse skill levels.

Adaptive move predictions. We now evaluate Maia-2's ability to predict compare across diverse skill combinations using the Cross-skill Testset . As shown in Figure 2, Maia-2 consistently outperforms both Maia 1100 and Maia 1900 in almost all combinations of active player and opponent player skill levels. In particular, although Maia 1100 and Maia 1900 demonstrate competent performance within their respective domains of expertise, their predictive accuracy decreases substantially outside of these targeted skill levels. This is because Maia-1 models are static and cannot respond to varied skill levels and adjust their predictions accordingly. In contrast, our proposed unified modeling approach with skill-aware attention enables Maia-2 to adapt its predictions to account for the skill levels of both the active player and the opponent player, so that varying skill level configurations correspondingly can result in better aligned human move predictions. More results on comparisons with other Maia-1 versions can be found in Figure 6 in the Appendix.

Move quality. One of our key motivations for creating a unified model of human chess behavior is to guide the development of future algorithmic learning tools. As such, understanding the mistakes that people make is of fundamental interest. Can Maia-2 predict mistakes better than Maia-1? Figure 10 in the Appendix shows the move prediction accuracy on the Grounded Testset as a function of move quality, measured by win-rate loss, which is calculated following the same procedures as prior studies [1, 25]. All models generally decrease in their ability to predict worse moves, since humans are generally trying to avoid mistakes, and high-quality moves are more certain whereas lower-quality moves can be more random and thus hard to predict. Nevertheless, Maia-2 outperforms all versions of Maia-1 across most of the move quality range, demonstrating the effectiveness of our unified modeling approach for human move prediction.

We are also interested in how certain the models are about their predictions of various move qualities. Figure 3.(A)(top) shows the probabilities that Maia-2 (x-axis) and Maia-1 (y-axis) attribute to the moves people actually played in Grounded Testset . The concentration of points in the lower right quadrant (closer to P(Maia-2) = 1 and P(Maia 1900) = 0) suggests that Maia-2 assigns a higher probability to the true move than Maia-1 does, indicating superior predictive performance. Conversely, the less dense upper left quadrant indicates fewer instances where Maia 1900 outperforms Maia2. Remarkably, while this consistently occurs across all move qualities, the distinction is more pronounced for Blunders and Errors compared to Optimal moves. Additionally, the bottom row of Figure 3.(A) shows the log odds ratio between P ( x, y ) and P ( y, x ) in the top row. The abundance of blue points below the diagonal indicates that Maia-2 is almost always more confident in the correct move than Maia-1 is, indicating an across-the-board improvement in move prediction. Maia-2 offers superior and more confident prediction across diverse move qualities. We also conduct an ablation study as shown in Table 3, we point the readers to Appendix A for more information.

4.2 Move Prediction Coherence

Maia-2's accuracy across the spectrum of human skill is certainly desirable, but perhaps an even more important dimension is prediction coherence as skill varies. A central drawback of Maia-1 is that it

(A)

(B)

models players at different skill levels independently from each other, which results in particularly volatile predictions: the same position might elicit very different predicted behavior from models of adjacent skill levels. This is problematic because we know from personal experience that this type of volatility is rare: players don't change that much as they improve. This limits Maia's ability to perform well in downstream tasks such as serving as a teaching aid, as its understanding of one skill level bears little resemblance to its understanding of the next. In reality, players move from one skill level to another by making small, consistent adjustments. Does Maia-2 reflect this behavioral coherence?

Prediction smoothness. We measure the coherence of Maia-2's predictions by testing for smoothness features in its entire set of predictions. Call a model's treatment of a position monotonic if the predicted probability of the correct move increases with skill monotonically. In the Grounded Testset of 100K positions, we find that Maia-1 only treats 1% of them monotonically. In stark contrast, however, Maia-2 treats 27% of them monotonically, clearly demonstrating that Maia-2 is much more coherent. Similarly, call a model's treatment of a position transitional if it predicts a suboptimal move for some prefix of skills and then transitions to an optimal move for all subsequent skill levels. Again, Maia-2 treats substantially more positions transitionally-around 22% of them compared with 17% for Maia.

It's important to note that Maia-2 is deliberately designed to encourage coherence across skill levels without rigidly enforcing it. Our objective is not to impose coherence as a hard constraint, which might obscure legitimate differences in player behavior between skill levels, but to create a model architecture that naturally encourages coherence where the data supports it.

Move prediction agreement. As a first test, we measure move prediction coherence as we vary active player skill and opponent player skill in Maia-2. The results shown in Figure 3.(B) reveal several trends. First, increasingly varying either the active or opponent rating results in lower agreement, suggesting that Maia-2 smoothly varies its predictions with skill. Second, comparing the two heatmaps reveals that Maia-2 has clearly learned that varying one's own skill has much larger effects than varying the opponent's-changing one's own skill against a fixed opponent can change the decision up to 22% of the time, but changing the opponent's skill while fixing our own skill will only change the decision up to 6% of the time. This is intuitive, as players must change their decisions in order to play at a higher level, while in theory one's opponent shouldn't affect one's decision. Of course, humans are not optimal agents and sometimes take their opponent's skill level into account when deciding on a move-willfully or not-which is reflected in our results.

Chess concept understanding. Human chess players of varying strengths differ in their ability to recognize important features and patterns on the board, e.g., stronger players are adept at discerning

1.0

middle_game_evaluation

1.0

middle_game_piece_value

0.68

has_bishop_pair_active

0.65

can_capture_queen_active

PreSkill-Aware Attention

-

PreSkill-Aware Attention

0.9

0.9

0.67

PostSkill-Aware Attention

PostSkill-Aware Attention

0.60

0.8

0.8

0.66

0.7

ccura

0.7

ACC

0.65

Accurac

0.55

0.6

AC

0.6

0.64

0.50

0.5

PreSkill-AwareAttention

0.5

一

PreSkill-Aware Attention

0.63

0.4

PostSkill-Aware Attention

0.4

Post Skill-Aware Attention

0.62

0.45

1200

1600

2000

1200

1600

2000

1200

1600

2000

1200

1600

2000

Player Skill Level

Player Skill Level

Player Skill Level

Player Skill Level

(a)

SkillDependentConcepts

(b)

(c)

SkillIndependentConcepts

(d)

subtle nuances. We now turn our focus to a critical question: does Maia-2 vary in its ability to capture human chess concepts when given different skill levels? Following the chess concepts probing strategy for AlphaZero [22], we show how Maia-2's grasp of various concepts varies with skill. The left two plots in Figure 4 show concepts for which Maia-2 clearly distinguishes between skill levels, with higher-skill players paying more attention to them than lower-skill players. These are general board evaluations as given by Stockfish [6], or aggregate piece values. Note that pre-skill-aware attention is always flat because by construction it cannot vary with skill, since skill-aware attention has not been applied yet. The two plots on the right depict concepts that live closer to fundamental chess rules, and as such are less dependent on player skill. For skill-dependent concepts, the figures reveal an increasing trend in mastery level after skill-aware attention, aligning with the increase in dedicated skill levels. Meanwhile, the model's mastery level decreases after passing through the skill-aware attention modules, potentially adjusting for the imperfections of human players. Conversely, the skill-aware attention blocks are not responsive to skill-independent concepts.

5 Discussion

Human Study. In addition to human move matching, we also consider engagement, another dimension of human study. In particular, we implement a randomized experiment on Lichess: human players challenge our bots, and we randomize whether players play against Maia-1 or Maia-2. Our result is that our higher move-matching and our vastly improved coherence, across all skill levels, come at no cost to human subject engagement, and in fact slightly increase engagement: players rematch Maia-2 almost 1 percentage point more than Maia-1 (41.2% vs. 40.3%). Although engagement is not our main objective, this is further promising evidence that we have achieved a more human-aligned model that coherently captures human style across different skill levels.

Ethical Considerations. We believe Maia-2 poses limited risk while offering large potential benefits. Our data is highly aggregated, with almost 1 billion games being used for training, and chess as a domain is generally low-risk. Meanwhile, helping people improve in chess could lead to increased cognitive skills, confidence boosts, and help with general life satisfaction. Our vision is for Maia-2 to power AI partners and training aids; it cannot currently replace skilled human tutors and coaches.

Limitation. Our work has limitations. First, we are excited by the applications that Maia-2 will enable, such as more relatable AI partners and AI-powered learning aids, the development of which is out of scope for the current work. Maia-2 does not yet incorporate search, although previous work has demonstrated that with proper regularization it can help improve move prediction performance [26]. Relatedly, we group the strongest players in a single bucket, although modeling the very best players in the world remains difficult due to the complexity and depth of their moves.

References

Reid McIlroy-Young, Siddhartha Sen, Jon Kleinberg, and Ashton Anderson. Aligning superhuman ai with human behavior: Chess as a model system. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 1677-1687, 2020.

Feng-hsiung Hsu. Ibm's deep blue chess grandmaster chips. IEEE micro , 19(2):70-81, 1999.

William Clark, Jan Golinski, and Simon Schaffer. The sciences in enlightened Europe . University of Chicago Press, 1999.

Frederic Friedel. Reconstructing turing's "paper machine".

Claude E. Shannon. A mathematical theory of communication. Bell Syst. Tech. J. , 27:623-656, 1948.

Tord Romstad, Marco Costalba, Joona Kiiski, and et al. Stockfish. stockfishchess.org, 2023. Accessed: 2024-01-05.

David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature , 529(7587):484-489, 2016.

Xidong Feng, Yicheng Luo, Ziyan Wang, Hongrui Tang, Mengyue Yang, Kun Shao, David Mguni, Yali Du, and Jun Wang. Chessgpt: Bridging policy learning and language modeling. arXiv preprint arXiv:2306.09200 , 2023.

Feng-Hsiung Hsu. Behind Deep Blue: Building the computer that defeated the world chess champion . Princeton University Press, 2002.

David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv , 2017.

Tom Zahavy, Vivek Veeriah, Shaobo Hou, Kevin Waugh, Matthew Lai, Edouard Leurent, Nenad Tomasev, Lisa Schut, Demis Hassabis, and Satinder Singh. Diversifying ai: Towards creative chess with alphazero. arXiv preprint arXiv:2308.09175 , 2023.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023.

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems , 35:22199-22213, 2022.

Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. Autogen: Enabling next-gen llm applications via multi-agent conversation framework. arXiv preprint arXiv:2308.08155 , 2023.

Max Hager. LLMChess.

Nicholas Carlini. carlini/chess-llm.

Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615 , 2022.

Andrew Lee, David Wu, Emily Dinan, and Mike Lewis. Improving chess commentaries by combining language models with symbolic reasoning engines. arXiv preprint arXiv:2212.08195 , 2022.

David Noever, Matt Ciolino, and Josh Kalin. The chess transformer: Mastering play using generative language models. arXiv preprint arXiv:2008.04057 , 2020.

Thomas McGrath, Andrei Kapishnikov, Nenad Tomašev, Adam Pearce, Martin Wattenberg, Demis Hassabis, Been Kim, Ulrich Paquet, and Vladimir Kramnik. Acquisition of chess knowledge in alphazero. Proceedings of the National Academy of Sciences , 119(47):e2206625119, 2022.

Lisa Schut, Nenad Tomasev, Tom McGrath, Demis Hassabis, Ulrich Paquet, and Been Kim. Bridging the human-ai knowledge gap: Concept discovery and transfer in alphazero. arXiv preprint arXiv:2310.16410 , 2023.

Reid McIlroy-Young, Yu Wang, Siddhartha Sen, Jon Kleinberg, and Ashton Anderson. Detecting individual decision-making style: Exploring behavioral stylometry in chess. Advances in Neural Information Processing Systems , 34:24482-24497, 2021.

Reid McIlroy-Young, Russell Wang, Siddhartha Sen, Jon Kleinberg, and Ashton Anderson. Learning models of individual behavior in chess. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 1253-1263, 2022.

Athul Paul Jacob, David J Wu, Gabriele Farina, Adam Lerer, Hengyuan Hu, Anton Bakhtin, Jacob Andreas, and Noam Brown. Modeling strong and human-like gameplay with kl-regularized search. In International Conference on Machine Learning , pages 9695-9728. PMLR, 2022.

Michael L Littman. Markov games as a framework for multi-agent reinforcement learning. In Machine learning proceedings 1994 , pages 157-163. Elsevier, 1994.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Mate-in-1 Puzzle

of rating 1500

Maia-2 with Increasing Player's Skill Levels

1100

❌

1300

❌

1500

✅

1700

✅

Maia 1100

✅

Maia 1300

❌

Maia 1500

❌

Maia 1700

❌

Maia 1900

✅

1900

✅

Stronger Players

0.24

0.13

0.22

0.29

0.45

1200

1600

2000

Opponent Player's Skill Level

48.0

47.5

47.3

46.1

49.2

48.4

48.8

48.1

47.5

49.5

49.3

49.0

49.1

48.4

47.8

50.1

50.0

49.9

49.8

49.2

48.7

48.6

50.4

50.8

50.6

50.4

50.1

50.1

49.6

51.0

50.9

51.5

50.9

51.0

50.7

50.8

50.2

50.5

51.5

51.6

51.5

51.5

51.4

50.5

51.9

51.6

52.0

50.8

51.8

51.5

51.9

52.2

52.2

50.6

51.6

51.8

52.0

52.8

Maia 1100

51.3

51.4

51.5

50.4

52.1

51.8

52.6

52.1

51.8

51.4

52.1

52.1

52.9

52.5

51.9

51.1

51.7

52.5

52.8

52.9

52.6

52.5

50.3

51.5

52.4

53.0

53.0

53.6

53.4

50.3

51.1

52.5

52.4

53.2

53.5

53.9

48.8

49.9

51.8

52.6

52.9

53.4

53.9

49.3

51.5

51.9

52.7

51.7

53.7

50.3

51.3

52.2

52.6

51.1

50.2

51.1

51.5

52.7

Maia 1500

51.1

52.0

52.5

52.9

51.1

51.5

52.8

52.9

53.3

49.4

51.2

51.6

52.9

53.0

54.0

49.0

49.9

51.4

52.0

52.7

53.5

54.4

47.8

49.5

50.9

51.6

52.2

53.3

54.2

47.5

48.3

50.4

50.5

51.8

52.9

53.9

46.2

47.3

49.6

50.3

50.7

52.0

53.0

46.3

48.7

49.2

50.3

49.8

52.3

47.7

48.4

49.4

50.2

49.0

47.3

48.3

48.8

50.2

Maia 1900

47

48

49

50

51

52

53

54

55

Move Prediction Accuracy

1200

1600

2000

Active Player's Skill Level

1200

1600

2000

Opponent Player's Skill Level

50.1

50.3

50.4

49.8

50.8

50.6

51.4

51.0

50.8

50.1

50.9

50.9

51.6

51.3

51.2

50.1

50.5

51.3

51.5

51.6

51.6

51.9

49.5

50.6

51.3

51.7

51.8

52.4

52.4

49.6

50.1

51.5

51.3

52.0

52.4

52.9

48.4

49.2

51.0

51.5

51.7

52.3

52.8

48.7

50.7

50.9

51.7

50.8

52.6

49.8

50.5

51.3

51.7

50.2

49.7

50.4

50.8

51.9

1200

1600

2000

Active Player's Skill Level

51.3

52.0

52.5

52.9

52.1

51.8

52.8

52.9

53.3

51.4

52.1

52.1

52.9

53.0

54.0

51.1

51.7

52.5

52.8

52.9

53.5

54.4

50.4

51.5

52.4

53.0

53.0

53.6

54.2

51.0

51.1

52.5

52.4

53.2

53.5

53.9

50.2

50.5

51.8

52.6

52.9

53.4

53.9

50.5

51.9

51.9

52.7

51.7

53.7

51.5

51.9

52.2

52.6

51.1

51.6

51.8

52.0

52.8

1200

1600

2000

Active Player's Skill Level

52.4

53.1

53.8

53.6

52.7

53.3

54.2

54.2

54.3

51.5

53.0

53.2

54.3

54.5

55.0

51.6

52.2

53.4

53.8

54.4

54.9

55.7

51.1

52.1

53.1

53.7

54.2

55.2

55.6

51.1

51.5

53.4

53.1

54.0

54.8

55.6

50.0

50.6

52.8

53.4

53.3

54.5

55.1

50.2

52.2

52.6

53.4

52.3

54.8

51.4

52.0

52.9

53.3

52.2

51.2

52.1

52.4

53.9

Maia-2

47

48

49

50

51

52

53

54

55

Move Prediction Accuracy

A More Experimental Results

Case study: Smoothness. We evaluate the smoothness of Maia-1 and Maia-2 by a case study in puzzle solving: a Mate-in-1 puzzle of 1500 skill level is presented to both Maia-1 and Maia-2, where smoothness can be evaluated by checking whether the predictions are monotonic and transitional as skill level increases. Call a model's treatment of a position as monotonic if the predicted probability of the correct move increases with skill monotonically, we can observe from Figure 5 that the Maia-2 predicted probabilities of the best move (in green arrows) increase monotonically from 0.22 to 0.45 as the skill levels rise from 1100 to 1900, while Maia-1 predictions are rather turbulent. Similarly, we call a model's treatment of a position transitional if it predicts a suboptimal move for some prefix of skills and then transitions to an optimal move for all subsequent skill levels. As shown in Figure 5, Maia-2 can mimic weaker players to whom the puzzle is hard to solve, while stronger Maia-2 with skill level configured above or equal to 1500 can successfully solve the puzzle. However, Maia 1100 surprisingly solved the puzzle, while the stronger Maia-1 models, e.g., Maia 1700 failed to make the optimal move. Therefore, in the considered case, as opposed to Maia-1, Maia-2 yields smooth predictions provided that its treatment of this position is monotonic and transitional .

1200

1600

2000

Player's Skill Level

0.08

0.10

0.12

0.14

0.16

Average Blunder Rate

50

55

60

65

70

75

Average Centipawn Loss

Average Blunder Rate

Average Centipawn Loss

Quality of predicted moves. Given the same chess position but increasing skill levels, a coherent, skill-aware human move prediction model should make progressively higher-quality moves. We measure the average centipawn loss (a standard move quality metric, the lower the better) and the average blunder rate (fraction of times an egregious mistake is predicted) in positions randomly sampled from December 2023 games. The centipawn loss is determined by comparing the moves predicted by Maia-2 against the top move as evaluated by Stockfish at depth 20 (human grandmasterlevel play), while blunders are classified as moves resulting in a win-rate loss of 10% or more. As shown in Figure 7, both the centipawn loss and the blunder rate exhibit a smooth, monotonic decrease as player skill rises, demonstrating Maia-2's capability of adjusting its predictions coherently to align with the increasingly skilled players.

Value head. As a proxy of model evaluation given a board position, we train the model value head which is potentially significant for a wide variety of downstream tasks. The value head is trained as a regression task from -1 to 1 indicating from losing to winning positions, and finally normalized to a continuous value between 0 and 1 similar to AlphaZero. The correct label for value head is the actual game results. To check the evaluation quality of our model value head, we calibrate the value head results with actual game outcomes, and generate a quantile-quantile plot in Figure 8 where the win probability is discretized uniformly into 100 bins from 0 to 1.

0.0

0.2

0.4

0.6

0.8

1.0

Predicted Win Probability

0.0

0.2

0.4

0.6

0.8

1.0

Empirical Win Probability

Ablation study. To understand which components of our architecture are most responsible for the performance gains, we conduct an ablation study on the Maia Testset . We train a version of Maia-2subset using a naive method of incorporating encoded skill levels without the proposed skill-aware attention module ('w/o Att'). In this model, the skill level encodings e a and e o are directly concatenated with flattened P encoded, without bridging them with the skill-aware attention (skipping Section 3.3), and directly connected to prediction heads. As shown in Table 3, Maia-2subset consistently performs better with skill-aware attention, demonstrating the necessity of modeling the complexity and non-linearity in player's skill development in sophisticated ways and the effectiveness of our proposed unified modeling approach with skill-aware attention to model such nuances. We also train a version of Maia-2subset without the auxiliary information head ('w/o Aux'). These results show that infusing auxiliary information as labels during model training also results in a significant performance improvement, although not as dramatically as skill-aware attention.

end_game_evaluation

end_game_piece_value

has_bishop_pair_opponent

capture_possible_on_d3_active

1.0

1.0

0.68

0.65

一

0.9

0.9

0.67

一

Post Skill-Aware Attention

Post Skill-Aware Attention

0.60

0.8

0.8

racy

0.66

0.7

0.7

ACCU

0.65

0.55

0.6

0.6

0.64

0.5

0.5

0.63

0.50

0.4

Post Skill-Aware Attention

0.4

Post Skill-Aware Attention

0.62

0.45

1200

1600

2000

1200

1600

2000

1200

1600

2000

1200

1600

2000

Player Skill Level

Player Skill Level

Player Skill Level

Player SkilI Level

(a)

SkillDependentConcepts

(b)

(c)

Skill Independent Concepts

(d)

0.00

0.05

0.10

0.15

0.20

0.25

10

20

30

40

50

60

70

Move Prediction Accuracy

Maia 1900

Maia 1500

Maia 1100

Maia-2

Win-rate Loss

Accuracy as a function of move quality. Figure 10 shows the move prediction accuracy on the Grounded Testset as a function of move quality, measured by win-rate loss, which is calculated following the same procedures as prior studies [1, 25]. All models generally decrease in their ability to predict worse moves, since humans are generally trying to avoid mistakes, and highquality moves are more certain whereas lower-quality moves can be more random and thus hard to predict. Nevertheless, Maia-2 outperforms all versions of Maia across most of the move quality range, demonstrating the effectiveness of our unified modeling approach for human move prediction. Maia-2's overall gains are not constrained to any specific move quality type, but are spread across the entire range.

B Reproductibility

Implementation details. To maintain a consistent perspective from both sides of players, we implemented board flipping to train and test Maia-2; that is, positions with black to move were mirrored such that all analyses could be conducted from the white side's viewpoint. We further refined our dataset through game and position filtering, selecting only rapid games from Lichess with available clock information and disregarding the initial 10 plys of each game as well as positions where either player had less than thirty seconds remaining. The filtration is significant for eliminating the noise introduced by rushed decisions under time constraints, which could skew the true representation of a player's skill. The choice of exclusive rapid games is also informed by the distinct rating systems across different game types, which necessitates the separation of data to maintain rating consistency. We report all hyperparameters involved in training Maia-2 in Table 5. It took approximately 13 days to train Maia-2 with 2 × A100 (80G) GPUs under our default settings.

Data Balancing. Chess games between players of significantly different skill levels are relatively rare but help us understand how players of lower skill levels approach games against far stronger opponents and vice versa. While previous work has ignored these games completely, they play a central role in our approach. Since games between players of similar skill levels vastly outnumber more uneven matchups, we use a data balancing strategy to effectively train our unified model for aligning players across all skill levels, in which games between players of different skill levels are

over-sampled. To be precise, we pre-process the data in chunks of N chunk games each. We then scan each data chunk to find games satisfying various (active player skill, opponent skill) combinations. Each skill combination can include at most N range games. We continue scanning the data chunk until all the skill combinations have N range games or the data chunk is fully consumed. Note that the higher the balancing factor N range N chunk , the less likely it is that rare skill combinations will fully reach N range, which will lead to less balanced data overall. On the other hand, if the balancing factor is too small, fewer games will be selected from each data chunk, which is data-inefficient. We choose a fair compromise between data efficiency and balanced data so that our training data encompasses a broad spectrum of skill levels without biasing excessively towards the more frequently-occurring equal-skill matchups.

Data Filtering. Online chess platforms feature a variety of game types, including blitz, rapid, and classical, each representing games played at different time controls (amount of time given to each player for the whole game). We use data from Lichess, a well-known large open-source chess platform, and its open database. In Lichess, since each game type is given a separate rating, ratings across different game types are not comparable (e.g. a rating of 1800 in 'Rapid' is significantly weaker than a rating of 1800 in 'Blitz' on Lichess). Previous work [1, 25] mixes player ratings across these game types together for training and evaluation. Instead of mixing data across game types, we focus on Rapid games only, which are medium-length games that lie between the fast-paced decisions of 'Blitz' games and the slower, more strategic considerations of 'Classical' games. 'Blitz' and 'Bullet' games, characterized by their quick pace, are composed of many decisions made under time pressure, introducing randomness that may not accurately reflect player intentions and skills. In contrast, Classical games are played less frequently, leading to data scarcity. Rapid chess is an ideal compromise between quality of play and quantity of data. In addition, we follow the procedures in [1] to filter valid positions within each game.

Chess position representation. We follow the well-established prior works [10, 1] to represent chess positions as multi-channel 8 × 8 matrices, including:

Piece Representation: The first 12 channels categorize the board's pieces by type and color, with one channel each for white and black Pawns, Knights, Bishops, Rooks, Queens, and Kings. A cell is marked 1 to denote the presence of a piece in the corresponding location, and 0 otherwise.

Player's Turn: A single channel (the 13th) indicates the current player's turn, filled entirely with 1s for white and 0s for black, providing the model with context on whose move is being evaluated.

Castling Rights: Four channels (14th to 17th) encode the castling rights for both players, with the entire channel set to 1 if the right is available or 0 otherwise.

En Passant Target: The final channel (18th) marks the square available for en passant capture, if any, with 1 and 0s elsewhere.

Chess concepts probing. Given a board position, we vary the skill level injected to Maia-2 to extract the P encoded ∈ R C patch × 8 × 8 as the learned respresentation of the ResNet-based backbone served as the control group which remain constant to varying skill ratings, and the output hidden states P ∈ R C patch × d att after skill-aware attention directly connected to model heads. We randomly pick 500,000 positions from Lichess December 2023 database and calculate Stockfish built-in or custom implemented concepts using the Forsyth-Edwards Notation of the positions.

We carefully choose P encoded and P as internal representations for the reason that P encoded is the comprehensive understanding of chess board for the backbone network right before skill-aware attention, and P is the final hidden states that directly influence model decision and evaluation. Following the work of Alphazero concept probing [22], for chess concepts with continuous values we train Lasso regressors with coefficient for L 1 -regularization selected by 5-fold cross-validation, and for binary concepts we train logistic regressors with downsampling. During evaluation on unbalanced test set, we measure the performance of continuous-valued probes with the coefficient of determination r 2 , and score the binary-valued probes with the Macro F1.

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction clearly outline the main contributions and scope of the paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations of our work in Section 5.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: We do not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We include details of model design and model training in Section 3, experimental setup in Section 4, as well as hyperparameter settings and implementation details in Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We include the link to our code including the data processing pipeline in the abstract.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We include dataset details and hyperparameter settings in Section 4 and Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: We train Maia-2 with a huge amount (9.1B) of chess positions. Therefore, it is hard to evaluate Maia-2 multiple times with different train/test splits.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We report the required computational resources in Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We conduct our work under the guidance of NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss the potential social impacts of our work in Section 1 and Section 5. Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [Yes]

Justification: We discuss the limited risks of our work as a human-like model in Section 5.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We include all assets used in our paper in main texts and references.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We release our code with documentations and comments.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our work does not involve crowdsourcing.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our work does not involve crowdsourcing.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a unified modeling approach called Hermes for aligning human and AI performance in chess. Hermes effectively captures human chess styles across different skill levels, providing a coherent reflection of player improvement. The approach incorporates a skill-aware attention mechanism that dynamically combines player skill levels with encoded chess positions, allowing the model to adapt to varying player skills. Experimental results demonstrate that this unified framework significantly enhances the alignment between AI and human players, facilitating a deeper understanding of human decision-making and the development of AI-guided teaching tools.\n\nSTRENGTHS:\n1. This paper is easy to read and follow, with detailed descriptions and accompanying code.\n2. The method is simple yet effective, demonstrating remarkable results in various settings.\n\nWEAKNESSES:\n1. The method is actually straightforward, as it only employs the multi-head attention mechanism, a general technique applicable in various settings and applications.\n2. The motivation for using attention and the reasons for its effectiveness are unclear.\n3. The concept of skill used in this paper is not well-defined, which may limit its applicability to the chess environment only.\n\nQUESTIONS:\n1. How is skill defined in this paper? Is it manually designed or given by the chess setting? How does it handle this setting, and why can't we implicitly obtain different skill levels?\n2. How could this method be expanded to settings other than chess, such as those used in multi-agent systems[1]?\n3. How can this method be expanded to different chess variants or adapted for a large-scale chess model?\n\nRef:\n[1] Yuan L, Zhang Z, Li L, et al. A survey of progress on cooperative multi-agent reinforcement learning in open environment[J]. arXiv preprint arXiv:2312.01058, 2023.\n\nLIMITATIONS:\nA/N", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-27"}

comments: SUMMARY:
This work explores developing a unified model to predict human moves in Chess. To address the coherence challenges, the authors propose to use skill-aware attention with channel-wise patching to encode skill levels and board positions into a neural network model. Experimental results show their proposed model achieves comparable or better performance in prediction accuracy and coherence compared to SOTA models.

STRENGTHS:
This work is overall very solid. The proposed model is clearly presented, with enough details for reproduction. Evaluation is also convincing and thorough.

WEAKNESSES:
A minor problem is about the skill level encoder. The proposed Hermes model encodes both players’ skills, but Maia model only encodes one player’s, which adds an advantage to Hermes model and makes comparisons in human move prediction potentially unfair.

QUESTIONS:
1. About the volatile predictions of Maia model, is this problem more due to prediction errors or the volatile nature of human playing data? For example, a move that relies on several subsequent moves to be effective might be a bad choice for middle-level player, but a good choice for both low-level (where the opponent doesn’t know how to counter) and high-level players (where the active player can manage the subsequent changes). I’m not an expert in Chess, just wondering whether this is a possible cause of the incoherent problem. 
2. In Figure 10, is Maia-2 actually the proposed Hermes model? This evaluation of win-rate seems indirect. Why not directly pair Hermes-1500 (by setting the active player level to 1500, for example) with Maia-1500 and evaluate the final outcome, to determine which model is better if the goal is to evaluate move quality?

LIMITATIONS:
No significant limitation.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a unified modeling approach named Hermes for aligning human and AI behaviors in chess. It addresses the limitations of previous models by integrating a skill-aware attention mechanism that dynamically adapts to various skill levels of players. The Hermes model aims to enhance AI-guided teaching tools and provide deeper insights into human decision-making in chess. The model is evaluated based on move prediction accuracy and coherence, showing improvements over existing models.\n\nSTRENGTHS:\nThe paper presents an approach to human-AI alignment in chess through the introduction of a skill-aware attention mechanism that dynamically adapts to players’ skill levels. This technique addresses the non-linear nature of human learning and significantly improves the coherence of AI models across different skill levels. The paper is well-structured and clearly written. The evaluation of the Hermes model is thorough, demonstrating notable improvements in move prediction accuracy and coherence over existing models like Maia and traditional chess engines such as Stockfish and AlphaZero.\n\nWEAKNESSES:\nDespite its strengths, the paper has several limitations. While the skill-aware attention mechanism seems \"innovation\", the overall novelty of the paper is somewhat limited as it builds on existing models and techniques without introducing fundamentally new concepts in AI or chess modeling. Additionally, the paper does not adequately address potential biases introduced by relying heavily on data from a specific online platform, which may affect the generalizability of the model. The experiments mainly report move prediction accuracy and move prediction coherence, which are not sufficient to support the model’s practical effectiveness in gameplay scenarios. Most importantly, the paper lacks human-AI experiments that would substantiate the claims of alignment. \n\nThus, I find the method lacks novelty and the experiments are insufficient to support the claims, leading me to recommend the paper for rejection.\n\nQUESTIONS:\nSee weaknesses.\n\nThe font in figures is too thin.\n\nLIMITATIONS:\nThe paper discussed the limitations in appendix.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work tackles the problem of modeling chess agents at varying skill levels. Prior work learns separate models for each skill level, so the authors introduce their method “Hermes” which uses skill-aware attention to adapt the predictions based on the skills i.e. chess ratings of both players in the game. Technically, Hermes uses a categorical embedding for different buckets of player ratings, and uses multi-head attention to project the game state to the player move, some auxiliary game information, and a value head. The active and opponent skill embeddings are projected and added to the query matrix in the attention step, to make the model skill-aware. The authors train and evaluate the model on large datasets of chess gameplay across varying ratings, and conduct in-depth behavior analysis of their models compared to the baselines. They demonstrate that Hermes slightly outperforms the baselines with higher confidence predictions. Further, it enables additional capabilities such as modeling chess players over the entire spectrum of skill levels, and monotonic improvement of predicted moves as the skill increases.\n\nSTRENGTHS:\n1. The paper tackles an important problem of skill-aware modeling of human behavior. The authors instantiate this in the setting of chess games between players of varying skill levels. The intuition of the overall method is well understood, the framework and the parameters are clearly mentioned, and the results are shown over the standard datasets and compared to recent baselines.\n2. The paper is well written. In the paper, the processing steps for the dataset, including filtering and balancing are explained clearly. The authors do not discard rare games between players at very different chess ratings. Is using these rare games helpful for the model? It would be nice to see some ablation showing the effect of this additional data on model performance.\n3. Hermes uses auxiliary information prediction to align the model better. This is very intuitive as forcing the model to predict auxiliary information from the current state should improve its understanding of the game concepts. I am curious to know if this has been explored in the past work. Else, it would be interesting to see if this could improve the performance of baseline agents\n4. The paper presents an in-depth analysis of the performance and predictions of Hermes under different conditions. It shows that Hermes is more confident in its predictions than the baseline that learns individual models at individual skill levels. Further, Hermes performs similarly across all pairs of skills showing that it captures and models the players' ratings over the entire spectrum, while the baseline is accurate only around the particular skill level they are trained on. Finally, they also show that the model implicitly differentiates in its predictions on the moves that are skill-dependent and independent.\n\nWEAKNESSES:\n1. The paper focuses on improving the predictive model performance for chess games, but the improvement over the baseline is very small (2%). Further, the table doesn't include any error ranges and has no statistical testing to show if the performance improvement is statistically significant.\n2. The paper does not cite some recent work on modeling agents with varying skill levels [1]. I would recommend the authors further include and cite works on modeling diverse agents and partners, specifically in multi-agent learning literature.\n3. In Hermes, the players’ skill embeddings are projected and added to the query matrix in the attention step, to make the model skill-aware. However, there are no experiments or intuition to explain the significance of this architectural choice. Over simple conditional networks (or more recent works such as FiLM [2]), it would be nice to see what is the significance of this particular attention mechanism.\n4. Some corrections regarding the manuscript: Figure 3 refers to an Errors (middle) figure which is missing. I believe there is a typing error in Figure 5, where the top row has Hermes labels which should be Mia.\n\n[1] Jacob, Athul Paul et al. “Modeling Boundedly Rational Agents with Latent Inference Budgets.” ArXiv abs/2312.04030 (2023): n. Pag.\n\n[2] Perez, Ethan et al. “FiLM: Visual Reasoning with a General Conditioning Layer.” AAAI Conference on Artificial Intelligence (2017).\n\nQUESTIONS:\n1. How does the granularity of skill clusters affect the model's performance?\n2. What is the training and test split over the dataset?\n3. Could the model interpolate between varying skill levels? I would be curious to see what would be its performance on unseen skill levels.\n4. Does the Q-Q plot for Value head prediction only include accurate predictions?\n5. Is there a qualitative analysis of where the improvement in Hermes over the baseline comes from? Does it improve all skill levels or is it restricted to a low number of buckets?\n\nLIMITATIONS:\n1. The paper focuses primarily on modeling chess agents, so, a discussion on the broader impact of this method would be valuable. Here, the model assumes access to the ground truth skill embedding of the player. However, It would be interesting to see if it is possible to extract the skill embedding from the past moves i.e. implicit modeling of player/opponent skill.\n2. The model's accuracy is around 54%, which is still pretty low for the model to be deployed. The evaluation metrics are limited to the model's prediction accuracy. A human study or evaluation against skill-specific agents could further demonstrate Hermes' effectiveness in modeling skills at all levels.\n4. The paper has limited citations to past work on modeling diverse agents, so it is difficult to assess the novelty and the impact of the contribution. Further, one of the main contributions is the skill-aware module, so I would suggest the authors to include some baselines or discussion to explain the intuition and benefits of using this mechanism over other simpler baselines.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}