Improving the Strength of Human-Like Models in Chess
Authors: Saumik Narayanan
Kassa Korley
Chien-Ju Ho
Siddhartha Sen
Venue: Submitted to ICLR 2023 (2023)
Keywords: Human-like AI
Curriculum Learning

Abstract: Designing AI systems that capture human-like behavior has attracted growing attention in applications where humans may want to learn from, or need to collaborate with, these AI systems. Many existing works in designing human-like AI have taken a supervised learning approach that learns from data of human behavior, with the goal of creating models that can accurately predict human behavior. While this approach has shown success in capturing human behavior at different skill levels and even identifying individual behavioral styles, it also suffers from the drawback of mimicking human mistakes. Moreover, existing models only capture a snapshot of human behavior, leaving the question of how to improve them---e.g., from one human skill level to a stronger one---largely unanswered. Using chess as an experimental domain, we investigate the question of teaching an existing human-like model to be stronger using a data-efficient curriculum, while maintaining the model's human similarity. To achieve this goal, we extend the concept of curriculum learning to settings with multiple labeling strategies, allowing us to vary both the curriculum (dataset) and the teacher (labeling strategy).  We find that the choice of teacher has a strong impact on both playing strength and human similarity; for example, a teacher that is too strong can be less effective at improving playing strength and degrade human similarity more rapidly. We also find that the choice of curriculum can impact these metrics, but to a smaller extent; for example, training on a curriculum of human mistakes provides only a marginal benefit over training on a random curriculum. Finally, we show that our strengthened models achieve human similarity on datasets corresponding to their strengthened level of play, suggesting that our curriculum training methodology is improving them in human-like steps.

Under review as a conference paper at ICLR 2023
IMPROVING THE STRENGTH OF HUMAN-LIKE MOD- ELS IN CHESS
Anonymous authors Paper under double-blind review
ABSTRACT
Designing Al systems that capture human-like behavior has attracted growing at- tention in applications where humans may want learn from, or need collab- orate with, these Al systems. Many existing works in designing human-like Al have taken supervised learning approach that learns from data of human behav- ior, with the goal of creating models that can accurately predict human behavior. While this approach has shown success in capturing human behavior at different skill levels and even identifying individual behavioral styles, it also suffers from the drawback of mimicking human mistakes. Moreover, existing models only capture snapshot of human behavior, leaving the question of how to improve them-e.g. from one human skill level to a stronger one-largely unanswered. Using chess as an experimental domain, an existing humman-like model to be stronger using a data-efficient curriculum, while maintaining the model's human similarity. To achieve this goal, we extend the concept of curriculum learning to settings with multiple labeling strategies, al- lowing us to vary both the curriculum (dataset) and the teacher (labeling strategy). We find that the choice of teacher has a strong impact on both playing strength and human similarity; for example, teacher that is too strong can less effective at improving playing strength and degrade human similarity more rapidly. We also find that the choice of curriculum can impact these metrics, but to smaller ex- tent; for example, training marginal benefit over training on a random curriculum. Finally, we show that our strengthened models achieve human similarity datasets corresponding their strengthened level of play, suggesting that our curriculum training methodology is improving them in human-like steps.
investigate the question of teaching
curriculum of human mistakes provides only a
INTRODUCTION
AI systems are growing increasingly capable at solving tasks, making decisions, and assisting hu- mans in a wide number of domains. In games such as Chess, Go, and Poker, AI has demonstrated clear superiority
human performance (Silver et al. 2018 Brown & Sandholm, 2019). In do- mains such as marketing, transportation, medicine, law, hiring, and finance, Al decision making is capable enough be deployed alongside humans in many real-life scenarios, and in some cases, AI performance is sufficient to take over entirely.
More recently, researchers have looked into developing AI with the explicit goal of replicating hu- man decision making, in contrast to simply optimizing for Al performance. There are several rea- sons for doing this. In domains such as autonomous driving, developing human-like like AI can result in increased safety for both humans and AI, by providing an understanding of human driving tenden- cies (Hecker al. 2019). situations where Al assisting or educating humans, humans exhibit a higher level of trust when the AI is human-like like (Wang al. 2019 Li et al. 2021 Kim et al. 2022).
and higher levels of satisfaction when interacting with human-like AI (Amigó et al. 2006 Ragot et al. 2020; Jenneboer et al. 2022). In gaming, human-liie like AI can offer opportunities to practice against models of real opponents (Mcllroy- Young et al. 2022b a), and is generally more enjoyable to play with in competitive cooperative play (Zhang al. [02T Soni & Hingston 2008).
Modern efforts to create human-like Al are improving rapidly, but these systems naturally lead to reduced task performance result of focusing human similarity. For example in chess,

---

Under review as a conference paper at ICLR 2023
researchers have used a supervised learning approach on human game data to create models that match human decision- maaiing at different skill levels (Mellroy-Youur et al. 2020 these models can even be fine-tuned to match the decisions of individual player and distinguish between in- dividual playing styles MMllroy-Young et al. 2022b, (021). Since these models strive to match human behavior at different skill levels, they also adopt the human tendencies and mistakes made at those levels. Moreover, each of these models only represents a snapshot of human behavior, with no investigation of how to improve the models, e.g., from one human skill level to stronger one.
In this paper, we attempt to address these questions by developing a methodology for improving the strength of human-like models, while retaining their human similarity. Specifically, given a human- like model at a particular skill level, our goal is to improve the model's strength in a tett-efficiiett manner, without degrading its human similarity by too much. Thus, we evaluate these human-like models along trade-off of three metrics: strength, data efficiency, and human similarity ($3.2).
Chess is an ideal domain for our study due to the availability of Al systems at both ends of the strength vs. human similarity spectrum. Chess engines like Stockfish and Leela (the open-source replication of AlphaZero (Silver et al., (018)) dominate the best human chess players in strength, while Maia Chess (Mellroy- Young et al. (2020) provides a framework for training models that can accurately mimic human behavior at different skill levels. Additionally, using chess affords us a massive amount of public human data, which can be used for training or further analysis. Finally, chess has a well-defined ruleset, which allows us to evaluate the playing strength of any chess model obtain using computationally cheap simulations.
In devising a methodology for strengthening human-like models, we considered two main ap- proaches. The first is to leverage state-of-the-art chess learning framework, such the deep reinforcement learning framework of AlphaZero, but with an additional constraint to match human moves. Such a constrained optimization approach, however, is difficult to integrate into existing human-like models. Additionally, this process is far removed from any real human learning process. Instead, we chose a methodology that more closely mimics how human might learn: starting with an existing human-like model, which we call the student, we fine-tune the model using customized training curriculum. This approach gives us control over how we trade off human similarity for playing strength, while also developing training curricula that are interpretable to humans. We im- plement this methodology by replicating and extending Maia's framework for training human-like models and fine-tuning them on individual player data (Mcllroy- Young et al. 2022b), except train our own models and fine-tune tune them on different training curricula that we design.
The core idea behind our approach is quite similar to the concept of Curriculum Learning developed by Bengio et al. (2009), which looks at selecting the most efficient ordering of the data for training model. However, we extend this concept in an important way, by not just looking at the selection of the data, but also considering the strategy for labeling the data. That is, we distinguish between the input dataset, which we call the curriculum, and the labeling strategy, which we call the teacher. In the case of chess, the curriculum corresponds to a set of chess positions, and the teacher could be any chess model that suggests which move to make in each position. Whereas traditional curricluum learning assumes single, "correct" label, in our work we consider chess engines and human- like models of varying strength, which may have different opinions about which move to make in a given position. In our evaluation, we explore what happens when we vary both the curriculum and the teacher ($4.1) I). vary only the teacher ($4.2). or vary only the curriculum ($4.3)
Our results lead to several interesting conclusions. We find that the choice of teacher has strong impact on both playing strength and human similarity, and is not the case that the strongest teacher is the best choice. For example, teacher like Stockfish is too strong and different when the student is a human-like model of novice strength. Instead, using a stronger human-like model as a teacher leads to much better outcomes in strength, human similarity, and training efficiency. We also find that the choice of curriculum can impact these metrics, but to a smaller extent. For example, training on a curriculum of positions the student made mistakes on provides only a marginal benefit over training on random positions the student encountered. Finally, we show that our strengthened models achieve human similarity on datasets corresponding to their strengthened level of play ($4.4), suggesting that our curriculum training methodology is improving these models in human- -aman-like steps.
2

---

Under review as a conference paper at ICLR 2023
2 RELATED WORK
Chess has been a useful domain for measuring the capability of AI, and researchers for half a century have worked on developing stronger chess engines. The first breakthrough occurred in 1997, when chess supercomputer, Deep Blue, defeated the reigning world chess champion in a match (Campbell eet al. 2002). By the mid-2000s, chess engines based on alpha-beta game tree search could con- sistently beat the strongest humans on standard computer hardware, and this trend continues to the current day with Stockfish, currently the strongest chess engine in the world.
With the announcement of AlphaZero, Silver et al. (2018) showed that chess engines trained using deep reinforcement learning could rival traditional alpha-beta engines. Recently, MMllroy- Young et all.[1220)) (2020) released the Maia Chess engine, which repurposes the AlphaZero architecture to use supervised learning on human games to mimic human play directly, instead of trying to learn op- timal play. Maia was trained on massive public human chess dataset provided by the website Lichess.org, and showed the ability to mimic human behavior at a variety of different skill levels in chess. Further work by Mcllroy- Young et al. (2022b) expanded the capability of Maia mimic indi- vidual chess players as well as distinguish between individual playing styles (Mcllroy- Young et al.) 2021]. Overall, the domain of chess contains some of the most sophisticated work on human-like AI, further supporting its position as the "Drosophila of AI" (McCarthy) 1990).
However, research human-like AI extends far beyond chess. Jacob et al. (022) have developed human-like Al for games like Go and Diplomacy. Hecker et al. (2019) show that autonomous
vehicles can be trained drive in a human-like manner using quantitative measure of human similarity. Hu & Dong 202T) investigate the problem of making existing object detection algorithms more human-like in order to pass Turing tests and defeat adversarial attacks.
In general, optimizing AI towards human similarity will necessarily lead to reduced task perfor- mance, especially in scenarios where Inon-human-like AI achieves superhuman performance. In this paper, we choose to handle this trade-off by extending the Curriculum Learning concept of Ben- gio et al.] (2009) to support varying both the curriculum and the labeling strategy (teacher). When training an existing human-like model a particular curriculum and teacher, we adopt the fine- tuning methodology of Mcllrooy-Young et al. (2022b), which itself draws on recent methods from the transfer learning literature (Sun et al. 2019: Yosinski et al. 2014 Kornblith et al. 2019)
While we intend curriculum training to be a domain-agnostic framework for efficiently developing strong and human-like AI systems, our choice of chess allows to draw from existing chess peda- gogical literature to inform our choices of specific curricula to investigate. There strong empirical evidence supporting the notion that human chess training can build knowledge improve future skill (Vaci et al.) 2019). There are many avenues that human chess players can take to study, e.g., openings, endgames, tactics, game analysis, etc. However, there wide disagreement as which of these, if any, is the most optimal path for training (Gobet & Jansen 2006).
3 METHODOLOGY
The goal of this work is to investigate data-efficient approaches for improving the strength of a given human-like model, while maintaining its human-like property. In this section, we first describe the procedure we use to train human-like models in the domain of chess. We then define our perfor- mance metrics, which allow us to evaluating the trade-off between data efficiency, model strength, and human similarity when improving a given model. Finally, we introduce our curriculum train- ing approach, which extends the concept of curriculum learning multiple teachers, enabling us to examine the impact of varying both the curriculum (dataset) and the teaching advice (label strategy).
3.1 TRAINING HUMAN-LIKE MODELS IN CHESS
To develop human-like models, we replicate and extend the Maia framework developed by Mcllroy- Young et al. 2020). We acquire a database of human chess moves from the online platform Lichess.org, where each move is labeled according to the player's Elo (skill) rating. We then group the moves by Elo rating, and use supervised learning to train neural network predict what move a human would play in a given chess position. For example, we train a model on moves made by players rated between 1150 1249, and we denote this model 1200. The neural network
3

---

Under review as a conference paper at ICLR 2023
is slightly modified version of Maia's neural network, which itself is adapted from Leela's architecture. Notably, we significantly optimized the preprocessing pipeline and trained a broader set of models, going up Maia-2600. Further information about training procedure can be found in the appendix.
3.2 METRICS
Given human-like model, we aim to improve its strength in a data-efficient manner, while main- taining human similarity. We discuss the metrics we use to evaluate these performance measures.
measure the amount of data used train or update the model. Note that while we have access to an abundant amount of data in the domain of chess, if are to generalize our research to other domains with practical data limitations, minimiz- ing the amount of data usage will play an important role ($5).
Data efficiency: Data size. For data efficiency,
Human similarity: Move-matching accuracy. To evaluate whether the model maintains its human-like playing style, evaluate how well the updated model can predict human behavioral data. In general, we perform this evaluation specifically on the behavioral data used to train the orig- inal model, as this provides direct way measure the deviation from the original model. However, in the case of chess, we also have access to human data at broad range of skill levels, so we also present results to evaluate our updated models against this additional data ($4.4).
To provide more insight into this metric, we trained set of human-like models (following the procedure $3.1) from Maia-1208 to Maia-2600, and measure their move- move-matchie matchingg accuracy on the datasets with player ratings from 1200 to 2600. The results, shown in Figure Ta demonstrate that Maia-K model is the most accurate predicting the data of players of rating K. In particular, it is not the case that higher-rated models are better at predicting lower-rated datasets. This result replicates the findings MMllroy- Young et al. (2020) and provides evidence that move-matching accuracy is reasonable measure to evaluate whether the updated model is still human-like
2600m
45
46
48
50
52
54
54
2600m
70
65
61
58
54
51
50
2400m
44
46
47
49
51
53
54
53
2400m
65
61
57
52
50
49
Maia Model Rating
2200m
45
47
48
50
52
53
53
52
Maia Model | Rating
2200m
73
68
63
58
55
50
48
46
2000m
47
49
50
51
52
52
51
50
2000m
69
64
59
55
50
45
43
42
1800m
50
51
52
52
51
50
48
1800m
67
61
55
50
45
42
39
39
1600m
50
51
51
51
49
48
46
1600m
50
45
41
37
35
35
1400m
50
51
51
50
49
48
46
44
1400m
50
45
39
36
32
31
30
1200m
50
50
49
49
47
46
44
43
1200m
50
43
37
33
31
27
26
26
1200d
1400d
1600d
18000
20000
220000
2400d
2600d
1200m
1400m
16600m
1800m
10000
22000m
2400m
2600m
Player Rating (Data)
Maia Model Rating
(a) Move matching accuracy.
(b) Match win percentage.
Figure 1: Illustrating the metrics move-matching accuracy and match win percentage.
Strength:
Match win percentage. To evaluate the strength of the updated model, have the updated model play against the original model in a large number full-length chess games (until checkmate draw is reached), and measure the match win percentage of the updated model. This win percentage is then used as a proxy for measuring the model strength.
To provide more insight into this metric, have the models from Maia- 1200 Maia-2600 play against each other. The match win percentages (based on 5120 games played between each pair of models) shown in Figure 1b We that increase the distance between two models,

---

Under review as a conference paper at ICLR 2023
the win rate of the higher-rated model increases. We also that as ratings go up, the win rates tend to be closer to 50%. In general, these win rates correlate with the general notion of chess Elo ratings, but the magnitudes should correspond to a 75% win rate, but in our models it corresponds to between 51 % and 58% win rate. Jacobb et al. (202)) showed that higher-rated Maia models trained using standard imitation learning were suboptimal, and claimed that this would lead to weaker play than expected. Our results support this claim, by showing that the difference between (for example) a Maia- 880 and Maia- 1600 is smaller than the difference between an 1800-rated human and 1600-rated human.
not the same. In the standard Elo system, a 200 point difference
3.3 CURRICULUM TRAINING
We formally introduce the notion of curriculum training the data-efficient fine-tuning of an isting human-like model, with the dual objectives of improving the model's strength and preserving the human similarity of the original model.
In curriculum training, we start with a human-like model M, which we call the student. Our goal is to choose a pair of curriculum X and teacher Y to update the given model. curriculum is specified by the input distribution X. For example, X could be a distribution of chess positions that players of a certain rating have encountered in their games. A teacher specified by the labeling strategy Y, which specifies the labels for each input drawn from X. For example, Y could be the move particular human would make in the given position, or the move a superhuman engine would make.
After specifying the curriculum and teacher (X, Y), we draw data points {(z, v)} from the provided distribution and use them to update the model M. Following the curriculum learning approach, we update the model by performing gradient descent using this data. Note that curriculum training notion extends the standard curriculum learning notion developed by Bengio et al. (2009) to settings with multiple teachers (labeling strategies). To evaluate the performance of curriculum training, we vary the amount of data used to update the model M and measure the strength (match win percentage) and human similarity (move-matching accuracy) of the updated model, as per $3.2
In this work, we use the domain of chess to evaluate the effectiveness of curriculum training. We pre- trained a of Maia models serve as the initial human-like model M and the potential teachers Y. We vary the choice of teachers Y to be different ratings of human players or human-like models, or a superhuman chess engine, and examine their impact. We also examine the impact of choosing curriculum X based on the potential mistakes that the student might make, using the assistance of superhuman chess engine, stronger-rated Maia models, or human expert.
4 EXPERIMENTS
In this section, we examine the effectiveness of currieulum training in the domain of chess. We adopt the Maia- 1200 model as our student (i.e., the initial human-like model we wish to improve), and update it using different designs of curriculum X and teacher Y, varying both X and Y ($4.I). varying only Y ($4.2), and varying only X ($4.3). We evaluate the effectiveness of curriculum training on our three performance measures: data-efficiency, human similarity, and model strength. We then extend our evaluation of human similarity to cover datasets of higher-rated players, to measure the progress of the student with respect to higher-rated human-like models ($4.4).
4.1 EXPERIMENT 1: MIGHER-RATER PLAYER DATA
We first examine the most natural design of the curriculum and teacher pair: updating the initial human-like model Maia- 1200 using data from higher-rated human players. Specifically, let Xn be the distribution of positions encountered by R-rated players, and YR be the moves made by R-rated players in these positions. (Incidentally, this is the dataset used to train the Maia-It Rt models.) In this experiment, we set our student M as Maia-1200, and we vary the rating It from 1400 to 3000 and use (Xx, Ya) as the (curriculum, teacher) to update the student. To investigate the efficiency of this design, we gradually increase the amount of data used to update the student model. For each resulting model (updated using a different amount of data and different (X n, Yn)), we measure its human similarity (i.e., move matching accuracy) and strength (i.e., match win percentage).
5

---

Under review as a conference paper at ICLR 2023
For each design from R 1400 to 3000, fine-tune the student for up 10,000 batches of data drawn from (XR, YR). where batch is a set of 1024 data points. Due to a lack of data for higher-rated players, the size of the training curriculum is smaller for R. 2600, 2800, and 3000 (with 7300, 900, and 100 batches, respectively). We evaluate win percentage with a match size of 1024 games, and move-matching accuracy using 10 batches. We compute these evaluations every 10 batches between batch-0 and batch-100, and every 100 batches between batch-100 100 and batch- 10,000. We repeat each of evaluations 10 times, for total of 10,240 games and 100 batches per evaluation, and report the moving average (with window size of 50) of these measurements.
1400d
0.650
1600d
1800d
Playing Strength (Win%)
0.625
2000d
2200d
2400d
0.600
2600d
2800d
3000d
0.550
10
0.525
100
300
0.500
1000
0.44
0.45
0.46
0.47
0.48
0.49
0.50
Human Similarity (Move Matching Accuracy)
3000
10000
Figure 2: The results of using higher-rated player data to fine-tune lower-rated student model. Each curve represents the change in human similarity and strength when fine-tuning the Maia-1200 model on increasing amounts of data from players of particular rating. For each curve, label the points that are trained using different numbers of batches, from 10 batches to 100,000 batches.
Experiment results. The results, displayed in Figure 2 demonstrate the trade-offs of our three performance metrics for different pairs of curriculum and teacher (X YR). First of all, if our goal is purely to increase strength without concerns for human similarity or data efficiency, training using the highest rated curriculum leads to the strongest model.
If our objective is to increase the model's strength while maintaining the model's human similarity above selected threshold (without concern for data efficiency), using the closest-rated data for fine- tuning leads to the optimal outcome. For instance, if we want to maintain move-matching accuracy of at least 49%, the optimal strategy is to train on 1400-rated data curriculum for 10,000 batches, which gives model with % win probability. We could more efficiently achieve 55% win probability by training on a 1800-rated curriculum for 1000 batches, but at the cost of a much lower move-matching accuracy. The drawback of using the closest-rated curriculum is that training low rated data will eventually hit platcau-na not only do we require an order of magnitude more data achieve the same performance, but the overall gains are limited. Figure Ia suggests that training a 1200-student on 1400-rated data can only achieve 57% win rate, even with an infinite amount of data, while higher-rated curricula easily surpasses this limit.
If our objective is to maximize strength gains with a constraint on data usage, we see an interesting and non- non-trivial result: at many points during fine-tuning, the optimal curriculum is not the highest- rated or lowest-rated one, but at point in between. For example, at batch-30, the 2000-rated curriculum optimal, while at batch- 100 and batch-300, the 2400-rated curriculum optimal. Past this point, it appears that the 2600-rated curriculum is optimal at all batch sizes. However, it would not be surprising if the trend continues: if we had access to the 3000-rated curriculum at higher batch sizes, the high-rated model might perform worse than 2600-rated 2800-rated curriculum.
Overall, these results suggest that the strength of teacher matters significantly when training mod- els. Note that in this experiment, we have simultaneously updated the curriculum X and the teacher Y. In order to separately examine the effects of curricula and the effects of teachers on the fine- tuning process,
two additional experiments investigate these factors separately.

---

Under review as a conference paper at ICLR 2023
4.2 EXPERIMENT 2: EFFECT OF TEACHERS
To understand the effect of teachers on training, we hold X constant and vary Y. For these results, chose fix X the of board positions that the student was initially trained on. In this case, X would be the distribution of board positions encountered by 1200-rated players.
However, when fix the input distribution X, we are unable vary Y as in the previous experi- ment, as we might not have data on which moves, say 1400-rated players, would actually make in our selected X distribution. As an approximate solution, we use Maia models that we have trained to serve as the teacher Y. Thus, we teach Maia-1200 using curriculum where X is the set of posi- tions encountered by 1200-rated humans, and Y are the moves that Maia-1400 would make in these positions. We also introduce additional possible teacher, Stockfish, the strongest chess engine in world. This allows us to approximate much higher rated humans, since no models higher than Maia-2600 available. Using this curriculum design, we ran another experiment with the same training procedure and evaluation methods as Experiment 1.
1400
0.675
1600
1800
Playing Strength (Win%)
0.650
2000
2200
0.625
2400
0.600
2600
sf
0.575
10
0.550
100
0.525
300
0.500
1000
3000
0.42
Human Similarity (Move Matching Accuracy)
0.44
0,46
0.48
0.50
10000
Figure 3: The results of varying the teacher Y Stockfish (a superhuman chess engine) and Maia models of different ratings, while fixing the input distribution X to the student's positions.
Experiment results. The results are displayed in Figure 3 We see that, similar to the earlier experiment, move-matching accuracy is monotonically decreasing with respect to teacher rating. We also the same non-trivial result that the most data-efficient curriculum is often not the highest or lowest-rated teacher, but somewhere the middle. In particular, the most efficient teacher is Maia-2200, Maia-2400. Maia-2600, and these three points tend to be clustered closely together.
Interestingly, we see that Stockfish performs noticeably worse than most of the human-rated models. This could purely be because Stockfish's 's strength outclasses all humans, much of the move advice Stockfish teaches is out reach for the Maia-1200 student. An alternate hypothesis is that Maia- 1200 is unable to account for the stylistic differences between a human-like model and alpha- beta model like Stockfish. It would be an interesting future direction to clarify this question, e.g., by running curricula taught by lower-rated chess engines. Overall, these results confirm our observation in Experiment that the teacher has significant effect in fine-tuning a human-like model.
4.3 EXPERIMENT 3 EFFECT OF CURRICULA
We now hold the teacher constant and vary X, which board positions we teach from, examine the effect of curricula. From the previous experiment, see that Maia-2200, Maia-2400, and Maia- 2600 are the three most efficient teachers for retaining human similarity and improving playing strength, and all very similar to each other overall. Accordingly, take the middle of these three, and choose our teacher to be Maia-2400 for the next set of experiments.
One possible class of curricula could analyze is the distributions of board positions encountered at different levels. However, these distributions do not qualitatively differ very much from one another. Instead, specific domain knowledge in chess develop our sets of curricula

---

Under review as a conference paper at ICLR 2023
based fixing mistakes made by the student. We list three categories of mistakes, and choose one representative curriculum for each category to analyze.
Mistakes identified by superhuman chess engine. We choose the position of blunders, which is defined as move which lowers expected win probability by over 15% predicted by Stockfish.
Mistakes identified by human expert. We have collaborated with International Master (IM) on this work. The IM analyzed 80 simulated games played by Maia models at various strengths, and determined that Maia tends play suboptimally in the opening phase of the game, especially positions where Maia should "connect the rooks". As this is also a common problem for human chess players, we selected this our representative curriculum.
Mistakes identified by higher-rated human-like model. The final category is positions where higher-rated Maia models disagree with the move that the lower-level Maia-student would make. While this is not conclusive evidence of mistake, is possible indication that the lower-level Maia is playing suboptimally. We selected positions where Maia- 1200 and Maia-1400 differ, which we denote 1400-inflection positions.
Using these curricula and the Maia-2400 teacher, ran another experiment with the same training procedure and evaluation methods as Experiments and 2. For context, we also present a baseline curriculum which is just a random set of positions encountered the student's rating level.
1400i
0.675
blunder
random
Playing Strength (Win%)
0.650
rooks
0
30
0.600
100
0.575
300
1000
0.550
3000
10000
0.525
0.42
0.43
0.44
0.45
0.46
0.47
0.48
0.49
0.50
Human Similarity (Move Matching Accuracy)
Figure 4: The results of fixing Y to Maia-2400 while varying to positions identified by a super- human chess engine (blunder), human expert (rooks), and human-like AI (1400-inflection).
Experiment results. The results displayed in figure If our objective is to maximize strength gains with constraint on the data usage, both the blunder curriculum and the 1400-inflcction cur- riculum more efficient at teaching the base model than random curriculum, while the rooks curriculum underperforms in this regard. One hypothesis is that the rooks curriculum (human expert recommendation) is too specific in the positions it tries to teach. While it is true that both human stu- dents and Maia models tend to make mistakes in the opening phase of the game, it is likely that these decisions are game deciding, hand, the blunder and 1400-inflection curricula (positions identified by superhuman and human-like Als) look mistakes that the model makes across the entire distribution of general board states, but the emphasis mistakes leads to better outcomes than emphasis. If our objective is to increase the model strength while ensuring the model maintains human similarity above a selected threshold, the blunder curriculum outperforms random while the 1400-inflection and rooks curriculum do not. However, the difference in performance small across all curricula.
it may be less critical look these positions. On the other
Overall, our results demonstrate that, among a general set of curriculum we have examined, altering curriculum only brings marginal benefits compared with using random curriculum.
4.4 EXPERIMENT 4 MEASURING HUMAN SIMILARITY
8

---

Under review as a conference paper at ICLR 2023
In the previous of experiments, we evaluated human similarity by using the 1200-rated human dataset. Because the student model originally trained and evaluated on this dataset, it makes sense way measure how
stockfish
42
43
44
44
44
44
45
44
2600m
44
46
47
48
49
49
50
49
2400m
44
47
48
49
50
51
49
to this same dataset much the updated model has deviated from the orig- inal model. In domains where human data is labeled according to individual skill, this is indeed the only option for evaluation, and thus our previous approach should generalize to domains with human- like models trained in this way.
Teachers
2200m
45
47
49
50
50
49
2000m
47
1800m
48
50
50
51
48
46
1600m
48
50
50
50
49
48
47
45
1400m
49
50
50
49
4B
47
46
44
However, thanks to the rating metadata included in the human chess dataset, we can go one step further
random
44
47
48
49
51
49
Curricula
in our analysis. After model undergoes curriculum training, the model will naturally degrade in its ini- tial performance, because differently. However, the updated model might in fact play more like higher-rated model because we have taught it to play better.
blunder
44
46
48
49
50
50
50
49
rooks
44
46
47
48
49
49
50
48
teaching it to play
1400
43
45
46
47
49
49
50
48
1200
1400
1600
1800
2000
2200
2400
2600
Player Data Rating
To investigate this, took the 11 models trained in Experiments 2 and 3 after learning the full 10,000- batch curriculum, and analyzed their move-matching accuracy against higher-rated datasets, to see if the updated models play like higher-rated humans.
Figure 5: The results of analyzing how simi-
lar curriculum-updated models are to human datasets at a range of skill levels.
Experiment results. The results are displayed in Figure Overall, we find that the updated models are more similar to higher-rated datasets than the original dataset. Looking at the models from Experiment where we varied the model's teacher, there is very strong correlation between the rating of the teacher and the most similar dataset of the updated model. For instance, the model taught by Maia- 1600 is most similar to the 1600-rated dataset, while the model taught by Maia- 2400 is most similar to the 2400-rated dataset. Interestingly, each of the models taught using targeted curricula and Maia-2400 teacher all perform most similar the 2400-dataset. This provides more credence to our earlier observation that the teacher is much more important than the curriculum.
CONCLUSION AND DISCUSSION
Despite growing effort to develop human-like models that capture human behavior at different granularities (such as skill levels), surprisingly little attention has been paid to how these models be trained improved. In this paper. propose generalized framework for improving human-like model while maintaining its human similarity, using a novel extension of curriculum learning which considers both the curriculum data train on, and the teacher used to label this data. We measure performance as trade-off between the improved model's strength, its human similarity, and the data efficiency of the curriculum.
We evaluate this framework in the domain of chess through a series of experiments. At high level, find that the choice of teacher is critical for maximizing the efficiency of strength gains. In many cases, the best teacher is not the strongest teacher. Notably, Stockfish (the world's strongest chess engine) is worse teacher than even a low-rated human-liks model. Our results also suggest that the choice of curriculum is less important than the choice of teacher. Although we did find evidence that efficient curriculum design is possible, remains an open question how best to select a curriculum for training human like models in chess.
is important to note that any insights we gain from training human-like models do not necessarily apply to training real humans. These are, after all, neural network models, and not human brains. Nevertheless, the results in $4.4|suggest that our curriculum training methodology is able to improve human-like models in a way that ensures human similarity their improved level of play, path similar to human learning. This could provide insights into training real human students and is a topic plan to explore future work.

---

Under review as a conference paper at ICLR 2023
ETHICS STATEMENT
This work develops a methodology for improving the strength a given human-like model, while maintaining its human-like property. The human-like models create trained aggregate human data grouped by coarse skill levels, similar to the models trained by Mcllroy- Young et al. (2020) (though our strongest human-like models surpass theirs in target Elo rating). We do not train or make use of any individualized models in this work.
While there many potential societal benefits creating human-like models (as discussed in the introduction), there are also potential ethical concerns associated with creating, deploying. or using such models, especially when the models capture individual human behavior. McIlroy et al. refer to such models as "mimetic" and explore these ethical concerns in Mcllroy-Young et al. (2022a). For example, a human-like model of a target individual may leak private information about that individual, may allow others to prepare or plan for future interactions with that individual (by practicing with the model), and my even allow the individual to multiply their own output and interactions (by using copies of the model "proxies" of themselves). Moreover, the presence of such models could introduce new forms of human-Al interactions, raising new questions about how humans perceive, trust, and evaluate these Al systems. In general, would need more careful domain-specific consideration in order to address these questions.
In this work, we train human-like models using publicly available dataset from Lichess.org, For future research that would like to extend our methodology to other domains and/or examine its impact real humans, proper review and approval from an IRB would be required.
REPRODUCIBLILIT STATEMENT
Our All the data needed to reproduce our results is available at https::/database..ichhe. anonymized source code is available https://annnymouusppeepe..eeeeee sropeeescieneeefcceetttiieeee DBBE For guidance on how to use this repository to reproduce our code, we have provided README.X file. Details on how the models were trained available in the Appendix.
REFERENCES
Enrique Amigó, Jesús Giménez, Julio Gonzalo, and Lluís Mårquez. Mt evaluation: Human-like human acceptable. In Proceedings of the COLING/ACL 2006 Main Conference Poster Sessions, pp. 17-24, 2006.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th anmual international conference on machine learning. pp. 41 41-48, 2009.
Noam Brown and Tuomas Sandholm. Superhuman AI for multiplayer poker. Science, 365(6456): 885-890, 2019.
Murray Campbell, A. Joseph Hoane Jr, and Feng-hsiung Hsu. Deep blue. Artificial intelligence, 134 (1-2):57-83, 2002.
Fernand Gobet and Peter Jansen. Training in chess: A scientific approach. Education and chess, 2006.
Simon Hecker, Dengxin Dai, and Lue Van Gool. Learning accurate, comfortable and human-lide like driving, arXiv preprint 19033.9903.000 2019.
Bao-Gang Hu and Wei-Ming Dong. A design of human-like robust ai machines in object identifica- tion. arXiv preprint arXiv:2101.02327 2021.
Athul Paul Jacob, David J. Wu, Gabriele Farina, Adam Lerer, Hengyuan Hu, Anton Bakhtin, Ja- cob Andreas, and Noam Brown. Modeling strong and human-like gameplay with kl-regularized search. In International Conference on Machine Learning, pp. 9695-9728. PMLR, 2022.
Liss Jenneboer, Carolina Herrando, and Efthymios Constantinides. The impact of chatbots tomer loyalty: A systematic literature review. Journal of theoretical and applied electronic com- merce research, 17(1):12-222) 2022.
10

---

Under review as a conference paper at ICLR 2023
Jihyun Kim, Kelly Merrill Jr., Kun Xu, and Stephanie Kelly. Perceived credibility of an ai instruc- tor in online education: The role of social presence and voice features. Computers in Human Behavior, 136: 107383, Nov 2022. ISSN 0747-5632. doi: 10.1016fj.chb.202.107733.
Simon Kornblith, Jonathon Shlens, and Quoc V Le. Do better imagenet models transfer better? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2661- 2019.
Lan Li, Tina Lassiter, Joohee Oh, and Min Kyung Lee. Algorithmic hiring in practice: Recruiter and hr professional's perspectives on Al use in hiring. In Proceedings of the 2021 AAAI/ACM
Conference on AI, Ethics, and Society, pp. 166-176, 2021.
John McCarthy. Chess as the drosophila of ai. In Computers, chess, and cognition, pp. 227-237.
Springer, New York, NY, 1990.
Reid Kellroyy-Yunng. Siddhartha Sen, Jon Kleinberg, and Ashton Anderson. Aligning superhuman ai with human behavior: Chess as a model system. In Proceedings of the 26th ACM SIGKDD International Conference Knowledge Discovery Data Mining, pp. 1677-1687, 2020.
Reid McIlroy- Young, Yu Wang. Siddhartha Sen, Jon Kleinberg, and Ashton Anderson. Detecting
individual decision makiing style: Exploring behavioral stylometry in chess. Advances in Neural Information Processing Systems, 14:24482-24444 2021.
Reid Hellroy-Youngg Jon Kleinberg, Siddhartha Sen, Solon Barocas, and Ashton Anderson. Mimetic models: Ethical implications of AI that acts like you. In Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 479-490, 2022a.
Reid Mcllroy- Young, Russell Wang, Siddhartha Sen, Jon Kleinberg, and Ashton Anderson. Learn- ing models of individual behavior in chess. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 1253-1263, 2022b.
Martin Ragot, Nicolas Martin, and Salomé Cojean, Ai-generated vs. human artworks. a perception bias towards artificial intelligence? In Extended abstracts of the 2020 CHI conference on human factors in computing systems, pp. 1-10, 2020.
David Silver, Thomas Hubert, Julian Schrittwieser, loannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, and Thore Graepel. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419): 1140-1144, 2018
Bhuman Soni and Philip Hingston. Bots trained to play like human more fun. 2008.
Qianru Sun, Yaoyao Liu, Tat-Seng Chua, and Bernt Schiele. Meta-transfer learning for few-shot learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 403-412, 2019.
Nemanja Vaci, Peter Edelsbrunner, Elsbeth Stern, Aljoscha Neubauer, Merim Bilalić, and Roland H Grabner. The joint influence intelligence and practice on skill development throughout the life span. Proceedings of the National Academy of Sciences, 116(37): 116337):133331111 18366, 2019.
Dakuo Wang, Justin D. Weisz, Michael Muller, Parikshit Ram. Werner Geyer. Casey Dugan, Yla Tausezik, Horst Samulowitz, and Alexander Gray, Human-ai collaboration data science: Exploring data scientists' perceptions of automated ai. Proceedings of the ACM on Human- Computer Interaction, 3(CSCW):1-24 2019.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? Advances in neural information processing systems, 27, 2014.
Rui Zhang, Nathan J. McNeese, Guo Freeman, and Geoff Musick. "An ideal human" expectations of AI teammates in human-Al teaming. Proceedings of the ACM Human-Come Computer Interaction, 1(CSC 2021.
A APPENDIX
11

---

Under review as a conference paper at ICLR 2023
A.1
MAIA MODEL TRAINING
Data Pipeline
The raw data used for this work was downloaded from http://dattabaseliccesss The specific files used were from the months 2021-06 to 2021-03 for the training data, and 2022-04 for the test data. We also use the 2022-04 data for evaluating the result of our curriculum experiments.
al. 2020) published code for preprocessing the data, we decided to write our own code from scratch using a tensorflow dataset pipeline. Using this custom pipeline, we were able to store the entire preprocessed dataset using only 231GB, while the original pipeline would have used over 3TB. In addition, this training pipline massively sped up training Limetrainiig a full model using the existing published code would take around 30 hours an NVIDIA V100, while the new pipeline's efficient handling of IO reading and GPU allocation means we can train the same model in under 9 hours using the same GPU. We have made this code available, and hope that these improvements make the training process more accessible to the public.
While Mcllroy-Yourg
Model Training
The majority of the details of our model are identical to the models used in Mcllroy- Young et al. (2020). We use the same training size (400,000 batches), optimizer, and learning rate schedule. The architecture of neural network is almost identical, except that remove the value head, resulting in purely supervised learning.
Nevertheless, there are few differences between the prior models and work. First, restrict the training and test datasets to exclusively use blitz data (i.e., games between 3 and minutes), while the prior work included all games above 3 minutes. Since games with 3-8, 8-20, and above 20 minutes are scored using different rating pools, conflating these different systems could lead to less fine- graingr understanding of human behavior, since the model may get confused by conflated rating levels.
We also discard game history while training models. While this sacrifices the prediction accuracy slightly (by 3%). it aligns better with our goal of desigming a curriculum, since a curriculum may span positions from various games (at varying points each game), which may not preserve the context of game's history. We also expand the models to incorporate higher-rated styles. The prior work trained models rated between 1100 and 1900, while we added additional higher-rating models until the rating of 2600 to facilitate richer discussion of curriculum design and model improvement. lack of data above 2600 prevents us from training Maia models at higher levels.
12


Reviews:
[{"review_id": "NrkvwWBSDRF", "reviewer": "Reviewer_9kEs", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors analyze an approach to improve the playing strength of human-imitating chess engines, while trying to maintain the human-like play. The basic method is to use transfer learning (one-step curriculum learning) by retraining an existing human-like model with data from better players or a conventional chess engine. The evaluation focuses on the effect of the difference in playing-strength between initial model and curriculum data, in relation to the human similarity. Additional, a subset approach is evaluated, using only specific chess positions or a random subset.", "summary_of_the_review": "The paper does not propose a novel method, but performs a detailed analysis of an existing approach applied to the domain of chess. This is a niche topic, but substantial evaluations and conclusion may still justify acceptance. However, in case the reviewers assumption concerning the Maia-1200 set and the human similarity metric is correct, all conclusions become questionable. Therefore, the evaluations are not sufficient for acceptance. Furthermore, the stated clarity issues should be resolved.", "strength_and_weaknesses": "The authors offer a quite extensive evaluation of the proposed approach as well as the basic models used. This evaluation includes a consideration of differences between the used positions and the source (teacher) of the proposed move. Therefore, the evaluation is beyond what is usually expected. However, the used definition of the \"human similarity\" metric is problematic. For one, expecting an exact matching of ground truth moves is very strict and a similarity metric should be preferred. Although, this is likely not a trivial task, because similarity to human play can potentially be only evaluated by human experts. However, the phrase \"behavioral data used to train the original model\" suggested, that in most experiments, only moves from the Maia-1200 set are considered. (Also see clarity) In case this understanding is correct, the metric only evaluates deviation from novice play - not from human play. Therefore, the correlation between increase in playing strength and decrease of similarity becomes obvious. A potential resolution could be comparing to any human play (independent of the playing strength or up to the teachers playing strength). This should be an approximate if any human would play like the engine. \nAnother problem is, that it seems, that learning approach has not converged, after using all batches once (Fig 2,3). Without convergence, the used learning rate becomes an interesting question, because it implicitly defines the trade-off between initial weights and new data.\nFurthermore, Section 3.3. is a bit lacking - considering it is the introduction of the main approach. (See clarity)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "LjDiEdQAUwd", "reviewer": "Reviewer_aYkk", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper extends the concept of curriculum learning and investigates the question of teaching an existing human-like model to be stronger using a data-efficient curriculum, while maintaining the model’s human similarity. The authors then applied the method to chess and experiments show that the choice of teacher has a strong impact on both playing strength and human similarity. Based on their experiments, they claim that strengthened models maintain human similarity on datasets corresponding to their strengthened level of play.", "summary_of_the_review": "Overall, the paper has poor novelty and the experimental results seem too obvious and have no new findings. Please refer to “Strength And Weaknesses” for more details.", "strength_and_weaknesses": "Strength:\nThis paper studies curriculum learning and human-like models in the widely-studied domain of Chess. They provide detailed experiments to find the relation between strength and similarity.\n\nWeaknesses:\nHowever, there are no new techniques in the methodology, and the experiment results are not surprising. To be more specific, they used transferring learning to fine-tune a 1200m model with datasets of different ratings. It is very obvious that the playing strengths will increase and the human similarity will decrease when comparing the fine-tuned model with the original one. Besides, in subsection 4-4, the author said “the model taught by Maia-1600 is most similar to the 1600-rated dataset, while the model taught by Maia-2400 is most similar to the 2400-rated dataset”. This conclusion does not give any new findings. Overall, there are no new findings and the novelty is poor.\n\nOther comments:\n* Fig. 1 and Fig. 5 only provide the decimal numbers, while the readers might be interested in the actual difference in smaller grains.\n* In experiment 4.1, the author said “at least 49% … the optimal strategy is to train on 1400-rated data curriculum for 10,000 batches … with a 55% win probability”, and “a 55% win probability by training on a 1800-rated curriculum for 1000 batches but at the cost of a much lower move-matching accuracy.” But according to Fig. 2, the move-matching accuracy is about 48.6% for 1800-rated curriculum of 1000 batches, which is not “much lower” as stated in the context. The results seem to have not much difference when comparing different settings.\n* The author should explain how a move is generated by the trained model, is it by MCTS or just by the move with the maximum policy?\n* The author claims to propose a framework for curriculum learning, but according to the contents of the paper, this seems exaggerated.\n* In subsection 4.3, the author should provide the total number of positions for the three categories of mistakes. Especially for human experts, the 80 simulated games seem to be much smaller than the other two categories which can be generated by programs.\n* In Fig. 1 and Fig. 5, what does the color mean?\n* On page 5, “50% .” -> “50%.”\n* In Data Pipeline section of the appendix, “from 2021-06 to 2021-03”, is it typo for “2021-06 to 2022-03”?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "HbLsZ6fV33f", "reviewer": "Reviewer_xpsR", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The topic of the paper is finetuning models trained to play chess like humans (defined as: making the same moves as in a dataset of human games) to improve its performance (ie. chess rating) without losing the human-like behavior. Authors introduce a framework called curriculum training, and \"use it\" to update the model trained on human games with a certain rating to play like humans of a higher rating.\n\nCurriculum learning consists of an idea that the game positions (which authors refer to as curriculum, and RL terminology would call states) can be chosen independently from the strategy of choosing target moves (which RL terminology would call action/policy and authors refer to the teacher or labeling strategy).\n\nAuthors choose various settings where the models trained on games of humans of a given strength are finetuned in a supervised way to reproduce moves of higher-ranked humans:\n\n1. Where both states and actions come from the human recorded data (Experiment 1), in a way similar to Behavioral Cloning / Imitation Learning [1]\n2. Where the states come from the human data, but the actions are coming from a model trained to imitate human moves (Experiments 2, 3), which has similarity to distillation [2, 3]\n\nIn these experiments, authors study the effect of increasing the amount of data and the strength of the humans in the target dataset / model on two metrics:\n1. accuracy of the updated model to predict the moves of the humans of the dataset the was originally trained on (with lower score) (human likeness) \n2. performance of the updated model in games with similar models\n\nIn Experiment 4, authors shortly discuss human likeness, as measured on the target dataset.\n\nThe phrasing of the paper doesn't make it clear whether the data used to measure the human likeness is the same as the one the models have been trained on, or whether a proper train/test split was applied.\n\n[1]: Abbeel et al. [Apprenticeship Learning via Inverse Reinforcement Learning](https://ai.stanford.edu/~ang/papers/icml04-apprentice.pdf)\n\n[2]: Hinton et al. [Distilling the knowledge in a neural network](https://arxiv.org/abs/1503.02531)\n\n[3] Furlanello et al. [Born again neural networks](https://arxiv.org/abs/1805.04770)", "summary_of_the_review": "I find the results of the paper presented in a compelling way, but I don't find them interesting enough for the ICLR community. I think that the methods employed by a typical reader of ICLR papers if they wanted to improve a model while keeping it human-like would be the same before and after reading the paper, nor do I see any other benefits they would gain from the lecture.\n\nNits:\n\n1. Ethics statement: \"and m*a*y even allow the individual\"\n2. In sec. 4.3 authors write \"from the previous experiment, we see that Maia-2200, 2400, and 2600 are the three most efficient teachers for retaining human similarity and improving playing strength.\" However, based on the curves, it seems to me that there is no clear winner there, and the 1400 is offering a better \"human similarity\".\n3. I suspect that the result of \"training on a curriculum of errors to provide a marginal benefit over training on random positions\" is highly domain-specific, and may not extend beyond the particular datasets tested.", "strength_and_weaknesses": "## Strengths\nThe writing of the paper is exemplary, making it relatively easy to understand. I particularly liked the sec. 1 motivation behind the need for human-like models, even though they may have a worse performance than the models not taking this into account.\n\n## Weaknesses\nI find the evaluation of the human likeness of the models to be performed on the original datasets (of the original training, not the finetuning) unacceptable. Doing this makes the metric to count both \"unhuman\" and \"human, but of higher-rating\" as unhuman behavior, which doesn't follow the motivation of improving the models while keeping them human. While the sec. 4.4 shows partial results on the target rating, I find presentation of the original rating results as confusing, and the sec. 4.4 to not be detailed enough to override this assessment.\n\nIf the proper train/test split was not used, the results may suffer from overfitting (ie. the reported human likeness would not constitute a good approximation of a real move-matching accuracy on new positions).\n\nThe presented results provide no new insights for me, and are predictable:\n\n1. As human likeness measures accuracy of reproducing the original data, it drops when finetuning on different data [4]. The higher the rating gap (=> bigger domain shift), the bigger the catastrophic forgetting.\n2. As we train on the data with humans of a better rating, the performance (assuming enough data) improves, as the model gets to reproduce the better moves.\n3. The initial speed of improvement is lower for datasets that are more different (ie. higher rating) from the original, due to less common features being reusable in the new domain. This implies that, for a fixed sample budget, to get the best final performance there will be a tradeoff between low rating (=> the domain is close and there are a lot of reusable features), and high rating (=> overall better moves to reproduce). Authors also provide a correct intuition in the paper behind this: \"After a model undergoes curriculum training, [it] will naturally degrade in its initial performance, because we are teaching it to play differently\".\n\nI don't find the results in Experiment 3 very informative, as it's hard to understand the difference between the handcrafted data distributions (is the score lower because the data is of less variance, or coming from different-rating source, or the chosen positions don't cover the space of positions well enough to improve the model, etc.).\n\nThe benefits of introducing the curriculum training framework are not clear to me: the concrete method for updating the network (finetuning) is prior work, as is generating the actions from previous trajectories or other models. The paper makes no claims about any of the instantiation of the framework (using trajectories or other models' actions, or changing the states only) as being empirically better then the others, nor does it compare to a natural baseline of training on the new dataset from scratch.\n\n## Actionable feedback\n\n1. Present the results in the experimental section on the target domain (ie. it's human if it's close to what the human of a given strength would play)\n2. Clarify whether a train/test split was used in the paper and if not, provide the results where the evaluation data is separate from the training one.\n3. Provide a clearer value for an ICLR reader: what it is actually supposed to learn from the paper? The reader would be aware of the techniques for improving the models from the literature, is one of the methods better than the others, in what situations? Which of the claims are only applicable in the domain of human-like models or domain of chess? Is there anything else to take home?\n\n[4]: https://en.wikipedia.org/wiki/Catastrophic_interference", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "C7TZFlmFZE", "reviewer": "Reviewer_XUrX", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a curriculum learning paradigm to improve the strength of human-like chess-playing models. The paper studies how the occurrence of problems at a level and the labelling strategy affect the learning process. Through various experiments and analysis of the strength of the teacher and the selection of problems, the authors find that 1) the strength of the teacher labelling the problems matters much more than the curricula of problems 2) The most efficient teacher to improve performance is not the most or the least expert but an intermediate one. ", "summary_of_the_review": "The paper proposes a new dimension of axis for exploring curriculum design, teacher expertise. This is quite an interesting insight but the paper in its current form does not sufficiently support its claims.", "strength_and_weaknesses": "Strengths\n\n- **************Clarity**************: The paper is well written and easy to understand, and the sections naturally follow from one to the other.\n- **********Novelty:********** The paper is relatively novel in its approach towards understanding curriculum learning and building human-like models. While the zone of proximal development (ZPD: problems with medium difficulty are the most useful) have been studied by measuring problem difficulty, the paper looks at teacher/annotator proficiency.\n- The analysis in the paper is thorough, and the graphs and methods are easy to understand.\n- The insight of accounting for labeller/ teacher proficiency is quite powerful.\n\nWeakness\n\n- The interpretation of the result (most data-efficient curriculum is often not the highest or lowest-rated teacher, but somewhere in the middle) in Experiment 2 is not well supported. The authors themselves suggest the alternate hypothesis that seems more likely: Maia- 1200 cannot account for the stylistic differences between a human-like model and an alpha-beta model like Stockfish. The alternate hypothesis seems likelier as the most similar stylistic teachers with the highest ratings are the best teachers. Without an experiment to test this, one of the central claims of the paper is not on firm grounds.\n- I am unsure how the results in this paper might extend to a domain outside of chess: either for teaching people or for building stronger human-like models. The discussion section is lacking here, with just one application domain in the paper! For example, the claim that “that the teacher is much more important than the curriculum.” might not extend to other domains.\n- Experiments 2,3 and Figures 2 and 3 show human similarity based on the games of 1200 rated players. When improving the strength of a model, one might want to see a resemblance closer to the target rating rather than the base rating. So, if I wish to have a certain similarity with a 1600-rated human, would it be better to train using a 1600 agent or a 1400 agent or an 1800 agent? Using matching accuracy from 1200-rated games does not seem right (this is also seen in experiment 4).\n\nMinor\n\n- The caption or the graph in figure 5 does not mention the metric shown.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject