Learning Chess Blindfolded

1 INTRODUCTION . Recently , transformer-based language models such as GPT-3 have stretched notions of what is possible with the simple self-supervised objective of language modeling , becoming a fixture in state of the art language technologies ( Vaswani et al. , 2017 ; Devlin et al. , 2019 ; Brown et al. , 2020 ) . However , the black box nature of these models combined with the complexity of natural language makes it challenging to measure how accurately these models represent the world state underlying the text . Motivated by the above issues , we propose training transformer language models for the game of chess . Chess provides a simple , constrained , and deterministic domain where the exact world state is known . Also , chess games can be transcribed exactly and unambiguously using chess notations ( Section 2 ) . In fact , the form of chess notations allows us to probe our language models for aspects of the board state using simple prompts ( Section 3 ) . Due to the simplicity and precision of chess , we can evaluate language model predictions at a more fine-grained level than merely comparing it to the ground truth . For example , even if the next move prediction doesn ’ t match the ground truth move , we can still evaluate whether the move is legal given the board state , and if it is illegal , we can determine the reason why ( Appendix D ) . Moreover , since the state can be exactly modeled , we can evaluate models using counterfactual queries as well . The proposed evaluation sets and metrics are described in Section 5.3 . A side benefit of working with chess is that we have access to nearly unlimited data that is coupled with the exact board state at every turn . This board state is a form of grounding for the move sequence and allows us to compare training on move sequences alone to training with access to varying amounts of explicit state . Thus , modeling chess using language models may have implications for the debate surrounding the ability of language models to capture meaning if only exposed to text ( Bender & Koller , 2020 ) . To test the impact of chess board grounding on learnability and data efficiency , we can train language models with varying degree of access to the board state ( Section 4 ) . Finally , while chess represents a controlled domain , it is by no means trivial for a language model . To illustrate the challenges of language modeling for chess , consider the left board shown in Figure 1b , where white is next to move . In order to generate a valid next move , the language model needs to ( a ) infer that it is white ’ s turn , ( b ) represent the locations of all pieces , both white and black , ( c ) select one of the white pieces which can be legally moved , and finally ( d ) make a legal move with the selected piece . Thus , a language model has to learn to track the board state , learn to generate moves according to the rules of chess , and on top of that learn chess strategies to predict the actual move . We find that when given enough training data , transformers can learn to both track piece locations and predict legal moves at high levels of accuracy . However , when testing predictive ability for long move histories or when only given small training sets or when the model has access to limited history ( Appendix C.1 ) , predictive ability suffers . These challenging settings can provide an interesting testbed for future development of language models , and moreover because of the probing properties , errors can be diagnosed in great detail . In these more challenging settings , we show that providing parts of the board state ( during training time only ) can lead to significant improvements in accuracy . Our results also provide some key insights on transformer language models : ( i ) They are robust to various ways of incorporating explicit supervision about the board state when given enough training data . ( ii ) In particular , they are robust to changes in input distribution where additional tokens , related to board state , are added to input sequence only during training ( Section 4.1 ) . In contrast to LSTMs , transformers achieve this robustness even with smaller training sets ( Appendix F ) . ( iii ) The model performance strongly relies on access to the whole sequence history as the performance drops on limiting this access to a fixed-size window of previous tokens ( Appendix C.1 ) . To summarize , our contributions are to : • Propose chess as a testbed for evaluating world state tracking capabilities of language models . • Show that by selecting ( and tweaking ) the appropriate chess notation , we can probe language model for aspects of the world state using simple prompts ( Section 3 ) . • Propose a suite of probing tasks to evaluate language models for chess on world state tracking ( Section 5.3 ) . These probing tasks go beyond simple exact match , and use a more fine-grained evaluation , and allow for automated error analysis ( Appendix D ) . • Show that given enough training data , transformer language models can learn to track piece locations and predict legal moves at high levels of accuracy . • Evaluate the effect of grounding by training and evaluating a spectrum of transformer language models with varying degrees of access to the world state . We find that grounding helps in challenging settings of our proposed probing tasks . • Provide insights on transformer language models such as their robustness to incorporating the world state in various ways , their dependence on access to the whole history , etc . 2 BACKGROUND . Chess Preliminaries . Figure 1a1 shows how squares are indicated in chess notations via a combination of lettered columns and numbered rows . Chess notations use this square naming convention to denote the movement of pieces . As our notation , we choose Universal Chess Interface ( UCI ) notation , which combines the starting square and the destination square to represent a move.2 The move in Figure 1b is 1Source https : //en.wikipedia.org/wiki/File : SCD_algebraic_notation.svg 2For more details see https : //en.wikipedia.org/wiki/Universal_Chess_Interface represented as f1b5 in UCI where f1 indicates the starting square and b5 denotes the ending square . While another notation , SAN , is the standard choice for gameplay , we prefer UCI ( see Appendix A for our reasons for choosing UCI over SAN ) . 2.1 RELATED WORK . Simulated Worlds and Grounding . There have been several prior efforts in relating simulated worlds to natural language . The bAbI framework simulates a world modeled via templates to generate question-answering ( QA ) tasks ( Weston et al. , 2015a ) . The recent TextWorld framework facilitates generating , training , and evaluating interactive text-based games ( Côté et al. , 2018 ) . bAbI and TextWorld are similar to our work in the sense that the true world state is , by construction , available . The key difference with bAbI is that the it provides explicit world-state supervision in the form of training data for QA . With TextWorld we differ in : ( 1 ) their objective ( reward maximization vs. maximizing the probability of the next observation ) , ( 2 ) how they are ultimately evaluated ( final reward vs. world state tracking ) , and ( 3 ) whether we can directly probe the model ’ s knowledge of the entire state . The world models of Ha & Schmidhuber ( 2018 ) also maximize the probability of the next observation , but differ along the other two dimensions . Similarly the work by Hermann et al . ( 2017 ) and Hill et al . ( 2017 ) on developing and using 3D world simulations for learning grounded language has only partial overlap with the objective function , and differ along the other two aspects . Our work is related to work on grounding in that we are interested in comparing model performance when it does not have access to grounding information to when it does ( Bruni et al. , 2014 ; Kiros et al. , 2014 ; Ororbia et al. , 2019 ) . However , unlike the work of Ororbia et al . ( 2019 ) , for instance , the goal is not to improve performance of language models using access to more of the world state , but to assess how much of this state has been recovered by the model from just learning the LM task . Cloze Tasks for Natural Language Models . There has been a plethora of prior work on developing and using cloze tasks for evaluating natural language models ( Hermann et al. , 2015 ; Hill et al. , 2016 ) . These cloze tasks can range from testing general text understanding ( Paperno et al. , 2016 ) to targeting particular aspects of natural language , such as commonsense/pragmatics ( Mostafazadeh et al. , 2016 ; Ettinger , 2020 ) , narrative understanding ( Mostafazadeh et al. , 2017 ) , factual knowledge ( Petroni et al. , 2019 ) , etc . Creating these tasks often requires human curation , 3 and the evaluation is typically limited to exact match . In contrast , we propose cloze tasks/prompts targeting the world state , which can be precisely automated for chess , and can be evaluated at a fine-grained level . Probing . One of the goals of this work is to probe the language model ’ s board state tracking capability . A typical solution used by prior work is to train a probing model on top of a pretrained model ( Ettinger et al. , 2016 ; Alain & Bengio , 2017 ; Adi et al. , 2017 ; Tenney et al. , 2019 ; Hewitt & Liang , 2019 ) . This setup is time-consuming as it requires training probing models for all tasks . Moreover , the complexity of the probing model can also affect the conclusions ( Pimentel et al. , 2020 ) . In our case , we show that by appropriate choice of notation , probing for board state can be accomplished via simple prompts ( Section 3 ) . Deep Learning for Chess . Deep networks have been used in prior work to predict/mimic the next move given the true game state David et al . ( 2016 ) ; Oshri & Khandwala ( 2015 ) . Using just self-play and the rules of chess , AlphaZero achieves superhuman performance starting from random play ( Silver et al. , 2018 ) . The focus of these work is the quality of game play given the true board state while we just use chess as a testbed for evaluating the tranformer LMs world state tracking capabilities . Recently there have been several work focusing on transformer language models for chess ( Presser & Branwen , 2020 ; Cheng , 2020 ; Noever et al. , 2020 ) . These work are similar to ours in the sense that input is limited to the move sequence and not the true board state , but the focus of these work is again the quality of game play rather than how well is the model aware of the underlying board state . 3Automated cloze tasks without any human filtering can end up with instances which even humans can ’ t answer ( Hill et al. , 2016 ) 3 LANGUAGE MODEL PROMPTS AS BOARD STATE PROBES . Chess notations provide a simple alternate solution of using language model prompts as board state probes . For example , the prompt “ e2e4 e7e5 g1f3 b8c6 d2d4 h7h6 f1 ” ( the underlined move sequence leads to the left board state in Figure 1b ) can be used for next-token prediction with a language model trained on UCI notation . The generated token can be interpreted as the ending square predicted for the bishop at f1 . This prediction can then be used to determine the level of board state awareness of the model . A prediction of g1 may indicate that the model does not recognize that the piece type at f1 is a bishop , as such a move is not possible for a bishop . If the model predicts g2 , it may mean that the model is not aware that another piece is currently located at g2 . UCI notation is sufficient for assessing ending square prediction . However , it does not allow us to test starting square prediction directly . That is , we would like to give a language model the prompt “ e2e4 e7e5 g1f3 b8c6 d2d4 h7h6 N ” , where N represents knight , and expect it to generate a valid starting position for a knight of the correct color . To allow testing the model with such prompts , we propose randomly including piece types in moves during training with some fixed probability p. We refer to this strategy as “ randomly annotated piece type ” ( RAP ) and use the nomenclature “ UCI + RAP p ” to indicate that with p % probability , piece type is part of the move notation during training . Note that for p = 0 , the notation reduces to UCI . When testing with these starting square prediction prompts , we only include piece type for the prompt , not for any moves in the history . Thus , using RAP during training allows us to probe , at test time , where the model thinks each piece is , given any game history ’ s prefix ; by simply providing the desired piece type ( e.g. , N ) the model outputs the predicted starting square for a piece of that type .

Summary: This paper explores learning chess from raw notation as a benchmark for the ability of language models to track world state. Chess is an interesting benchmark, as a set of moves can be unambiguously linked to a world state, there are large amounts of data available and the model can easily be probed for its board tracking abilities. The contributions of this paper are twofold: (i) introducing blindfolded chess as a benchmark for grounded language learning and world state tracking, as well as a suite of probing tasks to evaluate models; (ii) empirical evidence that transformer language models can learn both the rules of the game and to track board state.