SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xidong Feng *
Yicheng Luo
Ziyan Wang
Hongrui Tang
Mengyue Yang
Kun Shao
Huawei Noah's Ark Lab
David Mguni
Yali Du
Jun Wang
When solving decision-making tasks, humans typically depend on information from two key sources: (1) Historical policy data, which provides interaction replay from the environment, and (2) Analytical insights in natural language form, exposing the invaluable thought process or strategic considerations. Despite this, the majority of preceding research focuses on only one source: they either use historical replay exclusively to directly learn policy or value functions, or engaged in language model training utilizing mere language corpus. In this paper, we argue that a powerful autonomous agent should cover both sources. Thus, we propose ChessGPT, a GPT model bridging policy learning and language modeling by integrating data from these two sources in Chess games. Specifically, we build a large-scale game and language dataset related to chess. Leveraging the dataset, we showcase two model examples ChessCLIP and ChessGPT, integrating policy learning and language modeling. Finally, we propose a full evaluation framework for evaluating language model's chess ability. Experimental results validate our model and dataset's effectiveness. We open source our code, model, and dataset at https://github.com/waterhorse1/ChessGPT.
In recent years, large language models (LLMs) based on transformer architectures [57] have showcased remarkable capabilities far exceeding their original design as simple language modeling tools. This was especially notable following the advent of ChatGPT [38]. Stemming from causal language modeling, a plethora of recent studies have concentrated on developing efficient and powerful LLM base models [16, 6, 55, 5, 53], supervised fine-tuned models [52, 12, 3, 24] and models [38, 23, 64, 51] leveraging Reinforcement Learning from Human Feedback (RLHF) [13].
Concurrently, there has been a growing trend in employing Large Language Models (LLMs) as foundational elements for decision-making systems. These systems either depend on the expressive capacity of transformer architectures to execute imitation learning, thereby modeling complex behaviors [11, 22, 4], or they harness the common knowledge embedded within LLMs to facilitate the policy learning process [60, 17, 15, 2]. However, the dynamic interplay between policy learning and language modeling has been scarcely addressed. Human decision-making typically involves both: we draw upon historical policy interaction to refine our policy and also employ our thoughts for strategic
*Work done during internship at Huawei
1
Game Dataset
Online Game
Pro Game
Modeling
General Text
Language Dataset
Wiki
Blog
Consolons of the
Annotated Game
Mixed Dataset
Youtube Video
4
Instruction-tuning
Conversation Dataset
Result "1-0"  
OK 4kA - 0 T"
Reply 3 Share -
Reddit Conversation
ChessGPT-Base
Policy Learning By
Causal Language
Language Learning By
ChessGPT-Chat
Chat Ability By
Supervised Fine-tuning
ChessCLIP
Aligning by Contrastive Learning
[FEN]
mbqk2r/1pp2pp1/p3p2p/8/1b1PN3/5N2/
PPP2PPP/R1BQK2R w KQkg - 1.9
Figure 1: Our work provides a comprehensive pipeline that encompasses data, model, and evaluation framework, aiming to foster research on the interaction between policy learning and language learning. Data: Our dataset is organized into four categories: game, language, mixed, and conversation datasets.
[PGN] 1. d4 d5 2. Nf3 Nf6 3. e3 a6 4.
Nc3 e6 5. Bd3 h6 6. e4 dxe4 7. Bxe4
Nxe4 8. Nxe4 Bb4+
Modeling Ability
Value Judgement
ChessGPT
White takes
large advantage
Checkmate in
one move with ...
Rg5# can
checkmate in
one move
Model: Leveraging this rich dataset, we present two models: ChessGPT and ChessCLIP. Evaluation: Our evaluation framework is structured around three key dimensions: modeling capability, value judgement, and policy proficiency. Refer to Appendix C for image sources.
consideration, mostly in natural language form. Based on this logic, we argue that the study of natural language understanding and policy learning should not be isolated. To advance exploration in this realm, we choose one classic game: Chess, as a practical testbed for initial steps in this direction.
Chess, one of the oldest and most universally played board games, presents an ideal testbed due to the wealth of both policy data and language data. In terms of policy data, it is reported that over ten million games are played daily on Chess.com, the most frequented online chess platform. Regarding language data, a myriad of chess-related knowledge is readily accessible online in various forms and mediums, ranging from game analysis, books, puzzles, and news, to online tutorials, wikis, and even YouTube videos. Building on these resources, we have constructed a comprehensive pipeline dedicated to research on chess policy learning and language modeling, illustrated in fig. 1. Specifically, we provide the following three components:
Datasets. We curated a large-scale game and language dataset for chess. Our dataset comprises numerous game data from online chess databases recording how humans and chess engines game replay. It also includes a language dataset that encapsulates chess knowledge in a natural language
format, as well as a mixed game-language dataset, which offers the most straightforward interrelated data including articles, discussion, or commentary (language) on specific chess game replay (game). Models. We introduce two models, ChessCLIP and ChessGPT, leveraging our datasets. These models showcase the potential for AI to learn from a mixture of replay data and language knowledge. Evaluations. We design an extensive set of tasks to evaluate our models' abilities from three distinct perspectives: modeling ability, to gauge the model's proficiency in tracking game state; value judgement ability, measuring the model's capacity for value assessment and chess knowledge; and policy ability, to test the model's capability in decision-making. Our experimental results confirm that our models consistently outperform other LLM baselines in all evaluation tasks.
We illustrate our full pipeline in fig. 1. Our work primarily pursues two objectives. Firstly, we construct the whole pipeline on chess as an initial step in promoting research on the interaction/interplay between policy learning and language learning, as well as on the potential of language as a tool for action and understanding. Secondly, our efforts have yielded valuable by-products: the ChessGPT/CLIP models. These models possess practical applicability - they could potentially serve as effective Chess AI assistants for humans.
The pursuit of creating artificial intelligence capable of playing chess can be traced back to the very beginning of the history of computer science [56]. Chess engines today achieve superhuman-level performance by utilizing human knowledge [9] or self-play [47]. Recently, there has been increasing interest in improving the interpretability [32] of these systems and their alignment with human behavior [33] besides strong performance. A chess engine that aligns with human behavior may unlock many exciting opportunities, for example, they can be used as a personalized tutor for chess beginners [33]. Some research efforts also concentrated on employing LLMs to learn policies in Chess [36, 50]. However, these studies mainly center on small-scale datasets or limited training.
There has been increasing interest in leveraging Internet-scale knowledge for creating agents capable of generalizing across many tasks and capabilities [60, 17, 44]. For example, MineDojo [17] introduced a framework on Minecraft for understanding how to enable artificial agents to learn in an open-ended environment. More recently, there has been a surge in research that treats LLMs as agents, aiming to harness their internet-scale knowledge for decision-making tasks [31, 58, 62, 21]. In contrast to these studies which typically rely on powerful LLMs like GPT-4 [8], our paper concentrates more on training, especially the interplay between language modeling and policy learning.
3 A large-scale game and language dataset for chess
We introduce a large-scale game & language dataset by collecting chess-related materials from the Internet. Our dataset can be mainly divided into four categories: (1) The Game dataset, encompassing online chess match replay data involving worldwide human players and chess engines of varying skill levels. (2) The Language dataset, principally recording chess-associated knowledge, analyses, discussions, and news in the form of natural language (3) The Mixed Game-Language dataset, incorporating both game data and human natural language elements (such as game analysis or comments) in alignment. (4) The instruction-tuning and conversation dataset, consisting of instruction data and conversation data related to chess. We include comprehensive dataset descriptions, statistics, and text examples in Appendix D, and the procedure of data collection and pre-processing in Appendix E.
3.1 Game dataset
Game replay data provide the most direct method for both humans and machines to grasp the play mechanics of Chess. In chess, these data are commonly stored in the Portable Game Notation (PGN²) format which is a standard plain text format as illustrated in Figure 2. A PGN starts with some headers that include metadata about the game. These headers include information such as the name of players, the Elo ratings, the opening play, and the game outcome. The headers are followed by a move text section that records the moves played by the two players in turn. The moves may be further annotated with comments enclosed in braces.
2We refer to Appendix B for readers who are not familiar with chess notation format.
Previous work [33] uses the moves recorded in PGNs for policy learning. The moves are interpreted as actions in a Markov Decision Process and the state position can be reconstructed by loading the PGN into a chess engine. However, PGNs may contain additional useful information beyond the individual moves made. For example, the Elo ratings in the headers may inform us about the relative strength of the players. Additional information included in the comments of the move text section can also be useful - some of the moves are annotated with evaluations generated by computer chess programs that predict the current advantage of the players. These additional annotations may be useful from a reinforcement learning perspective, e.g., for value function learning. For this reason, we curated the game dataset with all of this information intact to facilitate policy learning.
Lichess dataset We collect five months of online game data from the Lichess database [30], culminating in a total of 17.5 million game replay records for online game players.
Pro-player dataset In the Lichess dataset, the majority of player Elo-ratings range between 1000 and 2000. To diversify our game dataset with more skilled matches, we also incorporated an additional 440,000 game records from 245 professional chess players. These professionals typically hold notably higher Elo ratings within the range of 2000 to 2800.
CCRL Chess engines like StockFish and LeelaZero have attained a proficiency level far beyond what any human player can currently reach. Considering this, we additionally incorporate the Computer Chess Rating Lists (CCRL) [10], which
[White "Alice"]  
[Black "Bob"]  
[Result "0-1"]  
[WhiteElo "2100"]  
[BlackElo "2000"]  
[Opening "Sicilian Defense"]  
1. e4 { [%eval 0.1]} 1... c5  
2. Nf3 2... Nc6  
...  
13. b3?? 13... Nf4? 0-1
Figure 2: Replay example in Portable Game Notation (PGN) format.
is a dataset of chess games played by computer chess engines. The CCRL dataset comprises a considerable collection of chess games, specifically 3 million, all of which are played by computer chess engines and stored in PGN format. The Elo-ratings of chess engines fall in the range of 2800-3700.
Chess puzzles A chess puzzle represents a particular chessboard configuration, designed to present a distinct challenge or objective for the solver. Chess puzzles often require players to find the best move or sequence of moves to achieve a specific goal, such as checkmating the opponent's king, or finding a tactical combination. In our game dataset, we integrate 3.2M puzzles sourced from the Lichess puzzle dataset. Each puzzle within this collection is annotated with its rating, theme and solution.
Chess modeling dataset We observe that most chess rule descriptions are conveyed in natural language, posing a challenge for machine learning models since they statistically require a large volume of model data to accurately comprehend the chess rules [45]. To address this issue, we build a synthetic chess modeling dataset leveraging the python-chess library [41]. We collect chess game data from a one-month dump of the Lichess dataset, deliberately distinct from the month used in our own Lichess dataset. we design several model-based tasks including converting PGN to FEN, transferring UCI to FEN, and predicting legal moves, etc, resulting in 1.9M data samples.
3.2 Language dataset
Existing dataset Numerous existing datasets comprise general internet crawl data from platforms like CommonCrawl or Wikipedia. We establish a filtering pipeline to extract only chess-related language corpus from pre-existing language corpus, including C4 [43], Pile [19], Oscar [37], Wikipedia [18] and RedPajama [53]. These datasets extend the scope of our language data beyond mere game-play.
Chess blogs Numerous chess websites often publish insightful blogs, sharing their analyses and perspectives on various aspects of chess gameplay. Such blog data is incredibly valuable, as it encompasses game-specific analysis, forming a vital link between the concrete chess game data and its interpretation in natural language form. We manually select approximately 30 chess-related websites and scrape 73.2k blog articles.
Chess books Similar to chess blogs, chess books can provide long and detailed analysis of the game. We extract approximately 8k chess-related books from online library to enrich our language dataset.
Chess forums Chess forum serves as a platform for a large amount of chess-related dialogues and conversations involving a diverse range of users. These platforms encompass high-quality question-and-answer pairs, as seen in platforms like StackExchange, or more generalized discussions on
various chess-related topics, commonly found in dedicated chess-specific forums. We mainly scrape chess forum data from 5 chess-specific forum platforms and StackExchange, using requests and playwright. This process results in a collection of 140K posts, representing a wealth of diverse views, queries, and discourses related to the world of chess.
3.3 Mixed game-language dataset
Annotated chess game An annotated chess game is a chess game accompanied by written commentary and analysis. In an annotated game, each move made by the players is explained and evaluated, providing insights into the thought process, strategic considerations, and tactical ideas behind the moves. Here is an example of an annotated PGN with Sicilian Defense opening:
1.e4 c5 {The game starts with the Sicilian Defense, one of the most popular and aggressive responses to 1.e4. Black aims to control the center and create imbalances early on.}
These annotated games inherently maintain the correspondence between board state and human language, serving as an exceptionally high-quality data source to align a model with complex human intentions and judgements. We amass annotated games from seven sources, five of which are collected from the internet while the rest two are commercial datasets. In total, we collect 245K annotated games with 1.3M board-language pairs.
Youtube transcripts Drawing inspiration similarly from MineDoJo [17], a YouTube video can naturally serve as a mixed game-language dataset by aligning video clips with natural language transcripts based on timestamps. Rather than generating image-language pairs directly, we develop a pipeline that accurately applies OCR (Optical Character Recognition) to chessboard screenshots to generate FEN (Forsyth-Edwards Notation), a system that describes the chess state in a language format. We gathered around 83k chess videos, resulting in million-scale English transcripts and board-language pairs, thus establishing a substantial mixed game-language dataset.
3.4 Instruction-tuning & conversation dataset
Supervised fine-tuning is a crucial component to train large language model (LLM) to follow instructions [38, 34, 61]. In addition to the comprehensive chess materials mentioned before, we also collect instruction-tuning and conversation datasets which can be used to finetune the pre-trained LLM base model, thereby enhancing its instruction-following and dialogue capability.
Instruction-tuning data from GPT-4 Inspired by Alpaca [52], we use the self-instruct technique [59] to generate high-quality, instruction-following data through GPT-4 [8]. Specifically, we manually construct 200 seed prompts for chess-related questions or instructions. These prompts serve as few-shot examples, guiding GPT-4 towards more coherent and relevant generation. Finally, we generate around 4k instruction-response pairs using this pipeline.
Conversation data from Reddit The instruction data collected from GPT-4 are mainly in a single-step form, which means only one round of question-answer pair is included. To mitigate this issue, we collect multi-step conversation data about chess on Reddit. Reddit allows users to interact by commenting on posts and responding to other comments, creating a nested structure of responses. This nested structure can be easily converted to a conversation tree by treating the comment's reply as a child node for that reply. A rich source of conversation data can then be acquired by navigating from the root node to each leaf node via every available path. In all, we choose 6 chess-related sub-reddits and collect 410k human conversations about chess.
4 Large-scale pretraining
We will showcase two models - ChessCLIP and ChessGPT trained on the large-scale dataset.
4.1 ChessCLIP
CLIP (Contrastive Language-Image Pre-Training) [42] is a neural network trained on a variety of modalities (e.g. image, text). By conducting contrastive learning on a large amount of paired data, CLIP bridges the image and language modality, enabling the model to understand vision by language information and vice versa. Our mixed game-language dataset in Section 3.3 has a similar paired
structure because the annotation is naturally paired with its preceding game trajectories. Based on this subset, we can train a ChessCLIP to bridge the modality of policy and language. Specifically, by denoting the chessboard state  $S$  at timestep  $t$  as  $S_{t}$ , and the annotation language as  $L_{t}$ , the data pair at timestep  $T$  can be represented by  $\left(\left\{S_{t}\right\}_{t=T-k}, a_{T}\right)$ , where  $\{S_{t}\}_{t=T-k}^{T}$  is a stacked  $k$  history states and  $a_{T}$  is the last move.
We want to emphasize more on what ChessCLIP can do by aligning the policy modality and the language modality. Firstly, ChessCLIP offers a similarity metric given one PGN and a text description. Just like the application of large-scale image/text retrieval using CLIP, ChessCLIP can help users conduct PGN/text retrieval - search for game based on text or search for comments based on specific game. In addition, because of the low-dimensional feature of action space compared to vision or language space (there only exists a few legal moves for a given chess state), we can directly conduct search algorithms to maximize the similarity to generate action based on one text description using ChessCLIP. For example, given a chessboard state and a text description, ChessCLIP can generate a move by iterating through all legal moves and finding one move that returns the largest similarity. By the same logic, ChessCLIP can directly generate move sequences (multiple actions) using greedy search or beam search. We refer the reader to Appendix F.1.1 for more discussions.
Implementation details We preprocess the annotated PGNs to produce board/text pairs which we feed separately to the board and text encoders. In particular, for every move in the PGN, we extract the comments attached to the move as well as the board state. We encode the board positions and moves using the same scheme as those used by Leela Chess Zero (lc0) [26], which is similar to the encoding used by AlphaZero [47] for encoding positions and moves in chess. Concretely, the board positions are encoded as a  $\mathcal{R}^{8\times 8\times 112}$  feature map and the actions are encoded as a  $\mathcal{R}^{1858}$  vector. We instantiate a ChessCLIP model with a pair of text encoder and a board/action encoder. For the text encoder, we only fine-tune the last two layers of pretrained text encoder from OpenAI CLIP model. For the board/action encoder, we use a ResNet [20] architecture that conditions on the action encoding via a modified FiLM layer [40]. Please refer to Appendix F.1.1 for implementation details.
4.2 ChessGPT
The Generative Pretraining Transformer (GPT-3) [7] is an autoregressive language model that uses deep learning techniques to generate human-like text. GPT-3 is trained by casual language modeling, which aims to predict the next word in a sentence given all the previous words. Following the same logic, we train a GPT-like model using all chess materials introduced in Section 3. Unlike other policy behavior data in robots [27] or video games [35], the chess state and move data can be represented in merely textual format. Thanks to this feature, we can directly treat chess as a text game and the imitation learning objective for policy learning can be directly covered by casual language modeling over the game dataset provided in Section 3.1.
Implementation details We follow common implementations of training a domain-specific instruction-following LLM. Firstly we conduct base-model fine-tuning using chess corpus introduced in section 3.1, 3.2 and 3.3. Due to computational constraints, we choose to finetune the RedPajama-3B-base [53] model, which is an open-souce replication of LLaMA [55]. The base model adopts the GPT-NeoX [6] architecture, a GPT-3 [7] variant with a few modifications such as rotary positional embedding, parallel attention computation, and different initialization. The base-finetuning brings us our base model: ChessGPT-Base. After base-finetuning, we conduct supervised fine-tuning by supervised learning on question/conversation response using data introduced in section 3.4 and general conversation data from OASST1 [24], Dolly2 [14], Alpaca-GPT4 [39], and Sharegpt [46], forming our chat model: ChessGPT-Chat. We leave further RLHF (Reinforcement Learning from Human Feedback) training for future work. Refer to Appendix F.1.2 for more details.
5 Evaluation and benchmark
In this section, we present a comparative analysis between ChessGPT trained on our database with other baseline LLMs. The purpose of our experiments is to assess the performance of ChessGPT in three primary dimensions: Chess modeling ability, Value judgement ability, and Policy ability. The Chess Modeling capability focuses on the language model's proficiency in accurately tracking the game state and predicting valid moves. Regarding the Value judgement ability, we assess the model's precision in evaluating the worth of a chess game, encompassing the identification of advantageous
Table 1: Bigbench State Tracking in Chess
Tasks	LLM Models (%)
LLAMA-7B	RedPajama-Base	ChessGPT-Base	ChessGPT-Chat
Real Short	29.5 ± 1.4	23.2 ± 1.3	99.5 ± 0.2	98.5 ± 0.4
Real Med	39.3 ± 1.5	38.2 ± 1.5	97.7 ± 0.5	97.8 ± 0.4
Real Long	53.0 ± 1.6	51.9 ± 1.6	98.1 ± 0.4	97.6 ± 0.4
Syn Short	31.3 ± 1.4	24.9 ± 1.3	94.2 ± 0.7	92.3 ± 0.8
Syn Med	39.9 ± 1.6	37.7 ± 1.5	94.6 ± 0.7	88.9 ± 1.0
Syn Long	45.8 ± 1.5	42.2 ± 1.5	92.8 ± 0.8	85.1 ± 1.1