ChessGPT: Bridging Policy Learning and Language Modeling
Authors: Xidong Feng
Yicheng Luo
Ziyan Wang
Hongrui Tang
Mengyue Yang
Kun Shao
David Henry Mguni
Yali Du
Jun Wang
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Chess
Dataset
large language model
policy learning
language modeling

Abstract: When solving decision-making tasks, humans typically depend on information from two key sources: (1) Historical policy data, which provides interaction replay from the environment, and (2) Analytical insights in natural language form, exposing the invaluable thought process or strategic considerations. Despite this, the majority of preceding research focuses on only one source: they either use historical replay exclusively to directly learn policy or value functions, or engaged in language model training utilizing mere language corpus. In this paper, we argue that a powerful autonomous agent should cover both sources. Thus, we propose ChessGPT, a GPT model bridging policy learning and language modeling by integrating data from these two sources in Chess games. Specifically, we build a large-scale game and language dataset related to chess. Leveraging the dataset, we showcase two model examples ChessCLIP and ChessGPT, integrating policy learning and language modeling. Finally, we propose a full evaluation framework for evaluating language model's chess ability. Experimental results validate our model and dataset's effectiveness. We open source our code, model, and dataset at https://github.com/waterhorse1/ChessGPT.

ChessGPT: Bridging Policy Learning and Language Modeling
Xidong Feng University College London
Yicheng Luo University College London
Ziyan Wang King's s College London
Hongrui Tang University College London
Mengyue Yang University College London
Kun Shao
Huawei Noah's Ark Lab
David Mguni Huawei Noah's Ark Lab
Yali Du King's College London
Jun Wang University College London
Abstract
When solving decision-making tasks, humans typically depend information from two key sources: (1) Historical policy data, which provides interaction replay from the environment, and (2) Analytical insights in natural language form, exposing the invaluable thought process or strategic considerations. Despite this, the majority of preceding research focuses only source: they either historical replay exclusively to directly learn policy or value functions, or engaged language model training utilizing that powerful autonomous agent should cover both sources. Thus, we propose ChessGPT, GPT model bridging policy learning and language modeling by integrating data from these two sources in Chess games. Specifically, we build a large-scale game and language dataset related to chess. Leveraging the dataset, we showcase two model examples ChessCLIP and ChessGPT, integrating policy learning and language modeling. Finally, we propose a full evaluation framework for evaluating language model's chess ability. Experimental results validate our model and dataset's effectiivnness. We open source our code, model, and dataset https: //github//tubb..... wtt.//aaeeroorro//Ceeeeeeeee
language corpus. In this paper, argue
Introduction
In recent years, large language models (LLMs) based on transformer architectures [57] have show- cased remarkable capabilities far exceeding their original design as simple language modeling tools. This was especially notable following the advent of ChatGPT [38]. Stemming from causal language modeling, a plethora of recent studies have concentrated on developing efficient and powerful LLM
base models [16 6 55 53 supervised fine-tuned models [2
12
and models 38
leveraging Reinforcement Learning from Human Feedback (RLHF) 13
Concurrently, there has been growing trend in employing Large Language Models (LLMs) as foundational elements for decision-making systems. These systems either depend on the expressive capacity of transformer architectures
execute imitation learning, thereby modeling complex behaviors [1 22 or they harness the common knowledge embedded within LLMs to facilitate the policy learning process (60 17 [5] 2 However, the dynamic interplay between policy learning and language modeling has been scarcely addressed. Human decision decision-making typically involves both: we draw upon historical policy interaction to refine our policy and also employ our thoughts for strategic
Work done during internship Huawei
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Game Dataset
Language Dataset
STRATEGY ACTION
60
ALPHAZERO STOCKFISH
Data
Computer Game
Rodeling
Porum
Blog
Mixed Dataset
Conversation Dataset
GPT-4
Annotated Game
Youtobe
Instruction tuning
RoddRt Conversation
ChessCLIP
Causal Language
Modeling
ChessGPT-Bass
Language Learning By Causal Language Modeling
Model
ChessG
Chat Ability By Supervised Fine-tuning
3
Aligning by Contrastive Learinng
Modeling Ability
Value udgement
Policy
[TER] nrrracceeoottpppeeeeeeeoo
PPP27PF/RZ2QCCE.. xoke 11
one move with
Evaluation
White large advantage by playing Rís
Rg5á checkmate one move
ChessGPT
ChessGPT
deck
Figure 1: Our work provides comprehensive pipeline that encompasses data, model, and evaluation framework, aiming to foster research on the interaction between policy learning and language learning Data: Our dataset is organized into four categories: game, language, mixed, and conversation datasets. Model: Leveraging this rich dataset, we present two models: ChessGPT and ChessCLIP. Evaluation: Our evaluation framework is structured around three key dimensions: modeling capability, value judgement, and policy proficiency. Refer to Appendix C for image sources.
consideration, mostly in natural language form. Based on this logic, we argue that the study of natural language understanding and policy learning should not be isolated. To advance exploration in this realm, we choose one classic game: Chess, as a practical testbed for initial steps in this direction.
Chess, one of the oldest and most universally played board games, presents ideal testbed due to the wealth of both policy data and language data. In terms of policy data, it is reported that ten million games played daily Chess.com the most frequented online chess platform. Regarding language data, a myriad of chess-related knowledge is readily accessible online in various forms and mediums, ranging from game analysis, books, puzzles, and news, online tutorials, wikis, and even YouTube videos. Building on these resources, we have constructed a comprehensive pipeline dedicated to research chess policy learning and language modeling, illustrated in fig. Specifically, we provide the following three components:
Datasets. We curated large-scale game and language dataset for chess. Our dataset comprises numerous game data from online chess databases recording how humans and chess engines game replay. It also includes language dataset that encapsulates chess knowledge in natural language

---

format, as well as mixed game-language dataset, which offers the most straightforward interrelated data including articles, discussion, or commentary (language) on specific chess game replay (game). Models. We introduce two models, ChessCLIIP and ChessGPT, leveraging our datasets. These models showcase the potential for Al to learn from mixture of replay data and language knowledge. Evaluations. We design an extensive set of tasks to evaluate our models' abilities from three distinct perspectives: modeling ability, to gauge the model's proficiency in tracking game state; value judgement ability, measuring the model's capacity for value assessment and chess knowledge; and policy ability, to test the model's capability in decision-mann makiing. Our experimental results confirm that our models consistently outperform other LLM baselines in all evaluation tasks.
We illustrate our full pipeline in fig. [ Our work primarily pursues two objectives. Firstly, we construct the whole pipeline on chess as an initial step in promoting research on the interaction/in- terplay between policy learning and language learning. as well as on the potential of language as tool for action and understanding. Secondly, our efforts have yielded valuable by-products: the ChessGPT/CLIP models. These models possess practical applicability they could potentially serve as effective Chess AI assistants for humans.
2 Related work
The pursuit of creating artificial intelligence capable of playing chess can be traced back to the very beginning of the history of computer science [56). Chess engines today achieve superhuman-level performance by utilizing human knowledge [9] or self-play [47] Recently, there has been increasing interest in improving the interpretability [32 of these systems and their aligmment with human behavior (33) besides strong performance. A chess engine that aligns with human behavior may unlock many exciting opportunities, for example, they can be used as a personalized tutor for chess beginners 33 Some research efforts also concentrated on employing LLMs to learn policies in Chess
36 50
However, these studies mainly center on small-scale datasets or limited training.
There has been increasing interest in leveraging Internet-scale knowledge for creating agents capable of generalizing across many tasks and capabilities [60 17 44) For example, MineDojo [17 introduced a framework on Minecraft for understanding how to enable artificial agents to learn in an open-ended environment. More recently, there has been a surge in research that treats LLMs as agents, aiming to harness their internet-scale knowledge for decision- contrast to these studies which typically rely on powerful LLMs like GPT-4 [8], our paper concentrates more training, especially the interplay between language modeling and policy learning.
tasks [1 58 62 21] In
3 A large-scale game and language dataset for chess
We introduce large-scale game & language dataset by collecting chess-related materials from the Internet. Our dataset can be mainly divided into four categories: (1) The Game dataset, encompassing online chess match replay data involving worldwide human players and chess engines of varying skill levels. (2) The Language dataset, principally recording associatsd knowledge, analyses, discussions, and news in the form of natural language (3) The Mixed Game-Language dataset, incor- porating both game data and human natural language elements (such as game analysis or comments) in alignment. (4) The instruction-cuning and conversation dataset, consisting of instruction data and conversation data related to chess. We include comprchensive dataset descriptions, statistics, and text examples in Appendix D and the procedure of data collection and pre-processing Appendix E
3.1 Game dataset
Game replay data provide the most direct method for both humans and machines to grasp the play mechanics of Chess. chess, these data are commonly stored in the Portable Game Notation (PGN?) format which is standard plain text format as illustrated in Figure 2 A PGN starts with some headers that include metadata about the game. These headers include information such as the name of players, the Elo ratings, the opening play, and the game outcome. The headers are followed by move text section that records the moves played by the two players in turn. The moves may be further annotated with comments enclosed in braces.
We refer Appendix B for readers who not familiar with chess notation format.
3

---

Previous work [33] uses the moves recorded in PGNs for policy learning. The moves are interpreted as actions in a Markov Decision Process and the state position can be reconstructed by loading the PGN into a chess engine. However, PGNs may contain additional useful information beyond the individual moves made. For example, the Elo ratings in the headers may inform us about the relative strength of the players. Additional information included in the comments of the move text section can also be useful- some of the moves are annotated with evaluations generated by computer chess programs that predict the current advantage of the players. These additional annotations may be useful from a reinforcement learning perspective, e.g., for value function learning. For this reason, curated the game dataset with all of this information intact to facilitate policy learning.
Lichess dataset We collect five months of online game data from the Lichess database [0), culminating in a total of 17.5 million game replay records for online game players.
[White "Alice"]
[Black "Bob"]
[Result "0-1"]
Pro-player dataset In the Lichess dataset, the majority of player Elo-ratings range between 1000 and 2000. To diver- sify our game dataset with more skilled matches, we also incorporated an additional 440,000 game records from 245 professional chess players. These professionals typically hold notably higher Elo ratings within the range of 2000 2800.
[WhiteElo "2100"]
[BlackElo "2000"]
[Opening "Sicilian Defense
1. e4 { [leval 0.1] } 1... c5
N13 2... Nc6
CCRL Chess engines like StockFish and LeelaZero have at- tained a proficiency level far beyond what any human player can currently reach. Considering this, we additionally incor-
13. b3?? 13... NF4? 0-1
Figure 2: Replay example in Portable Game Notation (PGN) format.
porate the Computer Chess Rating Lists (CCRL) (10), which
is dataset of chess games played by computer chess engines. The CCRL dataset comprises considerable collection of chess games, specifically 3 million, all of which are played by computer chess engines and stored in PGN format. The Elo-ratings of chess engines fall in the range of 2800-3700.
Chess puzzles A chess puzzle represents a particular chessboard configuration, designed to present a distinct challenge objective for the solver. Chess puzzles often require players find the best move or sequence of moves to achieve a specific goal, such as checkmating the opponent's king, or finding tactical combination. In our game dataset, integrate 3.2M puzzles soureed from the Lichess puzzle dataset. Each puzzle within this collection is annotated with its rating, theme and solution.
Chess modeling dataset We observe that most chess rule descriptions are conveyed in natural language, posing a challenge for machine learning models since they statistically require a large volume of model data accurately comprehend the chess rules 45.. To address this issue, we build a synthetic chess modeling dataset leveraging the python-chess library [1]. We collect chess game data from one-month dump of the Lichess dataset, deliberately distinct from the month used in our own Lichess dataset. we design several model-based tasks including converting PGN to FEN, transferring UCI to FEN, and predicting legal moves, etc, resulting in 1.9M data samples.
3.2 Language dataset
Existing dataset Numerous existing datasets comprise general internet crawl data from platforms like
CommonCrawl or Wikipedia. We establish a filtering pipeline extract only chess-related language corpus from pre-existing language corpus, including C4
43 Pile [19, Oscar [37], Wikipedia [18]
and RedPajama (53) These datasets extend the scope our language data beyond mere game-play.
Chess blogs Numerous chess websites often publish insightful blogs, sharing their analyses and perspectives on various aspects of chess gameplay. Such blog data is ineredibly valuable, as encompasses game-specific analysis, forming a vital link between the concrete chess game data and its interpretation natural language form. We manually select approximately 30 chess-related websites and scrape 73.2k blog articles.
Chess books Similar chess blogs, chess books can provide long and detailed analysis of the game. We extract approximately 8k chess-related books from online library to enrich our language dataset.
Chess forums Chess forum serves as platform for large amount of chess-related dialogues and conversations involving diverse range of users. These platforms encompass high-quality question- and-answer pairs, seen in platforms like StackExchange, more generalized discussions

---

various chess-related topics, commonly found in dedicated chess-specific forums. We mainly scrape chess forum data from 5 chess-specific forum platforms and StackExchange, using requests and playwright. This process results in collection of 140K posts, representing a wealth of diverse views, queries, and discourses related to the world of chess.
3.3 Mixed game-language dataset
Annotated chess game An annotated chess game is chess game accompanied by written commen- tary and analysis. In an annotated game, each move made by the players is explained and evaluated, providing insights into the thought process, strategic considerations, and tactical ideas behind the moves. Here an example an annotated PGN with Sicilian Defense opening:
lee4 c5 (The game starts with the Sicilian Defense, one of the most popular and aggressive responses 1.e4. Black aims to control the center and create imbalances early on.]
These annotated games inherently maintain the correspondence between board state and human language, serving as exceptionally high-quality data source to align model with complex human intentions and judgements. We amass annotated games from seven sources, five of which are collected from the internet while the rest two are commercial datasets. In total, we collect 245K annotated games with 1.3M loard-language pairs.
Youtube transcripts Drawing inspiration similarly from MineDoJo 17 YouTuuee video can naturally serve as mixed game-language dataset by aligning video clips with natural language transcripts based on timestamps. Rather than generating image-language pairs directly, we develop a pipeline that accurately applies OCR (Optical Character Recognition) to chessboard screenshots to generate FEN (Forsyth-Edwards Notation), system that describes the chess state in a language format. We gathered around 83k chess videos, resulting million-scale English transcripts and board-langee board-language pairs, thus establishing substantial mixed game-langure dataset.
3.4 Instruction-tuning & conversation dataset
Supervised fine-tuning is a crucial component to train large language model (LLM) follow instructions [38 334 [1]. In addition the comprehensive chess materials mentioned before, we also collect instruction-tuning and conversation datasets which can be used to finetune the pre-trained LLM base model, thereby enhancing its instruction-followigg and dialogue capability.
instucttion-uning data from GPT-4 Inspired by Alpaca [52] we use the self-instruct technique [59] to generate high-quality, instruction-followe construct 200 seed prompts for chess-related questions or instructions. These prompts serve as few-shot examples, guiding GPT-4 towards more coherent and relevant generation. Finally, we generate around restruction-response pairs using this pipeline.
data through GPT-4 [8]. Specifically, we manually
Conversation data from Reddit The instruction data collected from GPT-4 are mainly in a single- step form, which means only one round of question-answer pair included. To mitigate this issue, we collect multi-step conversation data about chess on Reddit. Reddit allows users to interact by commenting on posts and responding other comments, creating nested structure of responses. This nested structure can be easily converted to conversation tree by treating the comment's 's reply as a child node for that reply. rich source of conversation data can then be acquired by navigating from the root node to each leaf node via every available path. In all, we choose chess-related sub-reddits and collect 410k human conversations about chess.
Large-scale pretraining
We will showcase two models ChessCLIP and ChessGPT trained the large-scale dataset.
4.1 ChessCLIP
a neural network trained on a variety of modalities (e.g. image, text). By conducting contrastive learning large amount of paired data, CLIP bridges the image and language modality, enabling the model to understand vision by language information and vice versa. Our mixed game- languae dataset in Section 3.3 has similar paired
CLIP (Contrastive Language-Image Pre-Training) 42
5

---

structure because the annotation is naturally paired with its preceding game trajectories. Based on this subset, we can train a ChessCLIP to bridge the modality of policy and language. Specifically, by denoting the chessboard state S at timestep /. as St, and the annotation language as It,, the data pair at timestep T can be represented by (((S))EE-=:: aT), I-r) where {Si) is a stacked history states and @T is the last move.
We want to emphasize more on what ChessCLIP can do by aligning the policy modality and the language modality. Firstly, ChessCLIP offers a similarity metric given one PGN and a text description. Just like the application of large-scale image/text retrieval using CLIP, ChessCLIP can help users conduct PGN/text retrieval search for game based on text or search for comments based on specific game. In addition, because of the now-dimensional feature of action space compared vision or language space (there only exists a few legal moves for a given chess state ), we can directly conduct search algorithms to maximize the similarity to generate action based on one text description using ChessCLIP. For example, given chessboard state and a text description, ChessCLIIP can generate a move by iterating through all legal moves and finding one move that returns the largest similarity. By the same logic, ChessCLIP can directly generate move sequences (multiple actions) using greedy search or beam search. We refer the reader to Appendix F.I.I for more discussions.
Implementation details We preprocess the annotated PGNs to produce board/text pairs which we feed separately to the board and text encoders. In particular, for every move in the PGN, we extract the comments attached to the move as well as the board state. We encode the board positions and moves using the same scheme as those used by Leela Chess Zero (lc0) 266.. which is similar the encoding used by AlphaZero 47 for encoding positions and moves in chess. Concretely, the board positions are encoded as a R R528x112 feature map and the actions are encoded as R888 vector. We instantiate ChessCLIP model with a pair of text encoder and board/action encoder. For the text encoder, we only fine-tune the last two layers of pretrained text encoder from OpenAI CLIP model. For the board/action encoder, we use ResNet (20] architecture that conditions on the action encoding via a modified FiLM layer 40) Please refer to Appendix F.I.I for implementation details.
4.2 ChessGPT
The Generative Pretraining Transformer (GPT-3) 7 is an autoregressive language model that uses deep learning techniques to generate human-like text. GPT-3 is trained by casual language modeling, which aims to predict the next word in sentence given all the previous words. Following the same logic, we train a GPT-like model using all chess materials introduced in Section Unlike other policy behavior data in robots [27] or video games (35]. the chess state and move data be represented in merely textual format. Thanks to this feature, we can directly treat chess as a text game and the imitation learning objective for policy learning can be directly covered by casual language modeling
over the game dataset provided in Section 3.1
Implementation details We follow common implementations of training domain-specific fostructionnfollooiing LLM. Firstly we conduct base-model fine-tuning using chess corpus introduced
in section 3.1 and 3.3 Due to computational constraints, we choose finetune the RedPajama-
3B-base [53] model, which is an open-souce replication of LLaMA (55). The base model adopts the
GPT-NeoX 6 architecture, GPT-3 (7) variant with a few modifications such as rotary positional
embedding, parallel attention computation, and different initialization. The base-finetuning brings us our base model: ChessGPT-Bars After base- fisettiinu,,, we conduct supervised fine-tuning by supervised learning on question/conversatiin response using data introduced in section and general conversation data from OASSTI (24). Dolly2 14, Alpaca-GPT4 (39), and Sharegpt 44).
forming our chat model: ChessGPT-Chat. We leave further RLHF (Reinforcement Learning from Human Feedback) training for future work. Refer to Appendix F.1.2 for more details.
5 Evaluation and benchmark
In this section, we present a comparative analysis between ChessGPT trained on our database with other baseline LLMs. The purpose of our experiments is to assess the performance of ChessGPT in three primary dimensions: Chess modeling ability, Value judgement ability, and Policy ability. The Chess Modeling capability focuses on the language model's proficiency in accurately tracking the game state and predicting valid moves. Regarding the Value judgement ability, we assess the model's precision in evaluating the worth of a chess game, encompassing the identification of ad advantagcous

---

Table 1: Bigbench State Tracking in Chess
LLM Models (%)
Tasks
ILAMA-7B
RedPajama--art
ChessGPT-Base
ChessGPT-Chat
Real Short
29.5 + 1.4
23.2 = 1.3
99.5 + 0.2
98.5 + 0.4
Real Med
39.3 + 1.5
38.2 = 1.5
97.7 + 0,5
97.8 + 0,4
Real Long
53.0 + 1.6
51.9 = 1.6
98.1 + 0.4
97.6 + 0.4
Syn Short
31.3 1.4
24.9 1.3
94.2 0.7
92.3
Syn Med
39.9 + 1.6
37.7 = 1.5
94.6 + 0.7
88.9 +
Syn Long
45.8 + 1.5
42.2 = 1.5
92.8 + 0.8
85.1 + 1.1
Table 2: UCI to FEN
LLM Models (%)
Tasks
LLAMA
RedPajama Base
ChessGPT- Base
ChessGPT-Char
Real Short
2.2 + 0.0
5.2 + 0.0
95.1 + 0.3
95.7 0.1
UCI to FEN
Real Med
2.3 0.1
4.0 + 0.1
89.9 = 0.2
88.6 + 0.3
Real Long
1.8 0.0
3.8 + 0.1
85.7 = 0.2
81.4 + 0.5
Real Short
6.0 0.1
2.8 + 0.1
96.7 + 0.1
95.8 + 0.1
PGN FEN
Real Med
5.6 +
3.4 0.1
94.8 0.1
93.6 0.1
Real Long
5.7 0.0
8.9 1 0.1
95.4 + 0.2
82.7 1.0
positions and the calculation of situation scores. Lastly, the Policy capability gauges the model's aptitude for generating optimal moves based on a given position. By thoroughly examining these sub-categories, we can comprehensively evaluate and contrast the efficacy of different models in chess-related tasks. We choose the following models as baselines: LLaMA-7B (55), RedPajama- Base-3B (53), and compare them with ChessCLIP, ChessGPPTBBae-3311 and ChessGPT-Chha--33 To help readers who are not familiar with chess, we provide task examples and illustrative figures to elucidate these evaluation tasks in Appendix G.1
5.1 Chess modeling ability
Chess state tracking We utilized Big-bench's State Tracking in Chess task [49 54
evaluate
language models' ability to track the state of chess games encoded in UCI notation. The task involves predicting the legal ending square given the game prefix and starting square of the current move. For example, if the input UCI notation is f224 d7d5 gl", the expected output would be 1")3"," ; the chess piece square gl can only move to those real and synthetic games, divided into short, medium, and long categories based on move count. The evaluation measures correctness across all games using a specified output regex. Notably, the ChessCLIP is unsuitable for modeling tasks, so we do not include it in the comparison.
" "f"",, positions. The task dataset includes
Table presents performance analysis of all models the task. Our Base and Chat models consistently outperformed baselines in all tasks. This indicates their strong ability to track the state of chess games. However, the ChessGPT-Cha. model exhibited slightly lower performance, suggesting a potential trade-off between language capabilities and state tracking. Nevertheless, the results underscore the effectiveness of our dataset-trained L.LM models for chess state tracking.
Board state tracking We performed additional evaluations involving UCI to FEN and PGN to FEN conversions. In the UCI to FEN experiment, the target was replaced with FEN format, while in the PGN to FEN experiment, UCI was converted to PGN format as input and the target was replaced with FEN format. The similarity was measured using Levenshtein distance, which normalized to range of to 1 (33) These evaluations focused on assessing the model's capability to track the overall state of the chessboard by representing the state of each chess piece using FEN notation.
Table 2 illustrates the results of these evaluations. evident that compared to tracking the state of an individual chess piece, tracking the entire chessboard state becomes more challenging. The similarity scores between the two baselines were consistently below 10%, indicating lack of global chess piece state tracking ability. However, the ChessGPT achieves an average similarity score
Thhe model size is 2.8B and 3B is rounded representation
7

---

Table 3: State value multi-choice
Models (%)
Prompt Setting
LLAMA
RedPajama
ChessGPT-Base
ChessGPT-Chat
ChessCIII
W/O { suffix
33.2 + 0.7
31.1 + 0.7
43.1 + 0.8
52.8 + 0.8
N/A
With { suffix
26.9 + 0.7
29.7 = 0.8
53.7 + 0.8
53.5 + 0.8
38.1 + 0.8
Table 4: Chess Annotation Multi-choice
Models (%)
Prompt Setting
LLAMA
RedPajama
ChessGPT-Base
ChessGPT-Chat
ChessCLIP
W/O { suffix With { suffix
29.8 + 0.8
27.4 + 0.7
33.2 0.9
35.7 + 0.9
N/A
29.6 0.8
28.4 + 0.8
38.8 + 0.9
34.7 + 0.9
63.6 + 0.9
higher than 90%. These results demonstrate that our dataset-trained model excels in capturing and reproducing the global chess piece state in both UCI to FEN and PGN to FEN conversions.
5.2 Value judgement ability
In this part, we evaluate the model's ability of value judgement. Specifically, we want to assess the model from two perspectives: (1) its ability to align with the true value function given a chessboard state (the true value are evaluated by chess engines in enough search depths) in the evaluation of State value multi-choice, and (2) ability to align with human judgement and human knowledge in the evaluation of Chess Annotation Multi-choice and Opening multi-choice.
State value multi-choice Here we evaluate the model's ability see whether can determine which side holds the advantage for a given PGN. We construct an evaluation dataset consisting of 3000 game snippets and utilize Stockfish 15 with a depth of to calculate the winning rate for the white pieces. By categorizing the winning rate into three intervals: 0 - 33% for black advantage, 34 - 66% representing balanced state, and 67 100% for white advantage, we construct the state-value multiple-choice task. During experiments, we discovered that an additional "(" suffix to the prompt can significantly enhance the performance of the base model. This due to "(" consistently serving as the initial symbol for annotation in annotated PGNs. Consequently, we carried out our evaluation under two distinct prompt settings and report our results w.r.t multi-choice grade shown in Table
Chess annotation multi-choice The annotations within an annotated PGN can be viewed as reflection of human evaluative judgement. To examine the degree which the model's value aligns with human value, we extract 3k game-language pairs from the annotation dataset as the test set. By randomly selecting three annotations from the test set as incorrect options, we construct the chess annotation four-choice task. We report the multi-choice grade results over two prompts in Table 4
Opening multi-choice A chess opening refers to the initial moves made by players at the beginning of a chess game. There are numerous chess openings, each with its own name, characteristics, and strategic goals. For example, the Sicilian defense: 1. e4 c5 is one of the most popular and aggressive chess openings for Black. We use the Lichess opening dataset (29] including 3.5k opening PGNs and their corresponding names, to formulate two tasks: (1) PGN2Opening five-choice task, which aims at choosing the correct opening name for given PGN, and reversely, (2) Opening2PGY five-choice task, aiming at choosing the correct PGN for given opening name. We report the result in Table
In general, our trio of models surpasses the performance of two baseline language models across these four tasks in all settings. This result confirms that models more effectively aligned with both the true value function and human judgemnent/knowledee. ChessGPTT-cht deliver outstanding performance in the state-value task and the opening task. Notably, ChessCLII displays a surprisingly high level of proficiency in the annotation task and the opening task. This result reveals the model's 's capacity to extract human judgement and knowledge solely from annotations, even without training in any actual chess games.
Both ChessGPT-Baa and
5.3 Policy evaluation
Checkmate one We incorporate the checkmate-in-one task from Big-Bench (49 into our evaluation methods. This task is designed to challenge the model's ability to identify move in given PGN
8

---

Table Opening2PGN and PGN2Opening
Models (%)
Prompt Setting
LLAMA
RedPajama
ChessGPT-Base
ChessGPT-Chat
ChessCIII
Opening2PGN
43.0 + 0.9
26.5 0.8
92.2 + 0.5
94.7 + 0.4 55.8 + 0.9
73.0 + 0.8 80.5 + 0.7
PGNZOpening
20.0 0.7
20.7 = 0.7
49.3 + 0.9
Table 6: Checkmate in One
Models (%)
Setting
LLAMA
RedPajama
ChessGPT Basse
ChessGPT-Chr
ChessCLIP
With suffix (ESM)
1.6 1.6100 0.2
0.0 0.0
71.4 1 0.7
56.8 0.8
N/A
With suffix (MC)
2.3
0.0
66.1 = 0.8
11.3 0.5
2.9 = 0.3
W/O suffix (ESM)
1.7 0.2
0.0 0.0
26,5 = 0,8
59.4 + 0.8
N/A
W/O suffix (MC)
2.2 0.3
0.0
13.6 = 0.6
15.4 - 0.6
N/A
that would result in a checkmate. By doing so, it measures the model's capacity to comprehend and apply the rules chess. The model is essentially required to discern move that not only places the opponent's 's king under attack but also ensures that the king cannot evade capture in the next move.
We also find adding an additional instruction suffix like /Now white/black can checkmate one) can largely enhance the base model performance. We report the result in two prompts with two metrics (exact-string match as ESM and multi-choice- grade as MC) in Table O our ChessGPT- Base model and ChessGPT-Cher model show really great checkmate ability by surpassing two LLM baselines by large margin. ChessCLIP does not perform well in this task, because there does not exist much annotation data regarding checkmate-in-one behavior in the annotation dataset.
General policy In order to assess the model's generalization ability, we introduced Elo Rating as a factor in the task, aiming to evaluate its capacity to identify PGN and related keywords and generate the appropriate next move within the specified skill level. Model's 's selection of the next legal move is assigned a move score, which is normalized based on the win rate observed in the raw data. Table prresents the results representing the performance of different models in selecting the most suitable move for white chess. Notably, all models surpassed the performance of the random policy (~2 50%) the Elo Ratings correspond relatively high skill levels among human players.
Table 7: Elo Rating 1700-2000
LLM Models
Move Score
LLAMA
55.1 + 1.1
RedPajama
56.4 + 0.9
ChessGPT-
59.6 + 1.0
ChessGPT-
60.3 = 1.0
Further analyzing the performance of different models across varying Elo Ratings is crucial for understanding the observed results. The minor variations in move scores for different Elo Rating scenarios in Table 8 indicate that ChessGPT-Base may struggle to effectively incorporate Elo Rating information into its poocision-making process. This could be due to the model's limited understanding of the nuanced characteristics associated with distinct Elo Ratings. The complexity of the task and the challenges in accurately accounting for diverse playing styles further contribute to the limited variations in move scores across different Elo Ratings. Consequently, neglecting this information can lead to the model learning an average policy for each Elo Rating, resulting subpar overall performance. Similar findings were observed in the black chess test, and to further validate this viewpoint, we conducted input attention visualization. Refer to Appendix F.1.2 for more details.
To clarify, the dataset we have presented encompasses a wide range of games and varying Elo ratings, as shown in Figure Z which possesses the potential to effectively capture and generalize intricate patterns and policies associated with different Elo levels. However, the current training method might not sufficiently emphasize these nuanced features. This highlights a potential direction for future research, which involves enhancing the model's ability to better integrate and utilize metadata such as Elo Rating and other auxiliary data. By addressing these aspects, the model's 's overall generalization can be further improved.
Table 8: ChessGPT-Base in Different Elo Rating Results
Elo Rating
Move Score
700-1000
59.4 1: 1.0
1200-1500
58.9 1: 0.9
1700-2000
59.6 + 1.0
2700-3000
59.8 + 1.0

---

5.4 Qualitative results
We also perform qualitative comparison between our models (ChessGPT--Cha and ChessGPT-Bars and the baselines. We ask the language models series of questions ranging from factual knowledge of chess as well requesting the models to perform some operational tasks related chess. We found that ChessGPT-bbas performed similarly to RedPajama: both models can sometimes produce factual answers for some of the questions but they failed to generate coherent answers when asked to perform tasks such as providing commentary chess moves or converting the PGN notation to FEN. ChessGPT- Chaa gives more factual answers and demonstrates better performance when prompted to generate analysis and perform other chess-related tasks. Refer to Appendix H for qualitative analysis.
6 Conclusion
In this paper, we introduce a new large-scale dataset and benchmark on chess to encourage of study of the interplay between historical policy data and natural language knowledge. We accompany our dataset with an evaluation framework for assessing language models' capability in chess. We showcase two models, ChessCLIP and ChessGPT, that demonstrate promising results for learning the interplay between language and action. Nevertheless, our results indicate that we are only beginning understand how to bridge the gap between policy learning and language modeling and we discuss more on the future directions of our dataset in Appendix We hope that our dataset and benchmark can make future research on policy and language alignment more accessible.
10

---

References
[1]
[2] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et Do as I can, not as say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691 2022.
[3] Yuvanesh Anand, Zach Nussbaum, Brandon Duderstadt, Benjamin Schmidt, and Andriy Mulyar. GPT4AIl: Training an Assistant-style Chatbot with Large Scale Data Distillation from GPT-3.5- 2023.
Turbo. ottps://githut http://ggttuuu..ccoooimi...cc. hatmm//mmiaaaagttaaaaaa
[4] Bowen Baker, lige Akkaya, Peter Zhokov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (vpt): Learning by watching unlabeled online videos. Advances in Neural Information Processing Systems, 04224639-2465- 2022.
[5] Stella Biderman, Hailey Schoelkopf, Quentin Authony, Herbie Bradley, Kyle O BrBrien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: Suite for Analyzing Large Language Models Across Training and Scaling, 2023.
[6] Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al. GPTT-NeoX-00B An Open- Source Autoregressive Language Model. arXiv preprint (xXxi:22204067455 2022.
[7] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 33:1877-1901, 2020.
[8] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with GPT-4. arXiv preprint arXiv:2303.127712 2023.
[9] Murray Campbell, A. Joseph Hoane, and Feng-hsiung Hsu. Deep Blue. Artificial Intelligence, 134(1):57-83, January 2002.
[10] CCRL. https://www cow.terrcesctttccccccccc org. uk/ccrl/
[11] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084-15097,
2021.
[12] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality, March 2023.
[13] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances neural information processing systems, 30, 2017.
[14] Dahoas.
Dolly2.
https: httee///aaggiingaaccc///sttttttt
databricks - dolly-151
[15] Yuqing Du, Olivia Watkins, Zihan Wang. Cédric Colas, Trevor Darrell, Pieter Abbeel, Abhishek Gupta, and Jacob Andreas. Guiding pretraining in reinforvement learning with large language models. arXiv preprint arXhv:2302.0669) 2023.
[16] Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume I: Long Papers), pages 220-335,10 2022.

---

[17] Linxi Fan, Guanzhi Wang. Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang. Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with snternet-scale knowledge. arXiv preprini arXivv2206.08883,, 2022.
[18] Wikimedia Foundation. Wikimedia Downloads. https: wikimedia. org
[19] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang. Horace He, Anish Thite, Noa Nabeshima, al. The Pile: 800GB dataset of diverse text for language modeling, arXiv preprint (efiv:2001.0002)) 2020.
[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, Las Vegas, NV, USA, June 2016. IEEE.
[21] Sirui Hong. Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau. Zijuan Lin, Liyang Zhou, Chenyu Ran, et al. Metagpt: Meta programming for multi-agent collaborative framework. arXiv preprint arXiv:2308 00352, 2023.
[22] Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in neural information processing systems, 34:1273- 1286, 2021.
[23] Dai Juntao, Ji Jiaming, Pan Xuehai, Sun Ruiyang, Wang Yizhou, and Yang Yaodong. Con- strained Value-Aligned LLM via Safe RLHF, May 2023.
[24] Andreas Köpf. Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, et al. Ope- nAssistant Danneeraiooss-eemrraiiiiingg arXiv:2304.07327, 2023.
Democratizing Large Language Model Alignment, arXiv preprint
[25] Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Seao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, Jörg Frohberg, Mario Saško, Quentin Lhoest, Angelina McMillan-Major, Gérard Dupont, Stella Biderman, Anna Rogers, Loubna Ben allal, Francesco De Toni, Giada Pistilli, Olivier Nguyen, Somaieh Nikpoor, Maraim Masoud, Pierre Colombo, Javier de la Rosa, Paulo Villegas, Tristan Thrush, Shayne Longpre, Sebastian Nagel, Leon Weber, Manuel Romero Muñoz, Jian Zhu, Daniel Van Strien, Zaid Alyafeai, Khalid Almubarak, Vu Minh Chien, Itziar Gonzalez-Dios, Aitor Soroa, Kyle Lo, Manan Dey, Pedro Ortiz Suarez, Aaron Gokaslan, Shamik Bose, David Ifeoluwa Adelani, Long Phan, Hieu Tran, Ian Yu, Suhas Pai, Jenny Chim, Violette Lepereq. Suzana Ilie, Margaret Mitchell, Sasha Luccioni, and Yacine Jernite. The bigscience ROOTS corpus: A 1.6TB composite multilingual dataset. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
[26] Leela Chess Zero. https://lczero: /rrg/
[27] Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):1334-133,, 2016.
[28] Liunian Harold Li*, Pengehuan Zhang*, Haotian Zhang* Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang. Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai-Wei Chang, and Jianfeng Gao. Grounded language-image pre-training. In CVPR, 2022.
[29]
Lichess chess opening names.
[30]
Lichess Developers. Lichess.
[31] Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating Ilms as agents. arXiv preprint arXiv:2308.0m 03688, 2023.
[32] Thomas MeGrath, Andrei Kapishnikov, Nenad Tomašev, Adam Pearce, Martin Wattenberg, Demis Hassabis, Been Kim, Ulrich Paquet, and Vladimir Kramnik. Acquisition of chess knowl- edge AlphaZero. Proceedings of the National Academy of Sciences, 119(47)::2206625119 19, November 2022.
12

---

[33] Reid Mcllroy- Young, Siddhartha Sen, Jon Kleinberg, and Ashton Anderson. Aligning Super- human AI with Human Behavior: Chess Model System. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD '20, pages 1677-1687, New York, NY, USA, August 2020. Association for Computing Machinery.
[34] Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannanch Hajishirzi. Cross-task gener- alization via natural language crowdsoureing instructions. arXiv preprint arXiv:2104.08773, 2021.
[35] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing Atari with deep reinforcement learning. arXiv preprint 112.5:13 2013.
[36] David Noever, Matt Ciolino, and Josh Kalin. The chess transformer: Mastering play using generative language models. arXiv preprint wxXi::2008.0405: 2020.
[37] Pedro Javier Ortiz Su' arez, Benoit Sagot, and Laurent Romary. Asynchronous pipelines for processing huge corpora on medium low resource infrastructures. In Piotr Bański, Adrien Barbaresi, Hanno Biber, Evelyn Breiteneder, Simon Clematide, Mare Kupietz, Harald L Luungen, and Caroline Iliadi, editors, Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-7) 2019, Cardiff. 22nd July 2019, pages 9 - 16, Mannheim, 2019. Leibniz-Institut frur Deutsche Sprache.
[38] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
[39] Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction Tuning with GPT-4, arXiv preprint arXiv:2304.0327 2023.
[40] Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron Courville. FiLM: Visual reasoning with a general conditioning layer. In Proceedings of the Thirty-Secon AAAI Conference on Artificial Intelligence and Thirlieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, TAAT18/AAATTTS February 2018. Pres.. [41] python-chess: a chess library for python.
18, pages 3942-3951, New Orleans, Louisiana, USA,
[42] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning Transferable Visual Models From Natural Language Supervision, February 2021.
[43] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv e-prints, 2019.
[44] Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent, arXiv preprint arXiv:2205.06173 2022.
[45] Julian Schritwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Si- mon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering Atari, Go, chess and shogi by planning with a learned model. Nature, 588(7839):044-699 2020.
ShareGPT, unfiltered
ShareGPT_Vicuna Vicuna
13

---

[47] David Silver, Thomas Hubert. Julian Schrittwieser, loannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre. Dharshan Kumaran, Thore Graepel, Timothy Lillicrap. Karen Simonyan, and Demis Hassabis. A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science (New York, N.Y.J, 162(6419 2018.
[48] Alexei Barantsev Simon Stewart. Selenium, March 2023.
[49]
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonss al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint MxKi::2006.06615. 2022.
[50] Andreas Stöckl. Watching a language model learning chess. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 202!), pages 1369- 1379, 2021.
[51] Tianxiang Sun and Xipeng Qiu. Moss, March 2023.
[52] Rohan Taori, Ishaan Guirajani. Tianyi Zhang. Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford Alpaca: An -ootruction-folloiing LLaMA model. attpss//ge thub. com/tatsu-la lastsuullaasntnatttaaa swtaaford_aaappae
2023.
[53] Together Computer. RedPajama: An Open Source Recipe to Reproduce LLaMA training dataset. http://gggghaabb.cccc/oogeeeceer
Data, april 2023.
[54] Shubham Toshniwal, Sam Wiseman, Karen Livescu, and Kevin Gimpel. Chess as Testbed for Language Model State Tracking. In Proceedings of the AAAI Conference on Artificial Intelligence, 2022.
[55] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie- Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
[56] Alan Turing. Digital computers applied to games. Faster than thought, 1953.
[57] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[58] Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprini arXiv: Arxiv-2305. 16291, 2023.
[59] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi.
Aligning Language Model with Self Generated Instruc-
tions, 2022.
[60] Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. arXiv preprini arXiv:2302.00 01500, 2023.
[61] Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.0166 2021.
[62] Hui Yang, Sifu Yue, and Yunzhong He. Auto-gpt for online decision making: Benchmarks and additional opinions. arXiv preprint arXiv:2306.02224, 2023.
[63] Li Yujian and Liu Bo. A normalized levenshtein distance metric. IEEE transactions on pattern analysis and machine intelligence, 199)1100 2007.
[64] Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. ChatGLM-6B, March 2023.
14

---

Appendix
Acknowledgement
17
Different Chess Formats
17
B.1
Universal Chess Interlace (UCI)
17
B.2 Standard Algebraic Notation (SAN)
17
B.3
Portable Game Notation (PGN).
17
B.4
Forsyth-Edwwid Edwards Notation (FEN)
17
Image sources
18
D Dataset analysis
18
D. Dataset statisties and metries
18
D.2 Dataset examples
19
Data Collection and Preprocessing
26
Online Chess websites
26
Online Chess forums
26
Annotated PGN
26
Existing Datasets
27
E.S
YouTTube transcripts dataset
27
E,6
Lichess dataset
28
Pro-player dataset
28
E.8
Chess books
29
CCRL
29
10
Chess puzzles
29
E.
11 Chess modeling data
29
12
Preprocessing
29
Licenses and datasel cards
30
Implementation and Evaluation Details
30
Implmenetation details
30
Evaluation details
32
G.1
Evaluation task examples and plots
32
H
Qualitative Analysis
36
Experimental set-up
36
H.2
General knowledge
37
H.3 Chess modeling
ChessCLIP visualization
46
15

---

J
Potential directions
46
Limitations and Potential Societal Impact
47
16

---

A
Acknowledgement
We thank Jiacheng Liu for his work on collecting chess-related data and chess book list.
B Different Chess Formats
B.1 Universal Chess Interface (UCI)
The UCI format widely used for communication between chess engines and user interfaces, represents chess moves by combining the starting and ending squares of a piece, such as "c2e4" to indicate moving the pawn from e2 e4, For example, the UCI notation for a full game would be:
e2e4 c7c6 g1f3 d7d5 e4d5 c6d5 d2d4 b8c6 c2c4 gêf6 bic3 c8e6 c4c5 g7g6 cif4 f8g7
f1e2 16e4 elg1 e6g4 f3e5
g4e2 d1e2 c6e5 c3e4 e5c6 e4d6 e8f8 fiel g7d4 f4h6 d4g7
h6g7 f8g7 d6b5 a7a6 b5c3 d5d4 c3e4 d8d5 a2a3 a8d8 b2b4 h7h6 e2
B.2 Standard Algebraic Notation (SAN)
SAN (Standard Algebraic Notation) is a widely used notation system in the game of chess for recording and describing moves. provides standardized and concise representation of moves that easily understood by chess players and enthusiasts. SAN, each move represented by two components: the piece abbreviation and destination square. The piece abbreviation a letter that represents the type piece making the move, such as "K" for king, "Q" for queen, "R" for rook, "B" for bishop, "N" for knight, and no abbreviation for pawns. The destination square is denoted by combination of letter (a-h) representing the column and a number (1-8) representing the row the chessboard. Additional symbols may be used indicate specific move types. The symbol "+" is used indicate a check, while "#" denotes checkmate. Castling moves are represented by "O-0" for kingside castling and "0-0-0" for queenside castling.
B.3 Portable Game Notation (PGN)
PGN is widely adopted format for recording chess games. includes not only the SAN but also additional information like player names, event details, and game results. PGN files are human-readable can be easily shared and analyzed. Here is an example of a PGN representation a full game:
[Event "World Chess Championship"] (Site "London, England"]
[Date "2023.05.20") [Round "1"] [White "Carlsen, Magnus"] [Black "Nepomiiachtchi, Ian"] [Result "1/2-1/2"]
1. e4 e5 2. Ne3 Nc6 3. Bb5 a6 4. Ba4 Nf6 5. 0-0 Be7 6. Re1 b5 7. Bb3 d6
8. c3 0-0 9. h3 Nb8 10. d4 Nbd7 11. Nbd2 Bb7 12. Bc2 Re8 13. Nf1 BF8
14. Ng3 g6 15. a4 c5 16. d5 c4 17. Bg5 h6 18. Be3 Nc5 19. Qd2 h5 20. Bg5 Bg7
Nh2 Qc7 22. Rf1 Nh7 23. Bh6 Bh8 24. f4 exf4
Bxf4 Nf6
Rael bxa4
27. N13 Nfd7 28. Bh6 Ne5 29. Nxe5 Bxe5 30. Rf3 Qb6 31. Kh1 Qxb2 32. Ref1 Re7
33. Bg5 Rd7 34. Bi6 Bxf6 35. Rxf6 a3 36. Nxh5 a2 37. Qh6 gxh5 38. R6f3 h4
39. Rf4 f5 40. Rxf5 Rg7 41. Rh5 1-0
B.4 Eowsyth-Edwards Notation (FEN)
FEN is a notation system used to describe the state of a chess game. It represents the positions of pieces on the chessboard, active color. castling rights, passant targets. and the half-move and full-move counters. Here is an example of FEN notation representing the starting position:
KQkq
0
In this FEN notation, each letter represents piece on the board, with uppercase letters representing white pieces and lowercase letters representing black pieces. The forward slash ("/") separates ranks, and the number after
17

---

each rank indicates the number of empty squares. The active color is represented by "w" for white "b" for black. The castling rights denoted by "K" (white kingside), "Q" (white queenside), "k" (black kingside), and "q" (black queenside). The passant target square indicated with corresponding square, "-" there no en passant target. The half-move and full-move counters specify the number of half-moves since the last pawn move or capture and the current move number, respectively.
These different chess formats serve various purposes, from representing individual moves (UCT) to recording entire games (PGN) and describing specific positions (FEN). Understanding and working with these formats is essential for tasks like parsing, analyzing, and exchanging chess game data in different contexts.
C
Image sources
Here provide the sources for images used in our Figure 1
Online game: https://liches.oorgtt
Pro game: http://wwww..aatttt..o//o/ss///e whwesraaaaeeseesseeoooeoeee www.d-wwp-dddddeede Computer g match
Modeling: http://wwwwccasss.coommmeerreees
Puzzle: httss:/cchesspuzzle..ee/
General text: http:///www.bbbb....////////////
Wiki: https//en..wkkippcitt.o.giiiiis
Forum: https:/wwww.cdess.comm/uuuu
Blog: https//www.cdhes.coombbbggg
Book: httk:///wwwwaaooooooo///////////
Annotated game: http:///wwwwp/wacccssss/o//////m
ruws
Youtube video: http://wwwwwyyouuccccow/wwwttttt
Reddit: http::/wwwwwaaalloo////////////o
and reach 2000 2000dceescomiivqqp///oootttss
Value judgement:
D Dataset analysis
D.1 Dataset statistics and metrics
InTable we present the dataset statisties breakdown for cach data subset, including its raw size, document count, and subset type.
Table shows the properties of the chess-specific language dataset that we use for training ChessGPT. For these datasets, we computed the average number of words (num. words) per example, character repetition ratio (char. rep. ratio), word repetition ratio (word. ratio.), special character ratio (special char. ratio), stopwords ratio and perplexity of the first shard for each subset in the language dataset. These metrics based of the criteria employed by Bloom 25 their pre- processing pipelines.
18

---

Table 9: Dataset statistics
Component
Raw size
Document count
Subset type
Lichess
19.9 GB
17.5 M
Game
Pro-player
0.37 GB
0,44 M
Game
CCRL
3,60 GB
3.00 M
Game
Chess puzzles
1.53 GB
3.19 M
Game
Chess modeling
0.94 GB
1.90
Game
C4-Chess
0.59 GB
0.18
Language
Pile-Chess
1.10 GB
0.10 M
Language
RedPajama-Chess
5.65 GB
0.52 M
Language
Oscar-Chess
3.13 GB
0.33 M
Language
WikiPedia-Chess
40.3 MB
11.4 K
Language
Chess Blogs
0.59 GB
73.2 K
Language
Chess Books
1.86 GB
8.36 K
Language
Chess Forums
1.05 GB
0.14 M
Language
Annotated Chess Games
0.66 GB
245 K
Mixed
Youtube transcripts
0.98 GB
83.0K
Mixed
GPT-4-Chess
0.95 MB
3.91 K
Conversation
Reddit
0.86 GB
0.41 M
Conversation
Overall
42.3 GB
28.1 M
N/A
Table 10: Metrics for the language dataset.
num. words
char. rep. ratio
word rep. ratio
special char
stopwords ratio
perplexity
datasct
ratio
Chess puzzles
49.6618
0.0104
0.0000
0.3246
0.4771
988.3734
Bscar-Chess
1441.434)
0.0596
0.0499
0.2242
0.4119
665.6499
Piee-Ches
2105.2455
0.0626
0.0205
0,2409
0.4227
497,3883
RedPajama-Chess
1581.5825
0.0532
0.0163
0,2273
0.4218
410.1236
StackFxchange-Cheee
578.3733
0.0591
0.0816
0.2617
0.4835
1257
Wikipedia-Chess
463.4980
0.0876
0.0052
0.2604
0.3079
236.9671
C4-Chess
510.6041
0.0479
0.0082
0,2131
0.4418
548.9471
D.2 Dataset examples
In this subsection, we offer dataset examples for each subset to help the readers understand its contents more intuitively and clearly.
19

---

Lichess
[Event "Rated Bullet tournament https://liche org/ http.oourrrammee//yy//00::: www.chesss.or/pp///gggggg.
[Site
[Date "2017.04.01"]
[Round "-"]
[White "Abbot"]
[Black "Costello"]
[Result "O-1"]
(UTCDate "2017.04.011
[UTCTime "11:32:01"]
[Whiteßlo "2100"]
[BlackElo "2000"]
(WhiteRatingDiff "-4"]
[BlackRatingDiff "+1"]
[Whitelitle "FM"]
[ECO "B30"]
[Opening "Sicilian Defense: Old Sicilian"] [TimeControl "300+0"]
[Termination "Time forfeit"]
1. e4 [eeaal 0.17] }
[eeval 0. }
2. Nf3 { [Keval 0.25] 2... Nc6 [%eval 0.33] }
3. Bc4 { [%eval -0.13] } 3... e6 [Xeval -0.04] }
4. c3 { [Keval -0.4] 4... b5? [Keval 1.18]
5. Bb3?! { [leval 0.21] } 5... <4 [%eval 0.32] }
Bc2 { ['eval 0.2] 6... a5 { [leval 0.6]
[Neaal 0.29] 7... cxd3 { [Keval 0.6]
8. Qxd3 [%eval 0.12] } 8.... Nf6 [%eval 0.52] }
9. e5 [Keval 0.39] 9... Nd5 [%eval 0.45]
10. Bg5?! [Keval -0. 44] } 10... Qc7 [%eval -0.12]
Nbd2?? { [Keval 33]
11...
[%eval -2.99]
12. Bh4 [Keval -3.0] 12... Ba6? [%eval -0. }
13. b3?? { [Keval -4.14] 13... Nf4? [%eval -2.73] } 0-1
Pro-player
[Event "URS-ch34"]
[Site "Tbilisi"]
[Date ?1966.??.??"]
[Round "9"]
[White "Bronstein, David I"]
[Black "Suetin, Alexey
[Result "1/2-1/2"]
(WhiteElo ""]
[BlackElo ""] [ECO "B97"]
1.e4 c5 2.NF3 d6 3. cxd4 lxx444 Né6 N.N a6
B. e6 7.f4 Qb6 8. QdQ Qxb2
9.Rb1 Qa3 10. Bxf6 gxf6 11.Be2 Bg7 12.0-0 Nc6
13. Nxc6 bxc6 14. Rb3 Qc5+
Kh1
exf5 exi5 1a. Qd4 18 Qxd4 Bxd4 19.Rd1 Bf2
20. Rxd6
0-0 21 .b6 Bxb6 22. Rxb6 Re8
23.Bf1 Be6 Kgl Bxa2 25. Rxa6 Rxa6 26.Bxad Bd5
27.Kf2 Re4 28.g3 Bc4 Rxc6 Re2+
30. Kg1 Bxa6 31. Rxa6 Rxc2 32. Ra5 kg7 33. Rxf5 Kg6
3g.Rg5+ Kf6 1/2-1/2
20

---

CCRL
[Event "CCRL 40/15"]
[Site "CCRL"]
[Date "2022.01 09922
[Round "806. 6. 381"1
[White "Stockfish 060122 64-bit"]
[Black "Dragon by Komodo 64-bit"]
[Result "1/2-1/2"]
[ECO "D30"]
[Opening "Queen's gambit declined"]
[PlyCount "115"]
[WhiteElo "3505"]
[BlackElo "3480"]
1. d4 (book) d5 (book) 2. c4 (book) e6 (book) 3. Nf3 (book) Nf6 (book) 4. g3 (book) (book)
(book) (book) cxb6 23/33 28s)
c5 (-0. 23/30 40s) 7. Bg2 24/30 11s} cxd4 (-0. 15/29 17s) 8. Nxd4
{+0.08/32 15s} Nbd7 (-0.21/25 18s) 9. Nc3 (90.2 18s) Nxb6
(-0.31/29 21s} 10. 0-0 2...27/30 12s} Bb7 {-0.21 21/28 20s} 11. Be3
(+0.41/31 14s) Ne4 12/27 19s} 12. Bg5 (+0.31/30 18a} Be7 {-0.16/22
29s) 13. Qa4+ (+0.24/29 18s) Qd7 {-0.08/30 20s) 14. b3 (+0. 07/33 22s}
Nd6 (+0.00/29 17s) 15. Rfci (+0. 19/31 28s) Re8 02/312 15s) 16. e3
(+0.13/33 24s) Nde4 {-0.02/38 16s) 17. Qxd7+ {+0.27/34 21s} Kxd7
{-0.05/34 31s} 18. Nxe4 (+0. 25/37 13s} Nxe4 f-0.17/32 22s) 19. Bxe7
{+0.29/32 22a} Kxe7 {-0.24/0 24/32 20a) 20. Bxe4 (+0.19/35 36a} dxe4
{-0.12/34 37s} 21. Kf1 (40. 13/38 88a) g5 (-0.03/33 27s) 22. Ne2
(+0.00/37 63s) Kas
21s} 23. Rxc8 (+0. 08/40 12s} Rxc8
{-0.08/37 25s) 24. Ke1 {+0. 12/39 20s} Ba5 (-0.05/34
40s} 25. Rc1
{+0.07/41 17s} Ra8 1-0.06/32 27a} 26. Kd2 5+0.05/33 16a) a5 {+0.00/3)
32s) 27. (+0.10/5) 18s)
{-0.05/37 20s) 28.
(+0.00 00.00//9 16s)
Ke5 000300/39 67s) 29. Re1 (70.07/43 56s) 15 (+0. 00/39 21s} 30. Na4
(+0.00/41 13s) ка6 00/40 19s) 31. Rc5 (+0.00/45 34s) Ra8 00/42
57a) 32. Rc2 {+0.000 00/49 20a} Rb8 {+0.00/44 13a) 33. Nb2 {+0. 12/37 17s}
h5 {+0.00/ 00/41 14a) 34. Nc4+ {+0. 11/41 29a} Bxc4 (+0.00/42 14s} 35. Rxc4
{+0.00/46 23s} h4 0+033 21s} 36. Ra4 f+0.00/48 20s} Rbs +00400/43
16s} 37. Kc3 (+0.00/ 00/50 29s} Ke5 00/40 18s) 38. b4 (+0.00/55 24s)
axb4+ 1+0.00/41 26s} 39. Rxb4 {+0.000 00/53 30s} Re5+ (+0. 00/43 14s) 40.
Кb3 (+0.00/53 26a} Ra5 {+0.00/4) 20s} 41. a4 (+0.00/50 20s} Kd6
{+0.00/45 31s} 42. Rc4 {+0.00/50 16a) h3 (+0.00/43 25a) 43. Kc2
(+0.00/65 34s} ka7 {+0.000 00/47 23s) 44. Wb3 (+0.00/47 17s) Rd3+
(+0.00/49 22s) 45. Kc2 (+0. 00/50 19s) Ra5 (+0.00/51
28s) 46. Rc3
(+0.00/52 35s} g4 {+0.000 00053 22s) 47. Kci {+0.00/5) 23a} Ra5 {+0.00 00/55
25a) 48. Rc4 {+0.000 00/558 21a} Ra6 (+0.00/55 21s} 49. Kc2 (+0.00 00.00/59 59s}
Rd6 00/57 28a} 50. a5 00/58 14a) Rd5 {+0.00 00/53 23s} 51. Ra4
(+0.00/60 27s) Kc7 000500/57 16s) 52. a6 (+0.00/60 17s) Kb8 00/60
18s) 53. a7+ (00.9 25s} Ka8 (+0. 00/65 42s) 54. Ra6 (+0.00/59 20s}
Rc5+ (40.00/66 16s) 55. Kb2 (+0.00/66 25s) Rd5 (+0.00/66 14s) 56. Kc2
[+0.00/62 17s} Rc5+ {+0.00/66 18s) 57, Xd1 (+0.00/67 17s) Rd5+
[+0.00/63 14s) 58. Ke2 000.00/101 20s, Draw by 3-fold repetition)
1/2-1/2
Chess puzzles
Try your hand at this chess puzzle. The board's FEN is 1r4k1 determine the optimal backRankMate, endgame, ma,,m,at,,aat,, matemmm, and the solutions are provided in both SAN format as d2d8, b8d8, d1d8 and UCI format as 27. Rd8+ Rxd8 28. Rxd8#.
27, and you need to
for the player. This puzzle focuses
21

---

Chess modeling
With the FEN board state 42 and move in UCI d8d6, what is the corresponding SAN move? The derived move is Rd6.
b
C4
Nothing improves your chess more than playing long time control tournament events. The Irish championships are being held in Dublin next weekend. There are lot of events choose from! We have message encouraging us to play from Irish Chess Union Chairman. All members of our club are registered (by the club) with the ICU. You are eligible to play and should consider playing!
Pile
Q: What is the theory behind center control? Center control is an important aspect of playing chess, most openings are built around controlling the center, but why? Is center control really that important for winning game?
RedPajama
Mark Dvoretsky strong player and fantastic coach Mark Israilewitsch Dvoretsky was born on December 1947 in Moscow. After finishing his studies Mathematics and Economics 1972 Dvoreteky focused on a career as chess trainer and among other things worked for Botvinnik's school of chess. As young player Dvoretsky achieved number notable
1973 he the Championship of Noscow and in 1974 he finished fifth the PeSR-Championship One year later, in 1975, he won the B-tournament in Wijk aan Zee. But he soon decided to focus on his career as a chess trainer Dvoretsky has trained countless strong players, and among his regular students are well-known players such Valery Chechov, Nana Alexandria, Sergei Dolmatov, Alexej Dreev and Artur Jussupow. Among the players who occasionally trained with Dvoretsky are Garry Kasparov, Viswanthan Anand, Veselin Topalov, Evgeny Bareev, Viktor Bologan, Loek van Wely and lots of others. One training method of Dvoretsky selected positions with both colors against
Leningrad.
students play often surprised his students by winning the same position with Black and with White. Dvoretsky was an International Master and FIDE Senior Trainer. He published number of textbooks, sometimes with Artur Jussupow co-author. ChessBase published digital version his "Endgame Manual", Dvoretsky was a firm part of chess life Moscow and popular guest at chess events all over the world. Russian Chess Federation
Oscar
If one wishes learn chess from some the greatest players in the world, but does not live in greater New York, then online lessons may be what he or she is looking for. Many of our coaches are experienced in teaching both group and solo online lessons. Online lessons orchestrated Skype using online chess program.
22

---

Chess Blogs
What Is The Elo Rating System? The Elo rating system measures the relative strength of player in some games, such as chess, compared to other players. Its creator, Arpad Elo, master who worked improve the way the U.S. Chess Federation measured their players' skill levels. He was a solid chess player himself, as you can see from this game he played against a young Bobby Fiacher.
physics professor in the United States and chess
[Event "New Western Open"]
[Site "Milvaukee, USA"] [Date "1957.07.04"]
[Round "2"] [White "Arpad Elo"]
[Black "Robert James Fischer"]
[Result "0-1"]
[EventDate "?"]
[ECO "B93"]
[WhiteElo "?"]
[BlackElo "?"]
[PlyCount "98"]
1. e4 c5 2. Nf3 d6 3. d4 cxd4 4. Nxd4 Nf6 5. Nc3 a6 6. f4 e5
7. Nf3 Qc7 8 Bd3
Nbd7 9, 0-0 b5 10. Qe1 Bb7 11. a3 g6 12. Qh4 Bg7 13. exf4
14. Bxf4 0-0 15.
Qg3 Ne5 16. Nxe5 dxe5 17. Bxe5 Qc5+ 18.
19. Bd6 Qxe3
20, bxc3 Nxg3
Bxf8 Rxf8 22. hxg3 Bxc3 23. Rb1 Bd4 24. a4 Bc8 25. axb5 axb5
26. Rxb5 Bxg4 27.
Kg2 Bxf2 28. Kxf2 Be6 29. Rc5 Kg7 30. Kf3 Kf6 31. Kf4 Ra8 32. g4 h6 33. g5+
hxg5+ 34. Rxg5 Rh8 35. Rg2 g5+ 36. Kf3 Rh3+ 37. Rg3 Rxg3+ 38. Kxg3 Ke5 39. c3
Ba7 40. Bc4 f6 Bd5 Be8 42. c4 Kd4 43. Kg4 Bg6 44. Kf3 Bh5+ 45. Kf2 Bd1
Kg3 Be2 c5 Kxc5 48. Be6 Kd4 49.
Chess Books
The following illustrative game apparently complicated, it this its motives\nonly. AnIn reality itis the fight against White's e4 pawn, which dominates. Shoosmith-Anllae Anlimzovitsch Ostend, 1907. 1.44 Nf6 2. d63.N13 Nbd7 N.Nc3e5 6. Be7 6. Bn03 7.0-0 exd4! (if 7...Re8, then 0.05 and Black will beAncramped for along time. For example, 7., Ree8 8.45 NeS\n9 Be3 Nxd3
10.Qxd3 Ná7 11. b4 a5 12.43, etc) Nxd44\8RR b3 Me5 10. Be2 (this
advance will soon be belnintelligible) 11.Bb2 Bd7 12.3 BE8 13. Ng6 14. Q4. c6\n15. Rae1 b5 (now the situation is clear: Black keeps an mnneye White's pawn and seeks
23

---

Chess Forums (Stack Exchange)
Q: there way to use handicaps in chess to bridge the gap betueen players of different skill levels? Handicapping is routine in the Japanese game Go (my best game). The basic strength unit is stone, and
difference represents full level of difference in strength. (about a 1500 player) once asked 2100 player how much handicap she would need to give me so that we would have an equal chance to win. "Probably a knight, maybe more," she answered. once took knight handicap against 2200 player and lost, but it much tighter, closer game than one with no handicap. That might auggest that a pawn equivalent to about 200 points of rating. Apparently handi doesn't do much for say, 50 point difference in strength (you just play and take your chances). But above that, there might be ways move two times out of three (as was earlier done professional Go) might do something. Or would it? Why hasn't handicapping been done much formally in chess, as in Go?
handicaps. Even giving someone the first
Annotated
[Event "Al1 about the Traxler Counter-Attack: Why to play Traxler instead passive move"]
[Site Phttps://Lichess.ortg
[Date [????.?????"]
[Round "?"] [White "?"]
[Black "?"]
[Result ""]
[Annotator "http:::/1iccees.orr/////
[ECO "C57"]
[0pening "Italian Game: Two Knights Defense, Fried Liver Attack"]
(UTCDate "2017.10.1 13"]
(UTсTime "16:33:43"]
(Variant "Standard"]
{ Hello everyone, please click the little heart to show that this study was helpful to you, spread the word and show your appreciation. 1. e4 2. Nif Ne6 Bc4 Nf6 4. Mg5 You may wonder why you should play Bc5 instead exactly what white is trying to do. } 4... d5 5. exd5 It seems you'll win the exchange. } 6... Nxd5 5 Na5 This is the best defense if you do play d5. 6. Bb5+ c6 7. dxc6 bxc6 8. Qf3 6. Nxf7 White has sacrifice of their own. This the fried liver attack. } 6... Kxf7 7. Qf3+ Ke6 8. Ne3 The knight is pinned. a3 Nxc2+ { Sacrifcing a rook. } 10. Kd1 Nxal 11. Nxd5 8... Nb4 9. potential for dangerous discovered attack } 11... Qh4 12. Nxc7+ { Double check. 12... Xd7 13. QE7+ Qe7 14. Nxa8 Qxf7 15. Bxf7 And Black's king position destroyed,
d5. This is just to show you
move with
white is a pawn up. White's knight may be hanging, but so is black's.
24

---

of drama today Youtube transcripts dataset
('text': 'sink field cup round number six coverage', 'start': 0.96, 'duration': 4. 4.319), {'text': "i'm going recap all four games for", 'start': 3.6, 'duration': 4.08), ('text': "you guys
on the', 'start': 5.279, 'duration'; 4.961), {'text' "hans niemann front so let's just jump", 'start': 7.68, 'duration': 4.56), ('text': 'right into the games three decisive', 'start': 10.24, 'duration': 3.92}, ('text': 'games the first game to fimish 'pomniacci with white pieces against', 'start': 14.16, 'duration': 3.6), ('text': 'fabiano carawana', 'start': 15.92, 'duration': 4.72), ('text': 'and we see a sicilian by fabi', 'start': 17.76, 'duration': 5.359), ('text: and he plays the tymanov variation with', 'start': 20.64, 'duration': 4.967, f'text': 'e6 sometimes this is called the sicilian', 'start': 23.119,
Janna', 'start'; 12.24, 'duration'; 3.68), ('text':
'duration'; 4.721), ('text'; "con it's a very solid sicilian i've", 'start'; 25.6, 'duration': 3.919}, ('text': 'played this quite a bit myself from the', 'start': 27.84, 'duration'; 3.279), ('text': 'black side', 'start'; 29.519, 'duration': 2.961), ('text': 'takes takes', 'start': 31.119, 'duration': 3.041), ('text': 'knight f6', 'start': 32.48, 'duration': 3.919), ('text': 'and now we get kind of this uh', 'start'; 34,16, 'duration': 4.559), ('tex'': 'taimanov set with the knight on f6', 'start': 36.399, 'duration': 4. 4.241), {'text': 'instead of queen c7', 'start': 38.719, 'duration: 3.68}, {'text": *so this sometimes transpose into', 'start'; 40.64, 'duration'; 3.2)
GPT4 instruction-tuning data
"conversations":
"author": "Human 0",
"text": "Explain the difference between the Italian Game and the Ruy Lopez.", "url": "gpt-4",
"score":
,, {
"author": "Human 1",
"text": "The Italian Game starts with e. e5 2.3 Ne6
3.Bc4, focusing targeting the f7 square.
The Ruy Lopez begins with 1.e4
2.3 Nc6 BbB,,, attacking the knight on c6 to undermine Black's pawn structure and exert pressure the e5 pawn. The Italian Game tends while the Ruy Lopez more strategic positional.", "url": "gpt-4",
more tactical,
"score":
]
}
25

---

Reddit Conversation data
"author": "Human 0",
"text": "Honest question for those with higher ELOs. First, aorry if vague question but I'll try my best to word it: At what rating, in your opinions, do you find people stop making \"silly\" mistakes? I'm 1500 rapid improve my play. I found that at these ratings blunders are rare (assuming adequate time). It seems like mostly people just don't see/miss tactics play inaccuracies rather than playing straight up mistakes/blundera
this is confusing
1300 blitz, trying to
you higher rated elo players feel the way? do you think the inaccuracies we make/tactics we miss are quite obvious, the same way I can see a blunder is obvious? Curious on the perspective.", "score": 8, "other "other_data" data": false, "url": "/x/cher ch/sc/ comments httpl////.cooottt//////////e/tee wtr_thosetttttttteeeeoe witpee__ithhhhighee olo//
"author": "Human 1",
"text"; "Im 2200 in blitz on lichess, of my last around 8 were decided by major blunders. In rapid, the amount of major blunders decreases lot, but they are still very common. When you get higher rated, you will still blunder because you also going be facing higher rated opponents. would rarely blunder but blunder very easily against 2300+ people.",
games,
played 1200, I
"score": 22, "other :atter_data;; false,
"url": httte////e//mmt/ttttt//////tette gywapvd/"
www.twitttthheeeeeeooeeeeee elos/
E
Data Collection and Preprocessing
E.1
Online Chess websites
We choose around 26 chess websites, including chess.com and Lichess. com scrape chess-related language corpus. We gather diverse range of chess and language data from various platforms, including blogs, news, and articles. Specifically, we focused on extracting relevant information from several topics including blogs, news, openings, chess terms, forums, and articles. These topics were carefully chosen they contain valuable texts pertaining chess background knowledge, news, and also PGN games
instances.
R We utilize Beautifulsoup and Playwright I parse HTML pages and locate all the and PGNs. We further transfer those PGNs into text which helps us build mixed game-language datasets these sources. record crucial details such as the URL, author. title, time, and the actual content of the articles.
E.2
Online Chess forums
We choose 5 chess forums and follow basically similar way with Appendix E.1 to scrape forum text data.
E.3
Annotated PGN
We collect annotated PGN from sources: Lichess studies, Chess publishing, Megabase, Pgnlib, path- tnchessmastery and gameknot.
Lichess studies Lichess Study provides a collection annotated PGNs. The annotations are embedded PGNs, explaining the insight of the moves. Users can conveniently search for studies based on keywords or specific topics like Sicilian Defense. Puzzles, or d5. To enhance the searching process, we collect comprehensive set of 54 popular keywords. Our implementation leverages Selenium's 4g built-in functions efficiently parse
https: ://::///www cutuuyy..o///otttaare//aaattttaa
26

---

HTML pages and simulate the searching process. Additionally, we use Lichess API request for PGN games associated with specific study ID.
Chess publishing This contains commercial annotated PGNs from Chesspublishing.com source this source of data.
do not open
Megabase This contains commercial annotated PGNs from Megabasc202ff so we do not open source this source of data.
Pgnlib We collect annotated PGN from Pgnlib
Pathtochessmastery We collect annotated PGNs from Path to Chess Mastery
Gameknot We use Sclenium
48 to scrape annotated PGNs from gamekno
E.4 Existing Datasets
We mainly extract all chess-related language corpus from existing dataset C4 43 Pile 19. Oscar 37
Wikipedia [T8] and RedPajama [53]. To extract chess-related language corpus, we first filter language corpus that contains the keyword "chess'. We further utilize the leherta-v3-bass-tasssuuree
mode which conduct zero-shot classification based on classes specified by users. We two classes: chess-related and non-chess-related for this model. We input the first 2000 characters of each text example to the model and set the threshold as 0.5.
E,5
YouTube transcripts dataset
To collect the youtube transcripts dataset, first gather 19 keywords and utilize scrapetub gather all related channels. We extract all videos from these channels and use same model mentioned in Appendix (E.4to filter all video transcripts and also the corresponding videos that are not relevant to chess. It is fairly casy to extract transcripts from videos and the main difficulty is to extract the FEN chessboard representations from videos. Here mainly utilize two steps to extract specific FEN from chess videos.
E.5.1 Extract Chess Board from videos
The first step is to extract the chessboard from videos. We utilize GLIP (28], zero-shot language- detection model. By setting the prompt as 'Chess, chessboard', we extract chess-related detection bounding boxes from videos. We conduct further filterings such as congth-width ratio filtering guarantec it is valid chessboard bounding box in most cases, which will be processed based on the second step.
E,5.2 Convert Chess board image to FEN text format
Our second step involves converting the chessboard image into FEN text. FEN format serves as great way to deseribe the current chess board state in text-hased format. The pipeline that converts the chess hoard image to FEN format includes three main procedures chess hoard dccomposition, piece classification, and the prediction which player the next turn.
The aim of this section is breakdown whole chess board into small tiles (8 rows columns), where each tile contains only one chess piece. To achieve this, we initially convert the RGB image grayscale, preparing for the line detection process. Subsequently, we make two convolutional kernels find horizontal and vertical gradient [3] The Hough Transform is then applied detect vertical and horizontal lines and filter out seven vertical and seven horizontal lines that fit the demand. Finally, divide the board into 64 tiles by having the position of the 14 lines.
Chess board decomposition
https://ww.cchespubbeeiinngg
http:://Bhop.chesbaseecco com http://aaeeeaooeee/oooooocc///// http://oouuttttggaaaaaaaaaeeee database 2023
http:///ggghoooaaattttt////cco/a compvision.ipymb
27

---

Table 11: Training hyperparameters for chess board FEN
Hyperparameier
Value
Batch size Number of epochs
32
10
Learning rate
0.001
Optimizer
SGD
Momentum
0.9
Piece classification To facilitate model training and evaluation, employ open source chess image dataset on Kaggld which contains 80k training images and 20k testing images. Each tile can be classified into one of the 13 categories (p. r, b, n, k, 9. P. R, B. N. K. Q. and space) which detailed in Appendix B.3. We implement a model in Py Torch which uses pre-trained R8sNet18 (20) due to its well-established performance in image classification tasks. To adapt the model to specific problem, we replaced the original fully connected layer with new layer consisting of 13 output neurons, corresponding the 13 pieces categories. We train the model on 40000 images with hyperparameters shown in Table[[]
After the training process, we evaluate the model on testset with 20k images (equivalent 128k tiles). Please refer to Table 12 for the final accuracy of each category.
Table 12: Validation accuracy of each chess piece
Color
Piece
Accuracy (%)
Black
pawn
99.98
rook
99.99
knight
99.98
bishop
99.98
queen
100.00
king
99.97
White
Pawn
100.00
Rook
99.98
kNight
99.98
Bishop
99.99
Queen
99.95
King 99.98
Space
100.00
Prediction of next turn As FEN format also includes the prediction of the next turn which is indicated by "w" for white, and "b" for black, the prediction of the next turn is accomplished by analyzing the main color of cach tile. We use Colorthicf, library for grabbing the color palette from images, to extract the main color from cach tilc since the background color of a tile will be highlighted if move is played on that tile. Henee, we find the highlighted tile by analyzing the tile color to know who is the current player and naturally infer who is the next turn.
Finally, we also provide final certainty percentile evaluate what extent the generated FEN is correct by calculating product of the accuracy the 64 tiles.
E.6 Lichess dataset
We collect months Lichess dataset from the Lichess database 30 : 2013-02, 2014-02, 2015-02, 2016-02, and 2017-02. In fact, there are much more data available and we leave more game data for future work.
E.7 Pro-player dataset
We collect our pro ploylayer dataset from PGN Mento
https://wwir pgnmentor..cmmm com/
28

---

E.8
Chess books
We select 100 chess-related keywords and search for all related chess books (around 9k books) the online PDF library. Because of the legal issues about books' copyright, choose not open-source this source of data, Instead, we only open source the list of books we use.
E.9
CCRL
We collect CCRL dataset without comments from the official website for three settings of time control: CCRL BLITZ, CCRL 40/2 FRC and CCRL 40/15.
E.10
Chess puzzles
We collect our chess puzzles from the Lichess puzzle datase
E.11
Chess modeling data
We design 11 modeling tasks to generate data:
Given PGN, generate FEN representation.
Given a list of UCI moves, generate FEN representation.
Given FEN and UCI move, transfer the SAN format.
Given FEN and SAN move, transfer the move to UCI format.
Given FEN, generate ASCII board.
Given FEN and UCT move, generate the next FEN.
Given FEN and SAN move, generate the FEN.
Given FEN, generate all legal moves in SAN format.
Given FEN, generate legal moves UCT format.
Given PGN, generate all legal moves in SAN format.
Given PGN, generate all legal moves in UCI format.
To generate the synthetic modeling dataset for these tasks, we utilize PGN data extracted from the first I million games of the Lichess database dump from March 2017. In order to encompass wider range of ELO ratings. divide the elo-rating into 9 intervals: 0-1000, 1000-1200 1200-1400, 1400-1600, 1600-1800, 1800-2000, 2000-2200, 2200-2400, ensuring that our dataset contains approximately 10,000 games for each ELO interval. Consequently, the dataset achieves balanced representation across different ELO ratings. Then we further utilize the python-chess library 41 to complete all the tasks design generate final synthetic modeling dataset.
2400-3000. Random sampling is employed to select games from each interval,
E.12 Preprocessing
We preprocess the data sources in three levels of granularity. For sources where existing preprocessed dataset available, we filter out the subset that contains chess-related information without performing additional preprocessing. For sources that retrieve from the Internet, we only parse portions of the HTML. that contains information about chess. We implement different parsers for the different sources we consider. As result, our data preprocessing can be more light-weight compared to previous work that extracts corpora from raw HTML web pages. For PGN games, we use the original PGN but filter out some annotations that are not useful for learning the model.
The different sources contain data in different formats. To facilitate training all datasets, we preprocess all datasets have the same jsonl format.
We implement the data- preprocessing step for cach source a separate Apache Beam pipeline which allows process the datasets in parallel into multiple shards. We note that a simple Apache Beam pipeline implementation provides no guarantecs that data processing will be in the same order as they were read from the files. As result, running the same pipeline twice will produce a set of shards that are shuffled differently. To provide determinism in data-procsssing pipeline, adopt deterministic shuffling strategy similar the implementation in TensorFlow Datasets (TFDS) to ensure reproducible data processing while maintaining scalability.
16 "bctppe//ccr...chessmmcc. htt.:/ssdddmmcoo////////////0
https::/dataaase...ccee wiw.:aaaaa.e.ccc.s.c.... http:///tttaaaoo..cceesssooooooo
29

---

We provide further details on the preprocessing used for each individual source below:
C4, Oscar, The Pile, RedPajama, Wikipedia Since these datasets are available in processed format, we do not perform any additional preprocessing.
StackExchange We the forums Chess StackExchange We preprocess the StackExchange data in the same way as done in RedPajama. Coneretely, for each question, we first clean up the text via simple regular expressions and remove some HTML tags in the text. Then we prefix the question with Q: and the answers with A: and then concatenate the questions with all answers.
Chess puzzle and Chess modeling data The original data format is in CSV format with key data, such as puzzle FEN and puzzle answer. We leverage some language templates transfer the CSV as natural language text.
Lichess database, CCRL and pro-player dataset We keep only games with known finish, i.e., (win, lose draw). We remove clk, arrow, evp annotations from the comments. We further remove Emojis from the comments. Afterward, each PGN is considered single string of text that is used downstream training.
Medium article, Chess forum, Chess books We run the same preprocessing pipeline as in
25
Annotated PGNs We conduct language filtering using Fasttex 19 the same preprocessing as we do in Lichess database in ChessGPT training preprocessing.
ChessCLIP preprocessing. And we conduct
Insturction data from GPT-4 No-further pre-processe pre-processing
Conversational data from Reddit We filter the conversations based on the language, response length, number of emojis, blacklist words, and scores.
We initially applicd the same data-processing procedure described in 22 for all of the data that we collected. However, we found that the filtering used in 25 can be too aggressive in removing useful examples as many of data sources include significant portion of chess notation that does resemble natural language (e.g.. chess puzzles). Therefore, we opted for more light-weight pre-processing and use the processing from [5] only in cases where the text includes significant portion of natural language description (blogs for example). In addition, for further protection of privacy, we anonymize user names and replace them with terms like 'Human O' in all conversation- like data, especially in chess forums and Reddit conversational data.
E.13 Licenses and dataset cards
For specific Licenses and dataset cards, refer to our open-source dataset repository: https://huggingfacc oo/Satasess/aattrroore////eeeee cataasssssssassaaaeeeeeeeaaaaaaa
F
Implementation and Evaluation Details
We open source all our models: ChessCLIP
ChessGPT-Bass Base htttps:/huugiigger co/ htt.://www.....eeeeee//////eeeaa chessspptreeeeeepptpeee........e Sertsppt--aee-eeee base- and ChessGPT-Chn.
(https:/ hhghhhggggggce co/ Watterrorrse/chesssstt--oo chat-vi). Refer these URLs for model licenses and model cards.
F1
Implmenetation details
F.1.1 ChessCLIP
For the ChessCLIP dataset, preprocess the annotated PGNs produce board/text pairs which feed separately to the board and text encoders. particular, for every move the PGN, we extract the comments attached to move as well as board state. While our YouTube transcripts dataset can also serve as training data for ChessCLIP, we have discovered that it consistently contains more noise compared to the annotated PGN dataset. To ensure the stability of our training process, we have chosen to exclusively utilize the annotated PGN datasets. The task of refining the YouTube transcripts for future training remains a part of ongoing work.
For the ChessCLIP model, instantiate ChessCLIP model with pair of text encoder and board/action encoder. For board/action encoder, we use a ResNet 20 architecture that conditions the action encoding via modified FiLM layer [40]. We encode the board positions and moves using the same scheme as those used by Leela Chess Zero (Ic0) [26]. which is similar to the encoding used by AlphaZero [7]] for encoding positions and moves in chess. Concretely, the board positions encoded as R x8881112 feature map and the actions encoded as R. 1688 vector. For the text encoder, follow the same architecture as with the original OpenAl
18 hhttpprr/ddees.stcccccccccagggga
30

---

CLIP model and we only fine-tune the last two layers of pretrained OpenAl text encoder. Our implementation is
based the open-source implementation 20 of CLIP. We show training alpper-arameters in Table
Table 13: ChessCLIP Training Hyperparameters
Hyperparameters
Value
Hyperparameters
Value
Hyperparameters
Value
Learning Rate
5e-4
Warmup Step
500
Weight decay
0.2
Batch Size Per GPU
512
Number of GPUs
8
Optimizer
Adam
Optimizer hetal
0.9
Optimizer beta2
0.98
Optimizer epsilon
1e-6
Precision
AMP
L.carnngg Rate Scheduler
Cosine
Epochs
40
We would like to highlight that ChessCLIP can serve direct move sequence generator when provided with a text prompt. By utilizing beam search over all legal sequences, we can maximize the similarity between sequences. This is a unique feature as cannot be achieved with the original CLIP model when generating images texts due to the high dimensionality of image and text spaces. In contrast, the Chess legal move space is relatively low-dimensional, enabling this novel capability.
F11.2 ChessGPT
We follow common implementations of training domain-specific instruction-following LLM. Firstly we conduct base-model fine-tuning using chess corpus introduced in section 3.1 3.2 and 3.3 Due to computational
constraints, choose to finetune the RedPajama-3B base (53] model, which open soencsouce replication of
LLaMA (55). We also limit our model max token length as 1024. The base- mnse-finetuning brings our model: ChessGPT--Bse.
After base-finctuning, we conduct supervised fine-tuning by supervised learning on question/convvrsation response using data introduced in section a3d and general conversation data from OASST1 (24). Dolly2 [14,
Apaca-GPPS (39], and Sharegpt [6). forming our chat model: ChessGPT-Che We call it ChessGPT-Chat instead of ChessGPT-SFT because some of our conversation datasets generated by RLHF-tuned LLM. We
convert all our Q/A or conversation data into the following two conversation formats:
Between two people: friendly, helpful chat between some humans, <lendoftexl O: (Human Ques- dion)<clendoftextl Human I: (Human Reeponse (essponse<<lendoffee <lendoftexl Human O: [Human 0 Quextion) <lendofixl..
Between multiple people (Reddit conversational data): A friendly, helpful chat between some /llenootext 1: (Human Re- sponse > Human 2: (Human 2 (lnnoffnn
humans.< <len:n/sseeeddfffe>>>>>> Humann 0: (Human 0
Our basc-training code refers llama-finctung and sft-training code refers the Alpaca 52 and Fastchat [2] The training hyperparameters for ChessGPT-Base and ChessGPT--Cha are shown in Table
and Table 15
Table 14: ChessGPT-Base Training Hyperparameters
Hyperparameters
Value
Hyperparameters
Value
Hyperparameters
Value
Learning Rate
8c-5
Warmup ratio
0.03
Weight decay
0.00
Batch Size Per GPU
3
Number of GPUs
Optimizer
Adam
Accumulation step
8
Max token length
1024
Acceleration
FSDP
Precision
bf16
Learning Rate Scheduler
Cosine
Epochs
Table 15: ChessGPT-CHe Training Hyperparameters
Hyporparameters
Value
Hyperparameters
Value
Hyperparameters
Value
Learning Rate
2e-5
Warmup ratio
0.03
Weight decay
0.00
Batch Size Per GPU
4
Number of GPUs
8
Optimizer
Adam
Accumulation step
8
Max token length
1024
Acceleration
FSDP
Precision
6
Learning Rate Scheduler
Cosine
Epochs
31

---

Here we also show more evaluation results.
General policy result. Table presentss the results of the general policy experiment using black chess, which align with the findings from the previous white chess experiment. The comparison between the two ChessGPT models across different Elo ratings reveals a lack of noticeable distinctions, indicating the model's limited sensitivity to the key information provided in the prompt. A more intuitive illustration of this observation will be provided in the subsequent paragraph. There are two notable points to highlight. Firstly, ChessGPT demonstrates improvements compared to its base model RedPajama and performs on par with LLAMA. However, it is worth noting that both baselines exhibit limitations in adapting different Elo ratings, as the generated values across various Elo ratings show considerable similarities.
Table 16: General policy evaluation in Black
Move Scores (%)
Elo Rating
LLAMA
RedPajama
ChessGPT-Base
ChessGPT-Chat
700- 1000
52.9 + 0,9
46.2 + 1.0
51.9 + 0.1
52.1 + 0.9
1200- 1500
53.2 + 0.9
46,9 0.9
53.0 1.0
52.4 + 1.0
1700-2000
52.1 + 0.8
0.8
46.6
4 1.0
52.0 1.0
52.0 + 1.0
2700-3000
53.6
+ 0.9
47.3 1.0
52.2 + 0.9
52.1 + 1.1
Words attention visualization. To evaluate whether the ChessGPT-Baas Basse model captures the key information in the general policy task, we conducted a visualization analysis of its self-attention mechanism. The visualization, as shown in Figure reveals that the model does attend to the "WhiteElo" and "BlackElo" values to some extent. However, the level of attention dedicated to these important features appears to be relatively weak. This suggests that the model's i ability to appropriately incorporate and utilize the Elo ratings during the generation process is not as strong as desired. Therefore, further investigation and improvement are necessary enhance the model's attention towards and understanding the provided Elo rating information.
In the following chess game, you play white: [Event "Rated Classical game http://ccees..ogoooommmmmmssssss [Date "2017.04.01"] [Round "-"] [White
"???"| [Black "???"] [Result "1-0"] [WhitElo "1781"| [BlackElo "781"|
(WhiteRatingDiff" "??"J [BlackRatingDifff "??"] [ECO "??"] [Opening "??"]
[TineControl "30010"] [Termination "Time forfeit"] 1. o4 e6 2. d4 d5 3. e5 c5 4. NI3 exd4
5. Nxd4 Ne6 6. Nxc6 bxc6 7. Nc3 Ne7 8. g3 Ng6 9. 14 Be7 10. Be3 h5 11. Bd3 f5 12. exf6 Bxf6
13. Bd2 Ne7 14. Qe2 Qé6 15. 0-O-O Bd7 16. Kbl Rb8 17. b3 Qa3 18. Bel Qa5 19. Bd2 Bxe3 20. Bxe3
Qxe3 21. Qd2 Qff 22. Rhel a5 23. a4 c5 24. Choose your next move based on your and your
opponent's Elo ratings""
Figure 3: Visualization of ChessGPT-Base Attention: The figure illustrates the attention space of ChessGPT for the General Policy experiment, generating a compound level next move based Elo rating. The highlighted areas represent the importance of attention, with color intensity ranging from black red, where red indicates the highest importance."
F11.3 Compute resources
We use 80G A100 GPUs for all experiments. It takes 5 hours train ChessCLIP using all A100 GPUs. And it takes hours to train ChessGPT-Baa model and 18 hours to train ChessGPT-Chat.
G Evaluation details
G.1 Evaluation task examples and plots
Here show task examples and plots for our evaluation tasks. Basically, the evaluation tasks consist of three parts: Task Prefix, which can be regarded as a description of the task and also main prompt we use for LLMs. Input, which is the question and the input of LLMs. And Target, which contains the answer of the question for exact string match tasks, target score, which provides the score for each available answer for multi-choo choiic tasks.
32

---

g
8
8
2
2
g
g
8
Figure 4: Example in Chess state tracking The input leads to the following board configura- tion.The The task is to predict the squares to which the piece at d8, the black queen, can be legally moved. Here the black queen at square d8 can legally moved to any of the squares ["b8", "b6", "c7", "c8"].
The Chess state tracking (Fig
Task Prefix: For each of the following (in- (it-progress) chess games, please complete the notation for the last shown move by filling the destination square.
Input: e2e4 e7e6 d2d4 d7d5 e4e5 c7c5 c2c3 b8c6 g1f3 g8e7 a2a3 a7a5 f1d3 c8d7 c1e3 d8
Target: ["b8", "b6", "c7", "c8"]
Board state tracking
Task Prefix: Could you produce the Forsyth-Edwards Notation (FEN) that corresponds to the provided PGN-based move list of the chess games?
Input: 1. d4 g6 2. 04 Bg73 3. e4 Nf6 4. Nc3 O-O 5. Be3 Ne8 13 Nc6 7. Qd2 e6 8. h4 d5 9. cxd5 exd5 10. Nxd5 Nf6 1. Nxf6+ Bxf6 12. e5 Bg7 13. Bb5 Bd7 14. Rc1 a6 15. Bc4 b5 16.
Bb3 B15 17. Rxc6 Qd7 18. Rc5 c6 19. Ne2 Rad8 20. Ng3 Be6 21. Bxe6 Qxe6 22. Ne4 Rd5
23. O-O 24. Rxd5 cxd5 25. Nc5 Qc6 26. Bh6 14 27. Bxg7 Kxg7 28. Re1 Qe8 29. e6 Qe7
30. Qa5 Qxh4 31. e7 Re8 Nxa6 Rxe7 Rxe7+ Qxe7 Nc7 Qe3+ 35. Kh2 Q12 36.
Nxd5
Target: BBlp:@01dchpMM//Pp255PPP333386
36
Board state tracking
Task Prefix: Could you produce the Corsyth-Edwardd Notation (FEN) that corresponds to the provided SAN-based move list of the chess games?
Input: e2e4 d7d5 b1 d5e4 c3e4 g8f6 e4c3 e7e6 f1c4 18b4 d2d3 b4c3 b2c3 e8g8 g1e2 c7c5 e1g1 b8c6 e2g3 e6e5 a2a4 c8g4 f213 g415 c1g5 15g6 d1 h7h6 g5h6 g7h6 d2h6 c6e7 f314 e5f4 f1 d8d6 a1f1 16h7 g3h5 g6h5 h6d6 h5g6 d6e7 a8e8 e7h4 e8e2 h4h6 e2c2 h6g6
g8h8 c4f7 c2
Target: 3rrkp008Biis002222442222424PPPP2
06
State_trackin
33

---

Chess state value
Task Prefix: Evaluate the following PGN to see whether black white takes advantage.
Input: 1. e4 e6 2. d4 d5 3. e5 c5 4. NI3 cxd4 5. Nxd4 Nc6 6. Nxc6 bxc6 7. Nc3 Ne7 8, g3 Ng6 9. 14 Be7 10. Be3 h5 11. Bd3 f5 12. exf6 Bxf6 13. Bd2 Ne7 14. Qe2 Qd6 15. 0-0-0 Bd7 16. Kb1 Rb8 17. b3 Qa3 18. Bct Qa5 19. Bd2 Bxc3 20. Bxc3 Qxc3 21. Qd2 Q16 22.
Rhe1 a5 23.
Target Score: ("Black has advantage.", 1, "The game is equal.". 0, "White has advantage.".
0}
Chess annotation
Task Prefix: Annotate the last step of the following PGN.
Input: 1. d4 Nf6 2. 04 e6 3. Nf3 Bb4+ 4. Bd2 a5 5. g3 O-O 6. Bg2 b6 7. O-O Ba6 8. Bg5 Be7 9. Qc2 Nc6 10. a3 h6 11. Bxf6 Bxf6 12. Rd1 Qe7 13. e3 Rae8 14. Níd2 g5
Target Score: "Karpov could have resigned here with a clear conscience.", 0, "White intends to further weaken Black's kingside with 19.h5." 0, "h0 21. 21.Bx66+ fxg6 22. 22.Qxe6 Gives White winning attack.": O, "Black overreacts to the positional strength of White's game. 14...96 would have been more solid.":
Chess opening (opening2pgn)
Task Prefix: Show me the PGN of the following opening.
Input: Amar Gambit Opening
Target Score: "1. Nh3 d5 2. g3 e5 3. 14 Bxh3 4. Bxh3 exf4": 1, "1 d4 d5 2. c4 e6 3. Nc3
c5 4. cxd5 exd5 5. dxc5 d4 6. Na4 b5": 0, "1. d4 N16 2. g4 Nxg4 3. 13 Ní6 4. e4": 0, "1. Nc3
2. b4": "1. d4 Ní6 g6 3. Nc3 Bg7 e4 d6 f3 O-O Nge2":
Chess opening (pgn2opening)
Task Prefix: Show me the opening name of the following PGN.
Input: 1. Nh3 d5 2. g3 e5 3. Bxh3 4. Bxh3 exf4. The opening name of this PGN is.
Target Score: ("Amar Gambit": "Tarrasch Defense: Tarrasch Gambit": 0, "Indian Defense:
Gibbins-Wend Weidenngeidnnnnenn Gambit, Maltese Falcon": 0, "Van Geet Opening: DO0fcsseldorf Gambit": "King's Indian Defense: S00e4misch Variation, Bobolso--Korchno-PPetriiia Variation": 0}
1. e4 e6 2. Ke2 d5 3. e5 c5 4. f4 Nc6
5. Nf3 Qb6 6. g4 Bd7 7. h4 Nge7 8. c3 Ng6
9. d4 cxd4 10. cxd4 Be7 11. Kf2 0-0 12. h5 Nh8
Bxh7+
13. Be3 Qxb2+ 14. Kg3 Qxa1 15. Bd3 Qxa2 16. Rh2 Qa1
17. Qc2 Nb4 18.
Figure 5: Example for Checkmate in one The goal of this task is to probe the ability of language models to play chess in standard algebraic notation. The input to the model is sequence of moves such that next possible move is a checkmate. For example, the chess game shown in the figure can checkmate the opponent in one step is "Bxh7+"
checkmate, _nn
34

---

Checkmate in (Fig
Input :1, e4 e6 2. Ke2 d5 e5 c5 4. 14 Nc6 5. NI3 Qb6 6. g4 Bd7 7. h4 Nge7 8, c3 Ng6
d4 cxd4 10. cxd4 Be7 11. Kf2 O-O 12. h5 Nh8 13. Be3 Qxb2+ 14. Kg3 Qxa1 15. Bd3 Qxa2
16. Rh2 Qa1 17. Qc2 Nb4 (Now white has checkmate in one move.)
Target Score: ("Qxa1": 0.0, "Bxh7+": 1.0, "Qd2": 0.0, "Qe2": 0.0, "Qd1": 0.0, "Qc3": 0.0,
"Qc7": 0.0, "Bb1": 0.0, "Bc2": 0.0, "Bf5": 0.0, "Bg6": 0.0, "Bf1": 0.0, "Bb5": 0.0, "Bxa6": 0.0,
"Bc4": 0.0, "Bb3": 0.0, "Bc1": 0.0, "Bd2": 0.0, "Bc3": 0.0, "Bxb4": 0.0, "Nbd2": 0.0, "Nc3":
0.0, "Na3": 0.0, "Rc2": 0.0, "Rg2": 0.0, "Rh1": "Ri2": 0.0, "Re2": 0.0, "Rd2": 0.0. "Rh3": 0.0, "Rh4": 0.0, "Kh2": 0.0, "Kh3": 0.0, "Kh4": "g5": 0.0, "15": 0.0, "Ng5": 0.0, "Nh2": 0.0.
"Nf2": 0.0, "Ne1": 0.0.)
General policy
Task Prefix: In the following chess game, you play black.
Input:
[Dae "2017.04.01"]
White "???"
Black "???"j
Result "0-1"
WhiteElo "983"]
BlackElo "2983"]
WhiteRRtingDDiff "??"
BlackRatingDDfff "??"
ECO "??"
Opening "??"
TimeControl "300+0"]
Termination "Time forfeit")
1. b3 e5 Bb2 Nc6 a3 Nf6 4. h3 d5 5. g3 Bd6 Bg2 0-0 e3 8. d3 Be5 9. d4 Bd6
10. Ne2 Ne7 11. c4 c6 12. Nbc3 N15 13. Qd2 a5 14. Qc2 Be6 15. cxd5 cxd5 16. Nb5 Rc8
17. Qd2 Qb6 18. Nxd6 Nxd6 19. Qd1 Rc7 20. a4 Rfc8 21. Ba3 N15 22. Bc5 Rxc5 23. dxc5
Target Score: ("Rxc5": 0.0, "Qa7": 0.02, "Qd8": 0.05, "Qc7": 0.083, "Qb4+": 0.11, "Qxc5":
0.13, "Qc6": 0.16, "Qa6": 0.19, "Nxe3": 0.22, "Nxg3": 0.25, "Nd4": 0.27, "Rc6": 0.30, "Ng4":
0.33, "Qxb3": 0.36, "d4": 0.38, "Nh4": 0.41, "Kh8": 0.44, "Nd7": 0.47, "h6": 0.5, '96". 0.52,
"Rf8": 0.55, "Ra8": 0.58, "Ne7": 0.61, "Qd6": 0.63, "h5": 0.66, "Re8": 0.69, "Ki8": 0.72, "Qb5": 0.75, "Rd8": 0.77, "Bd7": 0.80, "Ne8": 0.83, "Ab8": 0.86, "Nd6": 0.88, "95": 0.91,
"Rc7": 0.94, "Nh5": 0.97, "Nh6": 1.0]
G.1.1 Evaluation on chess modeling ability
The dataset used for the tasks includes real games from the I. Lichess December 2019 dump. The first 400,000 classical and rapid games were extracted from this dataset, ensuring that both players minimum Elo rating of over 2000 to maintain game quality. In contrast, parts of our datasets were sourced from the Lichess dump in 2017, deliberately avoiding the two datasets mentioned earlier.
G.1.2 Evaluation task construction
For state value multi-choice and general policy task construction, we follow a similar way as the chess modeling data to collect 3000/2000 PGN states from Lichess 2017-03 dump respectively. For general policy, calculate the winning rate for all legal moves using StockFish-15 with the depth of 18. The rank of the winning rate determines score one legal move choice general policy evaluation task.
35

---

H
Qualitative Analysis
In this section, we present some qualitative results between RedPajama, ChessGPT--Base and ChessGPT-CHan
H.1
Experimental set-up
We use the Beam search implementation HuggingFace transformers generating samples each model. Please refer to Table 17 for the hyperparameters used for the qualitative experiments. For each question, we sample three responses from each model. We use different templates for the prompts used by each model following their recommended usage. Concretely, for ChessGPT-Baa and friendly, helpful chat between some humans enddof text >>ummn O I endoftext >>Hmman 1: for ChessGPT-Chat. For each question. we replace O with the question and use the new prompts as input to the models.
<human>: O<<bo>>: for RedPajama, Q: O-A
Hyperparameter
Value
max tokens
128
top-k top-p
50
0.7
temperature
0.7
Table 17: Hyperparameters for qualitative analysis.
36

---

H.2 General knowledge
In this section, we ask each model some questions about general knowledge of chess. We found that the base models perform reasonably well these questions since the answers these questions be easily found online. However, we lind RedPajama sometimes produces factual answers. Detailed examples can
be found tables 18 and 21
Prompt
The quickest possible checkmate in: (Answer: 2 moves, Fool's mate)
ChessGPT-
Qh5# B: 1. Qf7# C: 1. Qd8# D: 1. Qh8# E: 1. Qg8# F: 1. Qh7# G: 1. Qg6# H:
Qe8# I: 1. Od7# J: 1. Qc8# K: 1. Qb7# L: Qa6# M: 1. Qb5# N: 1. Qa4# O: 1.
Two moves. B: Four moves. C: Six moves. D: Eight moves. E: Twelve moves. F: Sixteen moves. G: One move.
K+Q K B: K+B K C: K+N vs K D: K+N+N vs K K+B+N F: K+B+N
K+B G: K+R vs H: K+R+R vs K K+R+R vs K+R J: K+R+R vs K+R K: K+R vs K R+R+ vs K+R M: K+R+R VS K+R
ChessGPT-Chal
2 moves
2 moves
2 moves
RedPajama
chess
chess
chess
Human Error Analysis
ChessGPT-Base: For the prompt "The quickest possible checkmate is in: (Answer: 2 moves, Fool's mate)", the answer given by ChessGPT-Base is not completely accurate. Although it contains "Two moves.", the specific moves listed above are all wrong.
ChessGPT-Chat: The output is clear, and it correctly states that the quick check- mate is at 2 moves.
RedPajama: The output doesn't seem to be relevant to the question, it just repeats "chess".
Table 18: Question fastest checkmate.
37

---

Prompt
What is it called when a player can't defend an attack against their king? (Answer: checkmate)
ChessGPT-Bas
When player cannot defend their king against an attack, it is called a checkmate. The game is over and the opponent wins.
Back Rank Mate
Fork
ChessGPT-Chal
Checkmate
Checkmate
Checkmate
RedPajama
In chess, a check is when player's king is under immediate attack by the oppo- nent's pieces.
chess, called a "checkmate".
Checkmating
Human Error Analysis
The first answer is accurate and correctly describes the situation where a player's inability to defend their king's attack is called a "checkmate". However, the subsequent "Back Rank Mate" and "Fork" not relevant to the given hint.
ChessGPT-Base:
ChessGPT-Chat: The output is clear, and it correctly states that the quick check-
mate is at 2 moves.
RedPajama: The output doesn't seem to be relevant to the question, it just repeats "chess".
Table 19: Question on checkmate.
38

---

Prompt
What determines which player starts first in chess game? (Answer: white)
ChessGPT Base
White always moves first.
White moves first.
chess, the player with the white pieces always moves first.
ChessGPT-Chat
White and Black are both considered to have the advantage, and the player who is the first to move (the white player) is always the first player to move.
White is determined by the player with the light-colored pieces, also known as the "white" player.
In game of chess, the player with the white pieces should start first.
RedPajama
The first player to move is determined by the order in which the players are seated. The first player is seated to the left the board.
Thhe first player to move is determined by a sequence of moves called the "opening". This sequence of moves is determined by the player who moves first.
The first player to move the one who moves first the first game.
Human Error Analysis
ChessGPT-Base: All answers clearly state that white always goes first, which is accurate.
All answers mentioned that white or the player using white pawns should move first, which also accurate. However, the "both white and black are considered to have an advantage" mentioned in the first answer is inaccurate, because usually white considered to have small opening advantage.
ChessGPT-Chat:
RedPajama: None of the answers explicitly state that white goes first, and the first and second answers are completely inaccurate. The third answer was vague and unclear.
Table 20: Question starting player.
39

---

Prompt
Does DeepBlue defeat Garry Kasparov? (Answer: yes)
ChessGPT-Bass
No.
Yes.
No. It is a draw.
ChessGPT-Chaa
DeepBlue, a chess ppeess-plyyng computer developed by IBM, defeated Garry Kasparov in match in May 1997, in a six-game match. Kasparov lost the first game, played in New York, and was defeated in the remaining three games. The match was billed as contest between a human world champion and a machine, and was billed as a landmark in the history technology.
chess-playing computer developed by IBM, which became the first machine to win a chess game against a reigning world champion, Garry Kasparov, on May 3, 1997. However, Kasparov won three and drew two of the following six games, beating DeepBlue by score of 4-2, winning the match and the computer's first game in the match. The match was highly publicized, as represented a major milestone in artificial intelligence and computer chess. IBM's Deep Blue project was the result of large-scale effort to develop a chess-playing computer that could challenge the
DeepBlue
DeepBlue was a chess chess-playing computer developed by IBM, that became the first computer to beat reigning world champion in match, when it defeated Garry Kasparov in Game of their 1997 match.
RedPajama
Deep Blue defeated Garry Kasparov in a chess match in 1997.
Yes, DeepBlue defeated Garry Kasparov in 1997.
No, DeepBlue defeated Garry Kasparov in 1997.
Human Error Analysis
Mixed responses were provided. The first and third answers are "no", while the second answer is "yes". This is inconsistent and without further explanation.
ChessGPT-Base:
ChessGPT-Chat: All three answers mentioned 1997 game that DeepBlue had won against Garry Kasparov. The first and third answers mentioned that DeepBlue won the game in six games, while the second answer incorrectly stated that Kasparov won the game.
RedPajama: Both the first and second answers were "yes", pointing out that DeepBlue beat Garry Kasparov 1997. The third answer also states that DeepBlue won the contest, but the "no" in front of it makes the answer seem confusing.
Table 21: Question on Deep Blue.
40

---

H.3 Chess modeling
In this section, we present qualitative analysis of the model's performance to performing chess-related tasks. We provide qualitative comparison for three tasks:
Opening recognition
In this task, the models tasked detect chess opening. Example responses can be found in table 22
Prompt
1.e4 e5 2.14, What is the name of this opening? (Answer: King's Gambit)
ChessGPT PT--Bsee
It is called King's Gambit Accepted.
King's Gambit Accepted. G: King's Gambit Accepted, Cunningham Defense, Bertin Gambit, Haxo Gambit,
The King's Gambit.
ChessGPT- GP-Chat
The opening is called the King's Gambit.
The opening is called the King's Gambit.
The opening is called the King's Gambit.
RedPajama
The Sicilian Defence.
The French Defense
The French Defense.
Human Error Analysis
ChessGPT-Base: first and second answers called "King's Gambit Accepted", which is the name when the opposing side accepts a sacrificed pawn. And the third answer is "King's Gambit", which is broader category. Technically, it's only called "Accepted" after exf.exf4.
ChessGPT-Chat: All three responses call it "King's Gambit", which is correct.
RedPajama: All responses are wrong, calling it "Sicilian Defense" and "French Defense", neither of which have anything do with the openings offered.
Table 22: Question on chess openings
41

---

PGN to FEN conversion
this task, the models were tasked to convert chess game from PGN notation to FEN. Example responses can he found in table 23
Prompt
d4 exd4 Nxd4 Nf6 5. Nc3 Qe7 6. Bd3 d5 7. O- Nxe6 fxe6 10. Bxe4 Nxe4 11. Nxe4 Nd7 12. Bg5 Qb4 13. Qg4 Qd4 14. Qxe6+ Be7 15. Convert the PGN FEN (Answer:
1. e4 e5 2. Nf3 d6 3.
O dxe4 8. Re1 Be6 9.
wkq 1 15)
ChessGPT--Bse
e4 e4 e5 2. Nf3 d6 3. d4 exd4 4. Nxd4 Nf6 5. Nc3 Qe7 6. Bd3 d5 7. O-O dxe4 8.
Re1 Be6 9. Nxe6 fxe6 10. Bxe4 Nxe4 11. Nxe4 Nd7 12. Bg5 Qb4 13. Qg4 Qd4
14. Qxe6+ Be7 15. Rad1 Qxb2 16. Nf6+ Nxf6 17. Qxe7#
1. e4 e5 2. Nf3 d6 3. d4 exd4 4. Nxd4 Nf6 5. Nc3 Qe7 6. Bd3 d5 7. O-O dxe4 8.
Re1 Be6 9. Nxe6 fxe6 10. Bxe4 Nxe4 11. Nxe4 Nd7 12. Bg5 Qb4 13. Qg4 Qd4
14. Qxe6+ Be7 15. Rad1 Qb4 16. Qxe7+ Qxe7 17. Nf6+ Nxf
1. e4 e5 2. Nf3 d6 3. d4 exd4 4. Nxd4 Nf6 5. Nc3 Qe7 6. Bd3 d5 7. O-O dxe4
Re1 Be6 9. Nxe6 fxe6 10. Bxe4 Nxe4 11. Nxe4 Nd7 12. Bg5 Qb4 13. Qg4 Qd4
14. Qxe6+ Be7 15. Rad1 Qb4 16. Bxe7 Qxe7 17. Qxe7+ Kxe7
ChessGPT-Chaa
oee//pppopppic00007/00388PPP/1P9 kq- 16
(xxippotop/000111a3388PPPPPP11 w kq 16
(5X1ppon11//000111/0099PPPPp22 w kg 16
RedPajama
1. e4 e5 2. Nf3 d6 3. d4 exd4 4. Nxd4 Nf6 5. Nc3 Qe7 6. Bd3 d5 7. O-O dxe4
Re1 Be6 9. Nxe6 fxe6 10. Bxe4 Nd7 11 1. Bg5 Qb4 12. Qg4 Qd4 13. Qxe6+ Be7
14. Qxe6+ Kf8 15. Qd6+ Ke8 16. Qd5+ Kf8 17. Qd6+
e4 e5 2. Nf3
d4 exd4 4. Nxd4 Nf6 5. Nc3 Qe7 6. Bd3 d5 7. O-O dxe4
Re1 Be6 9. Nxe6 fxe6 10. Bxe4 Nd7 11. Bg5 Qb4 12. Qg4 Qd4 13. Qxe6+ Be7
14. Qxe6+ Kf8 15. 14 g6 16. NI3 Qd7 17. Kb1 Kg7
dxe4 Nxe4
Human Error Analysis
ChessGPT-Base: Provides too much information, showing game records to go on, but not FEN.
ChessGPT-Chat: The FEN provided is very similar to the requested answer, but with errors in the move counters.
RedPajama: Completely wrong, it gives the game record instead of FEN.
Table 23: Question on PGN to FEN conversion.
42

---

UCI to FEN conversion.
this task, the models were tasked to convert a chess game from UCI notation to FEN. Example responses can he found in table 24
Prompt
Convert the following UCI to FEN: g2g3 d7d6 f1g2 e7e5 g1f3 g816 d2d4 b8 (Answer: http:///wwppopp././/////////pp/p
b KQkq
0 4)
ChessGPT GT--ase
You can use the following code to convert the FEN string to a UCI string: # r/bin/python import chess import chess.pygame import chess.uci import chess.pgn
The FEN is ) fen wttppp/rrpppppphhaaaaaaa/////rrr
w KQkq 0 1' board
chess. (fard((e))) board.parse san(fen) # spinttoorrdddadrrbbbrrellll
KQkq 0 5.
Here is a link to the FEN representation of the above position: rn-
oxttr//pppppsnh//sspp///////22e2 b KQkq 0 5
ChessGPT-Chat
mDDp:///ppppppomm//////////////0 w KQkq 0
htpp::/wppppppps..e.e/ee//////pr w KQkq- 0
wwppp:/wwpppppphh....///s/eepppp b KQkq 0
RedPajama
Here is the FEN:g2k414 b8k4b5 d7k6d6 e7k6e5 f1k1f2 g8k8f6 UCI: g2k4f4 b8k4b5
d7k6d6 e7k6e5 f1k1f2 g8k8f6 Note that the FEN is a different notation than the UCI.
Here is the FEN: g2g3 d7d6 f1g2 e7e5 g1f3 g8f6 d2d4 b8 The following the chess pieces: g knight g2 bishop g3 knight queen d7 queen d6 queen f bishop bishop knight e7 knight e5 knight h knight h2 knight h3 knight h4 - knight h5 knight h6 knight
FEN: g2g3 d7d6 f1g2 e7e5 g1f3 g8f6 d2d4 b8 This standard position chess, but not in FEN.
Human Error Analysis
ChessGPT-Base: first answer gives code snippet rather than a direct answer. The second and third answers try to convert UCI to FEN, but the result is wrong. In terms verbosity, too much information was provided, including an irrelevant code snippet.
The third answer is correct. The first two answers have subtle errors and the chess colors (w/b) are mislabeled.
ChessGPT-Chat:
RedPajama: Completely wrong, it gives the game record instead of FEN.
Table 24: Question on UCI to FEN conversion.
43

---

PGN annotation
In this task, the models were tasked to generate commentary for given game recorded in PGN format. Example responses be found in table
Prompt
1. e4 e5 2. Nf3 d6 d4 exd4 4. Nxd4 Ní6 5. Nc3 Qe7 6. Bd3 d5 7. O-O dxe4 Re1 Be6 9. Nxe6 fxe6 10. Bxe4 Nxe4 11. Nxe4 Nd7 12. Bg5 Qb4 13. Qg4 Qd4 14. Qxe6+ Be7 15. Generate a step-by-step analysis
ChessGPT-Bas
This is a typical position the Scotch Game. White has lead in development and Black has a pawn weakness on e6. White's next move is 14. Bxe7.
believe you're looking for a program that will analyze the position and output the best move for both sides. have couple suggestions: ChessBase: This is commercial program, but it's free for the first 30 days. Scid vs. PC: This is free and open-source, but you need download the source code. have used both of these programs in the past.
This is a pretty easy one. 1. e4 e5 2. Nf3 d6 3. d4 exd4 4. Nxd4 Nf6 5. Nc3 Qe7 6. Bd3 d5 7. O-O dxe4 8. Re1 Be6 9. Nxe6 fxe6 10. Bxe4 Nxe4 11. Nxe4 Nd7 12. Bg5 Qb4 13. Qg4 Qd4 14. Qxe6+ Be7 15. Rad1
ChessGPT-Chat
1. e4 e5 (Both players have opened with pawn moves 1. e4 and 1... e5, controlling the center and opening lines for their bishops and queen) 2. Nf3 d6 (Black has played pawn move 2... d6, which is a passive response to White's 2. Nf3) 3. d4 (White has played pawn move 3. d4, attacking Black's pawn on and opening lines for the queen and bishop) exd4 (Black has captured White's pawn on d4 with his pawn on e5, surrendering the center to White) 4.
e4 e5 (Both players have opened with pawn moves to the e-file: 1. e4 and 1.... e5) 2. Ní3 d6 (Black has played the Philidor Defense, which is solid, flexible response to 1. e4. Black aims develop pieces and prepare for central pawn breaks) 3. d4 exd4 (Black captures the pawn on d4) 4. Nxd4 Nf6 (Black develops the knight and prepares castle kingside) 5. Nc3 Qe7 (Black plays a queen move, attacking
1xb. (Black's best move) 16. Rab1 Qe5 (Black's only move to avoid losing material) 17. Qxe5 Nxe5 18. Rxb7 (White has significant advantage due to Black's vulnerable king and misplaced knight on e5)
RedPajama
e4 e5 2. Nf3 d4 exd4 4. Nxd4 Nf6 5. Nc3 Qe7 6. Bd3 d5 7. O-O dxe4
Re1 Be6 9. Nxe6 fxe6 10. Bxe4 Nxe4 11. Nxe4 Nd7 12. Bg5 Qb4 13. Qg4 Qd4
14. Qxe6+ Be7 15. Generate step-by-step analysis
e4 e5 Nr3 d6 d4 exd4 4. Nxd4 Nf6 5. Nc3 Qe7 6. Bd3 d5 7. O-O dxe4
Re1 Be6 9. Nxe6 fxe6 10. Bxe4 Nd7 11. Bg5 Qb4 12. Qg4 Qd4 13. Qxe6+ Be7
14. Nd5 Nxd5 15. exd5 Nd7 16. c3 c6 17. Bf4 Qd8 18.
1. e4 e5 2. N13 d6 3. d4 exd4 4. Nxd4 Nf6 5. Nc3 Qe7 6. Bd3 d5 7. O-O dxe4 8.
Re1 Be6 9. Nxe6 fxe6 10. Bxe4 Nd7 11. Bg5 Qb4 12. Qg4 Qd4 13. Qxe6+ Be7
14. Generate seep-by-step analysis
Table 25: Question on PGN annotation
44

---

Human Error Analysis
The first answer is short review of the Scotch Game, but not step-by step analysis. The second answer mentions other software for game analysis, which not what the user asked for. The third answer just repeats the PGN given.
ChessGPT-Base:
The first two answers started a step-by-step game analysis but were interrupted and not completed, but the analysis given reasonable. The third answer provides follow-up move for 15...xbb2, but the user does not provide this step.
ChessGPT-Chat:
RedPajama: The first and third answers just repeat the PGN given, analysis is provided. The second answer tries to provide follow-up move, which is also irrelevant the question, since the user just requested an analysis, not follow-up move.
45

---

Casine simlarity between text and image features
plays
Black Carc-Garm Deleniee
921
414
0P5
500
B acc plays Scard nav De Delenue
6.43
plys Engi Opening
Black French Opening
834
921
631
634
Blick plays Philddr belansa
6.3
522
0.0
4.2
pays Qucon's
Black Queents Delennee
0.
4.0
934
512
White plays London System
400
plys
Figure 6: Similarity matrix of different chess opening PGN and text using ChessCLIP.
I ChessCLIP visualization
We present visualization demo to better illustrate ChessCLIP's capability. Here we choose the chess opening as test scenario and choose different chess openings. Ten chess opening PGNs and their corresponding names are arranged sequence, so diagonal
in entire similarity matrix should present such similarity matrix generated by ChessCL.IP Figure The results exactly correspond to our expectations, which successfully validate the effectiveness of our ChessCIIIP model.
highest similarity.
Potential directions
In this section we will describe several potential directions based on our dataset.
Fine-tuning Rescarchers can explore dataset augmentation techniques to further enhance the diversity and size of the provided dataset. By introducing variations in game conditions, player strategies, or opening positions, researchers can create augmented datasets that can be used to fine-tune existing language models or train new models specifically tailored for chess-related tasks. This can potentially lead to improved performance in tracking, value judgement, and policy evaluation.
Dataset Augmentation
2. Transfer Learning Chess Variants: The dataset and benchmark provided in this study serve valuable resource for transfer learning experiments chess variants. Researchers leverage the knowledge learned from Chess or Chess960. By fine-tuning or adapting pre-trained language models on the provided dataset, researchers can explore the capabilities of these models in handling different chess variants, including state tracking, value judgement, and policy evaluation
chess task and apply
variants such as Fischer Random
3. Reinforcement Leaming/Planning with Language Models: Combining reinforcement learning or planning with language models trained on the provided dataset opens exciting possibilities for improving chess chess-playing agents. Researchers develop reinforcement learning algorithms that
46

---

utilize the language model's state tracking ability to build more sophisticated and strategic agents. By training agents to interact with the language model in a dialogue-like manner, researchers explore the potential of language-based reinforcement learning for chess-related tasks, such as move generation and evaluation..
4. Explainable Al in Chess: Given the language model's ability to generate human-readabee outputs, researchers can investigate the application of explainable Al techniques in chess. By interpreting the model's generated moves or predictions, researchers can gain insights into the reasoning behind the model's decisions. This can lead to the development of explainahle Al systems that provide justifications explanations for their chess moves, aiding hoth players and analysts in understanding and learning from the model's decision-making process.
5. Multi-modal Approaches: Researchers can explore multi-modal approaches that combine textual and visual information for chess-related tasks. By incorporating board visualizations, game position images, move sequence visualizations along with textual inputs, researchers can develop models that leverage both textual and visual cues improve state tracking. value judgement, and policy evaluation in chess. This can open up avenues for multi-modal analysis and understanding of chess games, allowing models to capture and reason over both textual and visual representations simultaneously.
6. Chess Education and Tutorial Systems: The dataset can be utilized to develop educational tools and tutorial systems for chess players of different skill levels. Rescarchers can leverage the language model's expertise in state tracking, value judgement, and policy evaluation to provide interactive and personalized learning experiences. By tailoring the tutorial content and feedback based on individual player performance, researchers can create intelligent systems that assist in skill development and stralegic improvement in chess.
7. Adversarial Attacks in Chess: With the increasing use of language models in critical applications like chess analysis and becisinn-making, it becomes essential investigate potential vulnerabilities and develop defenses against adversarial attacks. Researchers can explore techniques to generate adversarial examples specifically targeted chess-related tasks. By identifying weaknesses language models' state tracking or policy evaluation abilities, rescarchers can enhance the robustness and security of these models.
8. Chess Game Generation: Researchers can utilize the provided dataset to develop models capable of generating new chess game sequences. By leveraging the language model's 's understanding of chess moves and game structures, researchers can explore generative models that can produce novel and diverse chess game sequences. This can be beneficial for various applications, including chess game analysis. training data generation, and even game generation for chess variants.
Overall, our dataset and benchmark offer numerous potential directions, ranging from dataset expansion and transfer learning exploring chess variants, education, analysis, and game generation. These directions have the potential to advance the field of chess-related language modeling and provide valuable tools and resources for chess players and enthusiasts.
K Limitations and Potential Societal Impact
The availability of comprehensive and diverse chess dataset presents both limitations and potential societal impacts that researchers should consider.
Limitations. While the chess dataset provided in this study is valuable, it is important to acknowledge its limitations. One limitation is the potential bias introduced by relying on historical Lichess matches from different time periods. This may result in variations in player strategies, popular openings, and game trends over time, potentially impacting the generalizability of the dataset. Additionally, it worth noting that the dataset predominantly focuses on standard chess, and may not encompass the full spectrum of chess variants. Researchers interested in exploring niche or less-popular variants may need to gather additional data from specific sources to ensure comprehensive coverage of different chess variants. These considerations are crucial ensure the validity and applicability of research findings based on the provided dataset.
Potential Societal Impact. The availability of comprehensive and diverse chess dataset can have significant societal impact. First and foremost, it can contribute to the development of more advanced and intelligent chess-playing agents. These agents can be utilized in various applications, such as chess analysis, training tools for players of different skill levels, and even as opponents for chess enthusiasts. The dataset can also facilitate the advancement of chess education by providing valuable resources for tutorials, interactive learning platforms, and strategic guidance. Additionally, the dataset can inspire research in the field of artificial intelligence, contributing to the development of innovative techniques that can be applied beyond the domain of chess. Lastly, the dataset can encourage the exploration of explainable Al methods in chess, enabling players to understand and learn from the reasoning behind the model's moves, therehy promoting transparency and trust in AI systems.
47


Reviews:
[{"review_id": "naXpkQ0NlK", "reviewer": "Reviewer_HpTJ", "summary": "", "questions": "", "limitations": "The issues I am concerned about have already been stated above.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. From my perspective, I didn't quite grasp the motivation behind the author's pursuit of this work. Could the author elaborate on it in detail? Additionally, a significant portion of the paper is dedicated to the background and the source of the Chess data, which, in my opinion, could be included in an appendix. Instead, the main body should focus on elucidating the significance, importance, and other core insights derived from the Chess data.\n\n2. The author's literature review appears to be lacking, as the references cited seem insufficient and inaccurate. In line 22, the author references three papers in the context of models leveraging Reinforcement Learning from Human Feedback (RLHF). I have no doubts about [20], \"Constrained Value-Aligned LLM via Safe RLHF,\" as it is closely related to RLHF. However, the relevance of the other two papers, [43], \"Tianxiang Sun and Xipeng Qiu. Moss, March 2023,\" and [43], \"ChatGLM-6B, March 2023,\" to RLHF is not apparent to me.\n\nHow does the author perceive the significance of the following two papers in the context of RLHF?\n\n- Deep reinforcement learning from human preferences\n- Training language models to follow instructions with human feedback\n\nThe entire paper appears to be roughly drafted, and there is considerable confusion in the references and literature review.\n\n3. After reading the entire paper, it seems that the author summarized relevant online chess data to train ChessGPT, a 2.8B parameter pre-trained language model (mentioned in the hugging face dataset description as Model Description: A 2.8B parameter pre-trained language model in Chess.). However, in the main body of the paper, at line 248, the model is referred to as a 3B model. What could be the reason for this inconsistency?\n\nFurthermore, I have the following two questions, and I kindly request the author to provide answers:\n\n- In line 41, it is mentioned, \"a large-scale game and language dataset for chess.\" How large is it?\n\n- In line 42, it is mentioned, \"from online chess databases.\" Could you please specify which databases are included in this reference?\n\nIn summary, the author collected online chess datasets (which is not a challenging task) and then trained ChessGPT and other models based on a 2.8B/3B parameter model. My question is, what is the contribution of this work? Is it about the dataset itself, training a Chat model, or does the dataset facilitate research in a specific aspect of LLMs? Does it enhance the essential capabilities of LLMs in any particular way? From my perspective, collecting data and training a model are relatively easy and straightforward tasks. Furthermore, the author did not promote any specific field from the perspective of datasets or benchmarks, which does not align with the primary purpose of benchmarks.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "SM9ic7Wx6u", "reviewer": "Reviewer_y51X", "summary": "", "questions": "", "limitations": "Yes, see \"Opportunities For Improvement\" section above.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- interesting and timely problem statement.\n- large scale dataset collection and synthesis which will be useful to the community\n- ChessGPT and ChessCLIP will empower future research\n- systematic evaluation framework", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "EJQgPJpI78", "reviewer": "Reviewer_9skW", "summary": "", "questions": "", "limitations": "The models, as currently presented, do not appear to have the ability to learn from their own gameplay experiences in contrast to RL algorithms, which is a limitation for the further enhancement of their policies. While this limitation could potentially be addressed in future work, the current models might not fully capture the dynamic nature of decision-making processes in the context of games.", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "The release of ChessGPT and ChessCLIP, along with the large-scale dataset and the source code, represents valuable resources for the wider AI research community. The dataset is an extensive multimodal dataset of chess games and language descriptions of chess strategy and commentary that will be very useful for training multimodal chess models like theirs. Additionally, their trained models immediately can become useful for AI chess instruction and coaching. This paper is also a strong proof-of-concept for integrating policy learning and language modeling in general and beyond chess. ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "xzuE13d3zo", "reviewer": "Reviewer_jbSR", "summary": "", "questions": "", "limitations": "They were adequately addressed in section G of Appendix.", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. Teaching language models to reason about chess is a very challenging task. It is also significant for the research community, as it is known that large language models are not great at deducing complex tasks that involves many different moving objects and long sequences of moves (such as chess).\n2. The collected dataset is very large in size (millions of games), and the data source comes from multiple different sources, thus making the data more diverse. This dataset can be very useful for researchers who want to explore the planning or long-term state tracking abilities of large language models.\n3. The authors managed to train a GPT-3 type model that achieves significantly better performance on chess-related tasks compared to baseline large language models such as LLAMA.\n4. The authors presented a new set of tasks for thoroughly evaluating a language model's ability in chess-related tasks, which will be very useful for future research to directly compare results in a standardized way.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: All reviewers agree to accept the paper.
Decision: Accept (Poster)