SlowGuess/NeurIPS_2023_OCR huggingface dataset:
1. d4 {book} d5 {book} 2. c4 {book} e6 {book} 3. Nf3 {book} Nf6 {book} 4. g3 {book} a6 {book} 5. c5 {book} b6 {book} 6. cxb6 \{+0.23/33 28s\} c5 \{-0.23/30 40s\} 7. Bg2 \{+0.24/30 11s\} cxd4 \{-0.15/29 17s\} 8. Nxd4 \{+0.08/32 15s\} Nbd7 \{-0.21/29 18s\} 9. Nc3 \{+0.29/30 18s\} Nxb6 \{-0.31/29 21s\} 10. 0-0 \{+0.27/30 12s\} Bb7 \{-0.21/28 20s\} 11. Be3 \{+0.41/31 14s\} Nc4 \{-0.12/27 19s\} 12. Bg5 \{+0.31/30 18s\} Be7 \{-0.16/29 29s\} 13. Qa4 + \{+0.24/29 18s\} Qd7 \{-0.08/30 20s\} 14. b3 \{+0.07/33 22s\} Nd6 \{+0.00/29 17s\} 15. Rfc1 \{+0.19/31 28s\} Rc8 \{-0.02/31 15s\} 16. e3 \{+0.13/33 24s\} Nde4 \{-0.02/33 16s\} 17. Qxd7 + \{+0.27/34 21s\} Kxd7 \{-0.05/34 31s\} 18. Nxe4 \{+0.25/37 13s\} Nxe4 \{-0.17/32 22s\} 19. Bxe7 \{+0.29/32 22s\} Kxe7 \{-0.24/32 20s\} 20. Bxe4 \{+0.19/35 36s\} dxe4 \{-0.12/34 37s\} 21. Kf1 \{+0.13/38 88s\} g5 \{-0.03/33 27s\} 22. Ne2 \{+0.00/37 63s\} Kd6 \{-0.03/36 21s\} 23. Rxc8 \{+0.08/40 12s\} Rxc8 \{-0.08/37 25s\} 24. Ke1 \{+0.12/39 20s\} Bd5 \{-0.05/34 40s\} 25. Rc1 \{+0.07/41 17s\} Ra8 \{-0.06/32 27s\} 26. Kd2 \{+0.05/33 16s\} a5 \{+0.00/35 32s\} 27. Nc3 \{+0.10/35 18s\} Rb8 \{-0.05/37 20s\} 28. Rh1 \{+0.04/39 16s\} Ke5 \{+0.00/39 67s\} 29. Rc1 \{+0.07/43 56s\} f5 \{+0.00/39 21s\} 30. Na4 \{+0.00/41 13s\} Kd6 \{+0.00/40 19s\} 31. Rc5 \{+0.00/46 34s\} Ra8 \{+0.00/42 57s\} 32. Rc2 \{+0.00/49 20s\} Rb8 \{+0.00/40 13s\} 33. Nb2 \{+0.12/37 17s\} h5 \{+0.00/41 14s\} 34. Nc4 + \{+0.11/41 29s\} Bxc4 \{+0.00/42 14s\} 35. Rxc4 \{+0.00/46 23s\} h4 \{+0.00/43 21s\} 36. Ra4 \{+0.00/48 20s\} Rb5 \{+0.00/43 16s\} 37. Kc3 \{+0.00/50 29s\} Ke5 \{+0.00/44 18s\} 38. b4 \{+0.00/51 24s\} axb4 + \{+0.00/41 26s\} 39. Rxb4 \{+0.00/53 30s\} Rc5 + \{+0.00/43 14s\} 40. Kb3 \{+0.00/53 26s\} Rd5 \{+0.00/43 20s\} 41. a4 \{+0.00/50 20s\} Kd6 \{+0.00/45 31s\} 42. Rc4 \{+0.00/50 16s\} h3 \{+0.00/48 25s\} 43. Kc2 \{+0.00/53 34s\} Kd7 \{+0.00/47 23s\} 44. Kb3 \{+0.00/47 17s\} Rd3 + \{+0.00/49 22s\} 45. Kc2 \{+0.00/50 19s\} Rd5 \{+0.00/51 28s\} 46. Rc3 \{+0.00/52 35s\} g4 \{+0.00/53 22s\} 47. Kc1 \{+0.00/57 23s\} Ra5 \{+0.00/55 25s\} 48. Rc4 \{+0.00/58 21s\} Ra6 \{+0.00/57 21s\} 49. Kc2 \{+0.00/59 59s\} Rd6 \{+0.00/57 28s\} 5o.. a5 [  $^{+}$  .  $^{*}$  .  $^{*}$  .  $^{*}$  .  $^{*}$  .  $^{*}$  .  $^{*}$  .  $^{*}$  .  $^{*}$  .  $^{*}$  .  $^{*}$  .  $^{*}$  .  $^{*}$  .  $^{*}$  .  $^{*}$  .  $^{*}$  .  $^{*}$  .  $^{*}$  .
Chess puzzles
Try your hand at this chess puzzle. The board's FEN is 1r4k1/4nppp/8/4Pb2/8/1P5P/r1PR4/3R3K w -- 0 27, and you need to determine the optimal move for the player. This puzzle focuses on backRankMate, endgame, mate, mateIn2, short, and the solutions are provided in both SAN format as d2d8,b8d8,d1d8 and UCI format as 27. Rd8+ Rxd8 28. Rxd8#.
Chess modeling
With the FEN board state 3r4/4Rpp1/2N2k1p/8/5B2/5BP1/P4PKP/8 b -- 6 42 and a move in UCI d8d6, what is the corresponding SAN move? The derived move is Rd6.
Nothing improves your chess more than playing long time control tournament events. The Irish championships are being held in Dublin next weekend. There are a lot of events to choose from! We have a message encouraging us to play from the Irish Chess Union Chairman. All members of our club are registered (by the club) with the ICU. You are eligible to play and should consider playing!
Q: What is the theory behind center control? Center control is an important aspect of playing chess, most openings are built around controlling the center, but why? Is center control really that important for winning a game?
RedPajama
... Mark Dvoretsky - strong player and fantastic coach Mark Israilewitsch Dvoretsky was born on 9th December 1947 in Moscow. After finishing his studies of Mathematics and Economics in 1972 Dvoretsky focused on a career as chess trainer and among other things worked for Botvinnik's school of chess. As a young player Dvoretsky achieved a number of notable successes: in 1973 he won the Championship of Moscow and in 1974 he finished fifth at the USSR-Championship in Leningrad. One year later, in 1975, he won the B-tournament in Wijk aan Zee. But he soon decided to focus on his career as a chess trainer Dvoretsky has trained countless strong players, and among his regular students are well-known players such as Valery Chechov, Nana Alexandria, Sergei Dolmatov, Alexej Dreev and Artur Jussupow. Among the players who occasionally trained with Dvoretsky are Garry Kasparov, Viswanthan Anand, Veselin Topalov, Evgeny Bareev, Viktor Bologan, Loek van Wely and lots of others. One training method of Dvoretsky was to play selected positions with both colors against his students - and he often surprised his students by winning the same position with Black and with White. Dvoretsky was an International Master and FIDE Senior Trainer. He published a number of textbooks, sometimes with Artur Jussupow as co-author. ChessBase published a digital version of his "Endgame Manual". Dvoretsky was a firm part of chess life in Moscow and popular guest at chess events all over the world. Russian Chess Federation ...
If one wishes to learn chess from some of the greatest players in the world, but does not live in greater New York, then online lessons may be what he or she is looking for. Many of our coaches are experienced in teaching both group and solo online lessons. Online lessons are orchestrated via Skype using an online chess program.
Chess Blogs
What Is The Elo Rating System? The Elo rating system measures the relative strength of a player in some games, such as chess, compared to other players. Its creator, Arpad Elo, was a physics professor in the United States and a chess master who worked to improve the way the U.S. Chess Federation measured their players' skill levels. He was a solid chess player himself, as you can see from this game he played against a young Bobby Fischer.
[Event "New Western Open"]  
[Site "Milwaukee, WI USA"]  
[Date "1957.07.04"]  
[Round "2"]  
[White "Arpad Elo"]  
[Black "Robert James Fischer"]  
[Result "0-1"]  
[EventDate "?"]  
[ECO "B93"]  
[WhiteElo "?"]  
[BlackElo "?"]  
[PlyCount "98"]
1. e4 c5 2. Nf3 d6 3. d4 cxd4 4. Nxd4 Nf6 5. Nc3 a6 6. f4 e5 7. Nf3 Qc7 8. Bd3 Nbd7 9. 0-0 b5 10. Qe1 Bb7 11. a3 g6 12. Qh4 Bg7 13. g4 exf4 14. Bxf4 0-0 15. Qg3 Ne5 16. Nxe5 dxe5 17. Bxe5 Qc5+ 18. Rf2 Nh5 19. Bd6 Qxc3 20. bxc3 Nxg3 21. Bxf8 Rxf8 22. hxg3 Bxc3 23. Rb1 Bd4 24. a4 Bc8 25. axb5 axb5 26. Rxb5 Bxg4 27. Kg2 Bxf2 28. Kxf2 Be6 29. Rc5 Kg7 30. Kf3 Kf6 31. Kf4 Ra8 32. g4 h6 33. g5+ hxg5+ 34. Rxg5 Rh8 35. Rg2 g5+ 36. Kf3 Rh3+ 37. Rg3 Rxg3+ 38. Kxg3 Ke5 39. c3 Bd7 40. Bc4 f6 41. Bd5 Be8 42. c4 Kd4 43. Kg4 Bg6 44. Kf3 Bh5+ 45. Kf2 Bd1 46. Kg3 Be2 47. c5 Kxc5 48. Be6 Kd4 49. Bf5 Ke3 0-1
Chess Books
The following illustrative game is apparently complicated, but it is this in its motives\nonly.\nIn reality itis the fight against White's e4 pawn, which dominates. Shoosmith-\nNimzowitsch, Ostend, 1907. 1.d4 Nf6 2.c4 d63.Nf3 Nbd7 4.Nc3e5 5.e4 Be7 6.Bd3\n0-0 7.0-0 exd4! (if 7...Re8, then 8.05 and Black will be\ncramped for along time. For example, 7...Re8 8.45 NcS\n9.Be3 Nxd3 10.Qxd3 Nd7 11.b4 a5 12.43, etc) 8.Nxd4\nRe8 9.b3 Ne5 10.Bc2 a6 (this advance will soon be\nintelligible) 11.Bb2 Bd7 12.3 Bf8 13.f4 Ng6 14.Qf3 c6\n15.Rae1 b5 (now the situation is clear: Black keeps an\nelye on White's e-pawn and seeks ...
Chess Forums (StackExchange)
Q: Is there a way to use handicaps in chess to bridge the gap between players of different skill levels? Handicapping is routine in the Japanese game Go (my best game). The basic strength unit is one stone, and a one-stone difference represents a full level of difference in strength. I (about a 1500 player) once asked a 2100 player how much of a handicap she would need to give me so that we would have an equal chance to win. "Probably a knight, maybe more," she answered. I once took a knight handicap against a 2200 player and lost, but it was a much tighter, closer game than one with no handicap. That might suggest that a pawn is equivalent to about 200 points of rating. Apparently handicapping doesn't do much for say, a 50 point difference in strength (you just play and take your chances). But above that, there might be ways to use handicaps. Even giving someone the first move two times out of three (as was earlier done in professional Go) might do something. Or would it? Why hasn't handicapping been done much formally in chess, as in Go?
Annotated pgn
[Event "All about the Traxler Counter-Attack: Why to play Traxler instead of a passive move"]
[Site "https://lichess.org/study/WLyfoXTJ/xdA6LWme"]
[Date"????.??.??"]
[White"?"]
[Black"?"]
[Result ""]
[Annotator "https://lichess.org/@/Dat_Das"]
[EC0 "C57"]
[Opening "Italian Game: Two Knights Defense, Fried Liver Attack"]
[UTCDate "2017.10.13"]
[UTCTime "16:33:43"]
[Variant "Standard"]
{Hello everyone, please click the little heart to show that this study was helpful to you, to spread the word and to show your appreciation.} 1. e4 e5 2. Nf3 Nc6 3. Bc4 Nf6 4. Ng5 { You may wonder why you should play Bc5 instead of d5. This is just to show you exactly what white is trying to do.} 4... d5 5. exd5 { It seems you'll win the exchange.} 5... Nxd5 ( 5... Na5 { This is the best defense if you do play d5.} 6. Bb5+ c6 7. dxc6 bxc6 8. Qf3) 6. Nxf7 { White has a sacrifice of their own. This is the fried liver attack.} 6... Kxf7 7. Qf3+ Ke6 8. Nc3 { The knight is pinned.} 8... Nb4 9. a3 Nxc2+ { Sacrificing a rook.} 10. Kd1 Nxa1 11. Nxd5 { A move with potential for a dangerous discovered attack } 11... Qh4 12. Nxc7+ { Double check.} 12... Kd7 13. Qf7+ Qe7 14. Nxa8 Qxf7 15. Bxf7 { And Black's king position is destroyed, and white is a pawn up. White's knight may be hanging, but so is black's.}
Youtube transcripts dataset
{'text': 'sink field cup round number six coverage', 'start': 0.96, 'duration': 4.319}, {'text': "i'm going to recap all four games for", 'start': 3.6, 'duration': 4.08}, {'text': 'you guys not a lot of drama today on the', 'start': 5.279, 'duration': 4.961}, {'text': "hans niemann front so let's just jump", 'start': 7.68, 'duration': 4.56}, {'text': 'right into the games three decisive', 'start': 10.24, 'duration': 3.92}, {'text': 'games the first game to finish was janna', 'start': 12.24, 'duration': 3.68}, {'text': 'pomniacci with the white pieces against', 'start': 14.16, 'duration': 3.6}, {'text': 'fabiano carawana', 'start': 15.92, 'duration': 4.72}, {'text': 'and we see a sicilian by fabi', 'start': 17.76, 'duration': 5.359}, {'text': 'and he plays the tymanov variation with', 'start': 20.64, 'duration': 4.96}, {'text': 'e6 sometimes this is called the sicilian', 'start': 23.119, 'duration': 4.721}, {'text': "con it's a very solid sicilian i've", 'start': 25.6, 'duration': 3.919}, {'text': 'played this quite a bit myself from the', 'start': 27.84, 'duration': 3.279}, {'text': 'black side', 'start': 29.519, 'duration': 2.961}, {'text': 'takes takes', 'start': 31.119, 'duration': 3.041}, {'text': 'knight f6', 'start': 32.48, 'duration': 3.919}, {'text': 'and now we get kind of this uh', 'start': 34.16, 'duration': 4.559}, {'text': 'taimanov set up with the knight on f6', 'start': 36.399, 'duration': 4.241}, {'text': 'instead of queen c7', 'start': 38.719, 'duration': 3.68}, {'text': 'so this can sometimes transpose into', 'start': 40.64, 'duration': 3.2} ...
GPT4 instruction-tuning data
{ "conversations": [ { "author": "Human 0", "text": "Explain the difference between the Italian Game and the Ruy Lopez.", "url": "gpt-4", "score": 0 }, { "author": "Human 1", "text": "The Italian Game starts with 1.e4 e5 2.Nf3 Nc6 3.Bc4, focusing on targeting the f7 square. The Ruy Lopez begins with 1.e4 e5 2.Nf3 Nc6 3.Bb5, attacking the knight on c6 to undermine Black's pawn structure and exert pressure on the e5 pawn. The Italian Game tends to be more tactical, while the Ruy Lopez is more strategic and positional.", "url": "gpt-4", "score": 0 } ]
Reddit Conversation data
{
    "author": "Human 0",
    "text": "Honest question for those with higher ELOs. First, sorry if this is a confusing or vague question but I'll try my best to word it: At what rating, in your opinions, do you find people stop making \"silly\\" mistakes? I'm at 1500 rapid and 1300 blitz, trying to improve my play. I found that at these ratings blunders are rare (assuming adequate time). It seems like mostly people just don't see/miss tactics or play inaccuracies rather than playing straight up mistakes/blunders. Do you higher rated elo players feel the same way? Or do you think the inaccuracies we make/tactics we miss are quite obvious, the same way I can see a blunder is obvious? Curious on the perspective.,
    "score": 8,
    "other_data": false,
    "url": "/r/Chess/comments/nhh2mn/honest_question_for_those_with_higher_elos/"}
}
{
    "author": "Human 1",
    "text": "Im 2200 in blitz on lichess, and of my last 10 blitz games, around 8 were decided by major blunders. In rapid, the amount of major blunders decreases a lot, but they are still very common. When you get higher rated, you will still blunder because you are also going to be facing higher rated opponents. If I played a 1200, I would rarely blunder but I blunder very easily against 2300+
    people.,
    "score": 22,
    "other_data": false,
    "url": "/r/Chess/comments/nhh2mn/honest_question_for_those_with_higher_elos/
    gywapvb/"}
}
E Data Collection and Preprocessing
E.1 Online Chess websites
We choose around 26 chess websites, including chess.com and Lichess.com to scrape chess-related language corpus. We gather a diverse range of chess and language data from various platforms, including blogs, news, and articles. Specifically, we focused on extracting relevant information from several topics including blogs, news, openings, chess terms, forums, and articles. These topics were carefully chosen as they contain valuable texts pertaining to chess background knowledge, news, and also PGN games in some instances.
We utilize Beautifulsoup<sup>4</sup> and Playwright [1] to parse HTML pages and locate all the texts and PGNs. We further transfer those PGNs into text which helps us build mixed game-language datasets from these sources. We record crucial details such as the URL, author, title, time, and the actual content of the articles.
E.2 Online Chess forums
We choose 5 chess forums and follow basically similar way with Appendix E.1 to scrape forum text data.
E.3 Annotated PGN
We collect our annotated PGN from 7 sources: Lichess studies, Chess publishing, Megabase, Pgnlib, path-tochessmastery and gameknot.
Lichess studies Lichess Study provides a rich collection of annotated PGNs. The annotations are embedded in PGNs, explaining the insight of the moves. Users can conveniently search for studies based on keywords or specific topics like Sicilian Defense, Puzzles, or d5. To enhance the searching process, we collect a comprehensive set of 54 popular keywords. Our implementation leverages Selenium's [48] built-in functions to efficiently parse
<sup>4</sup>https://www.cummy.com/software/BeautifulSoup/
HTML pages and simulate the searching process. Additionally, we use Lichess APIs<sup>5</sup> to request for PGN games associated with a specific study ID.
Chess publishing This contains commercial annotated PGNs from Chesspublishing.com<sup>6</sup>, so we do not open source this source of data.
Megabase This contains commercial annotated PGNs from Megabase2023<sup>7</sup>, so we do not open source this source of data.
Pgnlib We collect annotated PGN from Pgnlib8.
Pathtochessmastery We collect annotated PGNs from Path to Chess Mastery<sup>9</sup>.
Gameknot We use Selenium [48] to scrape annotated PGNs from gameknot<sup>10</sup>.
E.4 Existing Datasets
We mainly extract all chess-related language corpus from existing dataset C4 [43], Pile [19], Oscar [37], Wikipedia [18] and RedPajama [53]. To extract chess-related language corpus, we first filter language corpus that contains the keyword 'chess'. We further utilize the deberta-v3-base-tasksourced-nli model<sup>1</sup>, which can conduct zero-shot classification based on classes specified by users. We set two classes: chess-related and non-chess-related for this model. We input the first 2000 characters of each text example to the model and set the threshold as 0.5.
E.5 YouTube transcripts dataset
To collect the youtube transcripts dataset, we first gather 19 keywords and utilize scrapetube $^{12}$  to gather all related channels. We extract all videos from these channels and use the same deberta-v3-base-tasksource-nli model mentioned in Appendix E.4 to filter all video transcripts and also the corresponding videos that are not relevant to chess. It is fairly easy to extract transcripts from videos and the main difficulty is to extract the FEN chessboard representations from videos. Here we mainly utilize two steps to extract specific FEN from chess videos.
E.5.1 Extract Chess Board from videos
The first step is to extract the chessboard from videos. We utilize GLIP [28], a zero-shot language-based detection model. By setting the prompt as 'Chess, chessboard', we can extract chess-related detection bounding boxes from videos. We conduct further filterings such as length-width ratio filtering to guarantee it is a valid chessboard bounding box in most cases, which will be processed based on the second step.
E.5.2 Convert Chess board image to FEN text format
Our second step involves converting the chessboard image into FEN text. FEN format serves as a great way to describe the current chess board state in text-based format. The pipeline that converts the chess board image to FEN format includes three main procedures - chess board decomposition, piece classification, and the prediction of which player is the next turn.
Chess board decomposition The aim of this section is to breakdown a whole chess board into 64 small tiles (8 rows * 8 columns), where each tile contains only one chess piece. To achieve this, we initially convert the RGB image to grayscale, preparing for the line detection process. Subsequently, we make two convolutional kernels to find horizontal and vertical gradients<sup>13</sup>. The Hough Transform is then applied to detect vertical and horizontal lines and filter out seven vertical and seven horizontal lines that fit the demand. Finally, we divide the board into 64 tiles by having the position of the 14 lines.
$^{5}$ https://lichess.org/api  
 $^{6}$ https://www.chesspublishing.com/content/  
 $^{7}$ https://shop.chessbase.com/en/products/mega_database_2023  
 $^{8}$ https://www.angelfire.com/games3/smartbridge/  
 $^{9}$ https://www.pathtochessmastery.com/  
 $^{10}$ https://gameknot.com/list_annoted.pl?u=all  
 $^{11}$ https://huggingface.co/sileod/deberta-v3-base-tasksource-nli  
 $^{12}$ https://github.com/dermasmid/scapetube  
 $^{13}$ https://github.com/Elucidation/tensorflow_chessbot/blob/master/tensorflow_compvision.ipynb
Table 11: Training hyperparameters for chess board to FEN
Hyperparameter	Value
Batch size	32
Number of epochs	10
Learning rate	0.001
Optimizer	SGD
Momentum	0.9
Piece classification To facilitate model training and evaluation, we employ an open source chess image dataset on Kaggle<sup>14</sup> which contains 80k training images and 20k testing images. Each tile can be classified into one of the 13 categories (p, r, b, n, k, q, P, R, B, N, K, Q, and space) which is detailed in Appendix B.3. We implement a model in PyTorch which uses a pre-trained ResNet18 [20] due to its well-established performance in image classification tasks. To adapt the model to our specific problem, we replaced the original fully connected layer with a new layer consisting of 13 output neurons, corresponding to the 13 pieces categories. We train the model on 40000 images with hyperparameters shown in Table 11.
After the training process, we evaluate the model on a testset with 20k images (equivalent to 128k tiles). Please refer to Table 12 for the final accuracy of each category.
Table 12: Validation accuracy of each chess piece
Color	Piece	Accuracy (%)
Black	pawn	99.98
rook	99.99
knight	99.98
bishop	99.98
queen	100.00
king	99.97
White	Pawn	100.00
Rook	99.98
kNight	99.98
Bishop	99.99
Queen	99.95
King	99.98
Space	100.00
Prediction of next turn As FEN format also includes the prediction of the next turn which is indicated by "w" for white, and "b" for black, the prediction of the next turn is accomplished by analyzing the main color of each tile. We use Colorthief, a library for grabbing the color palette from images, to extract the main color from each tile since the background color of a tile will be highlighted if a move is played on that tile. Hence, we find the highlighted tile by analyzing the tile color to know who is the current player and naturally infer who is the next turn.
Finally, we also provide a final certainty percentile to evaluate to what extent the generated FEN is correct by calculating the product of the accuracy of the 64 tiles.
E.6 Lichess dataset
We collect 5 months of Lichess dataset from the Lichess database [30]: 2013-02, 2014-02, 2015-02, 2016-02, and 2017-02. In fact, there are much more data available and we leave more game data for future work.
E.7 Pro-player dataset
We collect our pro-player dataset from PGN Mentor<sup>15</sup>.
$^{14}$ https://www.kaggle.com/datasets/koryakinp/chess-positions
15https://www.pgnmentor.com/
E.8 Chess books
We select 100 chess-related keywords and search for all related chess books (around 9k books) on the online PDF library. Because of the legal issues about books' copyright, we choose not to open-source this source of data. Instead, we only open source the list of books we use.
E.9 CCRL
We collect our CCRL dataset without comments from the official website<sup>16</sup> for three settings of time control: CCRL BLITZ, CCRL 40/2 FRC and CCRL 40/15.
E.10 Chess puzzles
We collect our chess puzzles from the Lichess puzzle dataset<sup>17</sup>.
E.11 Chess modeling data
We design 11 modeling tasks to generate data:
- Given PGN, generate FEN representation.
- Given a list of UCI moves, generate FEN representation.
- Given FEN and a UCI move, transfer the move to SAN format.
- Given FEN and a SAN move, transfer the move to UCI format.
- Given FEN, generate an ASCII board.
- Given FEN and a UCI move, generate the next FEN.
- Given FEN and a SAN move, generate the next FEN.
- Given FEN, generate all legal moves in SAN format.
- Given FEN, generate all legal moves in UCI format.
- Given PGN, generate all legal moves in SAN format.
- Given PGN, generate all legal moves in UCI format.
To generate the synthetic modeling dataset for these tasks, we utilize PGN data extracted from the first 1 million games of the Lichess database dump from March 2017. In order to encompass a wider range of ELO ratings, we divide the elo-rating into 9 intervals: 0-1000, 1000-1200, 1200-1400, 1400-1600, 1600-1800, 1800-2000, 2000-2200, 2200-2400, and 2400-3000. Random sampling is employed to select games from each interval, ensuring that our dataset contains approximately 10,000 games for each ELO interval. Consequently, the dataset achieves a balanced representation across different ELO ratings. Then we further utilize the python-chess library [41] to complete all the tasks we design to generate our final synthetic modeling dataset.
E.12 Preprocessing
We preprocess the data sources in three levels of granularity. For sources where existing preprocessed dataset are available, we filter out the subset that contains chess-related information without performing additional preprocessing. For sources that we retrieve from the Internet, we only parse portions of the HTML that contains information about chess. We implement different parsers for the different sources we consider. As a result, our data preprocessing can be more light-weight compared to previous work that extracts corpora from raw HTML web pages. For PGN games, we use the original PGN but filter out some annotations that are not useful for learning the model.
The different sources contain data in different formats. To facilitate training on all datasets, we preprocess all datasets to have the same json format.