Generating Creative Chess Puzzles
Authors: Xidong Feng
Vivek Veeriah
Marcus Chiam
Michael D Dennis
Federico Barbero
Johan Obando-Ceron
Jiaxin Shi
Satinder Singh
Shaobo Hou
Nenad Tomasev
Tom Zahavy
Venue: NeurIPS 2025 poster ()
Keywords: chess puzzle
creativity
generative model
Primary area: applications

Abstract: While Generative AI rapidly advances in various domains, generating truly creative, aesthetic, and counter-intuitive outputs remains a challenge. This paper presents an approach to tackle these difficulties in the domain of chess puzzles. We start by benchmarking Generative AI architectures, and then introduce an RL framework with novel rewards based on chess engine search statistics to overcome some of those shortcomings. The rewards are designed to enhance a puzzle's uniqueness, counter-intuitiveness, diversity, and realism. Our RL approach dramatically increases counter-intuitive puzzle generation by 10x, from 0.22\% (supervised) to 2.5\%, surpassing existing dataset rates (2.1\%) and the best Lichess-trained model (0.4\%). Our puzzles meet novelty and diversity benchmarks, retain aesthetic themes, and are rated by human experts as more creative, enjoyable, and counter-intuitive than composed book puzzles, even approaching classic compositions. Our final outcome is a curated booklet of these novel AI-generated puzzles, which is acknowledged for creativity by three world-renowned experts.

Generating Creative Chess Puzzles
Xidong Feng
Vivek Veeriah
Marcus Chiam
Michael Dennis Federico Barbero
Johan Obando-Ceron Jiaxin Shi
Satinder Singh Shaobo Hou
Nenad Tomašev Tom Zahavy
Gooole DeepMind 1 University of Oxford Milaa University of Montreal
{xidong, tonzahavyy@googgee.
Abstract
While Generative AI rapidly advances in various domains, generating truly creative, aesthetic, and iounter-intuitive outputs remains challenge. This paper presents an approach to tackle these difficulties in the domain of chess puzzles. We start by benchmarking Generative Al architectures, and then introduce an RL frame- work with novel rewards based on chess engine search statistics to overcome some of those shortcomings. The rewards are designed to enhance a puzzle's 's unique- ness, counter- mounter-ntuittiveess.. diversity, and realism. Our RL approach dramatically increases founter-intuitiie puzzle generation by 10x, from 0.22% (supervised) to 2.5%, surpassing existing dataset rates (2. 1%) and the best Lichess-traite model (0.4%) Our puzzles meet novelty and diversity benchmarks, retain aesthetic themes, and are rated by human experts as more creative, enjoyable, and counter- intuitive than composed book puzzles, Our final outcome is a curated booklet (Appendix M) these novel Al-generated puzzles, which is acknowledged for creativity by three world-renowned experts.
approaching classic compositions.
The idea of computers achieving human-level intelligence, first proposed by the Turing test [90], has been a subject of ongoing debate since the dawn computing [64]. From outperforming humans in complex games [72, 59, 8, 58] to demonstrating remarkable abilities in fields like medicine [79], law, business [44]. mathematics [28]. and coding [49, 24] Al's capabilities are rapidly expanding. Initial signs even suggest that LLMS pass a basic version of the Turing test [42], prompting the question of what cognitive functions will remain uniquely human.
Creativity is often viewed as the "final frontier" for AI [15]. Despite recent Generative Al demonstrat- ing notable abilities across various domains, they continue to fall short in creative problem-solving. abstract reasoning, and compositional complexity [40]. For example, while LLM-generated poetry can seem indistinguishable from human creations [68], expert evaluations indicate a deficiency in complex structural elements [19]. More generally, LLMs continue to struggle with creative writing [83], and tend generate practical but less original business ideas than humans [7].
This paper investigates the application of Gen AI techniques for creating aesthetic and creative chess puzzles [2, 48]. Chess puzzles have a history in education, online entertainment, and computational creativity research [3], yet, the generation of creative chess puzzles is difficult due number of reasons. Chess puzzles have non-trivial pattern that only reveals itself when inspecting the solution. Moreover, a standardized definition of a chess puzzle is absent, making it challenging to objectively measure its creativity or aesthetic qualities. The limited and diverse nature of available chess puzzle data, which varies in puzzle types, objectives, piece utilization further complicates the process.
Our work begins by formalizing the definition: what makes a board position creative and interesting that engage chess players? We concentrate well-established aspects of creativity in chess [48, 2]: intuttieintti aesthetics and novelty. Counter-intuitive solutions are those that, at
Reseearch conducted during an intershiip at Google DeepMind
Ryan Pachauri and Thomas Tumiel contributed but were inadvertently omitted from the author list.
39th Conference Neural Information Processing Systems (NeurIPS 2025).

---

Reinforcement Learning with Machine Feedback
Puzzle Verification Reward
Search
Countee-nta Tiaenis
Chess engine (Boocfal and Aphalmn))
Navilyy
Uniquennde: (best action bralb
Append
Train
Sample
Lichess Dataset (4.4 sample))
Generative
Aesthetic checks
Model
Figure 1: Our approach begins by training a generative model on the Lichess dataset (Section 3.2). followed by RL training (Section 2.1). Each position generated by the RI. model verified for legality, uniqueness, novelty, and nounter-intuitivenss: using chess engine search statistics (Section 1). The positions are filtered for aesthetics (Appendix 1) and selected based on reward for our booklet.
first glance, seem are terrible out-of-mind, but are, in fact, brilliantly effective. Aesthetics refers to the visually or intellectually beauty or elegance of chess position and its solution. Novelty straightforward: players are often drawn to positions or solutions that are uncommon or haven" t been frequently in normal chess game. In Section 1, detail the logic of these metries and present how we can quantify them with computation. These proposed quantifications are not merely descriptive; they form the basis of the reward functions and evaluation metrics used throughout our study. To help readers better appreciate what makes a puzzle creative, Fig. offers a detailed analysis on booklet examples. Next, we prioritize chess puzzles with single, unambiguous solution: multiple 'correct' paths dilute the satisfaction of finding the sharpest, most brilliant line. This criterion also allows us leverage the large open-source Lichess Puzzler [52] dataset. As shown Fig. about the volume of games played online (2021-200 2021-2025), only 2.1 of annually one million puzzles meet our counter-intuitivenes criterion a comparatively small number for AI applications.
Given these chess puzzle metrics, we initiated our work by evaluating several Generative Al archi- tectures, including auto-regressive transformers [91]. latent diffusion models [70], masked discrete diffusion models [77], and MaskGIT [11] all trained the Lichess Puzzler dataset. We bench- marked these models by generating one million positions from each and assessing them based on
legality, uniqueness, counter-intuitivenes. and novelty (results detailed in Tables 2 and 3).
L
sPy sayy TRO JPN +980
2 +h40 24E Davg
190
2R0 F&N +NEU paal
Figure 2: Booklet examples of creative chess puzzle, generated with our methods, with unique, counter intuitive and aesthetic solution (written upside down). 1 (left): White undefends both rooks with Rg6+. After one of the hanging rooks is captured, White plays the slow Qa1 move, sacrificing the second rook. The remaining White queen and bishop coordinate very well after Qf6+, with an unstoppable attack. The double rook sacrifice counter intuitive and aesthetic for a human: both rooks are initially very active and it is surprising that the remaining queen and bishop are sufficient to win the game. The winning move in this position is both counter-intuitive and the only one for White. Even Stockfish needs a moment to identify it (see analysis on lichess), but then confidently confirms it's the sole path victory. For description of positions 2-4, please see Appendix I.

---

To tackle the limited availability of chess puzzle data, we further trained the transformer model with RL. This reward function included Lichess-style uniqueness check and counter-intuitvvnesss measure designed to identify positions solvable by strong engines but not weak ones. This latter component, tuned with human feedback (Section 3.1), was based on a linear combination of search statistics from StockFish [86] and AlphaZero, with the 'critical depth" (the depth StockFish first finds the final solution) being the dominant term. In addition, we found that RL with pure puzzle reward would quickly succumb "entropy collapse' by repeatedly generating one single high-reward puzzle. Therefore, we developed a diversity filtering mechanism (Section 2.1) that promotes exploration and diversity. This was proved essential for preventing reward hacking and largely increased training stability, enabling the continuous generation of novel and diverse puzzles (Section 3.4).
Our findings demonstrate that RI. markedly enhances the generation of counter-intuitive puzzles. The probability increased significantly from 0.22% (transformer trained on Lichess) to 2.5%, exceeding both the best Lichess-trained model's 's rate (0.4%) and the baseline in the Lichess training data (2.1%) while satisfying novelty and diversity criteria. While aesthetic features were not directly verified by the reward and thus not optimised for during the RL training, observed that aesthetic themes were well- well-preserved in the RL samples (Section 3.3). Crucially, expert evaluation confirmed the success of our approach, demonstrating that some of our generated chess puzzles are fun, creative and counter intuitive puzzle compositions from the chess literature. Our final result is booklet of curated chess puzzles in Appendix M; a few examples can be seen in Fig. 2. The booklet is further reviewed by three world-renowned experts and garners the acknowledgement for their creativity.
1 Quantifying the standard of creative chess puzzles
Originating in the ninth century, chess puzzles predate modern chess rules and present chess-related problems solvable through knowledge of the game. This work concentrates on legal chess positions with a unique, optimal, and ideally aesthetic solution sequence, excluding puzzles with multiple solutions, non-standard elements, special stipulations, retrogrades, missing information, shortest mates, or mathematical aspects. Mate-in-x puzzles are considered only if they have a single correct solution. For an overview of various chess puzzle types, see [17, 87, 31, 85].
To quantify the standard of creative puzzles, we focus on puzzles with a unique solution. Following this, we examine creative aspects of puzzles such as novelty, counter- intuitiveness, and aesthetics.
To confirm that a chess position represents a puzzle with a single unique solution, we utilize the StockFish engine, following the Lichess Puzzler methodology. This involves recursively examining the principal variation to compare the winning chances w(a s) of the best move a' against the second best move (see Appendix C for more details). A move is considered unique if the difference in these winning chances meets or exceeds empirically chosen threshold (Tani 0.5):
Uniqueness:
Funi-
max w(a s) aEA(s)\(a))
max w(a s) 2 Tuni-
(I)
aEA(s)
This is paradoxical concept, often linked to rule-breaking and unconventional thinking. can be challenging to define precisely. For instance, Wikipedia [96] defines creativity as "the production of novel, useful products" [61], while also acknowledging the existence of significant variations among different definitions. Similarly in chess, professional players often share beautiful combinations from their games, and some games are considered aesthetically pleasing. However, considered beautiful remains
Creativity:
articulating the specific reasons why these games or combinations challenging, even for experts [48]. Nevertheless, a few aspects of chess positions had been associated with creativity in the chess literature [66, 2, 48]. Its important to note that while these aspects are not necessary or sufficient conditions for creativity, they are often associated with it.
Novelty:
Creativity the generation or expression of new ideas perceived as valuable or interesting. A key aspect of this definition is the subjective nature of "newness," which depends on the context-an idea might be novel to Al agent, specific individuals, or universally. While genuine invention occurs, most chess creativity involves applying existing concepts in innovative ways. For instance, grandmasters engage in creative work during opening preparation by seeking lines that are challenging. suit their playing style, or exploit potential inaccuracies in computer evaluations. This form of finding contextually effective new ideas aligns closely with the concept of "novelty" in Al terminology.
To quantify the novelty of generated chess positions we propose several metrics. First, we measure syntactic similarity using the Levenshtein distance [47] calculated over the FEN string representing the board (piece configuration) and the Principal Variation ([12), PV) derived from StockFish For the board distance, the Levenshtein distance roughly corresponds to the number of pieces that have
3

---

to be changed to get from one position to the other. For any two board positions b and b', the board distance defined as disthoard (b, 6¹) = Lovenshtein(FEN((), (E)( (EE)) and the PV distance is defined as distPV (b, B¹) Levenshtein (PV PV()))/ max (length (PV(b)) length (PV())))
To further capture the distance in the semantic space, we also leverage the generative model's uncertainty scquence-level entropy metric for diversity. High entropy suggests the model finds the current state less predictable, potentially indicating more novel, complex, or unusual semantic context compared to low-entropy states representing common patterns. Take an auto-regressive example, for model Tт and a board sequence string sT, the sequence-level
transformer
entropy is calculated as the average token-level entropy: H + E (ss+11 (So.), where 8t indicates the i-th token in the sequence and S0. indicates the sequence of tokens.
Counter intuitiveness. In chess, intuition often obscures ad advatageouus yet seemingly unsound moves like sacrifices tempo loss [66]. Players may overlook certain options or even consider them [48]. As formal logic alone can also be misleading [66], counter-intuitiv chess puzzles are considered creative because they demand players to think unconventionally, and question their intuition [2].
To quantify chess move's counter-intuitivenee we compare evaluation scores from chess engine using different search depths. A "shallow" search approximates an intuitive assessment, while a "deep" search serves as proxy for more accurate evaluation. This concept is extended through multiple searches with varied search budgets (t). The resulting data allows to measure counter- intuitiveness by: the evaluation difference (Eq. (2)), the area under the curve (AUC) of this difference over search time (Eq. (3)), or the critical search point where the evaluation becomes stable (Eq. (4)).
Search Gap
Vgap
VT
Vol:
(2)
Vaue
V - Veldl,
(3)
Max Gap
1=
A U c
Vep min t, subject to Vr Vel S T,
(4)
Threshold_.
F Fent
Σ W.Vi
Iont Fent > Tent-
Timeit
Shallow
Critical
Deep
Point
Figure 3: Counter-intuitivenes calculation visualization and equations.
More broadly, we can represent coumter-intuitiveness score Fent as a linear combination of various search statistics v, (Eq. (5)). We will discuss candidates for V; in Appendix F and a method for determining the weights w, by tuning them a curated Golden Set of counter-intuitive positions collected from diverse sources Section 3.1. Ient will serve as counter-intuitivenes indicator.
Aesthetics. Chess beauty, like creativity, is an clusive concept, but is often associated with specific patterns, structures, or themes within the game. The notion of introducing order into concept like creativity, which is by definition opposed to order, seems paradoxical, but there is logic in this madness. Creativity is not such a mysterious non-consequent and unfathomable process as it appears to many people [2]. This is because many of these structures designed defy some human heuristics for what constitutes a good move (like not losing material or tempo). Other structures may be simply appealing to human players. Previous research has explored classifying chess positions by aesthetic themes. Notably, Iqbal al. [38] identified 17 such elements and showed that a corresponding score function aligned positively with expert aesthetic judgments. Lichess also defines themes for their puzzles. Our work combines elements from both these sources, along with themes drawn from chess literature [2, 48]. Using this consolidated set, we developed detectors to automatically classify puzzles across diverse aesthetic themes, as detailed in Appendix I.
Methods
Training discrete generative models. A chessboard has 64 squares that are either empty or occupied by one of six chess pieces. We represent it using the Forsyth- Forsyth-Edwards Edwards Notation ([13), FEN)

---

discrete sequence and adopt a thess-speciallizee tokenization [71] with a vocabulary of 31 tokens. Using this representation, we can use established discrete generative models to generate chess puzzles. The models were trained exclusively on the Lichess Puzzler [52] dataset (without any pre-training), excluding any chess compositions or other datasets. We experimented with different candidates including auto-regressive transformer [91], MaskGIT [11], latent diffusion model [70, LDMs] and masked diffusion model [77], more details can be found in Appendix D.
2.1 Reinforcement Learning from puzzle feedback
RL [84] enables us further train our generative model generate better chess puzzles by verifying the puzzle metrics (Section 1). For implementation simplicity, we focus on the auto-regressive transformer and utilize a critic-free variant of Proximal Policy Optimization [73, PPO]. This version estimates the value/advanttage function using Monte-Carlo samples [50, 1, 75, 43, 33].
Puzzle generation as an RL problem. Unlike scenarios with per-step rewards (e.g.. navigating a maze or playing an Atari game step-by-step), the relevant quality metrics for a chess puzzle can only be comprehensively evaluated after the entire puzzle has been generated. This feedback shows resemblance to the paradigm of Reinforcement Learning from Human Feedback [14, 62] and reasoning models that are trained with outcome- based reward [30, 41J. In both scenarios, the model generates a complete output (a puzzle or text response), and a reward signal, derived from evaluating this entire output (e.g., based on puzzle quality checks or human preference ratings/answer correctness). Thus, puzzle generation with autoregressive transformer aligns with token-level Markov Decision Process (MDP) with outcome reward. In this view, the state s; = (I1,... 2zg is the sequence generated up to step and taking an action da corresponds to selecting the next token Tt-1 from the vocabulary V. The transition deterministic: St+1 = St i 2 (concatenation). The (ft+1 1 |8;; 0). providing the probability transformer itself acts as the policy TT (8)) = Ptransformert (ormer(
for the next token. Rewards are typically sparse, with zero intermediate reward and final outcome reward Routcome assigned to the complete sequence 87. reflecting puzzle's 's quality or validity.
We define our outcome reward Routcome (8T) with the uniqueness and counter intuitiveness shown in Eq. (1) and Eq. (5). Specifically, a position is considered a qualified puzzle and receives a reward of +1 only if it satisfies uniqueness standard (Eq. (1)) and its nounter-intuiiieenss score (Tent in Eq. (5))
surpasses threshold Tene = 0.1. Unqualified but legal position receives a reward of O. Positions identified as illegal are penalized with a reward of 22, such that the final reward is
Rousoome (ST) = [1, 1,, If Tuni 2 Tuni A Tent 2 Tent 0, if Legal else - 2.
(6)
Realism. Ensuring the realism of generated chess puzzles is crucial. Unlike puzzles derived from actual games, composed or computationally generated puzzles sometimes feature unrealistic positions. While the precise impact of realism on puzzle appeal or educational value is debatable, it's often considered desirable. Previous work [37] addressed this via post-generation modifications and will be discussed in Appendix A. In contrast, we employ several RL--based techniques, drawing inspiration from RLHF practices [62] to maintain realism and prevent deviation from the the lichess dataset with three mechanisms: (a) KL Divergence constraint: We incorporate a token-level Kullback- Leibler (KL) divergence penalty r.- = - K LL [s+) [pretrained (a (s)) into the reward function. This discourages the RL policy Ty from straying far from the policy trained on the lichess data Ttickess. thus preserving similarity to the training data distribution. (b) Training data mixture: We collect 100k high-quality examples filtered from the lichess dataset using the Routcome check in Eq. (6). These puzzles are used to seed the replay buffer discussed in the next sub-section. (c) Piece regularization. We find that the model can easily learn to artificially inflate reward and entropy by adding pieces (e.g., two white queens or three black knights). Thus, we zero out the board's reward if the count of any piece exceeds its count in the beginning of the game.
RL. The RL setup detailed above, can in principle, be satisfied by an agent that produces a single high reward puzzle with very little or no diversity. As we will soon see in Appendix H. this indeed happens frequently when maximizing reward, even with explicit entropy regularization. To address this, we leverage novelty filters. First, we check that a generated position is novel by measuring its board and pv distance against all other qualified positions within the same batch Litra-batchi and preceding batches (inter-batch). A position b passes the novelty test if Thoard dist"ourd()) (b) 2 Thoard and Tpy- dist (b) 2 Trv. We maintain a replay buffer to store preceding qualified and novel samples and subsample a small set of it for each inter- batch checking. In each RL step, we sample 16 positions from this replay buffer to enrich the qualified
Creating novel puzzles with diversity-filteriin
5

---

samples in the training batch. Subsequently, we calculate all samples' sequence-level entropy in the training batch (including those from replay buffer samples) and mark position as novel if the board's sequence-leve entropy (Sr) surpasses the predefined entropy threshold: Lem = H (b) 2 Tent- Note that this filter is different from entropy regularization because it only penalizes low-entropy generations, thereby avoiding the direct incentivization of high entropy generation. Qualified positions (Routcome(s)) = 1) that pass these diversity filters are considered novel, receive a reward of and are added to the replay buffer. All other positions, regardless of their score, get reward of 0 -2 (if illegal). Thus the final reward with diversity filtering is defined as
Rdii
outcome (ST) =1
If Routcome (ST) Joward A IPv A Tent
else Routcome (ST)
3 Experiments
We begin with a study of the counter intuitiveness reward component in Section 3.1, which is then used, in addition to novelty and uniqueness, to compare and select the best model in supervised- training experiment in Section 3.2. Once we identified good generative models, we post-train them using RL to generate counter intuitivene puzzles. We ablate and analyze the RI objectives in Section 3.4. We then analyze the creativity of our generations by studying their aesthetics in Section 3.3. We conclude with a human expert study comparing our generations with baselines in Section 3.5. Lastly, we present our best cherry picked and auto-selected generations in a booklet that can be found in the supplementary. We refer the reader to Appendix D for implementation details.
3.1 Tuning the counter-intuitivener reward Fent
We begin our experiments with study of the counter intuitiveness component. We extract counter intuitiveness features from the search statistic of chess engines and then compute the reward as a weighted combination of these features. Explicitly, let v, be vector of search statistics compute for a chess position then, the counter intuitiveness reward (s) = w (s(s), where the weight vector w; contain values in the [0, 1] interval. The subscript represents reward candidates.
The Golden Set The weights w, are tuned on small, manually curated set of 39 positive and 45 negative TRAIN examples. The resulting weights were subsequently evaluated on a similar, independent TEST set containing 21 positive and 20 negative examples. These examples are chosen to capture human intuitions about the counter-intuitivensss. aspect of chess puzzles that human players find appealing or interesting, and are selected from a variety of sources including books [66, 2, 48], the Lichess Puzzler dataset, and preliminary samples generated from our AR model after filtering for uniqueness. See Table 8 and Table 9 in the appendix for the examples. Given a particular w, and positive and negative examples {sx 1 1= we first sort the examples using the iunnter-ntuittveensss reward to produce a sorted dataset D'. We then compute the Average Precision (AP) metric [97] as:
1
AP;
Precision(k)) Ipos
Precision (1:) = pos(j);
(8)
ripos
j=1
where repos is the total number of positive examples, 4poa (k) is an indicator function equaling if the example rank k in the sorted set D' is a positive one, and Precision(k) is the precision cutoff k, i.e. the proportion of positive examples at rank k. Intuitively, the better the counter-intuitivenee score is at ranking the positive examples ahead of the negative ones, the higher the AP metric should be. practice, we tune the weights w, to maximise the AP metric using a form of random search on a discretized grid with step size of 0.1, i.e. 0.0, 0.1, 0.2, 1.0.
TRAIN
TRAIN+TEST
TEST
Uniqueness
0.4768
0.4326
0.3788
(1) Search Features
0.6619
0.6222
0.5487
(2) SF through time
0.7280
0.7312
0.7401
(3) + AZ through time
0.7475
0.7403
0.7360
Table 1: The average precision (Eq. (8)) of counter-intuitiveness score functions.
Table presents comparison of the average precision achieved by various candidate counter- intuitiveness scoring functions on our curated Golden Set Each scoring function was optimized using grid search to maximise average precision the TRAIN subset. Each candidate score function

---

also leverage more StockFish and AlphaZero features than the ones before it. We selected the best performing configuration on the TEST set, (2) Search Features SF through time, as our counter- intuitiveness. This configuration yielded sparse weights, with only two features having non-zero values: StockFish critical point (Eq. (4)) with a weight of 0.8 and normalized negative capture material (the negative value of captured material, weighted according to standard valuation ([95), divided by 9) with weight of 0.1. A complete list of weights, other tuned configurations. details on the uniqueness baseline, correlation analysis with the lichess rating and popularity scores and the average precision of specific search features can be found in Appendix F.
3.2 Generative models
Models. For autoregressive transformer, MaskGIT and masked diffusion model, we leverage the same tokenization and 200M parameter transformer from [71]. The model is a decoder-only trans- former with causal masking, post-normalization and SwiGLU [76]. For the latent diffusion model experiment, we investigate the impact of different chessboard representations on the generative model's ' performance. Since latent diffusion models typically operate on 2D image data, we explore representing the chessboard in 2D format. We first train VAE model on the same puzzle dataset. For the further diffusion model training, we employ the U-net transformer architecture from [67] with the frozen VAE encoder. To ensure comparability, the U-Net Transformer's 's hyperparameters are configured to yield approximately 220M parameters. More details can be found in Appendix D. For each model the checkpoint with the lowest test loss is selected for the final evaluation. We compare our models with two baselines: the lichess dataset and standard game dataset (2024-11) [51].
Lichest-Gamme
Lichess Pupzzls
Transformer
MaskGTT
Latent Diffusion
Masked Diffusion
Logal
100.00
100.00
99.07
97.61
97.14
99.72
Unique
4.32
95.25
23.44
27.76
22.31
Counter intuidive
0.71
2.25
0.93
1.44
0.85
1.11
Puzzle
0.03
2.14
0.22
0.40
0.19
0.34
Table 2: Generative models benchmark. Metrics are reported as percentages (%), for IM generated
positions (generative models) and IM sub-samples (Lichess datasets). Puzzle refers to positions that
meet both the uniqueness and inuuter-ntuutiiveesss criteria.
Results. Table 2 and Table 3 summarize our benchmark results for puzzle quality and novelty scores, respectively. As expected, the lichess dataset, which served as the training data, exhibits the best performance across all metrics, acting as practical upper bound. All four generative models significantly outperform the standard game dataset baseline. Additionally, the high board/PV distance w.r.t the lichess dataset, as well as the self metric, suggest that they are capable of generating novel and diverse positions. Among the generative approaches, Masked Diffusion and the Transformer achieve high legality ratios, while MaskGIT and Masked Diffusion excel in uniqueness and counter- intuitiveness. The Transformer and MaskGIT demonstrate strong performance on the various diversity metrics. Latent Diffusion, however, yields relatively moderate results across most indicators.
Notably, it is challenging to declare single superior model, none dominates across all scores, perhaps because the metrics present trade-offs. For instance, inspecting Fig. 8 in the appendix, reveals that the AR transformer achieves higher puzzle scores and less novelty as it is trained for longer at the cost of overfitting to the training data. Our checkpoint selection strategy, based on minimizing test loss on a held-out set, might also influence these relative performances. We observed that the AR model's test loss begins increase after approximately 15k training steps, leading to earlier checkpoint selection. In contrast, MaskGIT and the diffusion models did not show a similar increase in test loss even after 1 million training steps. Consequently, this selection strategy might curtail the AR model's 's training before reaches its peak puzzle score potential.
3.3 Creativity
Novelty. Inspecting Table 3 indicates that the datasets sampled from the generative models exhibit both novelty (when compared to the Lichess data) and internal diversity (when compared other samples from the same model). For instance, the top model, MaskGIT, achieves a self-comparison board distance of 14.57, surpassing the diversity baseline of the Lichess puzzles (11.914). Its novelty score relative to Lichess is 11.44, which is close to the Lichess baseline (11.914), suggesting its outputs are distinct from the training data. The board distance roughly equates to the number of piece changes separating generated position from its nearest neighbour. Values around 10 thus suggest relatively high novelty. To explore this qualitatively, the booklet presents each generated puzzle next to its three closest matches from Lichess, allowing readers to visually assess its distinctiveness.
7

---

Lichess-Game
Lichess Puzzle
Transtormer
MaskGTT
Latent Diffusion
Macked Diffusinn
Board distance (Lichess)
8,984
N/A
9,013
11,444
8,928
Board distance (Sell)
10.715
11.914
11,486
14,570
11.393
10.859
PV distance (Lichess)
0.995
N/A
0.747
0.855
0.758
0.637
PV distance (Seli)
0
0.949
1,084
0.947
Table 3: Novelty benchmark. For each position, we compute the distance to the Lichess dataset and
the distance to other samples from the same model ("Self"). The final metrics are averaged over all unique positions. We use 100k samples per method for
and 40k (the number of unique positions in the Lcches-Gaame 1M positions) samples for self-distance.
Emergence of Aesthetics. Fig. 4 illustrates how the positions in each data set are distributed across various aesthetic themes (detailed in Appendix I), whereas Fig. 18 and Fig. 19 in the appendix show the distribution across positions with a unique solution, and across positions with a unique counter intuitive solution respectively. Puzzles from the Lichess dataset are shown in blue, those from our AR generative model in orange, and those from the RL agent green. Aesthetic themes were identified only after the puzzles had been created, so they did not inform or guide the training process.
Analysis of the figures reveals diverse aesthetic themes across all three datasets. Their appearance in the Lichess dataset indicates that aesthetically pleasing positions naturally occur in human games, without deliberate creation. However, theme prevalence varies, with certain mate patterns like Boden's 's Mate and Arabian Mate being less frequent in Lichess data, potentially due to the dataset's filtering for unique positions where multiple winning moves can exist in mate scenarios. The figures also show that the AR model largely mirrors the original dataset's 's distribution of aesthetic themes; some themes are even more common in AR samples due sampling variations. Notably, the RL distribution aligns with AR and Lichess, despite not being explicitly trained for aesthetics.
Connacing Uchoos 48 ard Et IN chect
Figure 4: Distribution of aesthetic themes in chess positions across different datasets.
3.4 Reinforcement Learning
For simplicity, we conduct the RL training phase with the auto-regressive transformer. Our first results performed Reward maximization with the reward in Eq. (6) and an entropy regularization, starting from the -tress- supervised model. However, as the results in Appendix H. suggest, this setup suffered from entropy collapse and non realistic positions. Appendix H.2 and Appendix H.3 explain how we solved these issues respectively. Further ablation studies can be found in Appendix H.4.
We conducted our RL runs and compared the performance of the RL model with that of the generative model baselines that were trained only on lichess. Fig. 5.(a)-(c present the results for two RL variants: one trained solely with RL updates and another incorporating auxiliary high-quality Lichess puzzle data during training. Fig. 5.(a) demonstrates that RL significantly enhances the model's performance, achieving an average puzzle score that surpasses even the strongest baseline, the Lichess puzzle dataset itself. Interestingly, despite the uniqueness score being a component of the reward function (via the qualification threshold), the uniqueness ratio remains relatively stagnant around 20% in Fig. 5.(c). Instead, the primary gains stem from substantial increase in the counter-intuitivenes. score. Consequently, the overall rate of qualified puzzles (passing both uniqueness and counter- intuitiveness thresholds) generated by our RL model becomes comparable or slightly superior to the Lichess puzzle baseline (Fig. 5.(b). It is crucial to note that this parity is achieved even though the Lichess puzzle baseline has been explicitly filtered for uniqueness in advance while RI does not.
Fig. 5h) also illustrates the evolution of diversity metrics throughout the training process. Across all measured aspects (c.g. board distance, PV distance, entropy), the generated puzzles demonstrate
8

---

1013
012
600
200
002
300
200
003
300
300
3039
450
1000
200
200
200
1273
0310
Figure 5: RL run surpasses the baselines in puzzle metrics and keeps improving diversity through the training. We smooth the curve in (a)-(d) for better visualization.
consistent upward trend training progresses. This observation strongly validates the efficacy of our proposed diversity filtering mechanisms. The final RL model not only produces a higher rate of qualified puzzles but also generates results that are significantly more diverse than the initial lichess-trained supervised model or models trained with less sophisticated RL setups.
3.5 Human study and booklet
We further conducted human study with 8 chess experts (Lichess Elo 2000-2400) to evaluate the quality of the generated puzzles and compare them with creative puzzles recognized in the chess literature (Full results are in supplementary materials). The study included puzzles from four different sources. (1) Lichess Puzzler (2) RI. (auto-selected) (3) Booklet (4) Chess books (human selected) [2, 48, 66]. 10 puzzles were selected from each source and were presented to the raters in random order. The selected puzzles were required to satisfy our puzzle check, i.c., they had to pass the counter-intuitiveness and uniqueness checks. We also verified that the book puzzles satisfy the theme function. The booklet puzzles were collected from the outputs of various methods. We then filtered the puzzles based on aesthetic themes and sorted the positions each theme by the counter intuitiveness reward. Experienced players (FIDE rated 2200-2300) reviewed the top 50 puzzles from this process and selected small number as potential candidates for the booklet, more details can be found in the supplementary material. For each puzzle we include the three closest puzzles from the Lichess training dataset, identified using edit distance, and reference book puzzles for each theme.
To reduce bias, the Lichess and RL puzzles were selected from sub sampled set of exactly million puzzles. Furthermore, to maintain an unbiased estimation across aesthetic themes were selected for the study (sacrifice, underpromotion, attacking withdrawal, novotny, interference, unprotected position and knight on the rim is dim). The puzzles for each source were selected with the same aesthetic theme distribution as the other methods. The themes were selected such that all the methods could satisfy these counts. For example, the Lichess data did not include all the themes from the books, and some of the book examples were not unique or did not satisfy the theme when analyzed with StockFish Given the aesthetic theme counts, the puzzles from each method were ranked according to the counter nntuitiveness-rewaan puzzles were randomly shuffled to further reduce bias. They were rated from 0 (poor) to 3 (perfect) across five different metrics: realism, difficulty, creativity, fun, and counter-intuitvvensss.
and the top samples were selected for the study. The
Puzzle Source
Realism
Difficulty
Creative
Fun
Counter Intuitiveness
Book puzzles
2.74 + 0.45
+ 0.33
2.39 + 0.39
2.22 0.3
2.05 + 0.2
Lichess RL (auto-selected)
2,53 : 0.41
2.41 1 0.34
1.7 + 0.2
1.49 1 0.17
1.7 : 0.18
2.28 1 0.38
2.59 1 0.24
2.1 1 0.37
1.96 1 0.19
1.98 1 0.19
Booklet
2.59 = 0.33
2.39
+ 0.15
2.48 + 0.25
2.56 - 0.24
2.12 + 0.2
Table 4: Avg ratings from 8 chess experts from rating task.
The study's findings, detailed in Table 4 with average scores and standard deviations across five metrics, reveal distinct puzzle performance. In terms of Realism and Difficulty, Book puzzles ranked highest, followed by Booklet puzzles, then RL (auto-sclected) puzzles, and finally Lichess
Results.

---

Puzzler puzzles. For Creativity, Fun, and Coumter-intuitivenss. Booklet puzzles led, with Book puzzles next, followed by RL (auto-selected) puzzles, and Lichess Puzzler puzzles again ranking last. Notably, these results are significant because they demonstrate that Al Al-generated puzzles surpassed the data they were trained on and even human-composed puzzles in certain aspects.
To set an even higher standard of evaluation, invited three world-renowned experts provide in- depth review of our booklet: International Master for chess compositions Amatzia Avni, Grandmaster Jonathan Levitt, and Grandmaster Matthew Sadler. The experts' assessments were exceptionally positive, commending the puzzles for their creativity, novelty, and aesthetic merit. They regarded the collection as a pioneering advancement in the human-Al partnership for chess composition. The complete, detailed review is provided here.
4 Discussion and limitation
This paper presents a significant step towards generating creative chess puzzles using GenAl and RL techniques. The research successfully demonstrates that an RL framework, equipped with novel reward functions promoting uniqueness, counter-intuitiveness,, diversity, and realism, can substantially enhance the generation rate of such puzzles. This is particularly compelling because existing methods, like those used by Lichess, depend on a finite stream of human games and involve intensive re-analysis of positions primarily for uniqueness. We found that among Lichess's unique puzzles, only about 2. 2.1% align with our definition of counter-intuitive yielding a limited set of approximately 20,000 new creative puzzles per year. Our AI-driven approach surpasses this, achieving 2.5% rate for unique, counter-intuitive puzzles. Furthermore, the efficiency of generating candidates with our comparatively small generative models (c.g., 200M parameters contrasts favorably with the engine analysis Lichess performs per puzzle. This underscores Al-powered generation scalable and promising path to continuously deliver a rich and varied stream of creative chess puzzles.
A critical observation from our experiments is that achieving a high reward does not automati- cally equate to high human-perceived creativity. The models exhibited tendencies towards "reward hacking." " such as entropy collapse (generating the same puzzle repeatedly). producing unrealistic samples (e.g.. boards with excessive pieces) or exploiting chess engine weaknesses (making the position artificially more complicated for engines, for example, by allowing the defender deliver meaningless checks). While diversity filtering and realism constraints helped mitigate these issues, there remain gap in our understanding of creative chess positions due to the subjective, multifaceted nature of human creativity. The iterative refinement of theme- detecting functions also highlighted the difficulty in encoding nuanced human intuition about what makes a theme's 's presence "significant" or "creative". This emphasizes the necessity of human expert evaluation the arbiter of creativity. The human evaluation further corroborates these findings, with experts rating the booklet puzzles as more creative, fun and counter-intuitive than the book compositions, while the auto-selected puzzles were rated lower than the book compositions but higher than the typical Lichess puzzles, and approaching the quality of classic compositions. The creation of a chess booklet featuring curated puzzles and the subsequent human expert study were invaluable components of this research. This feedback loop is essential for validating the quality of generated puzzles and refining the generation process.
For future work, we believe key components of our methodology are broadly generalizable and could be applied beyond the domain of chess in computational creativity. For instance, the counter- intuitiveness reward that contrasts shallow and deep search evaluations, is not specific to chess. This principle of rewarding 'surprise" or non-obvious solutions could be readily transferred to other domains that rely on search iterative reasoning, such as the game of Go, automated theorem proving, or even prompting 'deeper thinking' in large language models. Furthermore. the 'entropy collapse' we observed (detailed in Appendix H) is fundamental challenge when using RL for novelty-driven tasks. Our weessit--ilterinn framework, which actively maintains novelty and prevents distributional collapse, offers a generalizable blueprint for training generative models to produce a rich and varied stream of creative outputs in other complex fields.
10

---

Acknowledgements
We would like to thank Demis Hassabis for discussion and feedback on our work. We thank Aleksei Ostapenko, Arpit Hamirwasia, Daniel Körnlein, John Reid, Joon Lee, Kola Adeyemi, Matteo Tortora, Preet Sardhara, Sachin Ravichandran, Sagar Jha, and Temirlan Ulugbek uulu for their participation in evaluating our chess puzzles through various rating tasks, and particularly for their engagement in our Human study (Section 3.5). We also thank them for their valuable discussions. The feedback instrumental in directing
from these tasks for feedback on the booklet, David Abel for reviewing the paper, Simon Osindero, Arnaud Doucet and Clare Lyle for joining discussions and providing feedback, and Gabriela Fernandez-Cuerv for engaging with chess experts.
work. We would also like to thank Lisa Schut
References
[1] A. Ahmadian, C. Cremer, M. Gallé, M. Fadace, J. Kreutzer, O. Pietquin, A. Üstün, and S. Hooker. Back to basies: Revisiting reinforce style optimization for learning from human feedback in 1lms, arXiv preprint arXiv:2402.14740 147400 2024.
[2] A. Avni. Creative Chess. Everyman Chess, 1991.
[3] D. Balduzzi, S. Racaniere, J. Martens, J. Foerster, K. Tuyls, and T. Graepel. The mechanics n-player differentiable games. In International Conference on Machine Learning, pages 354-363. PMLR, 2018.
[4] M. Bedau. Measurement of evolutionary activity, teleology, and life. 1992.
[5] M. A. Bedau, E. Snyder, C. T. Brown, N. H. Packard, et al. A comparison of evolutionary activity in artificial evolving systems and in the biosphere. European conference on artificial life, pages 134.134. MIt Press Cambridge. 1997.
Proceedings of the fourth
[6] M. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos. Unifying count-based exploration and intrinsic motivation. Advances neural information processing systems, 29, 2016.
[7] L. Boussioux, J. N. Lane, M. Zhang, V. Jacimovic, and K. R. Lakhani. The crowdless future? generative ai and creative problem-solving. Organization Science, 35(5):1599-1607, 2024.
[8] N. Brown and Sandholm. Superhuman ai for multiplayer poker. Science, 365(6456): 885-890, 2019.
[9] Y. Burda, H. Edwards, D. Pathak, A. Storkey, T. Darrell, and A. A. Efros. Large-scale study of auriosity-driven learning. arXiv preprint arXiv:1808.0433 2018.
[10] Burda, H. Edwards, A. Storkey, and O. Klimov, Exploration by random network distillation. arXiv preprint 18Xiv: 1280. 2018.
[11] H. Chang, H. Zhang, L. Jiang. C. Liu, and W. T. Freeman. Maskgit: Masked generative image transformer. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11315-11325, 2022.
[12] Chess Programming Wiki. Principal Variation. htttps://wwx https///www.eeeespprrrraingggnn org/ Principal.Variation. Variation, 2025. Accessed: 2025-10-23,
[13] Chess.com. Forsyth-Edwardd Notation (FEN). https://www. https://ww.chhes.ccom chess. www.sehhesssseerreee.e.e
fen- chess, 2025. Accessed: 0025-10-23.
[14] P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg. and D. Amodei. Deep reinforcement learning from human preferences. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https: httttp:://prroceeinggg. neurips cc/ pdf.
paper htt.:///spppppe...t..eeeaaaaaaaa
[15] S. Colton and G. A. Wiggins. Computational creativity: The final fronticr? In ECA! 2012, pages 21-26. IOS Press, 2012.

---

[16] S. Copeland.
10
positions
chess
engines
just
don't under-
stand, 2021.
URL
https : / www. chess, com/ antnci/eceee//
10-positions- chess- cenisseengnnee--- just dont- maatt-uddestand.
[17] D. Corbit and S. Natarajan. Strategic Test Suite. attps://sites.
strategictestsuie abbout,,, 2025. Accessed: 2025-10-23.
[18] C. Daskalakis, A. Ilyas, V. Syrgkanis, and H. Zeng. Training gans with optimism. arXiv preprint /711v0/711.00141, 2017.
[19] E. Davis. Chatgpt's poetry is incompetent and banal: discussion of (porter and machery. 2024). Department of Computer Science New York University, 2024.
[20] B. De Kegel and M. Haahr. Procedural puzzle generation: A survey. IEEE Transactions on Games, 12(1):21 2020. doi: 10.1
[21] M. Dennis, N. Jaques, E. Vinitsky, A. Bayen, S. Russell, A. Critch, and S. Levine. Emergent complexity and zero-shot transfer via unsupervised enviromment design. Advances in neural information processing systems, 33:13049-13061 2020.
[22] P. Doggers. sciousness?, 2017.
Will this
position help understand
human con-
URL https: //www chess. com www.///wo//wwwieeeeeeeee
will sall-his--oon thissppsiiiion wesiiionnhllltee help-to- wowtttoooddrttnadd... humand-uuun----- cotenicooncciiiiccc
[23] S. Farle, J. Togelius, and I.. B. Soros. Video games as a testhed for open-ended phenomena. In 2021 IEEE Conference Games (CoG), pages 1-9. IEEE, 2021.
[24] A. El-Kishky, A. Wei, A. Saraiva. B. Minaiev, D. Selsam, D. Dohan, F. Song, H. Lightmmn,
I. Clavera, Pachocki, et al. Competitive programming with large reasoning models. arXiv preprint ar arXiv:2502..668
2025.
[25] M. Faldor, J. Zhang, A. Cully, and J. Clune. Omni-epic: Open-endedness via models of human notions of interestingness with environments programmed in code. arXiv preprint 2024.
arXiv:2405.
[26] X. Feng, Y. Luo, Z. Wang, H. Tang, M. Yang, K. Shao, D. Mguni, Y. Du, and J. Wang. Chessgpt: Bridging policy learning and language modeling. Advances in Neural Information Processing Systems, 36.7216-7262 2023.
[27] Fernando, D. Banarse, H. Michalewski, S. Osindero, and T. Rocktäschel. Promptbreeder: Self-referential self-improvement via prompt evolution. arXiv preprini arXiv:2309.1679 167977, 2023.
[28) Google DeepMind. AI achieves silver-medal standard solving International Mathemati- cal Olympiad problems. Google DeepMind Blog, July 2024. URL https: attps://deepmind: waalee/dssover/bbbeeee paoove//llg//aae /ttp///iooooo/ee/sssaaasessseeee solv-s--ees--- imo-mmombee problems cammsate aars-at--ss wellver-meddllleeee cessed: 2025-10-23.
Ac-
[29] M. Guid and I. Bratko. Detecting fortresses in chess. Elektrotehniški Vestnik, 79(1-2):35-40. 2012.
[30] D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q Zhu, S. Ma, P. Wang, X. Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501. 12948, 2025.
[31] Harvey. 30,000 Chess Puzzles.
2025. Accessed: 2025-10-23.
[32] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-685 16551, 2020.
[33] J. Hu. Reinforce++: A simple and efficient approach for aligning large language models. arXiv preprint arXiv:2501.0322 03262, 2025.
[34] E. Hughes, M. Dennis, J. Holder, F. Behbahani, A. Mavalankar, Y. Shi, T. Schaul, and T. Rocktaschel. Open-endedness is essential for artificial superhuman intelligence. arXiv preprint arXiv:2446 04268, 2024.
12

---

[35] A. Iqbal. Complexity in chess does not enhance aesthetics. 04 2017.
[36] A. Iqbal. A computer composes a fabled problem: Four knights vs. queen, 2017. URL
httpe///axwww..rraabbebaee 00931.
[37] Iqbal. computational method of optimizing chess compositions to enhance aesthetic appeal. In 202/ 2nd International Conference on Artificial Intelligence and Data Sciences (AiDAS), pages 1-6, 2021. doi: LLLT9/AADDAS538872222151177
[38] A. Iqbal, H. Van Der Heijden, M. Guid, and A. Makhmali. Evaluating the aesthetics of endgame studies: a computational model of human aesthetic perception. IEEE Transactions on Computational Intelligence and Al in Games, 4(3y:178 191 2012.
[39] A. Iqbal, M. Guid, S. Colton, J. Krivec, S. Azman, and B. Haghighi. The digital synaptic neural substrate: A new approach to computational creativity, volume Springer, 2016.
[40] M. Ismayilzada, D. Paul, A. Bosselut, and L. van der Plas. Creativity in ai: Progresses and challenges. arXiv preprint arXiv:2410.17218 2024.
[41] A. Jaech, A. Kalai, A. Lerer, A. Richardson, A. El-Kishky, A. Low, A. Helyar, A. Madry, Beutel, A. Camey, al. Openai ol system card. arXiv preprint arXiv:2412.1677 2024.
[42] C. R. Jones and B. K. Bergen. Large language models pass the turing test. arXiv preprint arXiv:2503.23674 2025.
[43] A. Kazemnejad, M. Aghajohari, E. Portelance, A. Sordoni, S. Reddy, A. Courville, and N. L.. Roux. Vineppo: Unlocking rl potential for Ilm reasoning through refined credit assignment. arXiv preprint arXiv22410.0167, 2024.
[44] S. M. Kelly. ChatGPT passes exams from law and business schools. CNN, January 2023. URL pettps:/editiinn cnn. ctw/2023/01/26///////// http///c///////////////// ttcp//hhtthttttceee/e pasppt--assss--e exams exams/inaa iinexx...xx./// html. Accessed: 2025-10-23.
[45] A. Khalifa, Bontrager, S. Earle, and Togelius. Pegrl: Procedural content generation via reinforcement learning. In Proceedings of the AAA/ Conference on Artificial Intelligence and Interactive Digital Entertainment, volume 16, pages 95-101, 2020.
[46] J. Z. Leibo, E. Hughes, M. Lanctot, and T. Graepel. Autocurricula and the emergence of innovation from social interaction: A manifesto for multi-agent intelligence research. arXiv preprint 1303:01903300722 2019.
[47] V. I. Levenshtein. Binary codes capable of correcting deletions, insertions, and reversals. Soviet Physics Doklady, 10(8):707-710, 1966.
[48] J. Levitt and D. Friedgood. Secrets of Spectacular Chess. Everyman Chess, 1995.
[49] Y. Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, Leblond, T. Eccles, Keeling, F. Gimeno, A. Dal Lago, et al. Competition-Ievel code generation with alphacode. Science, 2022.
11966662)::092-10000
[50] Z. Li, T. Xu, Y. Zhang, Y. Yu, R. Sun, and Z.-Q. Luo. Remax: A simple, effective, and efficient method for aligning large language models. arXiv preprint arXiv:2310.10500 2023.
[51] lichess.org. Lichess standard game dataset (2024-11), 2024. URL https: //l:://deee database.
lichess org/ /standard/licheess db standard_dd.ae caaaanddrdrrrddd__eeeeeee 20led__024211111 11.24-1........... zst.
[52] lichess.org. Lichess puzzle dataset, 2025. URL https: lichess org/ #puzzles.
[53] Ma, W. Liang, G. Wang, D.-A. Huang, O. Bastani, Jayaraman, Y. Zhu, L. Fan, and A. Anandkumar. Eureka: Human-level reward design via coding large language models. arXiv preprint arXiv:2310. 12931, 2023.
[54] R. Mcllroy- Mellroy-Young, S. Sen, M. Kleinberg, and A. Anderson. Aligning superhuman Al and human behavior: Chess as a model system. CoRR, abs/2006.01855, 2020. URL https:
// //arxiv. org/ 2006 01855.
13

---

[55] T. Merino, S. Earle, R. Sudhakaran, S. Sudhakaran, and J. Togelius. Making new connections: Llms as puzzle generators for the new york times' connections word game. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, volume 20, pages 87-96, 2024.
[56] E. Meyerson, M. J. Nelson, H. Bradley, A. Gaier, A. Moradi, A. K. Hoover, and J. Lehman. Language model crossover: Variation through few-shot prompting. ACM Trunsactions on Evolutionary Learning, 4(4):1-40, 2024.
[57] S. Milosz and P. Kapusta. Predicting chess puzzle difficulty with transformers. In 2024 IEEE International Conference on Big Data (BigData), pages 8377-8384. IEEE, 2024.
[58] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-533 2015.
[59] M. Moravčík, M. Schmid, N. Burch, V. Lisý, D. Morrill, N. Bard, T. Davis, K. Waugh, M. Johanson, and M. Bowling. Deepstack: Expert-level artificial intelligence in heads-up no-limit poker. Science, 356(6337):508 (11)))5585551,,,00 2017.
[60] T. Moskovitz, B. O'Donoghue, V. Veeriah, Flennerhag, S. Singh, and T. Zahavy. Reload: Re- inforcement learning with optimistic ascent-descent for last-iterate convergence in constrained mdps. In International Conference on Machine Learning, pages 25303-25336. PMLR, 2023.
[61] M. D. Mumford. Where have we been, where are we going? taking stock in creativity research. Creativity research journal, 15(2-3):107-120, 2003.
[62] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkín, C. Zhang, S. Agarwal, Slama, A. Ray, et al. Training language models follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[63] J. Polder-Holder, M. Jiang, M. Dennis, M. Samvelyan, J. Foerster, E. Grefenstette, and T. Rocktäschel. Evolving curricula with ragret-based environment design. In International Conference on Machine Learning, pages 17473-17498. PMLR, 2022.
[64] R. Penrose.
Shadows of the Mind, volume 4. Oxford University Press Oxford, 1994.
[65] S. R. Penrose. Sir roger penrose and dr. stuart hameroff: Consciousness and the physics of the brain, 2020. URL sattps://vvat hwwt:////wwwwww/w//////// youtube wwwcc//o//ao/eeeeeeeeeeeo http://wtw//===00///4F000
[66] T. H. Persson. Tiger's Chaos Theory. Quality Chess, 2024.
[67] O. Petit, N. Thome, C. Rambour, L. Themyr, T. Collins, and L. Soler. U-net transformer: Self and cross attention for medical image segmentation. In Machine Learning in Medical Imaging: 12th International Workshop, MLMI 2021, Held in Conjunction with MICCAI 2021, Strasbourg, France, September 27, 2021, Proceedings 12, pages 267-276. Springer, 2021.
[68] B. Porter and E. Machery. Ai-generated poetry is indistinguishable from human-written poetry and rated more favorably. Scientific Reports, 14(1):26133, 2024.
[69] J. Pourcel, C. Colas, G. Molinaro, P.- Oudeyer, and L. Teodorescu. Aces: Generating a diversity of challenging programming puzzles with autotelic generative models. Advances in Neural Information Processing Systems, 37:67627-67662 2024.
[70] Rombach, A. Blattmann, D. Lorenz, Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10699 2022.
[71] Ruoss, G. Delétang, S. Medapati, J. Grau-Moya, K. Li, E. Catt, J. Reid, C. Lewis, Veness, and T. Genewein. Amortized planning with large-scale transformers: A case study on chess. Advances in Neural Information Processing Systems, 17:65765-657999 2025.
[72] J. Schritwieser, I. Antonoglou, T.. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lockhart, D. Hassabis, T. Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 58847839)600--69, 2020.
14

---

[73] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, Proximal policy optimization 2017.
algorithms. arXiv preprint
[74] J. Secretan, N. Beato, D. Ambrosio, A. Rodriguez, A. Campbell, and K. O. Stanley. Picbreeder: evolving pictures collaboratively online. In Proceedings of the SIGCHI conference human factors in computing systems, pages
2008.
[75] Z. Shao, Wang. Zhu, R. Xu, Song. M. Zhang, Y. Li, Y. Wu, and D. Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.033: 003000 2024.
[76] N. Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202 2020.
[77] J. Shi, Han, Wang. Doucet, and M. K. Titsias. Simplified and generalized masked diffusion for discrete data. In Advances in Neural Information Processing Systems, 2024.
[78] D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, T. Lillicrap. K. Simonyan, and D. Hassabis. Mastering chess and shogi by self-play with a general reinforcement learning algorithm, 2017. URL https: //arxiv... org/ absaaass/////
[79] K. Singhal, T. Tu, J. Gottweis, R. Sayres, E. Wulezyn, L. Hou, K. Clark, S. Pfohl, H. Cole- Lewis, D. Neal, M. Schaekermann, A. Wang, M. Amin, S. Lachgar, P. Mansfield, S. Prakash, B. Green, E. Dominowska, B. A. Arcas, N. Tomasev, Y. Liu, R. Wong. C. Semturs, S. S. Mahdavi, J. Barral, D. Webster, G. Corrado, Y. Matias, S. Azizi, A. Karthikesalingam, and V. Natarajan. Towards expert-level medical question answering with large language models, 2023.
[80] H. Steingrimsson. Chess fortresses, a causal for state of the art symbolic [neuro] architec- tures. In 2021 IEEE Conference on Games (CoG), pages 1-8. IEEE, 2021.
[81] S. Stoiljkovikj, I. Bratko, M. Guid, and F. UNI. A computational model for estimating the difficulty of chess problems. In Proceedings of the third annual conference on advances in cognitive systems ACS, page 7, 2015.
[82] A. Summerville, S. Snodgrass, M. Guzdial, C. Holmgård, A. K. Hoover, A. Isaksen, A. Nealen, and J. Togelius. Procedural content generation via machine learning (pegml). IEEE Transac- tions on Games, 10(3):257-270, 2018.
[83] L. Sun, Y. Yuan, Y. Yao, Y. Li, H. Zhang, X. Xie, X. Wang, F. Luo, and D. Stillwell. Large language models show both individual and collective creativity comparable to humans. arXiv preprint arXiv:2412.03/1 22.223222333 2024.
[84] R. S. Sutton, A. G. Barto, et al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.
[85] The British Chess Problem Society. A Beginner's Introduction to Chess Problems. https: /www. theproblemte emist. aamist.org// htt//beginner....e www.iner.pllttpeeeeereeeeeeeeeee pl?type=b_int, 2025. Accessed: 2025-10-23.
[86] The Stockfish community, T. Romstad, M. Costalba, J. Kiiski, and G. Linscott. Stockfish Chess Engine.
10080ppesent. Version 17.1 (as of October
23, 2025).
[87] The YACPDB Project. Yet Another Chess Problem Database (YACPDB). https://www. yacpdb. org/, 2025. Accessed: 2025-10-23.
[88] G. Todd, S. Earle, M. U. Nasir, M. C. Green, J. J. Togelius. Level generation through large language models. In Proceedings of the 18th International Conference on the Foundations of Digital Games, pages 1-8, 2023.
[89] J. Togelius, G. N. Yannakakis, K. O. Stanley, and C. Browne. Search-based procedural content generation: taxonomy and survey. IEEE Transactions on Computational Intelligence and Al in Games, 3(3):172-186, 2011.
[90] A. Turing. Computing machinery and intelligence. 1950.
15

---

[91] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[92] G. Wang, Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, Zhu, L. Fan, and A. Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305. /6291, 2023.
[93] Wang. J. Lehman, J. Clune, and K. O. Stanley. Paired open-ended trailblazer (poet); End- lessly generating increasingly complex and diverse learning environments and their solutions. arXiv preprint arXiv:1901.01755. 2019.
[94] R. Wang. J. Lehman, A. Rawal, J. Zhi, Y. Li, J. Clune, and K. Stanley, Enhanced poet: Open-ended reinforcement learning through unbounded invention of learning challenges and their solutions. In International conference on machine learning, pages 9940-9990 PMLR, 2020.
[95] Wikipedia. Chess piece relative value. Attps://ent wttppppela...g////kk//eaeeepeppe relative_value 2025, Accessed: 2025-10-23.
[96] Wikipedia. Creativity.
2025. Accessed: 2025-10-23.
[97] Wikipedia.
Evaluation measures (information retrieval), 2025. URL https://en
http://vlluattionnmmssurreesss cessed 22-July-2004).
wikipedia a.
retriveval.. [Online; ac-
[98] T. Zahavy, Veeriah, S. Hou, K. Waugh, M. Lai, E. Leurent, N. Tomasev, L. Schut, D. Hass- abis, and S. Singh. Diversifying ai: Towards creative chess with alphazero. arXiv preprint arXiv:2308.091 2023.
[99] K. Zaidi and M. Guerzhoy. Predicting user perception of move brilliance in chess, 2024. URL https: //tp///avvv//////////////e/aaaaa hrgg/abs/2406 11895.
[100] J. Zhang, J. Lehman, K. Stanley, and Clune. Omni: Open-endedness via models human notions of interestingness. arXiv preprint
2023.
16

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We show method of quantifying chess puzzle in Section with experimental results in Section 3.1. Generative model and RL design are presented in Section 2 and Section 2. with experimental results in Section 3.2, Section 3.3 and Section 3.4. The human study result shown Section 3.5.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Refer Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
17

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: All experiment details and results are illustrated Section 3, Appendix D, Appendix F, Appendix G and Appendix H.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- moen-e dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
18

---

reproducible, they Justification: Our codebase is built heavily on internal infrastructure. We can open source them after adapt them. But detail our experimental settings and configurations, which should be enough to reproduce easily. Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/ public/guides https//ooeuuuiiionnneeeeeeeeeeee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: /nips. wwtppccccp/cccceccccccccccccccct
/wdeddeeeooeebaaaacccc ioniissionn ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All experiment details are illustrated in Section 3. Appendix D, Appendix F, Appendix G and Appendix H.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to a level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification:
We report the error bar on human analysis, detailed in Table
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
19

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We report our compute resources in Appendix I..
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: attps://neurrpe: cc cc/ppblic/Ethicsaliiiiiiic
Answer: [Yes]
Justification: All human participants in human study have received appropriate compen- sation.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification:
We discuss broader impact in Appendix K.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out, For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
20

---

The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
The paper poses no such risks
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question:
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We mainly leverage the Lichess Puzzler dataset and follow the license and terms of use.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ /atassts has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide the booklet with anonymous link htttps///driv google. com//iLe////ISSSJSSSAA/A wow.:oSSSYYbbIDRRRase...... supplementary material. They well documented.
and in our
21

---

Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the methods in this research? Note that if the LLM is used only for writing. editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
22

---

Please refer to our LLM policy htttps//hourips.cc////o /tt.://ss/c///ofeeeeere////////1
for what should or should not be described.
23

---

A Related work
While much computational chess research has focused on enhancing engine strength, exemplified by engines like StockFish, AlphaZero [78], and Leela, a line of work has identified puzzles that remain challenging even for top engines [22, 65, 16, 80]. This has spurred the development of specialized solvers like Crystal, specialized methods [29] and creative problem-solving approaches [98]. In parallel, large-scale puzzle generation for platforms like Chess.com and Lichess typically extracts tactical sequences from games, prioritizing criteria such as solution uniqueness (often a single best move per turn, barring final mating moves and achieving a clearly winning state, usually defined by material gain or forced mate (chess.com blog). Assessing puzzle difficulty computationally involves heuristic models analyzing simulated human search tree features [81] and machine learning models trained on platform data [99, 57]. Furthermore, tools like the novelty grinder identify moves that are strategically tactically sound but statistically rare in human play. Despite these advancements in analyzing solvability, difficulty, aesthetics, and novelty, the generation of puzzles that are specifically creative or counter-intuitive remains an underexplored area.
A particularly relevant line of prior research is the work by Iqbal et al. on the Chesthetica program [36, 38, 39, 371, which focused on computationally modeling and optimizing for human appreciation of beauty in chess. Chesthetica employed a detailed aesthetic score built upon 17 terms, combining core aesthetic principles (like sacrifice, paradox, economy) with specific tactical themes (such as pins and zugzwang) derived from chess literature. This score was validated against expert human judgments, comparing composed studies to game sequences. Significantly, this research also found that the aesthetic score did not positively correlate with computational complexity, measured via engine solving time, indicating that difficulty and beauty are distinct dimensions [35]. Differing from the objective of generating broader unique positions as in Lichess Puzzler and the present work, Chesthetica's 's generation capability was primarily aimed creating mate-in-x problems, utilizing realism heuristics. Its proposed refinement method involved applying seven sequential steps intended to preserve the core solution: 1) removing pieces, 2) substituting white pieces with weaker ones, 3) substituting black pieces with stronger ones, 4) moving the white king out of check, 5) seeking shorter mate, 6) relocating the white king closer to other pieces, and 7) preventing major black pieces from being immediately capturable. It is noteworthy that the generation process itself was not guided by optimizing the aesthetic score; rather, the aesthetic evaluation functioned primarily as filter applied after generation to select suitable outputs. This focus on generating realistic mate scenarios often resulted in compositions resembling endgames with relatively few pieces, as visually evidenced on the Chesthetica youtube channel.
The automated generation of puzzles for games more broadly has also been studied extensively in the field of procedural content generation (PCG) The Procedural content generation via machine learning (PCGML) [82, 88] community studies how to bring machine learning to bear on this problem, and many of these puzzles have been generated using RI.-generative policies in (PCGRL) [45, 55, 69, 20]. There is a hope that similar algorithms could be used as a source of problems to form an increasingly complex curriculum to train generally capable agents [46, 21, 23], exhibiting an open-ended [34] stream of increasingly complex and novel problems which replicates the open-ended complexity of evolution [4, 5]. Given its origin, many approaches to achieving open-endedness evolutionary algorithms [74, 93, 94, 63], especially building on the power of modern LLMS [92, 27, 56, 100, 25, 53, 26). The integration of evolutionary methods is a promising avenue to extend our approach.
While this research primarily utilized Stockfish and AlphaZero for analysis and reward formulation, future work could delve deeper into improving measuring counter- innuter-ittutiiveess. For example, by incorporating insights from other chess engines like Leels or Maia [54], which is designed to mimic human play more closely. This could lead to puzzles that not only counter-intuitive for strong engines but also align better with human notions of surprise or beauty.
24

---

B The Lichess dataset
The lichess puzzles were generated from 300,000,000 analysed games from the Lichess database. Interesting positions were re-analyzed with Stockfish 12/13/14/15 NNUE at 40 meganodes. The resulting puzzles were then automatically tagged. The number of games played at the platform, and the number of puzzles through time can be viewed in Fig. 6.
user-driven popularity metric for each puzzle, ranging from 100 (best) to t001 (worst), is calculated based on weighted upvotes and downvotes. The weighting considers factors like successful solves and the rating difference between the solver and the puzzle. Lichess Puzzler also assigns an Elo rating to each puzzle, treating each solving attempt as a rated game.
We train all our models purely on the public Lichess Puzzler dataset with cutoff' date on 2024-12-01 with 4.4M data points. We split train and test by 99% and 1 1% resulting in 4.36M train samples and 44k test samples correspondingly. Note that the "FEN" column the raw dataset the position before the opponent makes their move, so we apply the first move in solution to the puzzle board.
4e6
SPuzzles
3e6
2c6
1cb
2013
2013
2013
2003
Figure The amount of games played in lichess between 2021-2025 (red) and the amount of puzzles extract from them (blue). The amount games is divided by 100 so its in the same scale as the puzzles.
25

---

C Uniqueness
To verify that chess position has only solution StockFish and analyze its search results. We begin by confirming that the position has single solution, using the approach from the Lichess Puzzler project, which utilizes StockFish analysis. The first step is to see if there's checkmate sequence in the primary line of play within 15 moves.
If yes: check that the puzzle player's main tin-of--pay is the only one that leads to checkmate. This check is then recursively applied by playing the top move from the main variation and then recomputing the variations from the resulting position. any alternative mating sequence appears before checkmate, then it is not puzzle.
If not: check that the puzzle player's 's main variation's chance of winning w(als) is substan- tially greater than the second best variation, i.e. it is unique using Eq. (1) below. This check is then recursively applied by playing the top move from the main variation and recomputing the variation from the resulting position. If the criterion holds for at least the first move from the initial position, the it is considered puzzle.
Note that in both cases, the puzzle criterion is only applied to the puzzle player's 's turn. For the opponent, the top move from the main variation is always played. The chance of winning for non-checkmate variation compute using the equation below based on Stockfish's centi-pawn score.
Figure evaluates our implementation of the Lichess puzzle uniqueness check against 1% of the Lichess puzzles dataset (approx 44K examples) and set of 30K non-puzzle examples, for different values uniqueness thresholds. The value of 0.5 used for experiments this paper, where Lichess Puzzler uses 0.7. There are other minor implementation differences between the Lichess Puzzler version and ours. For example, Lichess Puzzler ignored positions that don't 't win by enough. These modifications were ignored in our version for simplicity.
Lichess Puzzle Checker Evaluation
1.00
0.95
9.3
0.90
This Negetive Rate
0.85
0.80
0.75
0.70
0.70
0.75
0.80
0.85
0.90
0.95
1.00
Thue Positive Rote
Figure 7: Evaluation of Lichess Puzzle Checker (Uniqueness) for different threshold values
26

---

D Implementation Details
D.1 Generative model implementation details
We detail our model and training implementation details.
Autoregressive Transformer We adopt the same architecture from [71], decoder-only transformer with causal masking, post-normalization and SwiGLU [76]. We use 8 heads, 16 layers, and an embedding dimension of 1024, resulting in 200M parameters in total. We adopt the AdamW optimizer, with le-4 learing rate and le-4 weight decay coefficient. We conduct the training with 1024 batch size and 100k steps. Finally, we pick the checkpoint by measuring the test loss on our 44k test set.
Latent Diffusion Model For the latent diffusion model experiments, investigated the impact of different chessboard representations on the generative model's s performance. Since latent diffusion models typically operate on 2D image data, we explored representing the chessboard in 2D format. Specifically, we padded the original sequence representation (length 77) a length of 80 and reshaped into an 8x10 grid. The first 8x8 section of this grid represents the chessboard squares, while the remaining 2x8 section encodes metadata such the current turn, castling rights, and move number.
Latent diffusion models require an encoder map the input into a continuous latent space. Therefore, we trained a separate Variational Autoencoder (VAE) to serve as this encoder for our 2D chessboard representation. For VAE training, the 10 grid input was first converted into an 8x10x31 one-hot tensor (corresponding to a vocabulary size of 31). The VAE was trained using a standard objective comprising a reconstruction likelihood loss and a KL divergence penalty. Following common practices in latent diffusion [70], we set the KL divergence coefficient small value (le-3). The VAE training proceeded for 50,000 steps.
Upon completion of VAE training, its parameters were frozen. We then trained the diffusion model itself, which operates in the latent space generated by the VAE encoder. We employed U-Net Transformer architecture, specifically adapting the design from [67]. The network features a downsampling path and an upsampling path. The downsampling path consists of three Cross- Attention 2D Downsizing blocks (each combining cross-attention and ResNet convolutional blocks) followed by one standard 2D Downsizing block (ResNet convolutional blocks only). The upsampling path mirrors this structure in reverse: one standard 2D Upsampling block followed by three Cross- Attention 2D Upsampling blocks. The channel dimensions progress through 160, 320, 640, and 640 across the stages. All attention mechanisms utilize 8 heads. This configuration resulted in a U-Net Transformer with approximately 220M parameters, making it comparable in scale to other generative model.
For training the diffusion model, we used the Adam optimizer with a learning rate of 2e-4. We used a batch size of 512. The training followed the Denoising Diffusion Probabilistic Models (DDPM) [32] framework. The diffusion model was trained for 300,000 steps.
Masked Diffusion Model We follow MD4 [77] to train a masked diffusson model by maximizing a variational lower bound of the log likelihood of data. The loss function is
- log (x < Lz = ['' a;
1 1-a Eg(ar(1o))
(n9)
Σ
( Ho dt,
(9)
iiz [mask]
We reuse the transformer architecture from the AR model to parameterize the denoising network pe- except that remove the causal attention mask use bidirectional attention. We randomly draw t t~ (0,,00 1) to estimate the integral. Then we corrupt the clean data point (a sequence of length 77) by independently masking each element with probability or, this corresponds to drawing sample from (2)) to estimate the inner expectation. Finally, we compute the desnoising loss via cross
We use a linear schedule a, = 1 t for entropy at the masked positions and weight them by I-or' our experiments
MaskGIT Our MaskGIT model is the same as the training of our auto-regressive model, we choose the masking rate per batch by sampling from a uniform distribution r ~ (0( 0.5), we then mask each token IID with probability r. During inference, the MaskGIT model works by iterative demasking. We find that the inference optimizations originally used in MaskGIT [11] for the image domain, prioritizing decoding order by a confidence in the generation, results in a bias for generating empty chess boards. Intuitively, if the most likely token at any board position is the empty square, so the
27

---

first several generated squares are likely empty. Conditional on this, it is much more likely much of the rest of the board will be empty. We address this by selecting which token keep randomly, keeping one token at a time until the entire board has been generated.
D.2 Generative model evaluation details
For the PV distance evaluation, we truncate the full principle variation sequence length to 6. We use IM samples for the puzzle score calculation in Table 2. For Table 3, use 100k samples per method for Lichess-distance and 40k samples for self-distance. We adopt 40k for self-distance calculation because the Lichess-Gand Ganes dataset only have around 4% uniqueness ratio, which corresponds to 40k samples from 1M generated positions. We need align the number of samples because we are calculating minimal distance rather than the average one.
D.3
RL implementation details
For the diversity filtering, we set Thoard as and TPV as 1. We find that the truncation length used in generative model evaluation is too strict for selecting samples in RL, thus reduce it to 1, leading check over board's first-move. the final RL run, we lower the training learning rate 3e-5 and we set the replay buffer diversity filtering subsample size as 2k.
28

---

E Evolutionary Search
Complementary generative models trained on pre-existing data, evolutionary methods provide us with the opportunity to go beyond the training data distribution and set up an optimization process that would explore the set of possible board configurations in relation to a given Dounter-intuiiveene objective, and possibly with additional imposed stylistic constraints. Our initial implementation of evolutionary search in the context of this task has been rather straightforward, in that we' ve not explored many of the perhaps more advanced custom ideas that have been proposed in the literature over the years, having instead focused on the more 'basic' setup, where we have prioritized finding the right way of incorporating the score and the constraints to produce the desired outcome.
In our ES approach, we would run flexible number of distinct ES 'workers', each running its own
local ES process. These processes would not communicate between each other, rather producing independent outputs. Each worker would start by sampling a random non-puzzle chess position (or alternatively, set of positions) from the provided pre-existing dataset. Different workers would initiate their local ES processes from different sampled seed positions, to facilitate diversity across the whole worker population. In principle, one could also alternatively start from randomly generated position instead, though starting from sampled pre-exiss pre-existing position enables us to potentially stay closer to the original distribution under a certain set of parameter settings.
Each ES process running on its own separate ES worker would run for fixed pre-specified number of iterations, this being a parameter of the process. Each process would also have its own separate buffer. In each ES iteration, candidate solutions from the previous generation would be sampled from the buffer, proportionally to their counter-intuitivnnss. uniqueness, and conformity to additional constraints. To trade-off between exploration and exploitation, this sampling would involve a temperature parameter, modulating its stochasticity. The temperature parameter would be annealed between the start and the end of each ES process, so to sample more stochastically initially, and more directly proportional to the score towards the end of the ES run. The number of sampled solution candidates is tunable parameter, as is the temperature. For each sampled solution from the previous generation, the ES process would generate and propose one or more mutations, new solutions obtained via a number of random edits on the board. These edits would involve random piece removals and additions, as well potentially playing randomly selected number of random moves from the sampled position. Given that it possible to have mutations result in illegal board positions, this process would be repeated if necessary for each mutation to ensure it conforms to the basic criteria of legality, so as always get the requested number of candidates for the next ES iteration. The exact number of edits for these mutations is another controllable parameter. Once the new candidate solutions have been generated through mutations from the sampled set, would compute their own fitness in terms of inauterrituuitiveeess, uniqueness, and constraint satisfaction. We would then use that composite score to sort the old and the new solutions jointly, and then discard the lowest performing candidates, keeping the ES buffer at fixed size throughout the process. At the end of the final iteration, the ES process would write the best generated solutions, reset all the counters and parameters, re-sample the starting position, and start from scratch.
As is clear from the description above, even in this basic setup there are many controllable parameters. Rather than being a weakness, this is actually form of strength. given that different parameter configurations work in different ways, leading to different types of resulting puzzle positions. For example, running broad but shallow search, with a large buffer, a small number of iterations, and a low mutation rate, results in largely on-distribution positions that tend to turn the original sampled non-puzzle chess boards into chess puzzles that still feel largely similar to the original. On the other hand, running the ES process deeper with a higher mutation rate results in positions that are far more OOD and unusual. As this is a spectrum, the exact configuration represents a creative choice.
While chess experts generally favor realistic puzzle positions, the pursuit of true novelty-suce such as inventing entirely new categories of chess and realistic configurations. Evolutionary Search (ES) emerges as a promising complementary technique in this regard. As detailed in the paper, ES can directly optimize for specific objectives and constraints, enabling exploration of puzzle space regions that models trained solely on existing data might not reach. Future research could fruitfully investigate hybrid models, for instance, by using Generative Al initialize ES populations or applying Reinforcement Learning to refine puzzles initially identified through ES.
venturing beyond currently known
29

---

F Search Features
The following features were used to compute the counter-intuitiveness score. Features that are mentioned to have penalty were added with negative weights.
Stockfish computes win rate for each move in given position, Win(a). Using this win rate we computed:
top move gap: the win rate gap between the top move and the second top move, this is similar to uniqueness.
top move miseval gap: the win rate gap given the top move between deep-search and shallow search.
AlphaZero [78] has a prior network that computes move probabilities Prprior Once AlphaZero completes its search, its action probabilities Pipost-seareh (a) are computed as the number of visits MCTS has assigned to action a divides by the total number of visits. Using these statistics we computed:
score policy: Probability gap of top action (identified by deep-search) between pre and post search. I.e., Let a" arg max Prpost-son search (a), then score policy = Pfpost-seared (a")
Piprior (a*).
score prior drop: How much the action probability of the action (with highest prior probability) drops in post search. L.c., Let a" = arg max Pryrior (a), then score policy = Pr Prpost- (a") Prprior(()) (a").
prior entropy: (penalty) The entropy of the action probabilities in the prior.
Mise
capture material: (penalty) the material captured by the top move, weighted according to standard valuation (Wikipedia), divided by 9
promote material: (penalty) the material promoted by the top move, weighted according to standard valuation (Wikipedia), divided by
giving check: (penalty) if the best move is giving check.
mate in one: (penalty) if the best move is giving checkmate in move.
check: (penalty) 1 if the player to play is under check.
The tuned weights of the (I) Search Features config in Table using the features listed above were
check 1.0
giving check = 0.4
score policy 0.1
capture material 1.0
For the Stockfish Through-Time features, we computed the critical point and area under the curve from Section (Eq. (3) and Eq. (4)).
(time, nodes, depth) AUC: It is calculated as the area under curve where: the x-axis represents the search resources used, including time, nodes and depth, and the axis refers to the absolute difference between the engine's current evaluated win rate for the final solution move and the final win rate evaluation obtained after the longest possible search duration.
(time, nodes, depth) critical point (cp) move: The time, number of nodes searched, search depth at which the Stockfish engine first identifies the move that is eventually determined to be the final solution move after the longest search duration.
(time, nodes, depth) critical point (cp) value: The time, number of nodes searched, or search depth which the Stockfish engine's evaluation (expressed win rate) for the final solution move first reaches a value that within a specified threshold of the final win rate evaluation obtained after the longest search duration.
30

---

TRAIN
TRAIN+TEST
TEST
score policy auc (az)
0.47
0.47
0,47
score policy cp (az)
0.46
0.45
0.44
move cp (az)
0.46
0.50
0.61
top move miseval gap (sf)
0,41
0.38
0.32
score policy (sf)
0.44
0.41
0.34
depth auc (sf)
0.46
0.42
0.32
depth ep move (sf)
0.73
0.74
0.74
depth ep value (sf)
0.53
0.45
0.34
time auc (sf)
0.49
0.46
0.38
time op move (sf)
0.64
0.63
0.60
time value (sf)
0.51
0.44
0.33
nodes auc (sf)
0.50
0.46
0.38
nodes cp move (sf) nodes cp value (sf)
0.65
0.64
0.66
0.51
0.43
0.33
Table 5: The average precision (Eq. (8)) of specific search features.
Using these weights, the weights (2) search Features SF through time that we reported in Section were computed.
The weights of (3) Search Features SF through time AZ through time were
az move cp = 0.2 az policy entropy depth cp move 1.0
0.3
score prior drop = 0.1
capture material 0.3
Finally, Table 5 present the average precision for standalone search features detailed above. Analyzing these results for StockFish reveals clear trends. Firstly, features derived from the search process over time consistently outperformed static positional features. Secondly, when comparing methods for summarizing through-time data, the "critical point (move)' approach yielded better results than the 'value' method, and both critical point strategies were superior to the AUC method. Thirdly, considering the search dimension (depth, nodes, time), depth- barth-bbaeddd metrics generally outperformed most through-time measures. This superiority of depth-
node-based and time-based metries based features explains why they received high weights in the tuned configurations.
To demonstrate that simply having a unique solution is not enough to fully represent the counter- intuitiveness of chess puzzles, also present uniqueness as an indicator according to Eq. (1). To break ties ties, we performed 100 random shuffles of samples and computed the average score. Finally, Table in the supplementary presents a correlation analysis between various nnnterrintuitivennss (described Appendix B) conducted on 1% of the Lichess dataset (approximately 36,000 examples). The results indicate positive correlation between counter-intuitivenn and rating, but correlation with popularity.
baseline. This metric is calculated
and the lichess rating and popularity scores
Rating
Popularity
Pearson
Spearman
Pearson
Spearman
Uniqueness
-0.044057
-0.044483
0.021340
0.023032
(I) Search Features
0.374290
0.406505
0.020066
-0.023285
(2) + SF through time
0.343795
0.291719
0.016591
0.013052
(3) AZ through time
0.333493
0.222792
0.021130
0.014161
Table 6: Correlation analysis between candidate score functions and rating/popularity Lichess
31

---

G Additional results
We report additional results AR training curves in Fig. 8.
472
-200
0.001
0.307
632
247
Figure 8: AR training curves. More AR training steps bring in higher uniqueness and counter- intuitiveness score, with the price of lower board distance and PV distance.
32

---

H Additional RL results
H.1
Vanilla RL fails with entropy-collapse or out-of-distribution samples
We initiate our experiments with vanilla Reinforcement Learning (RL. training. incorporating an entropy bonus and starting from the lichess-trained supervised model. Fig. 9 illustrates the progression of reward and model generation entropy during training under different entropy bonus coefficients.
With relatively low entropy bonus coefficients (c.g., coef=le-2 or 2c-2), the training exhibits distinct failure stages. Initially, after several hundred steps, the reward sharply increases towards 1.0, suggesting that nearly all generated positions are classified as qualified puzzles. However, this coincides with the generation entropy plummeting towards zero-a clear indication of reward hacking. The model learns maximize reward simply by collapsing its output distribution onto a single, easily generated qualified puzzle (an example is shown in Fig. 9, 3rd board). the later training phase, this severe entropy collapse leads to complete training failure, where the model's output degenerates further into predominantly illegal positions (reward = -2, see example in Fig. 9, 1st board).
Addressing this entropy collapse issue proves non-trivial. Simply increasing the entropy bonus (e.g., coef=4e-2) successfully mitigates the drastic drop in entropy. However, this approach yields negligible reward improvement (the reward never exceeds 0) and introduces severe out-of-distribution (OOD) issues. As illustrated by the 2nd board of Fig. 9, the model adopts undesirable generation strategies, such as placing an excessive number of pieces or creating densely packed piece clusters. While these tactics likely succeed in maximizing the entropy score component of the reward, they produce positions that lack the strategic interest and natural game flow expected by human chess players.
actepy
etopy eroppedioo
2 white kings. Bogal board
10
4.
21-
High quite
02-
Mode Oolepoe
1
Collapse to one 000 example
200
420
600
1005
200
105
660
2005
Figure 9: Vanilla RL with entropy loss fails with cntropy-collapse or out-of-distribttion samples.
H.2 RL with diversity filtering stabilizes training
We introduced few filtering mechanisms in Section 2.1, including board distance, PV distance, entropy-based filtering and intra/inter-batch checking. To analyze their effects, we investigate the impact of progressively applying them.
Our first experiment explores only adding entropy-based filtering on the vanilla RL setting. Fig. 10 indicates this method partially mitigates entropy collapse, preventing the rapid decrease towards zero observed previously, although it does not fully stabilize the training process.
Fig. 11 shows stricter constraints, which combines the entropy-based filtering mechanism with the inter-batch diversity filtering based on board distance. With an appropriate entropy filtering threshold Tent, this configuration yiclds the first observed stable training trajectory for both reward and entropy. However, manual inspection of generated examples identifies a new diversity hacking issue. As exemplified in Fig. 11, the model learns to generate positions with high superficial diversity (c.g.. large board distance) but identical core tactics and solutions (principal variations). This indicates the model circumvents the filter by manipulating puzzle-inelevant rewarded puzzle core.
irrelevant elements while preserving the
To address this more sophisticated reward/diversity hacking, we employ all filtering mechanisms, detailed in Fig. 12. It illustrates how different components and their configurations contribute to
33

---

Reward
Entropy
entropy mask-0
enorppy mosc-0
entreey mast-10:0
entroyy mask-0
1.0
entiopy masco 6o
ennipyy masc-0.45
-0.5
-2.0
-2.
0.2
-2.0
500
750
1820
1250
1500
250
1000
1250 1500
Taining Stopps
Figure 10: RL with pure entropy reward mask filtering.
08
07-
Entropy
Mode: Co lasse
0.2
03
Pi: Ggd+ Bugg 2. NYW
Diftreee toards but Edlling hooking"
300
250
150
20
Trining
Thinning Stepe
Figure 11: Reward entropy mask + inter- mater-batch board-distance filtering can stabilize training. However, further RL hacks the board-distance filtering.
training performance. With optimal hyperparameter settings (c.g., entropy-filtering threshold = 0.6, utilizing all Board//V/entropy filters), we achieve stable training and maintain high sample pass ratio (samples are both qualified and diverse) over 5,000 training steps. Consequently, we adopt this comprehensive filtering strategy as our optimal setup for ensuring generation diversity.
H.3
Make puzzles more human game like
Beyond entropy collapse, the second major challenge identified in vanilla RL (OOD) generation. To address this, we aim to maintain the RL. model's 's alignment with the distribution learned by the initial pretrained model, primarily by minimizing their KL divergence.
Experimental results in Fig. 13.a suggest that even without employing more complex filtering mechanisms like entropy filtering, maintaining proximity to the pretrained model via a KL penalty offers some mitigation against entropy collapse. When combined with entropy filtering and other filtering techniques (Fig. 13.b-d), the KL penalty proves crucial for alleviating OOD phenomena. Specifically, when higher entropy bonus is applied (e.g., entropy-coef-0.003, a low KL penalty coefficient (kl-coef=0.01) allows the generation entropy surge above 1.0, consistent with the entropy levels observed during OOD generation in Fig. 9. However, increasing the KL penalty (kl-coef=0.03) effectively counteracts this surge, leading lower KL divergence and entropy, thereby reducing the occurrence OOD generations.
We also explore another technique common in RLHF: incorporating auxiliary supervised data to help anchor the model and reduce KI. divergence. We curate high-quality subset of approximately 100k puzzles by filtering the original Lichess dataset based on our defined uniqueness and counter- intuitiveness metrics. These puzzles are then used to seed the replay buffer, ensuring that samples drawn during RL training include these high-quality, human-like examples. Fig. 13.d demonstrates that this strategy successfully assists the model further reducing its KL divergence from the pretrained model.
Finally, address specific reward hacking behavior related to OOD generation observed in Fig. 14.a-c. The model learns artificially inflate reward (Fig. 14.a) and entropy Fig. 14.b) by adding an excessive number of pieces to the board (c.g., two white queens or three black knights). Fig. 14.c reveals that without intervention, the proportion of generated puzzles containing more pieces than standard chess set can exceed 30% and tends to increase during training. While these positions might be technically legal, they often unrealistic, rarely occur standard gameplay, and generally
34

---

Entropy
Sample Pass
1.00
1.4
Bdeebis IM beisdy
Mointam diversity and passing rati
0.13
1.50
0.2
1203
2000
1000
5000
1000
2000
30C0
42:33
2003
4000
5000
Thans rg Sheps
Traniing Stops
Collapon bater
Thoin Stopp
Figure 12: Further add pv distance and entropy filtering for replay buffer's 's new sample selection. We conduct hyperparameter search on entropy mask Board/PV/Entropy -aaed replay buffer filtering.
C.E. with
Indeeraa showwrre
Peoeaaydd priveed
Figure 13: KL coef hyper- ppprr-prraettr search. It can effectively control the OOD phenomenon and make puzzle more human like.
less appreciated by human players. To discourage this, introduce piece count penalty: the reward for any generated position exceeding the standard piece count is zeroed out. The effectiveness of this approach is evident in Fig. 14.c, where the penalty successfully reduces the prevalence of such positions to approximately 5%.
H.4 More ablation studies
We conduct several additional ablation studies and analyses to further understand the training dynamics and justify methodological choices:
PPO Update Epochs: We investigate the effect of the number of PPO update epochs per data batch. Fig. 15.a presents results for different epoch numbers (epoch=1, 2, 4). While epoch=2 yields slightly better performance than epoch=1, increasing to epoch=4 leads to highly unstable reward progression. Unlike traditional RL applications prioritizing sample efficiency, our primary focus is on training stability and mitigating entropy collapse through effective diversity management. Therefore, to ensure maximum stability, forgo the common practice of multiple updates per batch (epoch > 1) and consistently use epoch=l in the analyses presented above and our final configuration.
We examine the rationale for continuous model training alongside diversity filtering mechanisms. Fig. 15.b provides crucial comparison: when applying the same filtering mechanisms but keeping the model parameters frozen, the rate of discovering novel samples (eligible for the replay buffer) progressively diminishes over sampling steps, eventually approaching zero. This demonstrates that continuous model updates are essential for the sustained generation of novel, qualified puzzles under our filtering regime. This finding reframes the training objective: instead of converging to single, static optimal policy, our approach functions as dynamic search process. The model continually adapts based on the regions of the puzzle space has already explored, enabling ongoing exploration. This contrasts sharply with standard RL aiming for convergence to a stationary policy and shares conceptual similarities with exploration strategies driven by intrinsic reward in RL [10, 6, 9].
Necessity of Continuous Training:
Late-Stage Training Oscillations: Despite the significant stabilization achieved through compre- hensive diversity filtering, Fig. 15.d reveals oscillatory behavior emerging in the very late stages of extensive training (e.g, more than 5k steps) our experiments. We observe periodic fluctuations where reward and entropy move in opposition. suggesting the model alternates between prioritizing reward maximization and satisfying diversity/entropy constraints. This behavior is reminiscent of dynamics observed in adversarial training [18, 3] or constrained optimization [60] when operating near a Pareto frontier, where small parameter adjustments (c.g., few gradient steps) can cause
35

---

24 Rivertt
2-5 Vecrten
Too
0013-
100
Figure 14: (a) shows the effects on adding piece count penalty. Model can easily learn to violate piece count standard (nearly 35% violating from a.3) to get high reward (a.1) and entropy (a.2). (b) Adding lichess samples into replay buffer can lower KL divergence.
significant shifts between conflicting objectives. This indicates model is effectively optimized to the boundary defined by the reward and the imposed constraints. Potential avenues for mitigating these late-stage oscillations include: (1) employing softer constraint enforcement mechanisms, such as adaptive Lagrange multipliers, instead of the hard entropy mask threshold, and (2) exploring optimization algorithms specifically designed to achieve last-iterate convergence, e.g. [60].
Necessity of Supervised training ("Zero-RL") To underscore the necessity of the initial supervised pretraining phase, we attempt train the model using RL directly from randomly initialized Transformer. Fig. 16 illustrates the results of this "Zero-RL" approach under various entropy bonus settings. The outcome unequivocally demonstrates the infeasibility of this strategy: the randomly initialized model consistently fails generate even valid board positions. Consequently, it cannot produce any samples suitable for puzzle quality evaluation, rendering subsequent RL optimization based puzzle scores impossible. This finding strongly validates the critical role of supervised training in providing a foundational understanding of chess rules and structures, upon which RL can then refine generation towards specific puzzle characteristies.
Mich Eoeriing sauutyy AnTUAt Cooie
X-21 LAbe dlige MALEEION (ROWNAA))
11.01 STADAIE CICUBAL Eirlngu!
Uti naaning
Conflcting erynopy
Tyoch11
03
1300
2006
Figure 15: Other ablation studies. (a) PPO training epochs. (b) Importance of continuous training. (c) ocsillation issue at the later stage of extensive training.
Figure 16: Zero-RL does not work
36

---

I Aesthetics
Theme-detecting functions. To identify aesthetic themes derived from chess literature, we developed Python functions, similar to the Lichess Puzzler project. Each function processes a chess position and its associated solution sequence (mainline). It returns a boolean value signifying whether the specific theme is present. If the theme detected, the function additionally outputs the particular move from the mainline that triggers the theme's identification.
Improving quality of theme-detecting functions. While chess literature illustrates aesthetic themes using carefully selected, expert-approved positions where the theme is central and arises naturally, our initial theme-detecting functions struggled to capture this level nuance. They could identify patterns but often failed to distinguish genuinely creative instances from technically matching but uninteresting ones.
To enhance the quality of detection, we undertook a manual review process, specifically aiming to filter out false positives positions where theme was present but lacked significance (e.g.. an underpromotion immediately nullified by capture or leading to basic tactic). This iterative refinement reduced the occurrence of uninteresting identifications, but eliminating subtle false positives completely was difficult due to the challenge of encoding human intuition about creativity.
Figure 17 presents three example positions generated our RL agent. These are false-positives that we identified while improving the Attacking- -ithhinwll theme. Position (a) demonstrates an initial flaw: flagging Rook withdrawal (Black, e5 e7) whose actual purpose was defensive (preventing checkmate by supporting g7), not calculated attacking retreat. Adjusting for this revealed other issues, like position (b), where a Bishop retreat (Black, h1 + e4) was purely reaction to save material. Addressing these specific motivations led to final implementation, capable of finding compelling examples (some of which are presented in the booklet. Yet, the challenge of capturing nuance persists. Position (c) shows a Knight withdrawal (White, + f3) that, while matching the theme's 's rules, isn't considered creative because the withdrawal itself isn't 't key element of the position's 's strategic narrative. We could not improve this further, as defining and measuring a theme's 'centrality' seems elusive.
8
7
2
X
A
A
3
A
9
(a) Moving the Rook from e5 to (b) Bishop retreat from h1 to e4 is e7 while looks like withdrawal reaction material. actually defensive.
(c) Moving the Knight f3 while withdrawal move is not the key element in this puzzle.
Figure 17: Palse-positive identified while iteratively improving the Attacking- Withdrawal theme- tagging funtion.
Table presents the list of themes that were used for tagging and filtering our generated puzzles. As described in Section 1, we implemented a number of themes from chess literature, namely from Levitt and Friedgood [48]. Avni [2] and Persson [66]. We also included small set of themes from Lichess Puzzler project.
Fig. 18 shows the distribution of puzzles from different puzzle sets across the aesthetic themes. The blue and orange bars corresponds puzzles from Lichess dataset and those generated from auto-regressive generative model. This figure shows that the generative model can produce puzzles across the different themes that represented in the training dataset. The quantity of puzzles
37

---

Secrets of Spectacular Chess
Creative Chess
Tiger Chaos Theory
Lichess Puzzler
Paradox-of-Mateit -f-Merriaal
Attacking Withdrawal
Knight on the Rim is Dim King on Tour
Anastasia's Mate
Underpromotion
Sacrifice Pieces to Stalemate Voluntarily Entering into Pin Exchange Material Down
Arabian Mate
Material Imbalance Draw
Attraction
Winning with Limited Material
Backrank Mate
Set-Play Paradox
Unprotected Position
Boden Mate
Reduces Discovered Checks
Castling
Zugzwang
Deflection
Slow Move
Double Bishop Mate
Switchback
Double Check
Fork
Dovetail Mate
Systematic Manocuvre
enPassant
Paralysis
Exposed King
Self-Paralysis
Hook Mate
Interference
Intermezzo
Novotny
Mate
Bristol
Pin
Loshinsky's 's magnet
Skewer
Smothered Mate
X- R-R Attack
Table 7: Aesthetic themes
Deauee
10.00
3.80
033
040
de and ane
Figure 18: Distribution of aesthetic themes in unique chess positions across different datasets.
10.00
2.50
0x0
0.0
Figure 19: Distribution of aesthetic themes in unique and counter intuitive chess positions across different datasets.
Mating patterns. The above figures show that our puzzle generation methods don't perfectly mirror the training dataset's distribution. Specifically, certain mate themes such as Anastasia's 's Mate, Double Bishop Mate, and Hook Mate are missing from puzzles created by the RI. agent. To understand why, we looked at the distribution of puzzles in the training data that had Dounter-inuitiveenss reward of 0.1 higher (the threshold used for training the RI. agent), as illustrated in Figure 19. This revealed that training examples for Anastasia's 's Mate, Double Bishop Mate, and Hook Mate were not rated as counter-intuitive As a result, their effective representation in the training data used by the RL
38

---

W
A
23
A
A
A
20
Figure 20: King on Tour puzzles from Persson [66] (left, center) and generated by Al (right).
agent zero, which explains why the agent didn't 'tt learn generate high-quality puzzles with these themes and why they are absent its output.
Creativity is subjective. Figure 20 presents a puzzle illustrating the 'King on Tour' theme, an idea Persson [66] describes by noting, "To take one's king for walk is something that most chess players will shun mechanically." This theme arises when player moves their King far from its castle while avoiding danger. Many chess experts, including the book's 's authors, view such positions as creative because their solutions defy traditional chess strategies. On the other hand, some experts find this theme less engaging or uncreative, often because it demands significant calculation. For instance, in the rightmost position of Fig. 20, White has strong winning move (Bf6). Black can only postpone defeat with few checks. but White has defensive option that involves taking the king on 'tour,' a sequence that might even take Stockfish a few seconds to identify as winning. This difference in expert opinion is common across various aesthetic themes, highlighting the subjective nature of creativity chess.
Omitted description of selected chess puzzles. We report the omitted descriptions for the positions 2- 4 in Figure 2. 2: Black's position at glance seems lost White has many more pawns. Surprisingly, Black can force a draw, starting with the Rxh2 rook sacrifice. After the White king recaptures, Black plays Qh6+, followed by Qh4 exploiting the pin on the pawn. Even though White can seemingly defend with Rf3, it turns out that Black force stalemate in this position by playing the second rook sacrifice Rxg3+ and then sacrificing the queen. The stalemate line is unique and counter intuitive as it involves number of sacrifices and slow moves. The fact that the result is forced stalemate is surprising and aesthetic. 3: The unique winning move involves retreating the bishop far away from the attack with Bb1+, It turns out that no matter what Black plays, the idea of creating a bishop-queen battery for White is unstoppable. The unique bishop retreat paired with the slow creation of the battery makes this puzzle counter intuitive and aesthetic to a human. 4 (right): The puzzle is a creative example of a checkmate in 3 moves. Black sacrifices the queen with Qg1+ with the idea of promooig, followed by 'Arabian' checkmate. The combination of the Queen sacrifice, ander-promotions and Arabian checkmate pattern makes the puzzle aesthetic and counter intuitive.
39

---

J
Golden Set
Table 8: TRAIN Set
Postive Exampiees
Negative Examppees
deris Lp Spil pip pa WWP.500///
minghenn (rugkber//prrrpii prppp htpppp/pp////pppppp wwwp://wwwwe.ww/wwii/..o..
KQOq
mnv Bpp In wwppppppppppptiecppp wtw.P///lpPP///110020200
022
hp httppppll///p/p/a/pp (tpp:/2NNNPPPPPP Pipp
0
806 (playil Py2gpphP PttPP/Pa0001PPPPPPP
Zkn San2n/We2tgg 221p21ppp2221111 (PPB)Bpp))))) 22P111100032200
18
KQjl -1
1N (aaP////P11100000 PPPP1// 19
ISk 10p15ppp (22e pi PttpppppppPPPPPpo SPP RR
Sk Se1/5plipp Petpllppppppppiapaa GP/8 te4P RR4K1 b-.3
sQlba hp2p 1/5p2/
w. 2.37
10
In /N22
0.34
1Qtak2 2p2 2222 http://////ppppasppppp (1 b--66
11
w 1F
u 00
12
w--01
13
263N (Ri 2e Ih/ 1hr1p RARRA VWQ1 w-
2q2rikish (mpiees (881 1086-49PPk w--0
14
iBS w-- 0
clrinik 14R in
w--0
15
21 kk (is// KAN 2522 0-01
Bad/2anii/ Ppl SPSPPPPPPPPPPP PIQ K6 w--0 D
16 Sp (AAA///N//b
irsoreg tnlkk/pp pp (ppp)BB 2/ W-0
K400 Q665p2/8 28 IoN2N2 1 p
w--01
BKK/ NPS M21Rp (P/lups000 gSS0eeh
31977 pil (INI (P220/77
0
1B-01/77/K (N442522 113221122211111120000000
1b (p5 http:a/gggpppppppppppppe pwwpg////2PPPPPNN
97268-13122 pp "PPPXXPPPPPPPPPN
Q212 Srin Ptl.cc0NNitp.PPP000
pp 2/ (/MRP//RRKK
Stop
bilkel/5p pip
P1 b-
1/P Www. 4b2
49330 Spakk
Rl RIRRI Pallpp AP Is P www.inipi-eee1ePP. PiMleiiillliiKKaaa Piloo/
4BQ2/20211 Poo3 1BC3PPP P1:2oinnppl
27
Sp (P pape 180 (PMP)/ PP53 h--
Ii (tt.rrrPhhi hi h-
28
222k 1:5pl haskishll pp pp PPP.PPPPPPPPPPP/.P..
3.33.0
Sun 122 2202PPPPPPPP 22 SPPP (5 222 NalgN1 QUINNINNPPPR PRij 1e4 PP (2PP
0.50
29
Takal ON PPP 1o 1.6 UXPSPMRK w-- 0.21
b-- 6
30
rib Inbi 2pp2ttpppp p BA NNP IK2K
1RR IR4R
-
2R5KI 1/ N.PPR Op kepp hptpp//pp/apppppppppppp (np3a22sppPpa
0.66
31
8/P 2p N8 Ul pPSP (PSK 32808 0--0.5
Int/7ppPPSSPP il peid in 1K225N w 7.20
dkI8APp1/PPtt PPPPPPP whi K. 1188p1 b
202022235 Lo (ttppo/peoohtpppp222p2222 httpppppp/tpppppppppppppp by mby Ink hophl 10 PRP111 (2tpp221pppp2222222 22p52 3--
pb laak Ah22
2e ht Magp:eeppppppPppp000000000 wwtp://AAPPPPPPPPP/NPAANA 0
Qpk lppe Py IP PPR- h..00 024
0.41
D2 Op1 wttpg/// 2r5tpspeiu Spips/ Py2iggg 2 piak pplppps hig Nagang 180 1/PPP5/R7
-1
2rinkópl Seh pwirg2PyyPPP SEPRRRAPPP
0:31
TOlignk2pppha 3k3 IbP 2p22 wttpp///w...pppp0/00....
w2w.2200PPPPPP 1828K1
44
6k ip iphelp Ip 1pal peas P200PPP (P/RRKI 0.3
1K 3P3l Nhpp 1 12P wttpp//ppppppppppp
http://ppppPPPPpppppp (PP0 1/ b-- 36
45
126//2 P11001 Pic3 1/3p2P1188 0.50
4r 4r3R/4R Py 2p httpaa/PPPPPPPPPPaPPapp
b-. 27
Table 9: TEST Set
Fudiive Examples
Negaslive Ksamples
8787216532 ONN U-
26r211/86pppp ptt
215
8/8/ 1 34dg///kP/PPPP//555232 -
w 2
rl/k Inghttza wttp:ppphny..00/////////P///
w- - D
b
2QS/B7//R (ttp:///RR1P/22222222
w 01
8/pape/2Q1 Ib (tteegggbikkgggeee0000 h IKCRI b b-
1e3rk http://piirrippppppp//li..
w -- 0
for 4nrl1/1 http://220ppppp111 httppppp///2///333PPPPPP
w 5.27
teank httpppppppppppppppppppp.....m
1NNRRP -- 011
(h2/st1p77382Ppp la p
0
5rk 1/mp// 2pg hpppp Pt/TP
/ 0-
8/1 StKKPsS/2dd11155pppppp 215p2//110lp55//////33333333
ritk Ittpp/pplpp//// httpp////NN/10000 (P P3PPsKK
1K0381kpl
pptppppppppppppppppppppp PP//3448
w--01
70% Sp1 1p NN3P2/2P1 (1/PPPPPPPPPRKK
aK4/Rptk11/P
3g2rk/00pp 122 1pl http://OOpPPPPPPPPaaaaaaaa PoPPaaaaa
Q22
2r3k1/ http://00lpplllllcloooo wtgp:e/ppppppppo/e/////PPP.
1 0
5Q2/p1
--
16 222.2244666RR////2N44 -
17
00
8/3 1/1 Rtt.ssss2-2P..22..s h- 80
18 1tt.PP/PP////NNP/P//PPP.....
wwwp 17
19 (tt.a//ssseee00haaaaaaaeaeeeeeee 0-
2k wtw.2//ppp./1gggsseeeeeee.rreree
20 2ttpyy/bpp///////33//.....
K Broader Impact
Our results of creative chess puzzle generation bring in positive societal impacts, including:
Providing engaging tools for learning chess strategy, tacties, and creative problem-solving, which can foster cognitive skills.
Educational Enhancement:
Increased Accessibility & Engagement: Making novel and diverse chess puzzles more widely available, potentially increasing participation and enjoyment of the game.
Contribution to AI Creativity Research: computational creativity, applicable beyond chess.
The methods developed can offer insights into
We believe our paper does not have potential negative societal impacts.
40

---

L Compute Resources
We spend most of the compute
CPUs. For the final RL experiment, the StockFish 28M chessboards in total with 4096 CPUs, corresponding to 175k CPU hours (each position takes 15-30s, we choose average 22.5s for calculation). Note that the full research project requires much more compute than this single run cost.
41

---

M Creative Chess Puzzles Booklet
though the quality of the positions was generally high, and the players found them be creative, engaging and fun, they have not reached the level of creativity that can be found in chess books or chess compositions.
This booklet contains chess puzzles created using Artificial Intelligence (AI) techniques. This is a brief, non-technical summary of the methods used; for more detail, please refer to the paper.
Some examples of shortcomings include brute calculation, very long mates, a lot of material exchange, or positions where the opponent can give a lot of checks to save time. However, the chess engines found these positions difficult (for example, is common to see the web version of stockfish change its evaluation after thinking for few seconds). They may be interesting study, and therefore some examples can be found in the "Puzzles adversarial to chess engines" section.
Generating millions of chess puzzles: Our first method involved training generative neural net- works (auto-regressive transformer, discrete diffu- sion and MaskGit) on a large open dataset (4M) of chess puzzles from Lichess to learn the distribu- tion of those puzzles. The dataset was constructed by selecting positions from Lichess games us- ing software. Each chess position was repre- sented as sequence in the FEN notation, and neural network was trained to predict the dis- tribution of the next character in the FEN string based on the characters that preceded it. The neural network was trained exclusively on this dataset; no other datasets, including chess com- positions, were used. The trained network was then employed as a generative model to sample chess puzzles, starting from the first character of the FEN and iteratively sampling the remaining pieces. These techniques are commonly used in generative neural networks and language models.
Improving selection of puzzles with aesthet- ies: To improve our process, we took inspiration from the chess literature, studied chess aesthetics and developed theme detectors. Although these detectors were not flawless and often classified uninteresting or poor occurrences of the themes, we were able to identify creative positions by re- viewing the top 50 reward samples per theme and consulting with 2200-2300 FIDE rated players.
Our research suggests that an AI can potentially create interesting chess puzzles, but it still lacks a complete understanding of the nature of creativity in chess puzzles and does not provide a definitive answer to this question.
The second technique used reinforcement learn- ing to train the neural network on its generated output. Reward functions were used to select the best samples, and the network iteratively im- proved at generating puzzles with higher rewards. The reward function had two parts: a uniqueness check, similar to the one used in Lichess, to en- sure there was only one winning move; and a counter-intuitiveness check, to ensure the posi- tion could be solved by a strong chess engine but not weak one. There are many specific details involved in what constitutes a weak vs. strong chess engine in this study. Indeed this was a significant part of this research. However, simple example would be to use a chess engine with a very short computational budget as the weak engine and one with "full power" settings as the strong engine.
The selected puzzles are presented across three sets. The first set is organized into twelve sections: eleven highlight specific aes- thetic chess themes, and the last features puzzles without a dominant theme. To allow for quality comparison, the thematic sections first show ex- amples from existing literature (when possible). followed by our generated puzzles.
Outline:
The second set comprises puzzles created through an evolutionary search process. The third set fea- tures puzzles observed to be particularly challeng- ing for the Stockfish chess engine.
While many of our chess puzzles have annotated solutions, those from our recent experiments cur- rently do not have comments or solutions. We plan to annotate those newer puzzles as time al- lows.
The third method utilized an evolutionary search process. In this process, new positions were cre- ated by randomly adding or removing pieces from the board. Each iteration's 's best positions, as deter- mined by the reward, were selected and continued to evolve. This process resulted in interesting, yet unrealistic, puzzles, which are presented in sep- arate section.
For the learning methods, which used the lichess data for training, we also present the three closest puzzles from the training dataset (this includes the auto-regressive pre trained model and the model that was trained with reinforcement learning, but not the evolutionary search method). Closeness was determined by calculating the edit distance between each generated puzzle and every puzzle in the training set; the three with the smallest distance were selected and reported in the book- let. Inspecting the differences between the gener-
Selecting puzzles by reward: For the booklet, approximately 4 million chess puzzles were gen- erated by sampling positions from the models described above- a similar size to the Lichess database. The positions were sorted according the reward function described above and evalu- ated by group of chess players at Google. Al-
42

---

ated positions and the closest puzzles may help asses the novelty of the generated position.
M.1 Sacrifice
The final section of this booklet presents of Book example:
our generated puzzles as FENs from the three sets. This table excludes solutions and comments, allowing readers to analyze and solve the puzzles without spoilers.
8
We hope you enjoy puzzles, and would be curious hear any feedhack or questions you might have.
7
6
5
4
3
2
1
b
e
9
[Analyse on Lichess]
Selected puzzles:
8
7
6
5
4
3
2
8
8
8
b
d
9
[Analyse on Lichess]
Closest FENs [1], [2], [3]
1. Nd8! The knight is sacrificed two ways, but cannot be captured due to Q17#. Black is forced to give up material. Qxg2+ Qxg2 Bxg2 3. Rf8+! An intermezzo sacrifice that aims to liquidate. Nxf8 4. gxf8=Q+ Kxf8 5. Nxc6+ K17 6. Kxg2 White plays the endgame up piece.
43

---

8
8
7
6
5
4
4
3
2
2
a
b
d
f
h
b
d
9
[Analyse on Lichess]
[Analyse on Lichess]
Closest FENS [1], [2]. [3]
Closest FENs [1], [2]. [3]
1. Qg8+ Nxg8 2. Rh7+ Nxh7 3. Ng6# A creative example smothered mate.
Rg6+! White gives up both rooks to open up the a1-h8 diagonal. Capturing either rook eventually transposes to the same position. hxg6 2. Qal! Kxf7 3. Qf6+ Kg8 4. Bh6 and White covers all the cheeks.
8
7
6
5
7
A
4
3
3
5
2
1
3
b
d
e
9
[Analyse Lichess]
1
Closest FENs [1], [2], [3]
a
C
h
1.
Re4 Setting up mating net. f5 2. Rh4+!
gxh4 g4+ fxg4 4. hxg4#.
[Analyse on Lichess]
Closest FENs [1]. [2]. [3]
1. Rb7+! White gives up the rook to open the diagonal for the queen. Rxb7 2. Nb5+ Kb8 3. Qa7+ sacrificing the queen to finish the game. Rxa7 4. Rc8#.
44

---

M.2
Underpromotion [48]
Book example:
8
7
6
7
5
3
4
5
3
2
3
2
b
d
f
9
9 h
[Analyse Lichess]
Closest FENs [1], [2], [3]
[Analyse on Lichess]
I... Bxa7 2. d6 c3 3. d7 The position looks like it is heading for draw it seems like the black bishop has to stop White's pawn. Black however has another idea in mind. c2! d8=Q cI=N! The only winning line! Black underpromotes with tempo. 5. Kdl e2+ 6. Kc2 exfl=Q Black can eventually escape the checks finding shelter on gl covered by the new queen.
Selected puzzles:
7
5
8
7
3
3
6
5
4
a
h
3
8
[Analyse on Lichess]
2
3
Closest FENS- [1]. [2],
1
1... Qgl+! Black gives up the queen to set up the underpromotion and mate. 2. Rxgl fl=N+! Kh1 Rh2#.
b
d
[Analyse on Lichess]
Closest FENs [1], [2), [3]
1. Qe6 Threatening mate. Bxe7 2. d8=N! Un- derpromoting to threaten smothered mate. White's best option is to give up material. QB 3. NE7+ Qxf7 4. Qxf7
45

---

8
6
5
4
4
3
2
3
3
2
a
b
d
f
h
b
d
9
[Analyse on Lichess]
[Analyse on Lichess]
Closest FENS - [1], [2]. [3]
Closest FENs [1], [2), [3]
1. Rxb8+ Rxb8 2. fxg6 Nd2 If 2... fxg6 Qc7 wins. 3. gxf7+ Kh7! 4. f8=N+! Kh6! Be4! An underpromotion followed by a stunning bishop sacrifice. Black cannot capture the bishop. Qe3 6. Nxe6 White ends up with large material advantage and should convert quickly with careful play.
1. Rd8! White exploits Black's back-rank is- sues. Qxd8! 2. exd8=N! Underpromoting knight is the only move that wins! Rel+ 3. Kf2 Bxd8 4. Re6 White up material with winning position.
M.3
Attacking Withdrawal [2]
Book example:
8
8
6
7
5
6
5
3
4
3
3
2
a
C
d
e
h
1
b
e
[Analyse on Lichess]
Closest FENS [1], [2], [3]
[Analyse Lichess]
1. c8=N! White underpromotes to knight. Nxc8 if 1... Qxc8 2. Qxc8 Nxc8 3. b7 or if 1... Ke6 2. Qxd6 Qxd6 Nxd6 2. Qb7+! Forcing matters. Kd6 3. Qxc6 Kxc6 4. b7.
46

---

Selected puzzles:
8
8
6
6
5
4
8
8
4
8
3
8
2
2
8
b
d
9
a
b
C
d
h
[Analyse Lichess]
[Analyse on Lichess]
Closest FENs [1]. [2], [3]
Closest FENS [1], [2]. [3]
1. Bbl! White withdraws the bishop from the attack with the goal of setting up a bishop- queen battery. It turns out that this plan is largely unstoppable. Qb4 Engine also gives Bg5, but after 2. R713 Black has to depart with the queen and is defending with less material.
Rg4 White retreats the rook and sets up the threat of 2. Qxh7+ Kxh7 3. Rh4#. g5 2.
Bxg8 Kxg8 3. h4 and White is quickly breaking through.
2. Qc6 Threatening both Qg6 and 3. Qxh6!. Black cannot defend. Rgd8 3. Qg6 Qxb1 Black has to give up the queen.
7
3
8
5
7
6
3
5
4
1
3
3
a
h
2
[Analyse on Lichess]
1
Closest FENs- [1], [2], [3]
b
g
Nc2! The white knight retreats and sacri- fices itself to gain valuable time. Qxe4+ 2. Kgl Qxc2 3. Rxb8+ Kxb8 4. a7+ Rxa7 5. Qxa7+ and White has an unstoppable attack.
[Analyse on Lichess]
Closest FENS [1], [2], [3]
1. Re3! The onyl winning withdrawing move. Rxg2+ 2. Khl gxf6 If 2... h6 3. Qxf8+! 3. Rg3+ Rxg3 Qxg3+ Kh8 5. Qd6! Black cannot defend the hanging rook and pawn on f6.
47

---

M.4
Knight on the Rim is Dim [66]
8
Book example:
7
6
5
8
7
4
6
3
8
4
3
2
4
8
b
d
9
8
8
2
[Analyse on Lichess]
Closest FENS [1], [2]. [3]
a
b
d
f
8
[Analyse on Lichess]
7
6
Selected puzzles:
5
M
4
8
3
2
6
b
d
9
4
[Analyse on Lichess]
2
1
Closest FENs [1], [2], [3]
a
b
C
d
f
[Analyse on Lichess]
Closest FENs [1], [2], [3]
1... Qg6+ 2. Kel Na5! Black places the knight the rim, threatening checkmate. 3. b3 axb3 White surprisingly has no good way to defend the threat of Qc2.
48

---

Selected puzzles:
8
7
8
8
8
6
4
5
4
2
3
2
a
b
C
d
e
b
d
9
[Analyse on Lichess]
Closest FENs [1], [2], [3]
[Analyse Lichess]
M.5
Sacrifice Pieces to Stalemate [2]
Closest FENs [1]. [2], [3]
Book example:
1... Ne3+! Rxe3 White is forced to accept the sacrifice due to the bishop hanging on g4. Rgl+! A second sacrifice! 3. Kxgl (3. Kh3 Qxf2 and White forced to give up the rook with Qb7 to stop mate.) Qxf2+ 4. Khl Qgl+ 5. Kxgl Stalemate.
8
6
8
4
7
6
2
5
1
4
a
b
d
f
h
3
[Analyse on Lichess]
2
1
b
g
[Analyse on Lichess]
Closest FENS [1], [2], (3)
1. Rxg7! White starts chain of sacrifices that surprisingly force draw. Bxg7 (1... Rxg7?? 2. Q18+! Rg8 3. Qxh6#) 2. Rxg7! Kxg7 3. Qc7+ Kxg6 4. Qh7+ Kxh7 Stalemate.
49

---

8
6
6
5
4
4
3
2
2
a
b
C
d
h
b
d
9
[Analyse on Lichess]
[Analyse on Lichess]
Closest FENs [1], [2]. [3]
Closest FENS - [1], [2]. [3]
1... Rxh2! 2. Kxh2 Qh6+ 3. Kg2 Qh4! Black sets up strong mate threats that White has to address. 4. Rf3 (4. Rh1?? Rxg3 5. Kf2 Rh3+ and Black wins.) Rxg3+! 5. Rxg3 Black has managed to set up the stalemate with the help of White's rook. Qh2+! 6. Kf3 Qe2+ 7. Kxe2 Stalemate.
M.6 Novotny [48]
Book example:
8
7
6
5
6
4
3
3
4
2
&
8
2
b
d
9
[Analyse on Lichess]
a
b
C
d
f
h
[Analyse on Lichess]
Closest FENs [1], [2]. [3]
50

---

Selected puzzles:
8
7
DCD
6
6
5
8
4
4
3
8
2
3
3
8
2
b
d
g
a
b
d
f
h
[Analyse on Lichess]
Closest FENS [1], [2]. [3]
[Analyse on Lichess]
Closest FENs [1], [2]. [3]
8
Re8! White sacrifices the rook two ways! Qxb3 (I... Raxe8 NI7+ and similarly 1... Rfxc8 Nf7+) 2. NF7+ d5! Blocking the queen trade. 3. Qdi+ Kh2 4. Qh5 Nd8+! 5, Kh8 Rxa8 White up rook and quickly ending the game.
6
5
4
3
7
2
6
b
d
9
4
[Analyse on Lichess]
8
Closest FENs [1], [2], [3]
2
8
a
b
C
d
h
[Analyse on Lichess]
Closest FENs [1], [2], [3]
Bí8! White sacrifices the bishop two ways!
g6 (exf. 2. Qxg7+! Rxg7 3. Rxf8+ Rg8 Rfxg8#) 2. Qxe8 Bxf1 3. Qf7! Q 4. Rf3! Qxf7 5. Rxf7 Black ends the combination with dominating position.
51

---

M.7 Interference [48]
8
Book example:
7
6
8
5
4
6
3
8
8
2
3
3
b
d
9
[Analyse on Lichess]
1
Closest FENS [1], [2]. [3]
a
b
C
d
e
h
1... Rd8! Black sacrifices the rook to interfere with White's pieces.
2. Bxd8 If instead 2. Rxd8
Qe5+ is mating quickly and if 2. Qxd8 simply Bxd8 Qxe8 3. Rxe2 Qxd8 Black remains up piece and should win confortably with correct play.
[Analyse on Lichess]
Selected puzzles:
M.8 Unprotected Position [2]
Book example:
6
8
7
4
6
5
2
8
3
4
8
3
3
a
b
C
d
8
f
2
A
[Analyse on Lichess]
b
d
9
Closest FENs [1], [2), [3]
[Analyse on Lichess]
1. Re6! White interferes with the coordina- tion of Black's queen and rook. Rxe6 Nxe6 Black cannot recapture due to back-rank is- sues. h6 3. Qxd8+ Kh7 4. Nf8+ Kg8 5. Ng6+ Kh7 6. Nxe5 White ends the combination with overwhelming material.
52

---

2. Bxe5 Bxe5 Rxe5 g6 4. Qg5! The critical move that makes this variation work. Qxf7 Selected puzzles:
8
8
6
6
5
4
4
8
8
3
8
2
3
2
8
b
d
9
a
b
C
d
h
[Analyse on Lichess]
Closest FENs [1], [2]. [3]
[Analyse on Lichess]
I... Bd3! Black leaves the queen hanging, but the queen cannot be captured due the mate threat on fl. 2. Rb8+ If 2. h3 g3! 3. Rb8+ Kg7 4. Qg4+ Bgó 5. Qxg3 Qa7+ and Black picks up the rook. Kg7 3. Qxg4+ Bg6 h3 Ba7+ Black
Closest FENs [1], [2], [3]
1... NI3+ 2. Khl Qg3! 3. gxf3 Qxh3+ 4. Kgl Re5 Ending the game with rook lift. 5. f4 Nxf4! 6. exf4 Qg4+ 7. Khl Rh5#.
picks up the rook.
8
8
7
6
5
5
4
3
8
3
2
3
b
d
e
9
a
C
d
h
[Analyse on Lichess)
Closest FENs [1], [2], [3]
[Analyse on Lichess]
1. Rxd5! e5 If Qxb2 2. Qxh7+! Kxh7 3. Rh5#
Closest FENs [1], [2), [3]
5. Re7 Q88 6. Qe5+ Kg8 7. Rxb7 Re8 White seems in trouble due to the back-rank issues, but there is a beautiful finishing move here, 8. Rg7+! Qxg7 9. Qe8+ Qf8 10. Qxf8 After the dust has settled, white remains up pawns and easily wins the pawn endgame.
Qg5! Defending cl and threatening 2. Qxg6+ gxh6 3. Rg8#. Black tries to hold the position. g6 2. Rh3 Kg7 3. Rxh6 Rel+ 4. Qxel Black forced to give up material to defend check- mate.
53

---

8
8
7
6
6
5
4
4
3
8
2
2
a
b
C
d
h
b
d
9
[Analyse on Lichess]
[Analyse Lichess]
Closest FENs [1], [2], [3]
Closest FENs [1]. [2], [3]
8
M.9 XRay Attack
Selected puzzles:
6
3
8
4
7
6
2
5
1
4
a
b
C
d
h
3
3
[Analyse on Lichess]
8
2
Closest FENs [1], [2]. [3]
1
b
e
9
[Analyse on Lichess]
Closest FENs [1], [2], (3]
1. Re6+ The queen defends the rook via an x-ray. Rxc6 The immediate Kb7 transposes to the main line. Qxd8+! Kb7 3. Qb8+ White insists on the sacrifice. Black is now forced to accept. Kxb8 4. Nxc6+ Winning the queen in the next move. White finishes the variation up a rook.
54

---

M.10 Paralysis [48]
Book example:
6
8
7
4
6
8
5
2
4
3
a
b
C
d
f
h
2
[Analyse on Lichess]
1
Closest FENS [1], [2]. [3]
b
9
Be4 NI3+ If Rdl+ instead, then 2. Kg2 Qh4 Rxg7! and White wins quickly. Kg2 Nh4 3. Nxh4 Qg5 Black attempts clever ma- neuver, rerouting the queen towards better attacking square hoping to complicate matters. 4. Rxg7! Qxg7 5. Qxg7 Bxg7 8. Bxg7 The x- ray attack White's bishop results in massive liquidation. Kxg7 9. Bxd3 White ends the com- bination with bishop and knight for rook, in a technically winning endgame although it will require careful play.
[Analyse Lichess]
Selected puzzles:
8
800
6
5
4
8
7
3
-
A
8
2
3
5
1
4
b
d
e
9
2
3
[Analyse on Lichess]
Closest FENs [1], [2], [3]
a
d
1. Ng7 White sets up 2. Rxh7+ 3. Qh4#. Black is forced to act quickly. Rxg7 2. Rxb3! White ignores the situation on the king side and in- stead aims to suffocate Black's pieces. Ne8 3. Qh4 Black's 's position is completely paralysed. We show line that highlights the theme. Qe5 4. Rb8! Qc6 5, Qg5! Black is never able to save the rook due to mate threats with Rxh7+ Kg8 6. Qxe5! Kh8 Rxe8+ Rg8 8. Rxg8+ Kxg8 Qb8+ Qe8 10. Qxe8#.
[Analyse on Lichess]
Closest FENs [1], [2], [3]
55

---

M.11 Bristol [48]
Book example:
8
7
4
6
5
2
8
4
3
a
b
d
h
2
1
[Analyse on Lichess]
6
0
g
Closest FENs [1], [2], [3]
1... Nf3 2. Rh5 g5! Black aims to trap the White rook. 3. Bd5 Nh4! 3. Rh3 g4: 4. Rh5 Kg6 and White loses the rook and is getting mated quickly. 4. Be4+ Kg7 5. Kgl Rxa2 White's rook is paralysed and Black should be able to eventually win this position without much resistance.
[Analyse on Lichess)
Selected puzzles:
8
7
6
8
5
4
6
W
3
5
A
8
2
1
3
b
9
[Analyse Lichess]
a
C
d
e
h
Closest FENs [1], [2], [3]
1. Ba3+ Nb4 1... Ne7 again met by 2. Bg8 and 1... Bb4 is followed by simply 2. Nxc6. 2. Bg8! The Bristol move, making space for the White queen. Be7 3. Bb2 White does not hurry with Bf7+ and first plays intermezzo solidify their position. Qc5 QI7+ Kd8 Ne3 White enjoys decisive advantage with safer king and very active pieces.
[Analyse on Lichess]
Closest FENs- [1], [2], [3]
1... a2 2. Rdl Bf5 Black threatens Bh1 which would secure the promotion. 3. Rfl Kd8! Black cannot immediately play Bb1?? due to Rxf7+. The slow Kd8 instead wins! 4. Ral Bbl White now completely paralysed. Black has the simple plan of pushing the and h pawns and wins this position without much effort.
56

---

M.12 King on Tour [66]
Book example:
8
7
6
7
5
4
8
8
5
3
8
2
3
3
2
8
8
b
d
g
[Analyse on Lichess]
f
h
Closest FENs - [1], [2]. [3]
[Analyse on Lichess]
Selected puzzles:
8
6
7
5
4
8
5
8
3
3
2
3
2
A
8
b
d
e
9
[Analyse on Lichess]
a
Closest FENs [1], [2], [3]
[Analyse on Lichess]
Closest FENS- [1], [2), [3]
57

---

Selected puzzles:
8
8
6
8
MM
7
8
3
6
4
8
8
5
4
2
3
2
8
8
a
b
C
d
f
h
b
d
9
[Analyse on Lichess]
Closest FENs [1], [2]. [3]
[Analyse on Lichess]
M.13
Switchback [48]
Closest FENs [1], [2]. [3]
Book example:
8
M.14
Uncategorized
6
8
4
8
7
3
3
6
2
8
5
4
1
a
b
C
d
f
h
3
3
3
2
[Analyse on Lichess]
1
b
8
9
[Analyse Lichess)
Closest FENs [1]. [2], [3]
I... d2 2. Be2 QíS! The only winning idea. Black sacrifices the queen to try to promote the pawn. 3. Bdl Qcl White's pieces are not coordinated and there is no good response.
58

---

N Puzzles generated with evolutionary search
8
The following puzzles are selected from the ones generated with evolutionary search.
As briefly discussed in the Introduction, this method contrasts sharply with our generative modeling approaches. Instead of learning from data, this method relies applying random tations and perturbations to a population chess positions, and directly optimizes for counter intu- itiveness check. The optimization process does not enforce realism constraints, which a feature of our generative approaches and the puzzles produced with this method often diverges sig- nificantly from expert-level games. 'The allows for generating beyond what present within the training data, demonstrating a powerful alterna- tive for creative chess puzzle generation.
8
8
4
A
8
2
1
a
b
C
d
h
[Analyse on Lichess]
Closest FENs [1], [2], [3]
8
7
6
8
5
4
3
2
b
9
[Analyse on Lichess]
I... Qxh3 Giving up the queen for the bishop is the right decision here. The other moves fail tactically. For example 1...Qc7 2. Rd7 1... Qb8 2. Qxb8 or 1... Qg5 cxb7 Rxd8 Ne6
Qf6 4. Nxd8 Rc7 5. Qa8 is winning for White
2. gxh3 Rxc6 3. Rxc8 Qg5+ 4. Khl Bxc8 5. Q13 Q15 Qxf5 Bxf5 7. Qd4 Kxf8 8. Qxb4
Nd6 And the pawns on the queenside can't be stopped.
59

---

8
7
6
6
5
4
4
8
3
2
2
a
b
d
f
h
b
d
9
[Analyse on Lichess]
[Analyse on Lichess]
Among the 3 possible captures on d7 (and other alternatives) only one is correct. 1. Bgxd7 Ne6+ 5. Kc8 Qc5+ 6. Bc6 Qe7 7. b8=Q+ wins
1... Qo4! Not I... Qxe5 2. Rxf6 Qh5+ 3. Kg8 and while Black can hold the draw, there is no path to a win here. Other tries don't work either, for example I... Qf5 2. Nd7! Qxd7 3. Rxf6 Qh3+ 4. Kg8 Qc8+ 5. Rf8 and this is a draw. Another try would be 1... Qc3, but then White just play Ne6 and draw would ensure shortly. 2. Rxf6 Qh4+ 3. Kg8 Qxf6 4. Nd7 Q15 5. NI8 Ke3 and Black can start bringing the king forward. If White tries 6. Kh8 Qe5 7. Kg8 Qh5 8. Neó Ke4 and the king keeps approaching.
c6 al=Q cxb7 Qa3+ 4. Kd8 for White White were to instead play 1. Kxd7, then Black would lose if following up with 1... a2, but wins in case of 1. Kxd7 Ne4 c6 Ne5+ Kc7 bxc6 4. Kxc6 Kd4 5. Bxh3 a2 6. b7 Nxb7 7. Kxh7 al=Q. It's important to note the following vari- ation: 1. Bgxd7 Ne4 2.. c6 Nd6 c5, which is why Ne4 can't save Black in the solution. Finally, in
example line that follows. If
of the other bishop capture: 1. Bexd7 a2 2. c6 al=Q and now 3. cxb7 wouldn't work (in contrast to the solution) 3... Qa3+ 4. Kd8 N17+ 52. Kc7 Qd6+ as an example line te knight check on f7 is possible as the bishop is no longer e8 other moves don't work either (c7 or alternative king moves).
7
5
8
8
7
3
6
2
5
4
h
3
2
8
[Analyse on Lichess]
1
b
d
Despite White's overwhelming material advan- tage, Black manages to salvage a draw. I... Qb6+ 2. Kc5 Qf6+ 3. Kd5 Qd6+ 4, Kxd6.
[Analyse on Lichess)
60

---

1... Qe2! Moving the queen from one square where it can be captured to another. 2. Qxe2 Qxg5 3. Rxc8 Qxh4 4. g3 Nxc8 and Black is winning.
Of all the possible pawn captures, only one is winning for Black. I... fxg4 Ne6+ Kd6 3. Nxg5 exf4 4. c5+ Ke7 a5 Bc4 and White can't stop the Black pawns. Interestingly, capturing the hanging White knight on the first move would have lost the game for Black. 1... Kxd8 2. gxh5 Ke7 3. exd5 exf4 4. h6 Kf6 5. h7 Kg7 6. d6 Be6 7. a5 Be8 8. a6.
7
8
7
4
6
8
5
2 A
4
3
a
b
d
f
h
2
[Analyse on Lichess]
b
d
9
In this position, there are many moves con- sider, and precise calculation is needed to es- tablish that only one of them wins for White. Kxf3 hxg4+ 2. Bxg4 dxe4+ Kxe4 c3 4. Kd3 exf4 5. Bxe6 Let's consider the alternatives. 1. dxe5 dxe4 2. fxg5 c3 3. Ke3 f2 4. Bf1 h4
[Analyse Lichess
In what is a very messy position, there is narrow path to win. I... Rd6+ (I... axb4 Bxh3 Rd6+ 3. Ka7 Bxh6 4. c8-Q+ Bg8 5. Bxg3 Rxg6 6. Be5+ would be winning for White instead. The other captures on move fail as well.) 2. Ka7 (2. Kxb7 Bd5+ 3. Ka7 Bxc8) Bxc8 3. Bxg3 Rxgó 4. Be5+ Kh7 5. Rxb3 Bc5+ 6. Kb8 Rg8 7. Rd3 Bb6 Rd8 Rxd8 9. exd8=Q
Bxh3 e3+ 8. Kxe2 e2 9, g7 el=Q 10. g8=Q Qxe5 is a draw instead. 1.
5. g6 c2 Kd2
gxh5 c3 h6 c2 3. h7 cl=Q h8=Q Qd2+
5. Kxf3 dxe4+ is winning for Black, because
6. Kxe4 is impossible due to the threat of 6...
Qe2# For the same reason 1. fxg5 fails. Finally,
1. exd5 c3 2. Ke3 exf4+ 3. Kd3 hxg4 4. Bxg4 f2
Bxd8 10. Kxc8 Kxh6 11. Kxd8 Kg5.
5. Be2 g4 is winning for Black.
8
7
3
7
6
5
5
8
3
8
4
3
3
2
2
6
d
a
h
[Analyse on Lichess]
[Analyse Lichess]
61

---

In what is aesthetically a very pleasing posi- tion, there are three different pawns that be promoted on the next move, and multiple additional promising captures to calculate as well for example, the hanging Black rook on cl. However, it turns out that there is only one winning move, and it is h8=N+ Kg7 2. Rxg6+ Kh7 3. b8=Q Rc2+ 4. Kfl.
on fl would throw away the advantage, which is rarely the case! This is especially puzzling at the first glance since the rook is otherwise hanging on f3. Yet, there are specific tactical reasons for why this resource not available. The correct capture is the capture on a8, one example line being: 1... Rxa8 2. Qd5 Rxg3 3. Nxg4 Be6 N16+ Kh8 5. Nxh5 Bxd5 6. Nxg3 Qxc5 where Black is winning. Obviously this line is not forced, but the alternatives don't affect the outcome. So, let's look at why Rxf1 fails specifically on the first move of the prob- em: I... Rxf1+ 2. Rxf1 Rxa8 Bxe3 Nxe3 Qxe3 Qc6 5. Qg5+ Bhgó 6. Nh5 would be win- ning for White. In contrast, with the rook still there, Black is able to generate simultaneous threats against e3 and g2 in the same variation, under the main line of the solutions: 1... Rxa8 2. Bxe3 Rxg3 3. Nxg4 Bdxg4 4. Qxc2 Rxg2+ 5. Qxg2 hxg2.
promotooo
7
8
5
3
8
1
7
a
C
9 h
6
5
[Analyse on Lichess]
4
Despite the White queen hanging on bl, Black needs find different motif win in this position. I... Bd6 2. Qxb2 Rg[3+ 3. g3 Bxg3+ 4. Kh3 Rh1+ 5. Kg2 Nh4+ 6. Kxhl Rfl#
3
8
2
8
8
6
7
5
[Analyse on Lichess]
3
It is possible eapture either of the two White queens on c3 and h7 respectively, but neither is the best move the best move involves ignoring the opportunity and giving check instead. I... Rxg2+ 2. Kh1 Nxe5 3. Qxg7 Nxg6 4. Nxf3 Qg3 is the winning recipe. If Black were have been tempted by material instead: 1... Rxc3 2. Nxd7 Qxd7 3. gxf3 Bh5 4. Bb2 Nxh7 5. gxh7 Rxe4 dxc4 Rh8
2
a
f h
[Analyse on Lichess]
Bxg7 Rxh7 8. Qxd7+
In this astoundingly chaotic and complicated position with many possible captures, there only one move that secures the win. Per- haps more interestingly, interjecting check
Kxd7 9. Bd4 Nxd4 10. Rfd1 Bd6 11. Rxd4 Keó 12. f4 Bc5 13. Rd3 Be2 14. Re3 is only draw as an example line that may follow from the capture.
62

---

to e2 instead fails due to: 1. Qe2 Rgl+ 2. Kxgl cl=Q+ and if 1. Qg5 cl=Q.
7
8
6
7
6
4
5
4
8
2
3
i
2
a
b
C
d
f
1
[Analyse on Lichess]
b
9
Black needs to find the correct capture, among the available options. 1... Nxd4 2. Bxd4 cxd2 3. axb4 exd4 4. Bf5 Qh3 is winning for Black, unlike the alternatives. If instead: I... Rxd4 Qxd3 Bxf8 4. Bxf7+ Kxf7 5. Qd5+ Kg7 6. Qd7+ Kh8! 7. Qxg4 or 1... exd2 2. NfS Qh3 3. Bxf7+ Kxf7 4. Ng5+ another example line (with possible deviations at several points): I... exd4 2. bxc3 fxg6 3. Ne6 Nce5 4. Neg5 dxe3 Qd1 Bf6 6. Bxb4 Bxg5 7. h4 Bxh4 8. Bxc3.
[Analyse Lichess]
To win, amidst the chaos, White needs to play a quiet king move in the centre of the board. 1. Ke4 Bxd7 2. Nxd7 If instead
bxc3 Rxd3
Qh7 Bg3+ Ke4 Nc5+ 3. Rxe5 Re6+ 4. Re5 Rxf6 S. Ngl+ Kd2 6. Nf3+ Ke2 7. Ngl+ the threats against the White king secure Black a draw.
8
7
Mi
6
7
5
4
5
3
4
2
8
1
2
b
C
e
9
d
h
[Analyse on Lichess]
Black has 5 different ways of capturing the
[Analyse on Lichess]
knight on e7 that giving check. However, the only correct decision is not to capture it at all! I... Kf8 2. Qxh7 Bf2+ 3. Khl Kxe7 4. exd8=Q+ Re8+ Kd7 Bd3 Bxd3 9. Bxd3 exfl=Q+ 10. Bxfl Qxe8 11.
Of several possible first moves, only one wins.
One possible continuation would be: 1. Qf2
Rexd8 5. Bxd6+ Rxd6 Rxc8
cl=Q 2. axb4 0g5+ 3. Bg3 Kxg7 4. Nxd1 Qd5+
5. Kh2 Nxe8 6. bxa7 Qxd8 7. Bxb7 Nc7 8. Ne3
Qd3 Ne3 12. Qb5+ Ke7 13. Qxb7+ Qd7 seems
Kg8 9. Ng4 Rxg3 10. Qxg3 Nfe6 11. Nh6+ Ki8
to be holding on, an example continuation. Alternatively, 1... B8xe7 2. Qxh7+ KI8 3. Rxe4
12. Qg8+ Kc7 13. Qx17+ Kd6 Moving the queen
63

---

dxe4 4. Qh8+ Qg8 5. Qxg8+ Kxg8 6. Ba2+ Kf8 7. Rh5 winning for White. Rxe4 comes up as a motif in some of the other lines, for alternative first-move captures.
Rbf2 Ra7 8. Qxe3 Rxe3 9. Rf5 With equal- ity. This ultimately choice between two possible captures on with interesting ideas in both lines. If (the wrong move) 1.Qxe2 then after 1...Nhxf2 Qxel Nxh3+ 3,Kh1, Black plays the beautiful move 3...Nf4! where after 4.Rxf4 Qxf4! (Another nice sacrifice) 5. exf4 NI2+ 6. Qxf2 Bxf2 Black suddenly up mate- rial. But the reason why Rxe2 works instead is even more subtle. So, let's look at that same line, just with the rook on e2. 1.Rxe2 Nhxf2 2. Qxel Nxh3+ 3. Kh1 NE4 4. Rxf4 Qxf4 and now instead exf4, White has a surprising option 5. Ne7+! Rxe7 6. exf4 Nxf2+ 7. Qxf2 Bxf2 8. Rxe7 because the rook e7, where it captured the knight, is not defended by the rook a8, White doesn't It end up down exchange.
7
4
2
8
8
a
b
d
f
h
7
[Analyse on Lichess]
6
1. cxd4 Nxg2 2. Kb6 Bc8 3. Be6 NF4 4. Bb7 5
8
Bxb7 5. Kxb7 Nd5 6. b5 Nb4 7. bxa6 Nxa6 8.
Kxa6 9, Kb6 f4 a6 f3 11. a7 12 12. a fl=Q 13. Qg8+ and White will pick up the e7 pawn and have a winning position. Alternative first-move captures don't work for White, for example: 1. Kxd4 Nxg2 2. Kc5 NE4 3. b5 Nd3+ Kb6 Nb2 5. Bb3 axb5
4
3
8
2
a6 Ne4+ Kxb5 Nd6+ 8. Kc5 Ne8 9. BdS Bd3 10. Bb7 Bxa6 11. Bxa6 Nd6 is draw, and capturing the knight instead loses: 1. dxe3 dxe3 Kb6 Bd3.
1
b
e
g
[Analyse Lichess]
The Black rook on d5 hanging with check,
but capturing it is, surprisingly, not best. 1. b6 Qb4 2. Rd6+ Rxd6 h8=Q Qxb6 4. exd6 With mate to follow. If instead Kxf7 Qf1+ 3. R16 Qc4 4. Rd6 gxh3 5. Ke7 Rgl it would Black who winning.
7
Rxd5+ Kc7+ 2.
4
2
8
8
a
b
C
d
h
[Analyse on Lichess]
Rxe2 Nhxf2 2. Qxel Nd3 Qdl Nxb2 4.
Rxb2 Bxe3+ 5. Nxe3 Ne3 6. QB Qxe3+ 7.
64

---

[Analyse on Lichess]
textbfClosest FENs [1], [2), [3]
8
8
7
4
6
5
M
2
4
3
a
b
C
d
f
2
1
[Analyse on Lichess]
6
9
There are many possible moves to consider in this chaotic and highly unrealistic board setup. Yet, there only one solution, and it involves a temporary forced rook sacrifice! 1... Rg4+ 2. Kxg4 Nxf6+ And
[Analyse on Lichess)
Closest FENs [1], [2], [3]
recapturing doesn't work because 3. gxf6 Bfe2+ 4. Kg5 NI8+ 5. Bg7 Be3+ 6. Kf5 f1=Q+ 7. Ke5 Qf4# 3. Kg3
Ne4+ Kg4 Bfe2+ 5. KIS fl=Q+ Ke6 Nxc5+
Rxc5 Bg4+ Rí5 Qxf5+ 9. Kd6 Qe5#..
8
7
O Puzzles adversarial to chess engines
6
5
The following puzzles were generated with rein- forcement learning. These positions are computa- tionally demanding, and often requires significant search-time for Stockfish to determine the opti- mal line. This is likely because of many pieces on the board, which causes Stockfish to evaluate a number of lines from large search tree.
3
M
2
A
b
d
g
8
[Analyse on Lichess)
7
Closest FENS [1], [2], [3]
5
3
A
a
C
h
65

---

8
8
7
7
6
6
5
4
8
8
4
3
2
2
a
b
C
d
f
h
b
d
9
[Analyse on Lichess]
[Analyse on Lichess]
Closest FENs [1], [2]. [3]
Closest FENs - [1], [2]. [3]
8
6
6
4
5
4
4
8
3
8
2
2
3
1
a
b
C
d
h
b
d
9
[Analyse on Lichess]
[Analyse on Lichess]
Closest FENs [1], [2]. [3]
Closest FENs [1], [2], [3]
66

---

8
8
6
6
8
5
4
8
4
3
8
8
2
2
E
a
b
d
f
h
b
d
[Analyse on Lichess]
[Analyse on Lichess]
Closest FENs [1], [2], [3]
Closest FENS - [1], [2]. [3]
8
DPO
7
6
6
8
5
4
8
4
M
3
2
2
1
a
b
C
d
f
h
b
d
9
[Analyse on Lichess]
[Analyse on Lichess]
Closest FENs [1], [2]. [3]
Closest FENs [1], [2], [3]
67


Reviews:
[{"review_id": "Wv23NE3AmH", "reviewer": "Reviewer_MudV", "summary": "This paper proposes a method for generating creative chess puzzles using a transformer model enhanced with reinforcement learning. By defining and quantifying creativity through uniqueness, novelty, counter-intuitiveness, and aesthetics, the authors design reward functions based on chess engine search statistics. Their RL approach significantly increases the generation of counter-intuitive puzzles while preserving aesthetic qualities. Human evaluations show that the generated puzzles rival or surpass human-composed ones in creativity and engagement.", "questions": "Q1) Typo in Section 2 \"(latent diffusion [60, LDMs] )\"?\n\nQ2) In Table 1, the reported precision scores (e.g., 0.747504, 0.731198, 0.740115) have 6 decimal places, which is unnecessarily precise. Consider using only 4 dp?\n\nQ3) A discussion on potential transfer to other creative domains or a proof of concept in a second domain would strengthen the paper's broader impact?\n\nQ4) The Golden Set used to tune the counter-intuitiveness reward contains fewer than 100 samples. Would it  limit the robustness and generalizability of the learned scoring function? Will expanding this dataset or incorporating crowd-sourced annotations could improve the reliability of the reward design?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "S1) This paper demonstrates high originality, introducing novel problem formulations and a metric that quantifies creativity in chess puzzles. It also has a good motivation as AI's role in creative domain is an important area.\n\nW1) While the approach is compelling, the study is heavily domain specific, relying on chess engines, Lichess data, and hand crafted puzzle heuristics, which raises questions about the generalizability of the framework.", "was_revised": true, "final_justification": "I think overall this work has a good originality, and W1 rebuttal is convincing. I also agree in Q4 that high-quality data collection for this task is uniquely challenging, its a good strategy to prioritize the data quality than quantity. I will maintain my score.", "rebuttal": "We thank reviewer MudV for your constructive comments and feedback. \n\n## **New human eval result with chess grandmasters (GM)**\n\nBefore the rebuttal, we’d like to add a new human evaluation result with chess GM on our booklet. The following excerpts are their comments, which further affirm the value of our work.\n\n- GM A: **I'm quite particular about what makes a good chess puzzle—I look for natural-looking positions with reasonable play. Yet, I found myself enjoying many of the AI-generated examples in the booklet.**\n- IM for Chess Compositions B: **This booklet adds novel, AI-generated puzzles to the existing chess literature. I found several puzzles that were original, creative, and aesthetically pleasing, with surprising solutions that were satisfying to solve.**\n- GM C: **The puzzles in this booklet represent a pioneering step in human-AI collaboration for chess composition and hint at what might be achieved in the future for composing chess puzzles.**\n\n## **Rebuttal**\n\n**W1: While the approach is compelling, the study is heavily domain specific, ..., which raises questions about the generalizability of the framework.**\n\nThanks for your feedback. We respectfully position our work as a Creative AI application, a contribution category (https://neurips.cc/Conferences/2025/CallForPapers) explicitly welcomed by the NeurIPS call for papers. We argue that for a topic as sensitive and nuanced as AI-driven creativity, a deep dive into a single, rich domain is not a limitation but a methodological necessity for drawing sharp, meaningful conclusions. As we cite in our introduction, the field has seen instances where shallow evaluations of creative systems yielded misleading results, a pitfall we were determined to avoid. \n\nChess composition represents an ideal domain for such an in-depth study. It is a field with a profound history, recognized world titles, and established aesthetic principles, creating a perfect testbed for exploring the intersection of computation and creativity. Our focused investigation allowed for a rigorous analysis, culminating in the curated booklet of compositions we provided for expert (GrandMaster) review. Furthermore, our work makes a crucial contribution to the field demonstrating the ability of AI techniques to compose questions, and not only to solve them. It is often the case that even though solutions to specific problems appear more domain specific, they are important stepping stones to generalize later.\n\nWe also wish to emphasize the novel adaptations and contributions required to make these methods successful for our task, and we believe that the NeurIPS community can gain from reading our lessons. Specifically, our system is general and most of our contributions can be extended to other domains:\n\n- **Novel Counterintuitiveness Reward and generalizability**: Our primary technical contribution is a novel reward function that models counter-intuitiveness by analyzing the search statistics of an MCTS planning algorithm, effectively quantifying 'surprise.' This technique is directly transferable to other domains where MCTS is applied, such as games, mathematics, and computer science. More broadly, we propose this concept generalizes beyond explicit search. By drawing an analogy between 'deep search' and the 'deep thinking' demonstrated by recent reasoning-focused models (e.g., O1, R1), our approach offers a pathway to measure the novelty or counter-intuitiveness of solutions in any domain that advanced language models can address.\n\n- **Diffusion Model & Auto-regressive model.** While the model architectures that we’ve trained are otherwise standard, the engineering setup enables this to be applied to discrete generative problems more broadly, whether it is games or language.\n\n- **Adapting RLHF for Novelty**: A naive implementation of RLHF-PPO proved insufficient, suffering from entropy collapse and failing to produce novel outputs (Section 2.1, with more details given in Appendix H). It is fairly new to point out this entropy collapse issue and only in very recent work [Wang 2025], [Cui 2025], researchers began to investigate the role of entropy and the potential for its collapse during the RL training of LLMs, as well as methods to mitigate it. In addition, our RL ablations detail the non-trivial modifications (specifically, measure novelty using various metrics and to either penalize or skip training on samples that are not novel) required to overcome this, offering generalizable lessons for any researcher seeking to optimize for novelty in conjunction with a reward signal. It is only in the latest LLM reasoning literature that techniques based on conceptually similar principles have started to be discussed. For instance, [Yu 2025] is using curriculum learning to actively filter out problems on which the model has already converged.\n\nWang, Zihan, et al. \"Ragen: Understanding self-evolution in llm agents via multi-turn reinforcement learning.\" arXiv preprint arXiv:2504.20073 (2025).\n\nCui, Ganqu, et al. \"The entropy mechanism of reinforcement learning for reasoning language models.\" arXiv preprint arXiv:2505.22617 (2025).\n\nYu, Qiying, et al. \"Dapo: An open-source llm reinforcement learning system at scale.\" arXiv preprint arXiv:2503.14476 (2025).\n\n**Q1: Typo in Section 2 \"(latent diffusion [60, LDMs] )\"?  Q2: the reported precision scores in Table 1.**\n\nThank you for pointing this out. LDM refers to the latent diffusion model. We will correct this and use 4 dp in the main paper.\n\n**Q3: A discussion on potential transfer to other creative domains .....?**\n\nThank you for your constructive comments! We will add a broader impact discussion section in the main paper. Here is a draft:\n\n**Broader Impact and Generalizability**\nWhile this work is demonstrated within the specific domain of chess, its underlying principles and technical contributions possess significant potential for broader impact. The framework's most direct application lies in other domains that leverage guided search over a discrete state space, including not only complex board games like Go but also fields such as automated theorem proving. For instance, since systems like AlphaProof utilize AlphaZero, our counter-intuitiveness metric—which compares shallow and deep search evaluations—could be directly transferred to identify non-obvious or elegant problems, serving as a powerful reward signal for generating more creative mathematical problems. Conceptually, this work can also generalize to recent test-time scaling trends (OpenAI O1, DeepSeek R1). By taking analogy from deep search to deep thinking, we can also calculate the counter-intuitiveness for a given question of any domains that LLM can tackle.\n\nIn addition, the diversity-filtering based RL framework is widely applicable in other domains. In developing our methods, we identified and provided a solution for the \"entropy collapse\" phenomenon, a critical issue that has only recently gained attention in the LLM community. Our solution—a dynamic diversity-filtering mechanism that promotes continuous exploration by penalizing non-novel samples—is domain-agnostic and can be applied to most tasks, from language to code generation, to prevent the policy from converging to a fixed solution. In summary, this research offers a transferable blueprint for quantifying and optimizing for non-obvious solutions and for building explorative RL systems capable of advancing the frontier of creative AI.\n\n**Q4: The Golden Set used to tune the ..... Would it limit the robustness and generalizability of the learned scoring function? Will expanding this dataset or incorporating crowd-sourced annotations could improve the reliability of the reward design?**\n\nWhile we acknowledge that a larger dataset would enhance robustness and performance, the effectiveness of our method with a small, curated set is also a key strength. This can highlight a few critical points:\n\n- **Effectiveness approximating crowdsource results.** Even though our curated set is small, in Appendix F, Figure 6 we have shown that there is a positive correlation between our counter intuitiveness reward and lichess puzzle’s rating score. This indicates that our method can indeed capture crowdsource human player’s difficulty preference.\n\n- **Efficiency of Engine-Based Feature Engineering.** Our approach does not learn a reward function from scratch. Instead, we are tuning a linear combination of a small set of powerful, semantically rich features (e.g., critical search depth) extracted from deep analysis by state-of-the-art chess engines. The purpose of the small Golden Set is simply to find the right weighting of these features that best aligns with human intuition. This stands in stark contrast to standard RLHF, which typically requires a dataset several orders of magnitude larger to learn a meaningful reward model. Our method's data efficiency is a significant practical advantage.\n\n- **The High Bar for Quality Data in this Domain:** We must also clarify that high-quality data collection for this task is uniquely challenging. Unlike tasks amenable to general crowdsourcing, reliably evaluating the \"counter-intuitiveness\" of a complex chess position requires substantial, specialized domain expertise. An annotation from a novice player would be of very low quality, as they may not be able to distinguish a brilliant, counter-intuitive move from a simple blunder. To ensure data quality, our human expert evaluators possessed Elo ratings between 2000 and 2400, placing them in the expert-to-master category. Sourcing a large number of annotations at this level of expertise is a significant undertaking. \n\nTherefore, while expanding the Golden Set with more expert-annotated examples is a valuable direction for future work, our current results demonstrate the power and efficiency of combining deep engine analysis with targeted, high-quality human guidance."}, {"review_id": "MtQSW9diay", "reviewer": "Reviewer_2cAd", "summary": "The paper studies the problem of generating creative, aesthetic, and novel chess puzzles using generative models. The paper first defines numerical metrics for measuring these unique aspects of chess puzzles, then benchmarks multiple generative schemes, trained on a collection of chess game datasets. The author then adapts the autoregressive transformer with RLHF-PPO based fine-tuning using reward functions designed to maximize realism, novelty, and counter-intuitiveness in the output. From multiple numerical experiments, ablation studies, and human evaluation, the author showcases the effectiveness of GenAI and RL techniques for chess puzzle generation and provides a booklet of generated puzzles as an end product.", "questions": "- Graph on line 122 need minor fixing\n- 3.3 and 3.4 might need to be introduced in chronological order in section 3 overview paragraph", "limitations": "- Despite successfully generating puzzles that are evaluated favorably in both numerical experiment and human evaluation, the RL model still requires significant design effort to ensure good convergence.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strength\n\n- Well-written paper with tight structure and clear logic flow\n- clear quantification of various measurement with articulated intuitions (such as quantification of puzzle uniqueness, novelty and counter intuitiveness) and solves unique challenge for chess puzzle generation\n- meticulously designed reward with experiment demonstrate appropriate of design choices\n- demonstration of interesting property such as emergent of aesthetics from AR and RL backbones\n\n### Weakness\n\n- the work mainly used well established generative technique (AR, diffusion, RLHF-PPO), therefore the contribution largely lies in engineering choices specifically for chess puzzle generation.", "was_revised": true, "final_justification": "The reviewer initial found this work to be good contribution and quite creative in the application aspect. The quality of the work is also really high, therefore will remain the recommendation of acceptance. However, given the work is application focused and optimized forward a really specific usage of chess puzzle generation, the reviewer does not believe this work qualifies for higher score than 5.", "rebuttal": "We thank reviewer 2cAd for your constructive comments and feedback. \n\n## **New human eval result with chess grandmasters (GM)**\n\nBefore the rebuttal, we’d like to add a new human evaluation result with chess GM on our booklet. The following excerpts are their comments, which further affirm the value of our work.\n\n- GM A: **I'm quite particular about what makes a good chess puzzle—I look for natural-looking positions with reasonable play. Yet, I found myself enjoying many of the AI-generated examples in the booklet.**\n- IM for Chess Compositions B: **This booklet adds novel, AI-generated puzzles to the existing chess literature. I found several puzzles that were original, creative, and aesthetically pleasing, with surprising solutions that were satisfying to solve.**\n- GM C: **The puzzles in this booklet represent a pioneering step in human-AI collaboration for chess composition and hint at what might be achieved in the future for composing chess puzzles.**\n\n## **Rebuttal**\n\n**Q1: The work mainly used well established generative techniques (AR, diffusion, RLHF-PPO), therefore the contribution largely lies in engineering choices specifically for chess puzzle generation.**\n\nWe respectfully position our work as a Creative AI application, a contribution category (https://neurips.cc/Conferences/2025/CallForPapers) explicitly welcomed by the NeurIPS call for papers. We argue that for a topic as sensitive and nuanced as AI-driven creativity, a deep dive into a single, rich domain is not a limitation but a methodological necessity for drawing sharp, meaningful conclusions. As we cite in our introduction, the field has seen instances where shallow evaluations of creative systems yielded misleading results, a pitfall we were determined to avoid. \n\nChess composition represents an ideal domain for such an in-depth study. It is a field with a profound history, recognized world titles, and established aesthetic principles, creating a perfect testbed for exploring the intersection of computation and creativity. Our focused investigation allowed for a rigorous analysis, culminating in the curated booklet of compositions we provided for expert (GrandMaster) review.\n\nWe agree that building on prior work is standard in an application paper. However, we wish to emphasize the novel adaptations and contributions required to make these methods successful for our task, and we believe that the NeurIPS community can gain from reading our lessons. Specifically, our system is general and most of our contributions can be extended to other domains:\n\n- **Novel Counterintuitiveness Reward and generalizability**: Our primary technical contribution is a novel reward function that models counter-intuitiveness by analyzing the search statistics of an MCTS planning algorithm, effectively quantifying 'surprise.' This technique is directly transferable to other domains where MCTS is applied, such as games (AlphaZero), mathematics (AlphaProof), and computer science. More broadly, we propose this concept generalizes beyond explicit search. By drawing an analogy between 'deep search' and the 'deep thinking' demonstrated by recent reasoning-focused models (e.g., O1, R1), our approach offers a pathway to measure the novelty or counter-intuitiveness of solutions in any domain that advanced language models can address.\n\n- **Diffusion Model & Auto-regressive model.** While the model architectures that we’ve trained are otherwise standard, the engineering setup enables this to be applied to discrete generative problems more broadly, whether it is games or language. In addition, our work provides compelling evidence that for this creative task, masked diffusion models can be superior to their autoregressive counterparts, contributing a valuable data point to the field.\n\n- **Adapting RLHF for Novelty:** A naive implementation of RLHF-PPO proved insufficient, suffering from entropy collapse and failing to produce novel outputs (Section 2.1, with more details given in Appendix H). It is fairly new to point out this issue and only in very recent work [Wang 2025], [Cui 2025], researchers began to investigate the role of entropy and the potential for its collapse during the RL training of LLMs, as well as methods to mitigate it. In addition, our RL ablations detail the non-trivial modifications (specifically, measure novelty using various metrics and to either penalize or skip training on samples that are not novel) required to overcome this, offering generalizable lessons for any researcher seeking to optimize for novelty in conjunction with a reward signal. It is only in the latest LLM reasoning literature that techniques based on conceptually similar principles have started to be discussed. For instance, [Yu 2025] is using curriculum learning to actively filter out problems on which the model has already converged.\n\nWang, Zihan, et al. \"Ragen: Understanding self-evolution in llm agents via multi-turn reinforcement learning.\" arXiv preprint arXiv:2504.20073 (2025).\n\nCui, Ganqu, et al. \"The entropy mechanism of reinforcement learning for reasoning language models.\" arXiv preprint arXiv:2505.22617 (2025).\n\nYu, Qiying, et al. \"Dapo: An open-source llm reinforcement learning system at scale.\" arXiv preprint arXiv:2503.14476 (2025).\n\n\n**Q2: Graph on line 122 need minor fixing. 3.3 and 3.4 might need to be introduced in chronological order in section 3 overview paragraph**\n\nThank you for your advice. We will update the graph and reorganize the order of section 3.3 and 3.4.\n\n**Q3: Despite successfully generating puzzles that are evaluated favorably in both numerical experiment and human evaluation, the RL model still requires significant design effort to ensure good convergence.**\n\nWe argue that such design effort, instead, highlights a core contribution of our work. The challenge of stabilizing RL in a complex, sparse-reward generative domain while keeping generation diversity is precisely the problem we aimed to solve. Applying standard RL is notoriously difficult here. Therefore, our primary contribution is not just the final model, but the robust set of training techniques we developed to make it possible.\n\nAs detailed in our response to Q1 (point 3), the value of this contribution is underscored by the fact that the challenges we overcame—such as entropy collapse—are now emerging as critical research problems in the RL training of Large Language Models. Our work provides a proven framework and generalizable lessons for tackling these very issues."}, {"review_id": "zz5HMRlzV1", "reviewer": "Reviewer_MyRD", "summary": "This paper trains generative models to synthesize chess puzzles. It first analyzes various kinds of generative models using dataset of chess puzzles and then fine-tunes the autoregressive generative model using RL for improved performance. The results show that the final RL-tuned model surpasses the existing puzzle dataset in various metrics (e.g. counter-intuitiveness). Another tangible outcome of the project is a manually curated booklet of 50 AI-generated puzzles.", "questions": "The results in Table 2 are rather surprising. One would expect a well-trained generative model to mirror the statistics of the training data, but the models here significantly underperform the original dataset in all metrics. Is there any reason why the generative models are so far off compared to their training data?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- This project represents a useful, creative and unique application of generative AI tools. To my knowledge, while there have been numerous works which develop chess-playing systems, generative approaches for creating puzzles have not been explored. \n\n- The paper takes a pretty sensible approach — training a generative model and finetuning with RL. The results demonstrate clear benefits from the second stage of RL-based training.\n\n- The final booklet of puzzles could be a valuable outcome on its own as the quality is comparable to puzzles found in typical chess books.\n\nWeaknesses:\n\n- The technical contributions of this work are somewhat limited. Beyond the interesting choice of the setup tackled, the approach is to simply train typical generative models on an existing dataset and finetune the auto-regressive one using RL (mirroring standard practices in training LLMs e.g. for code generation).\n\n- While the setup tackled in this work is unique, the final system that can generate puzzles is somewhat limited in scope. In particular, unlike chess engines which can start from custom positions, the puzzle generation system lacks the flexibility to help coach a player for specific scenarios. For example, this system can only be queried to generate a puzzle, but cannot be instructed to generate one of a specific kind “e.g. generate an opening position puzzle reflecting a defensive scenario”.\n\n- The performance of the final system, while comparable to puzzles in books, is not ‘superhuman’, and in fact requires manual effort in curation etc (for example the booklet required significant manual effort). Moreover, the uniqueness of the puzzles from the RL-based system is limited. As such, it is unclear whether random samples from the learned model are of the desired quality.", "was_revised": true, "final_justification": "I'd like to thank the authors for their detailed response. While the concerns about contribution still somewhat persist, I agree that this is indeed a novel and creative application of AI and would be interesting for the broader community to see. I also think the additional GM study further highlights the value of the system, and would like to update my rating to lean towards acceptance.", "rebuttal": "We thank the reviewer MyRD for your constructive comments and feedback. \n\n## **New human eval result with chess grandmasters (GM)**\n\nBefore the rebuttal, we’d like to add a new human evaluation result with chess GM on our booklet. The following excerpts are their comments, which further affirm the value of our work.\n\n- GM A: **I'm quite particular about what makes a good chess puzzle—I look for natural-looking positions with reasonable play. Yet, I found myself enjoying many of the AI-generated examples in the booklet.**\n- IM for Chess Compositions B: **This booklet adds novel, AI-generated puzzles to the existing chess literature. I found several puzzles that were original, creative, and aesthetically pleasing, with surprising solutions that were satisfying to solve.**\n- GM C: **The puzzles in this booklet represent a pioneering step in human-AI collaboration for chess composition and hint at what might be achieved in the future for composing chess puzzles.**\n\n## **Rebuttal**\n\n**Q1: The technical contributions of this work are somewhat limited... simply train typical generative models on an existing dataset and finetune the auto-regressive one using RL (mirroring standard practices in training LLMs e.g. for code generation).**\n\nWe respectfully position our work as a Creative AI application, a contribution category (https://neurips.cc/Conferences/2025/CallForPapers) explicitly welcomed by the NeurIPS call for papers. We argue that for a topic as sensitive and nuanced as AI-driven creativity, a deep dive into a single, rich domain is not a limitation but a methodological necessity for drawing sharp, meaningful conclusions. As we cite in our introduction, the field has seen instances where shallow evaluations of creative systems yielded misleading results, a pitfall we were determined to avoid. \n\nChess composition represents an ideal domain for such an in-depth study. It is a field with a profound history, recognized world titles, and established aesthetic principles, creating a perfect testbed for exploring the intersection of computation and creativity. Our focused investigation allowed for a rigorous analysis, culminating in the curated booklet of compositions we provided for expert review.\n\nFurthermore, our work makes a crucial contribution to the field demonstrating the ability of AI techniques to compose questions, and not only to solve them. It is often the case that even though solutions to specific problems appear more domain specific, they are important stepping stones to generalize later.\n\nWe agree that building on prior work is standard in an application paper. However, we wish to emphasize the novel adaptations and contributions required to make these methods successful for our task, and we believe that the NeurIPS community can gain from reading our lessons. Specifically, our system is general and most of our contributions can be extended to other domains:\n\n- **Novel Counterintuitiveness Reward and generalizability:** Our primary technical contribution is a novel reward function that models counter-intuitiveness by analyzing the search statistics of a planning algorithm, effectively quantifying 'surprise.' This technique is directly transferable to other domains where search is applied, such as games (AlphaZero), mathematics (AlphaProof), and computer science. More broadly, we propose this concept generalizes beyond explicit search. By drawing an analogy between 'deep search' and the 'deep thinking' demonstrated by recent reasoning-focused models (e.g., O1, R1), our approach offers a pathway to measure the novelty or counter-intuitiveness of solutions in any domain that advanced language models can address.\n\n- **Diffusion Model & Auto-regressive model.** While the model architectures that we’ve trained are otherwise standard, the engineering setup enables this to be applied to discrete generative problems more broadly. In addition, our work provides compelling evidence that for this creative task, masked diffusion models can be superior to their AR counterparts, contributing a valuable data point to the field.\n\n- **Adapting RLHF for Novelty:** A naive implementation of RLHF-PPO proved insufficient, suffering from entropy collapse and failing to produce novel outputs (Section 2.1, with more details given in Appendix H). It is fairly new to point out this issue and only in very recent work [Wang 2025], [Cui 2025], researchers began to investigate the role of entropy and the potential for its collapse during the RL training of LLMs, as well as methods to mitigate it. In addition, our RL ablations detail the non-trivial modifications (specifically, measure novelty using various metrics and to either penalize or skip training on samples that are not novel) required to overcome this, offering generalizable lessons for any researcher seeking to optimize for novelty in conjunction with a reward signal. It is only in the latest LLM reasoning literature that techniques based on conceptually similar principles have started to be discussed. For instance, [Yu 2025] is using curriculum learning to actively filter out problems on which the model has already converged.\n\nWang, Zihan, et al. \"Ragen: Understanding self-evolution in llm agents via multi-turn reinforcement learning.\" preprint arXiv:2504.20073 (2025).\n\nCui, Ganqu, et al. \"The entropy mechanism of reinforcement learning for reasoning language models.\" preprint arXiv:2505.22617 (2025).\n\nYu, Qiying, et al. \"Dapo: An open-source llm reinforcement learning system at scale.\" preprint arXiv:2503.14476 (2025).\n\n\n**Q2: While the setup tackled in this work is unique, the final system that can generate puzzles is somewhat limited in scope. In particular, unlike chess engines which can start from custom positions, the puzzle generation system lacks the flexibility to help coach a player for specific scenarios......”.**\n\nThank you for your feedback on potential future directions, such as conditional generation. It is greatly appreciated and aligns with our own plans for future work.\n\nWhile our current system doesn't yet have a dedicated “coaching mode”, it can produce counter-intuitive chess puzzles across a range of aesthetic themes. These themes can be used as further filtering mechanisms to identify puzzles specifically relevant for coaching a player. For example, we could filter for positions from a specific opening or with a particular aesthetic theme to create a customized puzzle set.\n\n**Q3: The performance of the final system, while comparable to puzzles in books, is not ‘superhuman’, and in fact requires manual effort in curation etc (for example the booklet required significant manual effort). Moreover, the uniqueness of the puzzles from the RL-based system is limited. As such, it is unclear of the desired quality.**\n\n- **Curation.** We need to clarify that humans also create puzzle books by curating positions. And our system has already saved a lot of manual efforts (we only have one chess expert curating the candidates).\n\n- **Quality.** Note that the uniqueness of our RL system is not low (the reason Lichess puzzle has such a high uniqueness ratio in Fig 4.c is because it is getting explicitly filtered for uniqueness in advance while our RL is not, mentioned in line 316). For the final quality, we would like to point out the feedback from the three chess experts who extensively reviewed our chess booklet. They agreed that some of the puzzles were creative, aesthetically pleasing, beautiful and counter-intuitive.\n\n- **Not superhuman.** As we mentioned in the final discussion section, unlike verifiable domains such as chess playing, human definition of counter-intuitiveness is subjective. Even for the same position, players may have varying opinions, even if they are of comparable playing strength, including grandmasters. Our estimate through chess engines and the amount of human data in our evaluation is therefore certainly approximate, and a proxy for a more general preference distribution. Such evaluations are common for example in the RLHF domain, where there would usually not be any claims of superhuman.\n\n**Q4: The results in Table 2 are rather surprising. One would expect a well-trained generative model to mirror the statistics of the training data, but the models here significantly underperform the original dataset in all metrics.....?**\n\nThis is an excellent observation. The gap you've noted is not due to a poorly trained model, but rather a direct consequence of our model selection strategy, which deliberately prioritizes generalization over memorization.\n\nHere's the breakdown:\n- **Our Strategy Avoids Overfitting:** We strictly selected the model checkpoint with the lowest test loss. This standard practice ensures the model learns generalizable patterns from the data, rather than simply memorizing the training set.\n- **Overfitting Would Mirror the Training Data:** Crucially, we observed that if we continued training beyond this optimal point, the model does begin to overfit. As shown in Appendix Figure 7, this overfitting causes the puzzle score to rise. However, this comes at a steep price: the model's test loss increases, and its generation diversity significantly decreases. This also correlates with recent work on LLM RL research [Wu 2025].\n- **The Brittle Nature of Puzzles:** This effect is amplified because puzzles are \"brittle\"—a single incorrect piece can invalidate the entire puzzle. A model that generalizes will inevitably produce small, nuanced variations that may not pass the strict criteria of a puzzle.\n\nIn essence, we made a conscious trade-off. We could have achieved higher puzzle scores by overfitting, but we chose to present a model that generalizes better and is more diverse, which we believe is a more meaningful benchmark.\n\nWu, Mingqi, et al. \"Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination.\" preprint arXiv:2507.10532 (2025)."}, {"review_id": "qXJafQkGkw", "reviewer": "Reviewer_EyYA", "summary": "This paper introduces a methodology for the generation of creative chess puzzles, combining the definition of quality metrics and an RL pipeline. There are extensive method descriptions and I like the human evaluation study.\n\nBut overall I find this paper has several problems:\n* It does not clearly specify what the contributions are and does not relate the approach to existing work (no related work section in the main paper)\n* It uses a lot of jargon, and stays vague or imprecise on several key technical details. \n* It is very applied to chess puzzle generation, and does not seem to introduce any technical novelty, except perhaps the definition of the counter-intuitiveness measure, which is again very domain-specific.", "questions": "see above", "limitations": "see above", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The paper proposes to solve chess puzzle but these are not really defined anywhere in the paper. Definitions themselves use a lot of technical jargon: eg ”excluding puzzles with multiple, solutions, non-standard elements, special stipulations, retrogrades, missing information, shortest mates, or mathematical aspects.” Do all chess puzzle have the same objective (check mate)? \n\nMetrics definitions\n\nThe section dedicated to give definitions for the different metrics does not completely achieve its aim.\n* Uniqueness: this definition is rather unclear, are we talking about the uniqueness of solutions or the uniqueness of moves? Is a unique solution a solution where each move is unique? And does the proposed criterion (Eq 1) really translates in qualifying uniqueness of a solution? How does the Stockfish engine work, and can we rely on it? If the second best move has a chance of leading to win >0, then it means there might be another solution right there?\n* Creativity: this paragraph doesn’t offer any precise metric, just some discussion about the difficulty of evaluating creativity.\n* Novelty: \n  * What are FEN strings? Why is it a good choice? What is the principal variation measure? \n  * about the entropy of the board representation: what is the model used? how is it trained?\n  * I’m not sure the Levenshtein distances capture any kind of semantic distances between board positions — pieces could be differently placed but the abstract configurations may be similar, and inversely they could be placed very similarly but the abstract configurations might be very different\n* Counter-intuitiveness: This is interesting, but here again a bit vague: this paragraph starts to define it, but stops before giving a final definition\n* Aesthetic: here again a lot of discussion but no precise definition\n\nGiven how central these terms are, it would perhaps be good to have a concise, descriptive descriptions of these features right from the introduction. This is done to some extent but using very vague terms: eg “Counter-intuitive solutions are those that, at first glance, seem are terrible or out-of-mind, but are, in fact, brilliantly effective.” I think having a more technical, practical definition would help here. The abstract also appears to use vague terms and technical jargon.\n\nOptimization\n* Wouldn’t the use of a reverse KL divergence loss limit the creativity of the policy by forcing it to only generate puzzle within the distribution of the base model?\n* I’m not sure I understand, is the policy pretrained with the lichens dataset with supervised finetuning, or are these 100k examples only added to the buffer at the beginning of the PPO phase? Does PPO even have a replay buffer? It’s an online policy, so not sure this would work so well to add completely offline data? This doesn’t seem to significantly help either (Fig 4), so not sure why it is introduced. \n* Is there any reason why we should apply RL here instead of doing supervised fine-tuning on positive examples: diverse set of unique and counter-intuitive solutions?\n* It would be interesting to see the ratio of the number of solutions accepted to the replay buffer as a function of time? If diversity crashes we’d have fewer and fewer novel-enough solutions to enter the replay buffer\n\nCounter-intuitiveness measure:\n* This seems like the main contribution of the paper, and it takes only a limited part of the paper and is not very clearly laid out.\n* The fact that it captures well human-judgement on counter-intuitiveness is good but does not prevent the measure from being hacked by the policy during RL training. I would like to see an analysis of highly counter-intuitive models and some evidence that the measure is not being completely hacked by the policy in some strange way — eg by adding elements that would confuse the short-budget search (eg more pieces = higher branching factor = slower search) more than the high-budget search.\n\nNovelty:\n* “Inspecting Table 3 indicates that the datasets sampled from the generative models exhibit both novelty (when compared to the Lichess data) and internal diversity (when compared to other samples from the same model)” —> I find it hard to interpret these measures in an absolute sense, what’s the threshold that allows us to say these are definitely “novel” vs “not novel” here? Using the Lichess dataset as a baseline needs to be a bit justified, how do we know this dataset is diverse? If it’s a collection of boards people have played, maybe people don’t play such diverse games? The examples in the booklet help with that, thanks for including these. \n\nAesthetic:\n* From Fig 3 it’s hard to say if anything special is going on. There is some aesthetic distribution in the training data, and some in the fine-tuned model and some in the RL-trained model. There is not obviously more in one or the other, it seems more or less random. Of course, it wasn’t optimized for. Is there a reason why not?\n\nConclusion:\n* It's not clear what the contribution and novelty of this paper are compared to previous work\n* The main contribution seems to be the definition of evaluation metrics / reward functions, which have a rather limited scope --- maybe this work belongs to a more specialized venue?\n* The text is not always clear, it's dense and uses a lot of jargon, while missing key details. A lot is sent back to a very long appendix.", "was_revised": true, "final_justification": "n/a (just edited for formatting for now)", "rebuttal": "We thank reviewer EyYA for your helpful feedback. \n\n## **New human eval result with chess grandmasters (GM)**\nWe refer the reviewer to other rebuttals for new evaluation results from GM, which further affirms the value of our work. \n\n## **Rebuttal**\n**Q1: The section dedicated to give definitions for the different metrics does not completely achieve its aim.**\n\nWe thank the reviewer for this feedback, which allows us to clarify our metric definitions. We will incorporate these clarifications into the main paper to enhance readability.\n\n- **Uniqueness**: We recursively apply the check to principle variation (line 78) – a solution is defined as unique if every move within its principal variation is the unique move. As formalized in Eq. 1, a move is considered \"unique\" only if it is substantially better than the second-best alternative. We provide Appendix C for a much detailed description of the check.\n\n- **StockFish**: StockFish is the leading chess engine in the world, widely accepted by experts as a superhuman \"ground truth\" for chess analysis.\n\n- **Creativity**: In general, when dealing with creativity there is not a single go to definition. Yet, there is a consensus in the chess community that we followed [2, 40] and used clear definitions throughout the paper. We choose the following three metrics as a more concrete representation (line 91, 115, 129, 130)\n\n- **Novelty**: FEN represents Forsyth-Edwards Notation (a commonly used notation to represent a board, line 141). And we indeed have a semantic measurement with model-based entropy (line 105-109). This is mainly used during the RL process using the AR model itself to filter low-diversity samples (line 197-199). In Appendix H.2 we demonstrate its necessity, showing how board-distance only is susceptible to reward hacking and why entropy-based filtering is essential (Figs 9 & 10).\n\n- **Aesthetics**: Our aesthetic metrics are based on established heuristics drawn from respected chess literature [2, 40]. Detailed definitions for each aesthetic theme, such as sacrifices, are provided in Appendix I.\n\n- **Counter-intuitiveness**: The intuitive explanation in our introduction is formalized by the definitions in Equations 2-5 (Line 122). These equations quantify counter-intuitiveness by comparing shallow vs. deep engine search evaluations. Crucially, this reward is then fine-tuned using our expert human preference data (Sec 3.1).\n\n**Q2: Optimization details.**\n\n- **KL Divergence and Creativity**: The reviewer raises an important point about the trade-off between creativity and realism. Generally, from our human evaluator and GMs’ feedbacks, players always favors creativity only if it is realistic. Thus, our KL penalty is a conscious and critical choice for realism (Lines 172-186). Without this constraint, the model is prone to generating 'out-of-distribution' (OOD) positions that, while potentially novel, are often unrealistic or nonsensical to human players. We detail more in Appendix H.3 and show OOD samples in Figure 8.\n\n- **PPO, Replay Buffer, and Offline Data**: We utilize a critic-free PPO variant with Monte-Carlo value estimate. We further tuned the pretrained model in the RL process (line 149), and show Zero-RL (RL with no pretraining) does not work (Figure 15). The replay buffer is mainly for enabling diversity filtering (Lines 181-183), a crucial mechanism to prevent entropy collapse. Regarding the 100k high-quality Lichess examples, their purpose is to anchor the model and mitigate KL divergence by seeding the training process with realistic puzzles (Figure 13(b)). There is a trade off here that it may bring in less puzzle score but more human-like positions. Indeed this data is off-policy, but you can treat it as auxiliary SFT co-training.\n\n- **RL vs. Supervised Fine-tuning on positive samples**:  we choose RL to further leverage negative samples. First, negative samples also contribute to policy learning [Zhu 2025]. Second, by treating non-novel puzzles as negative samples, we are explicitly preventing the model from entropy collapse or generating similar puzzles (Appendix H). SFT, which only learns from positive data, cannot directly address this.\n\nZhu, Xinyu, et al. \"The surprising effectiveness of negative reinforcement in LLM reasoning.\" arXiv preprint arXiv:2506.01347 (2025).\n\n- **Ratio of Accepted Solutions over Time**: We thank the reviewer for this excellent suggestion. We did, in fact, analyze this phenomenon in two experiments. Figure 11 demonstrates how entropy correlates with the sample pass ratio and the ratio keeps decreasing with more training steps. Figure 14.b demonstrates that without continuous training updates to the policy ('W/O training'), the number of novel samples accepted into the replay buffer rapidly diminishes. This confirms the reviewer's intuition and underscores the necessity of continuous adaptation to prevent the model from exhausting the novelty.\n\n**Q3: Counter-intuitiveness measure and reward hacking:**\n\nThe potential for reward hacking is a critical concern in any RL framework, and we commend the reviewer for raising this point. We anticipated this issue and incorporated several mechanisms to mitigate it.\n\n- **Explicit Acknowledgment**: Our \"Discussion and limitation\" (Lines 368-383), explicitly acknowledges this tendency, including the example of \"exploiting chess engine weaknesses (making the position artificially more complicated for engines...).\" To counter this, we introduced realism constraints, including the KL divergence penalty and a piece count penalty (Appendix H.3, Figure 13), which prevents the model from adding excessive material to confuse the engine.\n\n- **Human-in-the-Loop Validation**: The final arbiter of quality is human evaluation. Our human expert study (Section 3.5, Table 4) confirms that the generated puzzles are not only rated as highly counter-intuitive but also as creative and fun, surpassing even human-composed book puzzles in these aspects. This provides strong evidence that our measure successfully captures human-perceived quality rather than being exploited in a meaningless way.\n\n**Q4: Clarification on Novelty.**\n\nWe thank the reviewer for this insightful question, which allows us to further justify our choice of baseline and the interpretation of our novelty metrics. We agree that interpreting novelty metrics in an absolute sense is challenging. This is why our evaluation is twofold:\n\n- **Quantitative Interpretation and Baseline Justification**: Lichess puzzle dataset is the largest publicly available dataset of its kind, with its puzzle collection derived from an immense pool of over 600 million human games. Given this massive scale, the internal diversity of the Lichess dataset serves as a reasonable human standard of diversity. Quantitatively, a board distance value around 10 (at least 10 piece edits, Line 282) is considered novel.\n\n- **Qualitative Validation**: To move beyond abstract numbers and directly address the reviewer's point, our curated booklet presents each generated puzzle alongside its three closest Lichess matches. This allows for direct visual assessment of its distinctiveness. Expert feedback reinforces this; GM A noted one puzzle as \"an elegant combination of themes... not something I've seen before.\"\n\n**Q5: Aesthetic clarification.**\n\nThis was a deliberate choice. Our primary goal was to first establish a framework for generating puzzles that are sound (unique) and interesting (counter-intuitive). We deliberately chose not to hard-code human definitions of aesthetics, but instead to let the model discover on its own. The key finding (Fig 3) is that a diverse range of aesthetic themes emerged naturally from a process that did not explicitly optimize for them. This suggests our core metrics for counter-intuitiveness and realism are well-correlated with human preferred properties.\n\n**Q6: It's not clear what the contribution and novelty of this paper are compared to previous work**\nTo clarify, our key contributions are:\n\n- A formal framework to quantify and optimize creative qualities in chess, refined with expert human data. (section 1, section 3,1).\n- Engineering design for puzzle generator (Section 2, section 3.2, 3.3)\n- Diversity-filtering RL techniques. We present key designs and the rationale of our RL framework, and identify and tackle the entropy collapse issue with diversity filtering, which is crucial for stable RL training. (Section 2.1, 3.4). We put the full exploration journey in Appendix H.\n- The final booklet and human study, which validates our methods and showcases the potential human-AI collaboration  – AI provides coarse generation and selection which largely decreases the human efforts for book-level samples.\n\nWe refer you to our response to Q1 for Reviewer 2cAd to clarify novelty and generalizablity. Sorry that we can't include it due to space limits.\n\nOur work significantly extends prior art like Chesthetica [30-33], which focused solely on aesthetics over a decade ago, by tackling a multi-objective problem with a modern deep generative RL approach.\n\n**Q7: The main contribution seems to be the definition of evaluation metrics / reward functions, which have a rather limited scope --- maybe this work belongs to a more specialized venue...**\n\nWe respectfully position our work as a Creative AI application, a contribution category explicitly welcomed by the NeurIPS call for papers. We argue that for a nuanced topic like AI creativity, a deep dive into a single, rich domain like chess is not a limitation but a methodological necessity. It allows for rigorous analysis and avoids the pitfalls of shallow evaluation seen in prior work. Chess, with its profound history and established aesthetic principles, provides the ideal testbed for drawing sharp, meaningful conclusions about the intersection of computation and creativity.\n\n**Q8: Formatting issues.**\n\nWe will correct all formatting issues, including captions, external links, and figure readability."}]

Author rebuttal:

Meta-review: This paper proposes an RL framework for generating creative chess puzzles, introducing reward functions for novelty, counter-intuitiveness, and aesthetics, with strong quantitative and human expert validation.  Its strengths are originality, careful reward design, solid evaluation, and a valuable curated outcome.  Weaknesses lie in limited generalizability, modest technical novelty, and clarity issues.  Despite these, the work represents a distinctive and well-executed application of creative AI, justifying acceptance.  In discussion, reviewers’ concerns about scope and presentation were partly addressed, with clarifications on methodology and claims of broader applicability; most reviewers maintained or improved scores, leading to consensus that the contribution, while niche, is impactful enough for the conference.
Decision: Accept (poster)