A Reasoning-Based Approach to Cryptic Crossword Clue Solving
Authors: Martin Andrews
Sam Witteveen
Venue: ICML 2025 poster (2025)
Keywords: NLP
Cryptic Crosswords
Reasoning
Proof/Verification
Primary area: applications->language_speech_and_dialog
TL;DR: Our LLM prover/verifier for the reasoning steps in cryptic crossword clues obtains SoTA results

Abstract: Cryptic crossword clues are challenging language tasks for which new test sets are released daily by major newspapers on a global basis. Each cryptic clue contains both the definition of the answer to be placed in the crossword grid (in common with regular crosswords), and ‘wordplay’ that *proves* that the answer is correct (i.e. a human solver can be confident that an answer is correct without needing crossing words as confirmation). This work describes an LLM-based reasoning system built from open-licensed components that solves cryptic clues by (i) hypothesising answers; (ii) proposing wordplay explanations; and (iii) using a verifier system that operates on codified reasoning steps. Overall, this system establishes a new state-of-the-art performance on the challenging Cryptonite dataset of clues from The Times and The Telegraph newspapers in the UK. Because each proved solution is expressed in Python, interpretable wordplay reasoning for proven answers is available for inspection.

A Reasoning-Basse Approach to Cryptic Crossword Clue Solving
Martin Andrews
Sam Witteveen
Abstract
Cryptic
Clee
Cut over politictan or the French case (7)
EXAPLLE
#5: Zown
Cryptic crossword clues are challenging language tasks for which new test sets are released daily by major newspapers on global basis. Each cryptic clue contains both the definition of the answer be placed in the crossword grid (in common with regular crosswords), and "wordplay" that proves that the answer is correct (i.e. a human solver can be confident that an answer is correct without needing crossing words as confirmation). This work describes an LLM-based reasoning system built from open-licensed components that solves cryptic clues by (i) hypothesising answers; (ii) proposing wordplay explanations; and (iii) using verifier system that operates on codified reason- ing steps. Overall, this system establishes a new state-of-the-ar aatt performance on the challenging Cryptonite dataset of clues from The Times and The Telegraph newspapers in the UK. Because each proved solution is expressed in Python, inter- pretable wordplay reasoning for proven answers is available for inspection.
For ONT antver LATOSIAT
antuor candidate
definteion woroplay muggattions (Graas)--6 FT!
genezater
18 suggestions i1 shows herel:
candiderec::
defini Out up over
(the, (case)
Socop STTAN Sround
attwer: 211PPCD
ansicc EXAMPLE
LUSGAGE
fooralssation to Pythion (Comord S-Flash Somman 91-11)
ssusetonep) DISTIAL",
Retty forwoliaattnn us
ossest AM0"(111-1 "ERA"
acsect cs, s__ecbevviatinn "solttscten o9*) sa_syyooyyl tte, French", (LE")
tD SOL.
Python varification Cuses and strulates Functions)
Error twith
assection fallune "sp" does not
"INITEAL", coule be REVERSE
Figure I. Proving process: answer candidate + wordplay + LLM formalisation
There are decades of solved puzzles (each one containing over 20 clues) from multiple major newspapers available, and new puzzles are published daily. This contrasts with (for instance) IMO/AIME problems, where there a much lower number of novel problems available.
1. Introduction
The method this work explicitly reveals the reasoning (i.e. validated wordplay) required to solve each prob- lem. By construction, there one 'true' reasoning path, although it might be expressed in different ways by dif- ferent solvers.
There has been significant work in reasoning in the fields of mathematics (Jiang et al., 2023; Yang et al., 2023; Trinh al., 2024) and code generation (Ni al., 2023; Ridnik et al., 2024) which benefit from having strong verifiers to validate their answers. This work tackles the relatively under-studied reasoning task of cryptic crossword solving, which has the following qualities:
1.1. An example Cryptic Clue
Thousands of people find Cryptic Crosswords satisfy- ing intellectual challenge on a daily basis. Solving these puzzles requires understanding multi-layered language constructs, blending logic, wordplay, and contextual nu- ance. This provides unique challenge for evaluating and improving LLMs' capabilities in NLU and reasoning.
As a concrete example, and to clarify the terminology used, consider the following moderately complex cryptic clue "Cut up over politician on the French case (7)".
Solvers must parse the clue carefully to separate the de f miition on (which acts a regular crossword clue) and the supporting wordplay that can be used to arrive at the same answer from two directions. Arriving at the same answer by two paths constitutes the necessary proof that the correct answer has been found. See Figure 2a for a visual depiction of the reasoning involved.
Red Dragon AI, Singapore, Correspondence Martin An- drews <martin @ reddragon.ai>.
Proceedings of the nnnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
'The Times UK (6-March-2024), Cryptic #28857 clue 4D

---

A Reasoning- Approach to Cryptic Crossword Clue Solving
In this work, we take our cue from the effectiveness of provers coupled with verifiers for mathematical reasoning tasks (Jiang et al., 2023). We tackle the cryptic crossword clue solving task using an LLM to (i) suggest answer candidates; (ii) create informal proofs (i.e. coming up with wordplay): and (ii) perform a formalisation pro- cess (which rewrites the wordplay ay logic in Python). The proposed solutions (expressed as executable Python) are then checked for validity.
2.2. Cryptic Crosswords
In an 800 participant research study into the backgrounds of cryptic crossword solvers (Friedlander & Fine, 2016), the following observation was made about the skills required to solve these linguistic/reasoning puzzles:
"... ...crtttcc crossword skill therefore appears to be bound up with code-cracking and problem- solving skills of quasi-algebraic nature. Con- versely, lexical ability, although no doubt valuable, does not appear to be a critical discriminator of high expertise among elite solvers."
1.2. Contributions
The following are the main contributions of this work:
An open-license system for reasoning over Cryptic clues our pipeline (illustrated in Figure 1) enables 9B- scale local models to achieve mate-of-theealt results on the Cryptonite dataset.
Cryptic crosswords have received surprisingly little atten- tion from the machine learning community, despite being a notable language-oriented reasoning puzzle with global appeal. One possible reason that cryptic crosswords are much less common in the United States than 'regular cross- words'. See Anthony & Goodliffe (2024) and Webb (2024) for inspiring demonstrations of experts solving cryptic cross- words in real-time.
Local models for cryptic clue tasks We show how local LLMs can be fine-tuned to produce answer candi- dates, and wo rdplay suggestions, and then prompted to perform Wordplay formalisation. Following an approach akin to mathematical statement formalisation, but where there are less than 10 examples of "good proofs' available, our novel pipeline was specifically engineered to avoid 'reasoning steps' becoming stuck in dead ends.
The benchmark dataset used by this work is Cryptonite (Efrat et al., 2021) a large-scale dataset of Cryptic Cross- word clues from The Times and The Telegraph (major UK newspapers). The dataset contains 523,000 naturally sourced clues (published between 2001 and 2020), with the train, validation and testing splits being chosen so that a given answer can only appear in one of the splits.
Python domain-specifie verifier - Using the output of the formaliser, the verifier presented here deconstructs the Python AST, so that it can evaluate each assert statement on a line-by-line basis. We believe that this is somewhat novel, since it enables the verifier to not only indicate whether the proof is valid overall, but also point to specific failures (used to regenerate failed formalisa- tions) on all proof lines simultaneously.
While the dataset made available in Rozner et al. (2021) is also of interest, its clues are limited to those from the Guardian newspaper, and Connor (2024) notes in the Guardian's 's own blog "The Times hosts an annual crossword- solving competition and it remains, until such time as the Guardian has its own version, the gold standard." Moreover, the smaller number (142,000) of clues that the dataset con- tains have no orientation markings ("across/down") )) which are required to make sense of some wordplay.
To promote further study in this area, all code for training the models, the formaliser and domain-specine verifier is made publicly available.
2. Related Work
For more in-depth discussion of the decision to focus the Cryptonite dataset (and not perform testing on the Guardian dataset), please see Appendix A.3. In summary, while the 'Init' split presented in Rozner et al. (2021) has at- tractive properties (explored there, and in other works), this work specifically targets the reasoning side of cryptic clues, which involves fine-tuning models on Cryptonite (including Wordplay examples with carefully matched train/val/test splits). This precludes from doing the same kind of multi-dataset comparisons found elsewhere.
2.1. Regular Crosswords
Non-cryptic ("regular") crosswords are known throughout the world. and are the predominant type found in newspapers in the U.S.A. One key difference from cryptic crosswords is that individual regular crossword clues are generally not "standalone" there may be number of different answers that fit the given clue. The key solving regular crosswords is thus the interaction between answers (i.e. the crossing- words), which allows for planning/backtracking to enable solving rates in the high 90% range (Wallace et al., 2022).
1.2.1. RULEE-BSED SOLVERS
This work, in contrast, focuses on the solving of clues on a standalone basis, which requires elements of reasoning through the wordplay present in cryptic clues.
Williams & Woodhead (1979) is an early example of at- tempting to devise a formal language for describing cryptic clues. However, the linguistic elements of the clues tend to
2

---

A Reasoning-Bass Baann Approach to Cryptic Crossword Clue Solving
Cut up over politician on the French
case (7)
(b) Offer of support also broadcast (8)
position
position
synorym
EXA
synonym
synonym
syonym
synorym
synorym
anagram
reverse
MP
aXe
LE
EXAMPLE
PROPOSAL
PROP + OSAL
(c) Research done, primarily on most of magical beings (5)
(d)
Wader woman
has on 5
synonym
take first letter
take most letters of
synoyym
synonym
synnnym
literal
HERON HER + ON
elves
DELVE
=
ELVE
Figure 2. Clue solving illustrations. Answers are in green, definitions blue (dashed frame), wordplays in orange, and indicators in purple. Further textual examples can be found in Appendix A.1
thwart a strictly formal approach.
also touch on their illuminating Partial Correctness Metries (which are relevant when attempting full grids, which is not the main focus here).
A more flexible rule-based solver with manually-crafted probabilistic grammar was introduced in Deits (2015; 2022). Building on the assumption that a clue can usually be split into wordplas ay and definit mitiii tion, the solver tries to find the most probable parse such that the wordplay yields a semantically-sinne simillr result to the definiti ion. The logi- cal form of this DSL approach very appealing. However, it appears limited to solving clues where the wordplay ay is somewhat simple (due to the combinatorial explosion of possibilities created by longer/more complex clues).
In this work, building on the groundwork of Andrews & Witteveen (2024) and Andrews & Witteveen (2025), we use a pipeline of 9B-scale LMs to produce answer can- didates and wordplay suggestions, followed by third LM to formalise each proposed solution using Python code and then rewrite/update the solutions based on feedback from purpose-built verifier. In our results, focus the 'pure' Cryptonite benchmark: Accuracy is judged based on Top-1 basis (with the model's 'ss single answer being marked wholly correct or not), with no crossing letters being given. Framed as a reasoning task, if the model 'understands' the cryptic solution properly, the answer will be wholly cor- rect there should be partial marks.
The goal of this work is to use the flexibility of LLMs to enable a far wider range of clues to be attempted, with the aid of formaliser/verifier to check the solutions.
2.2.2. LLM- BASED SOLVERS
Cryptonite is challenging task for LLMs Efrat et al. (2021) reported that fine-tuning Lage (a 770M encoder- decoder model) on Cryptonite's 's 470k cryptic clue training set achieved only 7.6% test set accuracy, slightly below the 8.6% accuracy of the rule-based clue solver of Deits (2022). Interestingly, prior to 2024, even large-scale Language Mod- els scored very poorly on cryptic clues, likely due to (i) the misleading surface reading of the clues; (ii) the obliqueness of the definitions; and (iii) the reasoning steps required to prove the answer correct based on the wordplay.
2.3. Code & reasoning
To compensate for LLM approximate generation of logical reasoning, techniques like PAL. (Gao et al., 2023) exploit LLMs' facility for writing code to create verifiable reasoning chains. An important influence on this work was also the Draft, Sketch, and Prove framework (Jiang et al., 2023) which uses an LLM to draft and create proof' that are then verified formally.
As with the prior LM autoformalisation work Ye et al. (2023), chose to Python as the intermediate language into which the natural language statement was formalised. In our case, rather than using an external prover, our system formalises its proofs directly in Python, using callable func- tions, such as is assanngrram () This light-DSL. approach was essential for our NLP task, since we found (for instance) that LLMs have trouble recognising whether two sequences
Recent works, such as Sadallah et al. (2024) and Saha et al. (2024), tackle cryptic crosswords with more up-to-date local models and commercial LLMs. Saha al. (2024) reports results with 5- and 10-Shot prompting (without fine-tuning the models), but also includes a wide-ranging study of the capabilities of models for crosswords in general. We include experiments that bring the relevant baselines up-to-date, and
3

---

A Reasoning- Baacd Approach to Cryptic Crossword Clue Solving
"publ icationt::" "FT", "setter": "falcen", "author": "teacow", "num": 16, "ad": "D", "pattern": "8", "clue": "[Offer) of support also broadcast", "wordplay": "PROP (support) + (ALSO) (+broadcast)", "answer"; "PROPOSAL"
are most confident about; (c) 'having a hunch' of the final answer; and (d) gaining a full understanding of how clue's s wordplay works (such that the function of every element can be explained) as proof of the overall process.
Concretely, the system starts with the clue, and generates 20 answer candidates. For each unique candidate, the next step is to generate 10 guesses at wordplay to justify the answer. Then, each of these wordplay hypotheses are 'formalised' as Python program in a particular form, which can then be verified by executing it (with several retries in case of failure). Successful executions are taken as proof that the original answer was correct.
Figure 3. An example from the Wordplay dataset (in this wordplay, () * an anagram indicator). This clue's solution is diagrammed in Figure 2b
of letters are anagrams of each other.
Observations of the behaviour of GPT-4 using Chain-of- Thought prompts (Wei et al., 2022) suggest that even very ca- pable models tend fixate early on during the reasoning pro- cess, and are only rarely able of completely re- re-hypothesising. These LLMs also frequently becomes caught up with the literal 'surface' meaning of the clue, which is often mis- leading (deliberately on the part of the setter). Organising our system's pipeline to hypothesise candidate answers as the first step (so that the models must try to fit the reason- ing to the answer, with varying degrees of success) bakes re- re-hypothesisation into the process.
In contrast with the tool-integrated reasoning framework Gou et al. (2024), where an LLM for mathematical problem- solving was fine-tuned on 16,000 examples of formalisation, we found that our light-DSL was able to be used by LLMs based on its in-context description alone. For full prompting details, please see Appendix A.5.4.
Informed by the evolution from AlphaCode (Li et al., 2022), in which huge numbers of programs are generated and fil- tered in order to generate a valid solution, to AlphaCodium (Ridnik al., 2024), in which solutions are iterated upon and involving much less computation, this work uses a ver- ifier that can feed back 'hints' to the formalising LLM, so that the task of re-writing nearly-valid proofs is made easier.
3.1. Candidate answer generation
Our first step to solving a given cryptic clue is to gener- ate multiple answer candidates from the original clue, pattern and ad (across/down) fields. For this task, we fine-tuned a Gemma2 9B base model (Gemma Team & Google DeepMind, 2024) using the LoRA (Hu et al., 2022) implementation provided by the unsloth package (un- sloth.ai, 2024). The model was trained for 1 epoch on the Cryptonite training set of approximately 470,000 examples.
2.4. Wordplay dataset
The Wordplay dataset (Andrews, 2024)- an example from which is given in Figure 3- consists of data gathered from websites where cryptic crossword enthusiasts post solutions on daily basis for each of the major publications. Each completed puzzle is annotated by an individual, identifi- able author/solver that lists the approximately 20 clues with their de offfinitinn wordplay and answer fields. Note that each solver can chose their own 'standard' for writing out the wordplay, leading to a significant variation in wordplay annotation styles (even across time for an indi- vidual solver). The Wordplay dataset deliberately follows the train, validation, and test splits defined by Cryptonite.
For each clue being evaluated, we generate 20 valid answer candidates, where candidates that did not match the pattern were immediately rejected and regenerated, and those not contained in the crossword words list (Beres- ford, 2000) were filtered out?, The number of candidates was chosen to balance generation cost with likelihood of the correct answer appearing in the candidate list see Figure 7 for cumulative frequency analysis. The list of candidates was then grouped so that the frequency of each answer could be found enabling statistics to be collected.
3. Methods
The overall system described in the work is illustrated in Figure 1, and the code2 is available under an Apache 2 license. The order of operations for the pipeline was cho- sen based on watching human solvers- who report going through the following steps: (a) attempting to parse the clue in number of ways, trying isolate the definition from the wordplay: (b) seeing which parts of the wordplay they
3.2. Generation of definitiion and wordplay suggestions
To train the wordplay suggestion model, which trans- lates each answer candidate into multiple definition
'This rejection of invalid words here is not 'cheating' since we do not use the dictionary to suggest words, rather is only used weed out actively proposed non-words from a short-list.
httpps: /ttpss:///iitiii http://gggthtuhh.omot/aaa hwtp://///ddddd///ccc/c// crossword reao fier
4

---

A Reasoning-Basse Basen Approach to Cryptic Crossword Clue Solving
Gemma2 answer candidate
Ac: Boun-Bun ( PAction',
def prode (andwer- "DELVE",
"AMAGRAH, REMOVE_ FIRST, ALST LASTVE 1GOR6_ INSIDE, 9CBS DUTSSIE, REVERSE, SUBST2ING, HOMEPHONE")
zeseazch dose, prinacily on nost of naçical beling", pattern-15")
External deciaitiona
dar 4o_oynoayy
t99t__ynooyn::sE, pattera: +> boc1:
Gemma2 wordplay
Trum ic aymon... is reasonable synonyn for "phasse",
letters estionally matching 'pattern'
definition: zeseazch done, prineelly, on most of magical beings vordslays Diona] (prinarily) ELNE(o] (nacical beines, mest oi)
dac is_sbbceviation htt.asssstsaaastttte.e..eee ser__abreviattinnsstt
bool:
Delmeniows whellas "Leat
is
valid abbroviation or short Corm for "phrase"
act hop (phresesstts act setionsActioon -2 bool:
formalsaatio
Detwrnines wheller 'pliass" night signify the given "action"
oot ("prinssily", Actionn INSTEALA)) assert "noux" (11) *p*
dar io_anacran (1ettersaa word:ate) -> beol;
Trum if 'word! own be formed From "letters" 6... anagran)
assert is_eynongm (*sagiccl ("ragica beings", "EINES")
hompehhe aphrasseaat,, (eil_hunppooeesssk)) beol:
essert ootion type I"nost ot", RRRMMO TASTI assert "BLrES"(i-1) "EIVE"
Deternines whether "ceat_honophons sounda like "phsaze"
assert "D"-*EINE" "DRIVE"
assert ("researr"" "DELVE", 1at
Figure 5. External functions available via In-Context Learning
proof
limiting factor, we subsequently tested an unmodified 2-mmm22-9B-1 model on the same task.
Figure 4. Python proving: answer candidate wordplay LLM formalisation
In terms of the DSL itself, the back-end to the is.synonym and i hemoooo functions consists of calls to sim- ple language models. The action type function per- forms nearest-neighhour match against list of indicator words, and the is abbrrvviaton function performs look-up against list of abbreviations both sourced from Deits (2022). For string manipulation actions (such as 'RE- VERSE"), the LLM formaliser itself was capable of produc- ing correct string manipulation expressions unaided.
and wordplay suggestions, we make use of the Wordplay dataset of Andrews (2024). For this task, we fine-tuned another Gemma2 9B base model using LoRA. The model was trained on 4 epochs on set of approximately 16,800 examples (consisting of solution explanations of puzzles from The Times and The Financial Times from selected authors in the Wordplay dataset).
3.3. Python formalisation
3.4. In-Context Learning
Rather than create a dataset with many examples of formali- sation, here we use in-context prompting with less than 10 examples of the formalisation style required. In preliminary work, we concluded that the available Gemini-Flass LLM was not capable of using a (novel) cryptic crossword domain specific language ("DSL") through in-context learning with so few examples. In contrast, we found that the LLM could be prompted to produce Python code with relative ease, the approach taken was to frame a Saltaratvvvestll--SSSS as Python function calls within assert statements. The LLM was found to be able to reliably produce syntactically correct Python, and use the 'external functions' that had been de- scribed (as illustrated in Figure 5) to form logical sequences of declarations, which could then be parsed line-by-line by manipulating the Python abstract syntax tree ("AST"). An example of the Python DSL being generated by the formali- sation LLM is given in Figure 4, with the workings of the clue solution being illustrated in Figure 2c.
To produce Python code that could be sent to the prover, the LLM was prompted in an In-Context Learning ("ICL") manner. This consisted of the following parts:
Cryptic crossword rubric explain to the LLM what the principles were behind the fields such clue, definition. wordplay, etc.
2. 20-shot examples elue + wordplay
3. The 'external functions' rubric shown in Figure 5
4. Few-shot wordplay - Python formalisations
(6 examples given)
5. The actual clue, answer, definition and wordplay being formalised
Gemini-Flash did not appear to be particularly sensitive to the prompting style used, except in the 'handover step' (be- tween problem description and model generation) where several trials were needed to obtain the final function def- inition in the required format consistently. Further details of all the ICL prompts are given in Appendix A.5. For the Gemma2-9B-it formalisation runs, the same prompts were used unchanged (with no other tuning/training). In additional, a further Gemma2-9B model was trained on 448 valid Gemini-created proofs of ground- Wordplay examples.
To formalise wordplay into Python 'proofs' of the correctness of solutions, we used Google's 5-001 LLM pinned model version) during development. This model was initially chosen instead of frontier-tier model since the formalisa- tion task should not require much inventiveness/reasonigg the actual required steps are already present in the wordplay, the task is merely to translate to Python. To determine whether the choice of Gemini-Fle was
5

---

A Reasoning--aase Baang Approach to Cryptic Crossword Clue Solving
Assertiontrrer: ss_saabeviaticnt (' an Astist', "xA"): 'an Artist' does not have valid abbreviation;
"RA" la an asbreviation for artist, art_llezy, Royal Artillezy, gunners, painter Assertiong:rer: action_type ('goes crazy', Actien.ANAGRAN) "goen crazy' does not suggest Action.ANAGER but 'crazy' does
aasertionsrrer: action_type
('worked', typoookpps 'wo"ked" does not suggest Action, HOMCFROPM but may be ANNGGAA
HOMDFRONE))))
Figure 6. Illustrative
responses (with hinting)
Figure 7. Statistics anawer candidate list, as more candidates generated
from the verifier
4. Experiments
4.1. Gemma2 9B answer candidate generation
During the initial experimental phases of fine-tuning local models for the answer generation task it was discovered that bbase models scored more highly than -it models. This might be explained by observing that instruction fine- tuning may (to some extent) penalise off-the-wall answers, which may be essential for our task. In addition, we also observed that while the Top-1 candidate from a model gen- erating with temperature = 0.5 had high accuracy, it was beneficial to run candidate generation with & = 1.0 (even though the Top-1 accuracy was lower in this case) since having a wider spread of answer candidates was useful for our pipeline overall.
3.5. Proof Verification with Hinting
The system's 's verifier must decide whether a given formal- isation is valid, and report any errors found to iteratively improve the Python code as feedback to the LLM formaliser in cycle, as seen in Self-Debug (Chen et al., 2024), and AlphaCodium (Ridnik et al., 2024). Examples of assertion failures, with constructive hinting, are shown in Figure 6.
This cycle is repeated until a formalisation is validated (zero assertion failures, considered a 'SUCCESS' with the answer having been proved), or max_ree max_rewrites=2 is reached. no Python formalisation can be validated, then the fallback a answer is used (defined being the most fre- quent answer amongst the original candidates produced in the first stage of solving the clue).
Figure 7a shows that the probability of the gold answer being among the candidates produced is (unsurprisingly) monotonically increasing in the number of independent sam- ples. It also shows that this process is not yet asymptotically limited, although slowing down with increasing n.
3.6. Partial Correctness Metrics
Figure 7b shows that choosing the highest-frequeng answer candidate can be a very effective strategy. How- ever, there is clear limit to this idea: There is a significant probability that cryptic crossword answers in the long tail of potential answers. Indeed, intentionally creating mis- leading clue 'surface readings' is a hallmark of good cryptic clue setting.
One interesting direction explored in Saha et al. (2024) was the performance of LLMs on cryptic clues if some of the letters were known (as would be the case if an entire grid were being solved). The conditions examined were with 25%, 50% and 70% of letters 'known'. Based on observations working with the proposed system for single clues (and more general experience of crossword solving), we approached this problem in two ways.
4.2. Gemma2 9B wordplay candidate generation
For the 25% level of letters 'known', it was a simple matter to use the current system with candidate answers which didn't match the known letters filtered out. For the higher levels of known letters, we instead used the FastText em- bedding method of Mikolov et al. (2018) to find the near- est neighbour answer within The UK Advanced Cryptics Dictionary, Beresford (2000), by comparing against the em- bedding of the raw clue phrase itself.
Since wordplay is so flexible, it is difficult to evalu- ate it for accuracy against other examples (without, say, a large LLM to evaluate the differences). However, good wordp ay should result in good formalisations, so evalua- tion is available on an end-to-end basis.
One key assumption in the system proposed here is that correct answer should lead to interpretable wordpl lay, whereas an incorrect answer candidate should give rise to unformalissabe/nnvriiabbll wordplay. The following typical example illustrates how the correct answer leads readily to correct wordplay ay (the workings of this clue
The 'arrtill Correctness' results, since they are not the core thrust of this work but interesting in their own right, are given in Appendix A.5.7.
6

---

A Reasoning-Basce Approach to Cryptic Crossword Clue Solving
illustrated in Figure 2d), whereas trials with an incor- rect answer candidate (which was, in fact, the most fre- quent candidate for this clue) give clearly unverifiable wordplay:
for each e clue). These voting-based results would have ex- ceeded prior state-of-the-art results for open-licensed mod- els on their own.
Going beyond single models, the Cemini-Flass
Formal 1 iser demonstrates Top-1 exact-match perfor- mance of 32.5% for the Cryptonite Test set, establishing a new state-of the art result against the updated baselines (the Bayesian IRT results are that Gemini-Fiie has probability of 92% of being actually better than 4PT-4o).
clue: "wader woman has on (5)* definition: "(wader) woman has cn"
answer: "HERON" correct answer wordplay: "woman (HER) has on (ON)"
# incorrect answer - 3 trials shown answer: EGRET
Moreover, the results of the non-fine-tuned Gemma2 11122-9B--0 9B-it Formaliser also (marginally) beat the previous state-of-the-art results which is perhaps an even stronger statement about the capabilities the system described here, since in this case Gemma2-9B models have been used throughout the solving process, showing that it is be possible achieve very competitive cryptic crossword solving results through reasoning with open-licensed models. The Bayesian IRT results are that the Gemma-9B FT model has a probability of 81% of being actually better than GPT-4o on Hard clues, 57% overall.
wordplay: "weman (HER) on top of
(REG another word for on, as in
'do you have the heating on?")"
wordplay: "EG (woman has) RET (an)
woréplay: "woman (HER) has on/around (EG) wader bird"
4.3. Cryptonite Results (Top-1 exact match)
In this work, we focus our testing on using the Cryptonite dataset of Efrat et al. (2021) as a benchmark, with the Top- 1 exact-match results shown in Table 1, As in Saha et al. (2024), due to computational constraints, we performed sampling of the validation and test sets, using fewer than the full 26k examples available. The standard deviation of these figures is = =1.5% 1000 samples, and +3.3% at 200. To determine whether the systems presented here 'beat' GPT-4o, we performed a Bayesian Item Response Theory test (Fox, 2010) to estimate the probability that our results outperformed the GPT-40 (over the same samples).
The formaliser results are (surprisingly) relatively worse for Quick clues. This seems to be related the fact that the agreemen//irqquecc-
Gemma2 freq model is very strong on these clues, and any "contribution' from the formalising/verificttin procedure is likely to overrule a good base-line result, due to erroncous verification of 'proofs' that are not valid.
4.4. Ablations
The 5-Shot results in Table show that:
The lines in Table marked '(AB) are ablations. Both utilise the measurement of average logprob of the output tokens given by the relevant model.
GPT-4o (2024-11-25) gives stronger results than those of GPT-4-Turbo (2024-04-09) given in Saha et al. (2024) - so this is an updated baseline;
The first ('logprob answer') shows the results of using the candidate answer generation Gemma2-9B FT model from above, with the candidate answer being chosen from the list of 20 possibilities according to highest logprob. Since answers are typically very short, this method is similar to the frequency-based selection model.
The updated GPT-4o results show surprisingly strong performance on the validation split (unfortunately, the composition of this commercial model's training data is unknown);
The second ('logprob wordplay') shows the results of evalu- ating the Gemma2-9B FT model that generates wo wordplat hipllyy lay hypotheses, and choosing an answer based on the highest logprob according that generating model. Somewhat unex- pectedly, this was not as effective as might be assumed from the generated wordplay seen in Section 4.2 - where the wo wordplas ay for wrong answers looks absurd. Examining samples of the wordplay ay most favoured by pure logprob order, it seems that the generating LLM finds simply- but completely fictitious wordplay lay quite likely.
Gemini- Flassh000 (which was used in development of the formaliser) is not particularly good at solving cryp- tic clues in itself;
The Gemma2-9B model gets a large uplift from fine- tuning on the Cryptonite training set (compare the 5-Shot figures to the later Gemma2-9B FT ones).
The Germa 2 9B FT accuracy figures are for the first re- sult returned by the fine-tuned Gemma2 model. In contrast, the Gemma2-9B freq accuracy figures are for the most common (i.e. highest frequency) result among the Gemma2 answer candidates (for which 20 samples were generated
Both of these ablations demonstrate that the formalisation and verification steps are essential components in our system
7

---

A Reasoning-Basse Approach to Cryptic Crossword Clue Solving
Table Cryptonite results Standard splits, Top-1 answer accuracy
Validation
Test
Model
samples
Overall
Quick
Hard
Overall
Quick
Hard
Rule-based (*)
26k
8.3%
8.6%
13.5%
5.8%
TS-large (770M) FT (*)
26k
7.4%
7.6%
12.8%
3.4%
Gemma2-9Ban 5-shot
1000
5.7%
11.5%
5.2%
4.5%
10.5%
4.0%
Gemini-Flash 5-shot
1000
6.6%
12.5%
6.1%
6.5%
11.8%
6.1%
GPT-4o 5-shot
1000
29.8%
45.0%
28.5% %
27.6%
47.4%
26.0%
Gemma2-99 FT Gemma2-9B freq (#=20)
1000
21.7%
28.8%
21.1%
15.9%
38.2%
14.1%
1000
26.6%
31.3%
26.2%
25.5%
55.3%
23.1% 1%
(AB) logprob answer
500
23.9%
35.9%
22.9%
22.7%
55.3%
20.1%
(AB) logprob wordplay
200
21.0%
15.4%
21.4%
20.5%
46.7%
18.4%
Gemini-Flas Formaliser
200
28.0%
23.1%
28.3%
32.5%
46.7%
31.4%
Gemma2 9B-it Formaliser
200
26.0%
23.1%
26.2%
29.0%
46.7%
27.6%
Gemma2 9B-FT Formaliser
200
27.0%
23.1%
27.3%
29.5%
53.3%
27.6%
Rows (*) are as reported Elrat al. (2021); Hard columns are non nan-Quick clues
they cannot be shorteut by 'dumb ranker' in the pipeline.
Appendix A.1.7, have the true answer UNDERMINED being defined by 'damaged'. This would likely be too distant to be reasonable for a regular crossword, but the strength of the wordplay (the answer being literally given in the clue) confirmation enough to satisfy solvers. Setting this 'synonym distance hurdle' is an ongoing challenge.
4.5. Qualitative Error Analysis
In addition the numerical results presented in Table note the following qualitative aspects of system's performance:
The headline success rate is bounded above by the initial candidate answer generation process. If the system cannot guess the answer in its top-k (k = 20 ), the remaining process doomed. As shown in Figure 7a, even with higher top-k, this puts an upper bound on performance that is well below 100% correct. Having better candidate answer generation would be beneficial and this would directly feed through verification process
4.6. Known Limitations the System
While the verifier implemented for this work is effective, does not completely cover the following possible potential 'shorteuts' in the Python functions analyses:
The entire Python function might consist of comments, so that nothing could trigger an a assert. This has been partly countered by requiring the Python code to include at least 2 assert statements
While the proprietary models may output the correct final answer, it is often the case that their 'reasoning process' makes no logical sense (indicating, perhaps. that they have memorised c clue/answer pairs). In contrast, our method does give us useful human interpintterretaalle reason- ing for each solution
The Python function contains conditional execution, rout- ing around assert statements
Occasionally, the hint assert XY2 failed results in the re-write assert XYZ==Fal see, which is clearly not productive
significant source false negatives is the synonym function, which relies on sequence of steps: first we attempt a look-up an open-source thesaurus, then in dataset of 'regular crossword swers'. But the final fall-back is asking an LLM whether given phrases are synonyms. While the first two steps may vote positively (for easy matches), cryptic clues that the definition and the answer are more distantly related than regular crosswords. For instance, in
The proof may be logically disconnected, with left-hand- side terms being supported justified by right-hand- side terms in other lines of the code
common in
These issues do not appear insurmountable, given time and effort. should be noted that since the formalising LIM
8

---

A Rcasoning-Basce Approach to Cryptic Crossword Clue Solving
only being used In-Context there is little chance that the above issues are being systematically abused (which would almost certainly happen if there was leaming-inn-the-lopp in a Reinforcement Learning setting).
applying these techniques other reasoning-intensive NLP tasks.
Around the time of the initial submission of this work, Guo et al. (2025) publicly disclosed practical framework for learning to reason using Reinforcement Learning with an outcome-only reward scheme, which opens up a whole new avenue for investigation. While the OpenAl ol models had previously displayed reasoning traces, these were not considered in Table 1 Partly due cost/APl considerations, but also because the authors strongly feel that proprietary black-bor methods have limited research value.
5. Conclusions
The authors recognize the -omain-specificity of cryptic erossword solving. However, believe that it serves rigorous and complex test-bed for reasoning, requiring multi-faceted language understanding and logic. While the specific DSL developed here is tailored to crosswords, the underlying principles of our approach- decomposition, for- malization, and verification with feedback are intended to be more broadly applicable other reasoning tasks. Cryptic crosswords, with their clearly defined rules and solutions, allow for precise evaluation and iterative refinement of these principles.
Going forward, clearly a Reinforcement Learning approach would be very interesting to apply to the Cryptic Crossword domain, since these NLP reasoning problems are rather dif- ferent from the mathematical proof/ programming challenge tasks that are typical being tackled. Section 4.6 highlights potential issue with our verification approach when com- bined with RL, since there a clear opportunity for RL reward hacking unless the verifier made 'bullet-proof".
Our results have validated our overall approach to codifica- tion of the cryptic crossword problem domain Generating answer candidates and wordplay suggestions followed by production of code via an LLM-base formalisation pro- cess, verification using Python code analysis tools, and itera- tive prompting of the LLM formaliser proved quite effective. Generating multiple candidate answers, followed by multi- ple wordplay samples, can been framed as inference- -ime computation (using 9B models) rather than using a large pro- prietary model. Due to the verification element, our system can benefit directly from additional test-time computation. Beyond numerical improvements, key contribution is the verifiable reasoning process itself, offering interpretability not available from black-box models.
We look forward to exploring the Cryptic Crossword rea- soning task there are wide number of different avenues available.
We were happy to discover that our development work us- ing the Gemini-Flash LLM as a formaliser was directly transferable to the open-licensed Gemma2-it model for the same role, with little loss of performance, enabling the whole pipeline to be run locally. The weakest link in the chain was, predictably, getting the 'Aha' of wordplay cre- ation to work humans can still generate wordplay that is beyond the capabilities of current models.
The authors sincerely hope that this work sparks interest in the cryptic crossword domain, which presents an array of interesting and challenging reasoning problems on which fruitful research can take place, even for those with limited computation budgets.
5.1. Further Work
Although we haven't explicitly tested generalizability to other NLP tasks, we believe the developed techniques par- ticularly formal verification and iterative refinement offer valuable insights for improving LLM reasoning in complex NLU scenarios. Future research could fruitfully explore
9

---

A Rcaaoning-Bassee Approach to Cryptic Crossword Clue Solving
Acknowledgements
els), via API (the Gemini-Flash model, with a pinned version number). The training procedures outlined in the text, and therefore have some degree of repro- ducibility, The fine-tuned Gemma2 models will be made available any case, on publication.
Support for this research was provided by the Google Al Developer Programs team, including access the Gemini models and GPUs on Google Cloud Platform.
The authors thank the ICML reviewers for their time and valuable feedback.
Computational requirements
The Gemini LLM was accessed by API, and the total spend to create the results in this paper was less than $100 USD, with each prompt round-trip taking around seconds. The Fine-Tuning of the Gemma2 9B model took around 24 hours for a full Cryptonite training run, and hours for the Wordplay dataset runs. Thus, the single-GPU model runs totalled less than $50 USD.
Impact Statement
Societal impact
There are many current eryptic crossword enthusiasts that would potentially not welcome Al-enabled solvers 'take over' their favourite pastime. In particular, when taken further, this line of work would potentially disruptive to public leaderboards that rank people according to the time taken to solve puzzles 100% correctly.
More generally, this paper presents work whose goal to advance the field Machine Learning. The techniques de- veloped in this work extend work done other domains for machine reasoning to broader field that includes NLP/NLU tasks. That being said, we do not feel that there are signifi- cant societal consequences of our work that require specific highlighting.
Potential bias in favour of native English speakers
While the English language has a high capacity for ambi- guity and wordplay overall, making this type of crossword possible, cryptic crosswords also exist in other languages (Wikipedia contributors, 2024). addition, although soly- ing cryptic crossword answers may be very difficult (even for native English speakers), understanding the answer from given wordplay is much simpler.
Reproducibility
Datasets and Code
The following outline the efforts that have been made to ensure reproducibilityy
Datasets Resources such dictionaries used, and the Cryptonite and Wordplay datasets are available online, via the sources referenced in the main text.
System Code & LLM Prompts- Python code for the complete end-to-end system is available under an Apache license at https: http:/ggghhhgocoeoeeee / crypt i crosswordd- rend-reasson rensonnii in-veri f er. The prompting required for the LLM formalisation are also included in the Appendix.
Models used training The models referenced in this work are available with open weights (the Gemma2 mod-
10

---

A Seasoning-Baser Approach to Cryptic Crossword Clue Solving
References
Gao, L., Madaan, A., Zhou, S., Alon, U., Liu, P., Yang. Y., Callan, J., and Neubig, G. PAL: Program-aided language models. In Proceedings of the 40th International Confer- ence on Machine Learning, pp. 10764-10799 2023.
Andrews, M. Wordplay Dataset repository. https: // github.com/moee mtt./dddd/a/e//////// wttpaa/acryytttwooooooooo.
2024.
Andrews, M. and Witteveen, S. Proving that cryptic cross- word clue answers are correct. In ICML 2024 Workshop on LLMs and Cognition, 2024.
Gemma Team and Google DeepMind. Gemma 2: Improving open language models at a practical size, 2024. URL https: /ttpps://ariiiie httpiv/.rxv/////////o abs/ 240.000118.
Andrews, M. and Witteveen, S. Generating code to verify cryptic crossword reasoning. In ICLR 2025 Workshop on Deep Learning for Code, 2025.
Gou, Z., Shao, Z., Gong, Y., Yang. Y., Huang, M., Duan, N., Chen, W., et al. ToRA: tool-integrated reasoning agent for mathematical problem solving. In The Twelfth International Conference on Learning Representations, 2024.
Anthony, S. and Goodliffe, M. https://youtu.be
Cracking the
Cryptic (17-May-2024). 2024.
Beresford, J. R. The UK Advanced Cryptics Dictionary. Technical report, published online, 2000. https://
Guo, D., Yang. D., Zhang, H.,, Song, J., Zhang, R., Xu, R., Zhu, Q. Ma, S., Wang, P., Bi, X., et al. DeepSeek-RI: Incentivizing reasoning capability in LLMs via reinforce- ment learning. arXiv preprint arXiv:2501.12944 2025.
Chen, X., Lin, M., Schärli, N., and Zhou, D. Teaching large language models self-debug. In The Twelfth International Conference Learning Representations, 2024.
Hu, E. J., Wallis, P., Allen-Zhu, Z. Li, Y., Wang, S., Wang, L., Chen, W., et al. LoRA: Low-rank adaptation of large language models. In International Conference on Learn- ing Representations, 2022.
Connor, A. Devious humour and painful puns: Will the cryptic crossword remain the last thing Al can't conquer? Guardian UK Crossword blog, 2024.
Jiang, Q.. Welleck, S., Zhou, P., Lacroix, T., Liu, J., Li, W., Jamnik, M., Lample, G., and Wu, Y. Draft, sketch, and prove: Guiding formal theorem provers with informal proofs. In The Eleventh International Conference on Learning Representations, 2023.
Deits, R. rdeits/cryptics code repository. https:// bithub.comm/rdeit///ryttttt 2015.
Deits, R. Seppaiccooosooddsjjjddderrrrrrro
git wtt.ub/cmm/rreetteccrpp....ooooo 2022.
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F,, Dal Lago, A., Hubert, T., Choy, P., de Masson Autume, C., Babuschkin, I., Chen, X.. Huang, P.-S.. Welbl, J., Gowal, S., Cherepanov, A., Molloy, J., Mankowitz, D. Sutherland Robson, E., Kohli, P., de Freitas, N., Kavukcuoglu, K., and Vinyals, O. Competition-level code generation with AlphaCode. Science, 378(6624): 1092-1097, December 2022. ISSN 1095-9203. doi: 10.1126//cience..bq11...
Efrat, A., Shaham, U., Kilman, D., and Levy, O. Cryp- tonite: A cryptic crossword benchmark for extreme ambi- guity in language. In Moens, M.-F., Huang, X., Specia, L., and Yih, S. W.-t. (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 4186-4192, Online and Punta Cana, Do- minican Republic, November 2021. Association for Com- putational Linguisties. doi: 10.18653/v1/0021..om main.344. URL https: aochllathhllooyyrro 2021.emnip www.eemnlpmmm.....
Macnutt, D. S. Ximenes on the art of the crossword. Methuen, 1966.
Fox, J.-P. Bayesian Item Response Modeling. Statisties for Social and Behavioral Sciences. Springer New York, 2010. ISBN 978-1 1-4419-07411- doi: 10.1007/978-1. 4419-0742-4.
Mikolov, T., Grave, E., Bojanowski, P., Puhrsch, C., and Joulin, A. Advances in pre-training distributed word representations. In Proceedings the International Con- ference on Language Resources and Evaluation (LREC 2018), 2018
Friedlander, K. J. and Fine, A. The grounded expertise components approach in the novel area of cryptic crossword solving. Frontiers in Psychology. 7, 2016. ISSN 1664-1078. doi: 10.3389//psye URL
Ni, A., Iyer, S., Radev, D., Stoyanov, V., Yih, W.-t., Wang, S., and Lin, X. V. Lever: Learning to verify language- to-code generation with execution. In International Con- ference on Machine Learning, pp. 2610662611111 26128. PMLR, 2023.
http:://ww..frottiersinnoo
bbunaa///aawcallooo///rree/////s fpsyg.2016.00567.

---

A RRasoning-Basse Approach to Cryptic Crossword Clue Solving
Ridnik, T., Kredo, D., and Friedman, I. Code generation with AlphaCodium: From prompt engineering to flow engineering, arXiv preprint arXiv:2401.000 1000000000 2024.
language models. Advances in Neural Information Pro- cessing Systems, 36:21573-21612, 2023.
Ye, X., Chen, Q. Dillig, I., and Durrett, G. SatLM: Satisfiabiliy-aidded language models using declarative prompting. In Proceedings of NeurlPS, 2023.
Rozner, J., Potts, C., and Mahowald, K. Decrypting cryptic crosswords: Semantically complex wordplay puzzles as target for NLP. In Advances in Neural Information Processing Systems, volume 34, pp. 11409-11421, 2021.
Sadallah, A., Kotova, D., and Kochmar, E. Are L.Ms good cryptic crossword solvers? arXiv preprint arXiv:2403. /2094, 2024.
Saha, S., Chakraborty, S., Saha, S., and Garain, U. Lan- guage models are crossword solvers. arXiv preprint arXiv:2406.09044, 2024.
Trinh, T., Wu, Y., Le, Q. He, H., and Luong, T. Solv- ing Olympiad geometry without human demonstrations. Nature, 2024. doi:
unsloth.ai. Unsloth code repo. bttps://githiii inslothai/uns11 oth, 2024.
Wallace, E., Tomlin, N., Xu, A., Yang, K., Pathak, E., Gins- berg, M., and Klein, D. Automated crossword solving. In Proceedings of the 60th Annual Meeting of the Asso- ciation for Computational Linguistics (Volume 1: Long Papers), pp. 3073-3085, 2022.
Webb, D.
Epic crossword battle Times cryptic puzzle #29029, https: /outu.bb/ ssps 2024.
Expert vs.
2024.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F, Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thouglhe prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-2488 2022.
Wikipedia.
Cryptic crossword the free encyclopedia.
Wikipedia, https://
2024. [Online; accessed 1-Ju1y-2024).
Wikipedia contributors. Cryptic crossword regional variation. http:://eenwlkkjpddaoo///// Cryppicccoosoordeeeeiooaaaaaaaae 2024.
Williams, and Woodhead, D. Computer assisted analysis of cryptic crosswords. The Computer Journal, 22(1): 67-70, 1979.
Yang, K., Swope, A., Gu, A., Chalamala, R., Song, P., Yu, S., Godil, S., Prenger, R. J., and Anandkumar, A. LeanDojo: Theorem proving with retrieval- rerrieedl-augmenttd
12

---

A Rcasoning-Basce Approach to Cryptic Crossword Clue Solving
A. Appendix
A.I. Cryptic Crossword Background
The following borrows extensively from the description on Wikipedia (2024) (kudos to the authors there), to which we have added wordplay ay annotations notation typical of the F' FifteenSquare.com website (and in the Wordplay dataset in this work).
A.1.1. 1. BASICS
A cryptic clue leads to its answer only if it is read in the right way. What the clue appears to say when read normally (the surface reading) is usually distraction with nothing to do with the solution. The challenge is to find the way of reading the clue that leads to the solution.
typical clue consists of two parts:
The straight or definition. This in essence the same as any non-cryptic crossword clue: a synonym for the answer, usually exactly matches the part of speech, tense, and number of the answer, and usually appears the start or end of clue. For our annotations, the span that encompasses the definition is highlighted using curly braces.
The cryptic, subsidiary indication or wordplay. This gives the solver some instructions how get to the answer in another (less literal) way. The wordplay parts clues can be obscure, especially newcomer, but they tend utilise standard rules and conventions which become more familiar with practice.
Sometimes the parts of the clue joined with link word phrase such 'from', 'gives' or 'could be'. One of the tasks of the solver is to find the boundary between the definition and the wordplay, and insert mental pause there when reading the clue cryptically.
We list below several of the important styles of wordplay that are commonly used, cach with an annotated example. For a more comprehensive list, along with an outline of the 'Ximenean principles', please see Wikipedia (2024).
A.1.2. ANAGRAMS
An anagram is rearrangement of certain section of the clue to form the answer. This is usually indicated by codeword which indicates change, movement, breakage or something otherwise amiss. For example:
clue:
Chaperone shredded corset (6)
definition: (Chaperone) shredded corset
answer: ESCORT
wordplay:
(corset) (*shredded)
A.1.3. CHARADE
In charade, the answer is formed by joining individually clued words to make larger word (namely, the answer). For example:
clue:
lat Leader managing money
definition:
Lut Leader (managing money)
answer: BANKING
wordplay:
BAN (outlaw) + KING (leader)
A.1.4. CONTAINERS
container or insertion clue puts one of letters inside another. For example (also starting to add a little more indirection):
cluc: Utter nothing when there's vickedness about (5)
definition:
(utter) nothing when there's wickedness about
answer:
VOICE
wordplay:
O (nothing) with VICR. (wickedness) around it (about)
13

---

A Rcasoning-Basce Approach to Cryptic Crossword Clue Solving
A.1.5. DELETIONS
Deletion a wordplay mechanism which removes some letters of a word to create a shorter word. For example:
clue:
Bird is cowardly, about to fly away
definition:
(Bird) is cowardly, about to fly away
answer: RAVEN
wordplay:
RAVEN (cowardly) - 'c' (i.e. circa, about) (-fly away)
A.1.6. DOUBLE DEFINITION
clue may, rather than having definition part and a wordplay part, have two definition parts. For example:
clue: Not seeing window covering (5)
(Not seeing) [window covering) answer: BLIND
definition:
wordplay:
Double Definition (DD)
A.1.7. HIDDEN WORDS
With hidden word clues, the solution itself written within the clue either as part of a longer word or across more than one word. For example:
clue:
Found ermine, deer hides damaged (10) Found ermine, deer hides danaged)
definition:
answer: UNDERMINED
wordplay:
[fo|UND ERMINE D Dieer] (hides)
A.1.8. HOMOPHONES
Homophones are words that sound the same but have different meanings, such as 'night' and 'knight'. Homophone clues always have an indicator word or phrase that has to do with being spoken or heard. For example:
clue:
We hear twins shave (4)
definition: We hear twins (shave)
answer: PARE
wordplay:
"pair" (twins, "we hear")
A.1.9. REVERSALS
word that gets turned around make another is reversal. For example:
clue:
Returned beer fit for king (5) Returned beer (fit for a king)
definition:
answer: REGAL
wordplay:
(LAGER) (beer, <returned)
14

---

A Reasoning- Basnn Approach to Cryptic Crossword Clue Solving
A.2. Wordplay Dataset
The Wordplay Dataset used in this work is extracted from websites where cryptic crossword enthusiasts post solutions the puzzles published in major publications. Each completed puzzle is annotated by solver who provides the community with befinitisn wordplay lay and answer fields for each of the approximately 30 clues in that day's grid.
For UK papers, these enthusiast websites include:
t immesorrthttmmsscoom Times, Times Quick
www.Eifteenssuaaee..ett Independent, Guardian, Financial Times
bigdave44.com Telegraph, Sunday Telegraph
The following is an example from the Wordplay dataset, formatted in YAML (the workings of this clue are illustrated in Figure 2c):
title: Financial Times 16, 479 479 by FALCON
url: Einanclll-ccmms--111999-----ooop
author: teacow
clues:
clue: '(Offer) of support also breadcast
pattern: '8'
ad:
answer: PROPOSAL
wordplay: PROP (support) (ALSO) (+broadcast)
In the above:
clue is the original clue, as given solvers, but with the 'regular crossword' definition portion highlighted with curly braces;
pattern is the number of characters in the answer;
ad (across/down) is potentially significant, because some clues include directional hints such as 'before' or 'upwards' which are only meaningful if the orientation the answer within the grid is known;
answer is the cluc's final answer (not known to the solvers before solving); and
wordplay is informally annotated explanation of how the clue words act together to logically build the letters in the answer (the resulting grid letters typically being in upper case) here the syymol signifies that ALSO is to be anagrammed due to the anagram indicator (broadcast) in the clue.
The Wordplay dataset is publicly available as Andrews (2024). Note that care was taken to ensure that the train- ing/validation/tesst splits follow those the Cryptonite dataset (and the test set answers are deliberately scrubbed from the retrieved data by the provided scripts, to reduce the chance that they become training data for an over-eager crawling system).
A.3. Choice Cryptonite vs Rozner
At the start this paper's research program, the Cryptonite dataset of Efrat et al. (2021) was chosen as being the focus, over the approximately contemporaneous dataset from Rozner et al. (2021) (denoted Rozner here), for the following reasons:
Cryptonite was larger (523k clues, compared 142k in Rozner)
Cryptonite consists of clues from The Times and The Telegraph (whereas Rozner is the UK's Guardian). While these are all fine newspapers, it is clear that the cryptic crossword community (found online via websites for wordplay discussions, or YouTube channels) that The Times is considered the Gold Standard cryptic crosswords.
15

---

A Reasoning- Approach to Cryptic Crossword Clue Solving
Indeed, Connor (2024) - one of the Guardian's 's own cryptic blog posts directly states: "The Times hosts an annual crossword-solving competition and it remains, until such time the Guardian has its version, the gold standard." - In the authors' view, The Times deserves its role as Gold Standard due to (a) adhering to / upholding the Ximenean standard Macnutt (1966) for what is allowed in clues; (b) doing so for decades; and (c) maintaining high consistency of clue difficulty within puzzles
The Cryptonite dataset was made available for direct download even though the licensing is (politely) 'fuzzy', it remains a useable research dataset (and seems unlikely to be challenged by The Times, since it is not possible to reconstruct their full puzzles from the clues given as individual line-items, due to deduplication, for example)
The Rozner dataset required researchers to "scrape their own data', likely because while the data was being retrieved from public website, the data itself could reasonably be assumed to be copyrighted. This slight inconvenience had useful impact (please see below)
Unlike the Cryptonite dataset, the Rozner dataset does not include Across/Down markers for the clues which makes some of the clues difficult resolve (for instance EXAMPLE on the paper's first page can only be read correctly if one sees that is Down clue- which converts 'up" into a reversal indicator)
The Cryptonite dataset also includes 'is quick' annotations that show whether a clue was taken from a 'Quick Cryptic' crossword (these clues are typically easier, which enables a further degree of performance analysis).
The Cryptonite dataset splits were set in stone. Rozner, though, had a series of splits (random, disjoint, and 'init'):
The 'random' split was clearly shown to be a poor way of separating train/test due to close overlaps
-
The 'dijooint' split is similar in spirit to the Cryptonite methodology
The 'Init' split had the additional twist that common prefixes would only be found in their own splits. This had a catchy intuition, although it's not clear from a cryptic cluing perspective whether this has much genuine basis. While there are some prefixes that are common (eg: EX- easily clued by referring to divorce, etc), the impact seems overall marginal (particularly given the accuracy rate differences reported)
-
Our paper describes a system trained on Cryptonite clue/answer training data, and also (as a component) the Wordplay dataset (which abides by the Cryptonite splits too).
It would be possible to test our existing (Cryptonite trained) system on the Rozner 'Init' test set. However, while Saha et al. (2024) could have the flexibility run tests on either dataset (since no training was performed), running our current model on the Rozner 'Init' test set would be clearly mis-aligned vis-a-vis the data split.
But there is also structural reason against re-training the paper's system on the Rozner 'Init' split for (specifically) Wordplay. The Wordplay dataset generation process was guided by the principle of maintaining the Cryptonite splits, it would be a disaster if Rosner 'Init' Wordplay splits were to be made public. The reason: is very likely that the Cryptonite test set has a large intersection with the Rozner 'Init' training set (and conversely). As seems evident from the baseline improvements shown above, OpenAl likely trains on the Cryptonite training set (as they are welcome to do). However, since (as of November 2024) Saha et al. (2024) appears to have released (or re-released) the 'Init' training under MIT license, a commercial vendor such as OpenAl would be quite within their rights to also train that. Thus, commercial systems (against which reviewers are forcing academic papers to benchmark) will have been trained on the test sets (without commercial vendors explicitly 'cheating' - they will just be training on all the available training data).
In the authors' judgement, the reasoning paths that are being tested here through the cryptic crossword task are a prize cultural asset, generated over decades of human effort, and this should not be squandered. Hopefully, this explains the authors' decision to train on only the Cryptonite dataset: We don't want encourage the gathering and distribution of cross-contaminating datasets, specifically Wordplay datasets.
16

---

A Rcasoning-Basce Approach to Cryptic Crossword Clue Solving
A.4. Fine-tuning prompt
The following verbatim training example used for the fine-tuning of the Gemma2 9--ase model:
#E# Instruction:
Cryptic cluc wordplay generation civen the clue and the answer, return expert definition and wordplay annotations
### Input:
clue: "musical and ballet, oddly, that can be avoided
answer: EVITABLE evitable
### Response:
definition: musical and ballet, oddly, (that can be avoided) wordplay: EVITA (musical) B[a]L[]]E[t] (ballet, odd letters)
A.5. In-Context Learning Prompts for the Gemini LLM
The Gemini LLM prompted in-context with the concatenation the following sections:
Cryptic Crossword overview
Many-shot wordplay examples
Declaration of 'external' Python functions
6-shot formalisation demonstration
Actual problem statement (for continuation as Python proof)
After a verification failure: Error messages for the generated proof, with hints if available, and request to improve iteratively
The sections of the prompt are described more fully below, note that care was taken ensure that the chosen terminology was consistently throughout.
1. CRYPTIC CROSSWORD PREAMBLE
The following the rubric and wordplay ay preamble given to the Gemini LLM:
A Cryptic crossword question involves using the words in the given clue to yield an answer that matches the letter pattern. The clue will previde definition of the answer, as well as some 'wordplay' that can also used confirm the answer. Expert question solvers write informal 'proofs' using particular format.
For the definitien, the original clue is annotated with '() to dencte where definition is to be found. For the wordplay, the following conventions are loosely used:
The answer is assembled from the letters in CAPS
Words in brackets show the origins of letters in CAPS, short forms Action words are annotated illustrated;
often being synenyms,
+ (ETO N) (+mad snagram-signifierr) = TONE
+ (FO < (<back = reversal-signifeer) ROOF
+ [re]UsR (missing - removal-signifiee USR
is shorthand for "oouble Definition'
17

---

A Reasoning-Base Bastn Approach to Cryptic Crossword Clue Solving
A.5.2. MANY-SHOT WORDPLAY EXAMPLES
Around 20 examples from the Wordplay dataset are included in the in-context prompt:
For example:
clue:
"arrived with an artist, to get optical device (6) definition: arrived with an artist, to get (optical device]
answer: CAMERA
wordplay: CAME (arrived) RA (artist, short form)
clue:
A.5.3. EXTERNAL PYTHON DSL FUNCTIONS
Domain Specific Python functions descrihed in-context the LLM,, which appears able them without seeing their internal functionality. In fact, the actual implementation of the functions is extensive than described, since calls to these functions also track 'near misses' which can be fed back as hints during the re-write process.
The task is to produce a formal proef using python code, where the docstring will also include an informal proof The following functions that
aid.
used your output code:
Action=Enum Daatt-EnnnnAAiin ('Action',
'ANAGRAM, REMOVE FIRST, INITIALS, WEMOVELLAST '+ INOES_INSIDE, CCES_CUTSIDE,, REVERSE, SUBSTRING, HOMOPHONE')
External definitions
def is (p_syyonymmphhaseeesee
test_synonym:strr, patter::sr="//
bool:
# Determines whether 'test_synonyn is reasonable synonym for 'phrase',
# with letters optionally matching 'pattern' (est.aabbreiiaionnsst))) -> bool:
def is_ (pporeeiaationnprasseesss
# Determines whether 'test_ abbreviation' is
# valid abbreviation or short form for 'phrase'
def action_typ (phrase:str,, action:Action) -> bool:
Determines whether 'phrase' might signify the given 'action'
def is_anagram (letters:str, word:str) -> bool:
Determines whether 'word' be formed from 'letters' (i.e. an anagram)
def is homophone (phrase:str, test pust_hhorophonesstar -> bool:
# Determines whether Stees_hhmmpphere sounds like 'phrase'
A.5.4. FEW-SHOT FORMALISATION EXAMPLES
The following 3 (out of 6) of the few-shot formalisation examples given before the final test-case prompt:
The following examples of simple functions that prove that
each puzzle sclution is correct:
ppython
def proof (ONsWer="ONCE",
clue="head decapitated leng ago", pattern='4"):
***
definition: head decapitated (Long ago)
wordplay: [bICNCE (head decapitated remove first Letter of BONCE)
assert is_synenym ("head", "BONCE")
assert action_type (" "'decapitated" Action. REMOVR_FIRST)
and [1:]== assert is is_synenym (oyonnymm"oon ago", "ONCE", pattern='4") ()
""python
def proof (answer="DECTMAI"" clue="the point of medical treatment", p
18

---

A Reasoning-Baser Approach to Cryptic Crossword Clue Solving
definition: (the point) medical treatment
wordplay: (MEDICAL) (*treatment anagram)
assert is "s__synnymmm" ("the point", "DECIMAL", pattern='7") assert action typeenn_tpptteee (yrerenttyeettettteeett ("treatment", Action. Action.ANAGRAM) assert is_anagram ("MEDrrra""M ("MEDICAL", "DECIMAL") proof()
"python
def proof (ENSWer"""PPRMARRTT"
clue-"fat bags for every brand that's big seller", pattern=!11"):
definition: fat bags for every brand that's s la big seller) wordplay: SUET (fat) (bags = goes outside) of A
(PER (for every) + MARK (brand))
assert is is_synenym("fat" "SUET")
assert actien_type("bage assert "SUET" == "SD" + "ET" assert is ss_abbreviationn ("for every", "PER") assert is (...ynnny"""brra".". "MARK")
Action. IS_CUSSIEE
assert ""
"SUPERMARKET"
assert is_synenym(" big seller", "SUPERMARKET", pattern='11')
proof() ()
A.5.5. FORMALISATION INSTRUCTION
The following instruction is given before the final 'test-case" prompt illustrated in Figure 4:
Please complete the following in a similar manner, and return the whole function:
"python
def proof (answer=
A.5.6. PROOF VERIFICATION WITH HINTING
Examples of assertion failures, with constructive hinting, are shown:
Assertion assert: i__abbreviation inaiaattinnnn Artist', 'RA') :
'an Artist' does not have valid abbreviation;
'RA" is an abbreviation for artist, artillery, Royal Artillery, gunners, painter
AssertionErrors assert action tonettype( ('goes crazy',
'goes crazy' itself does not suggest ANAGRAMM but 'crazy' does AssertionError: assert action_type ('worked', aetion.HOMOPHONE 'worked' does not suggest Acticn. HOMCPHONE, but maybe Action.ANAGRA
Please re-implement the SOLUTION above (altering both the docstring and the python code required), taking care to fix each of the problems identified and return the whole function:
"'"python def proof (answer-...
Once the prover has fully parsed given output with zero assertion failures, the proof considered success (up to re-write iterations are allowed, more that that considered an overall failure to prove the answer).
19

---

A Reasoning-Basce Approach to Cryptic Crossword Clue Solving
Table 2. Partial Correctness Metries Results
Validation
Test
Model
known%
samples
Overall
Quick
Hard
Overall
Quick
Hard
GPT-4T ('Init')
25%
33.7%
GPT-4T ('Init')
50%
52.9%
GPT-4T ("Init")
70%
76.3%
Gemini-Flash
25%
200
37.0%
38.5%
36.9%
45.5
66.7%
43.8%
Gemma2-9B-ir
25%
200
37.5%
38.5%
37.4%
44.0%
66.7%
42.2%
FastText k=1 NN
25%
200
15.5%
15.4%
15.5%
21.0%
33.3%
20.0%
Fast k=1 NN
50%
200
52.5%
38.5%
53.5%
62.0 %
46.7%
63.2%
FastText k=1 NN
70%
200
79.0%
61.5%
80.2%
81.0%
100.0%
79.5%
A.5.7. PARTIAL CORRECTNESS METRICS RESULTS
Our results in Table 2, which corresponds to the Exploiting Partially Filled Grids section in Saha et al. (2024), are based on running models on Cryptonite splits, rather than their 'Init'. However, the percentage differences observed are likely large enough outweigh the shift between the two datasets.
The GPT-4T rows Table 2 are as reported Saha et al. (2024), and apply to the 'Init' test dataset (i.e. different from our Cryptonite numbers, but still comparable figures in terms of what being shown here).
The Gemini/Gemma2 rows show the effect of simply filtering the output of Gemma2-9B fine-tuned candidate answer proposal model, based on random letter pattern using the same formulation Saha et al. (2024) and then using the rest of our pipeline. Here, our approach beats the previously reported GPT-4T results, and is itself limited by our first stage Gemma2 model's 'Top-20' candidate answers only containing the correct answer only around 45% of the time.
Our FastText = kNN systematic approach is clearly very powerful particularly considering that it does not involve any large model, merely brute-force search. This only works because number of known letters for these rows are so high- indeed the 70% level would not allowed in crosswords that obey the Ximencan guidelines of Macnutt (1966).
If solving complete grids were the target of research, would certainly incorporate this kind of solution and overlay the reasoning component to choose from the short-list output (rather than just selecting the first entry, as here). Note that also the performance is bounded above, because the wordlist is not exhaustive we determined that 7.0% of the gold answers (on the Cryptonite test set) do not appear in the list. This may not be such an issue with the 'Init' dataset, since that wordlist likely more restricted.
20


Reviews:
[{"review_id": "Cf8YVLfrGI", "reviewer": "Reviewer_HqW1", "summary": "This paper presents a system for solving cryptic crossword clues using a collection of fine-tuned and ICL-prompted open-weight language models, as well as a custom Python-based domain-specific interpreter. The proposed system first samples a set of candidate solution words from a fine-tuned proposal model. Another fine-tuned model is then used to sample potential wordplay decompositions of the answer candidates. An ICL prompted model then translates these wordplay breakdowns into a set of Python assertions containing a mixture of vanilla expressions and special functions (e.g. `is_synonym(a, b)`, `is_homophone(a, b)`) backed by additional small LM calls. Two rounds of refinement are conducted on this translation based on interpreter feedback. The highest-frequency answer with a wordplay breakdown that passes its translated assertions is returned as the answer to the clue (or the highest-frequency answer if no wordplay passes verification).\n\nThe authors evaluate their method on the Cryptonite dataset, containing cryptic crossword clues from the Times and Telegraph UK newspapers. They compare their method to several single-stage few-shot prompted LMs, including GPT-4o and Gemini 1.5 Flash. On the Hard subset of the test split, their method outperforms all considered baselines.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Answer to \"How many wordplay/definition suggestions are generated per answer candidate?\"\n\nFor each clue, we generate 20 candidate answers (the cumulative probability graph for the upper bound of success after this is given in Figure 7a).  These are then deduplicated, and we create 5 definition and wordplay examples for each candidate answer (we previously experimented with 10 samples, but the difference was marginal).  Examples of wordplay for correct and incorrect candidate answers are given in Section 4.2.  From a human point of view, the wordplay explanations for incorrect candidate answers are clearly nonsense - but (as commonly seen using LLMs) the models tend to 'approve' of their own outputs.  Therefore, proving out the reasoning in a more concrete way is essential, so we use our novel 'formalization/verification' process on all of these outputs.  \n\nThank you for the question : We will certainly add this important experimental detail to the paper, as well as expanding the process flow explanation in the text of Section 3 (rather relying so heavily on Figure 1).", "claims_and_evidence": "The authors claim that their method achieves SOTA results on Cryptonite. Section 2.2.2 does mention that the lack of reasoning capabilities presented a problem for models prior to 2024, but makes no mention of the new generation of LLMs post-trained for reasoning (the only mention I could find is a namedrop on line 207), begging the question of how the proposed method would perform against DeepSeek R1, Gemini Flash Thinking, OpenAI O1 or Claude 3.7 Sonnet.", "methods_and_evaluation_criteria": "Yes, the methods and benchmark are appropriate.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The main comparison is sound. I also checked the evaluation setup for the partially-filled answer setting ($\\S$3.6, A.3.5.7), and nothing stood out to me as improper.", "relation_to_broader_scientific_literature": "The authors cite the prior rule-based SOTA system for cryptic crossword solving. Their general scheme of checking \"loose\" answer proposals from an LM has gained a lot of steam recently in the guise of \"reasoning models\", LMs post-trained with CoT+RL which can exhibit self-verification behavior. So far, public reasoning models have not been developed that are trained end-to-end to take advantage of autoformalization; such a direction is very promising. This work definitely serves as positive evidence of the power of this kind of architecture, even using small models and a semi-informal format.", "essential_references_not_discussed": "I think it might be worth discussing the connection to prior LM+autoformalization approaches - SatLM (Ye et al, 2023) comes to mind, especially since that work also deals with the LM language familiarity issue by using Python as an intermediate language to decode constraints from the model.", "other_strengths_and_weaknesses": "I thought the authors did a good job of motivating the problem area as a useful exercise, and in introducing the problem format, which is admittedly confounding at first glance.\n\nParts of the paper were a bit hard to follow or lacked important detail, namely the method description ($\\S$3); while Fig. 1 shows a clear overview of the modeling flow, Section 3 does not contain a complete text description of the pipeline; it contains a stepwise summary of the human strategy that inspired the method, and subsections 3.1-3.5 detail stages of the process and design considerations for those stages, but there is no concise description of the overall procedure, which would be helpful to lead us through the subsections.", "other_comments_or_suggestions": "No typos stood out to me during my reading. See \"Other strengths and weaknesses\" for a suggested change to Section 3.", "questions_for_authors": "One key question that I was unable to find in the text (although I may have missed it) is: How many wordplay/definition suggestions are generated per answer candidate? Parts of the text allude to checking multiple wordplay options, but I'm not sure if this is just due to considering multiple answer candidates.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VzUwGHyeTr", "reviewer": "Reviewer_c1BM", "summary": "The paper presents a reasoning-based system for solving cryptic crossword clues using open-licensed LLMs. It follows a three-step pipeline: (1) generating answer candidates, (2) proposing wordplay explanations, and (3) verifying solutions via a Python-based formalizer. The system outperforms prior methods on the Cryptonite dataset, achieving a Top-1 accuracy of 32.5%, surpassing both rule-based and previous LLM-based approaches. The key contribution is using Python assertions to validate reasoning, improving reliability and interpretability.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "In Section 4.5 (\"Known Limitations of the System\"), we chose to explicitly discuss the limitations of our Python-based verification system to ensure transparency.  However, these potential 'shortcuts' are mainly a cautionary note regarding the potential for a Reinforcement Learning loop (in future work), where we would expect an RL system to exploit them, whereas the current system only rarely showed such behaviour (which is why these loopholes were not eliminated during development).\n\nOn the other hand, we would be delighted to add a section containing a qualitiative analysis of the key false positive / negative failure modes of the system.  Notably:\n\n* The headline success rate is bounded above by the initial candidate answer generation process.  If the system cannot guess the answer in its top-k (k=20 here), the remaining process is doomed.  As shown in Figure 7a, even with higher top-k, this puts an upper bound on performance that is well below 100% correct.  Having better candidate answer generation would be beneficial - and this would directly feed through our verification process (which is a step that the proprietary models do not benefit from, and gives us human-interpretable reasoning for each solution)\n* A significant source of false negatives is the `is_synonym` function, which relies on a sequence of steps: first we attempt a look-up in an open-source thesauraus, then in a dataset of 'regular crossword answers'.  But the final fall-back is asking an LLM whether given phrases are synonyms.  While the first two steps may vote positively (for easy matches), it is common in cryptic clues that the `definition` and the `answer` are more distantly related than regular crosswords.  For instance, in Appendix A.1.7, we have the true answer `UNDERMINED` being defined by `damaged`.  This would likely be too distant to be reasonable for a regular crossword, but the strength of the wordplay (the answer being literally given in the clue) is confirmation enough to satisfy solvers.  Setting this 'synonym distance hurdle' is an ongoing challenge.\n\nClearly, this analysis deserves more space than was available in the submitted version of the paper, and would provide valuable additional insight to those interested in the issues being tackled.  Thank you for highlighting the need for this quantitative analysis.", "claims_and_evidence": "Most claims are well-supported, particularly the state-of-the-art performance (32.5% Top-1 accuracy) on Cryptonite and the effectiveness of wordplay verification. However, the Python-based verifier has limitations (e.g., bypassing assertions), meaning correctness is not fully ensured. The claim of broader generalization is weak, as the study focuses only on cryptic crosswords. More analysis of failure cases and alternative tasks would strengthen the paper’s broader impact.", "methods_and_evaluation_criteria": "Yes, the methods and evaluation criteria are well-suited for cryptic crossword solving. However, there is no evaluation of the boarder application of the proposed method.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Log probability-based ranking ablations suggest verification matters, but the study lacks detailed error analysis on failure cases.\nThe paper would benefit from qualitative examples of incorrect reasoning that passes verification.\n\nThe verifier can be bypassed via faulty assertions, but there is no analysis of how often verification fails or of its impact on final accuracy.", "relation_to_broader_scientific_literature": "The paper proposes an LLM-based and program-aided approach to significantly improve machine performance on the cryptic crossword solving task.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t6tkpMbUcQ", "reviewer": "Reviewer_FiS7", "summary": "This paper proposes a reasoning-based approach to solving cryptic crossword puzzles, integrating large language models (LLMs) with Python formal verification.\nThe system generates answer candidates, derives wordplay explanations, and translates them into verifiable Python code for validation.\nIt achieves a new state-of-the-art (SOTA) performance on the Cryptonite dataset.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer highlighting the identified limitations of our Python-based verification system. To clarify, these shortcomings were not overlooked, but rather explicitly discussed in Section 4.5 (\"Known Limitations of the System\") to ensure transparency.  Presenting these potential 'shortcuts' was a deliberate choice to enhance the paper's rigor and openness, crucial for fostering further research – especially as it may serve as a cautionary note regarding the potential for a Reinforcement Learning loop (in future work).\n\nAs to the reviewer's point about the challenges posed by complex and unconventional cryptic clues, while the paper doesn't explicitly claim that the LLM struggles with these in a way that fundamentally limits generalizability, we acknowledge that cryptic crosswords, by their very nature, present a spectrum of complexity, and some clues are indeed more challenging than others.\n\nAlthough we haven't explicitly tested generalizability to all broader NLP tasks, we believe the developed techniques – particularly formal verification and iterative refinement – offer valuable insights for improving LLM reasoning in complex NLU scenarios.  Future research could fruitfully explore applying these techniques to other reasoning-intensive NLP tasks.\n\nWe sincerely appreciate the reviewer's valuable feedback, which has allowed us to clarify these key points. Having addressed the verifier limitations and the scope of generalizability, we respectfully request that the reviewer re-evaluate the paper's rating based on this improved understanding of our work.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "No Theoretical Claims", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "The paper  contributes to the intersection of natural language understanding (NLU), reasoning, and programmatic verification within the domain of cryptic crossword solving.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Strengths:\n- The approach introduces Python formal verification, ensuring that each solution's reasoning process is transparent and verifiable. By converting wordplay explanations into executable Python code, the system eliminates the \"black-box\" problem of traditional LLMs.\n- The study demonstrates that fine-tuned open-source models (Gemma2-9B) can outperform proprietary models like GPT-4o in complex reasoning tasks. This makes the solution cost-effective, locally deployable, and a strong alternative for NLP applications.\n\nWeaknesses:\n- The Python-based verification system has weaknesses, such as bypassing assert statements or generating logically inconsistent proofs. This reduces reliability and may lead to incorrect solutions being accepted.\n- The approach is tailored for cryptic crossword solving, and its effectiveness in broader NLP reasoning tasks remains unproven. Additionally, the LLM struggles with highly complex or unconventional clues, limiting its generalizability.", "other_comments_or_suggestions": "None", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rbqjEVKdoL", "reviewer": "Reviewer_9S8J", "summary": "This paper presents a multi-stage pipeline approach to solving cryptic crossword clues, focusing on reasoning through the wordplay mechanisms that make these puzzles challenging. The system consists of:\n\n1. A candidate answer generator (fine-tuned Gemma2-9B model)\n2. A wordplay suggestion generator (fine-tuned Gemma2-9B model)\n3. A formalizer that converts wordplay into verifiable Python code (using either Gemini-Flash or Gemma2-9B)\n4. A purpose-built verifier that evaluates the Python code and provides feedback for refinement\n\nThe key methodological contribution lies in the design of the Formaliser and verification system. This framework systematically decomposes model-generated solutions into verifiable sub-tasks through Python scripts, where each verification component either leverages LLM reasoning (via the `is_synonym` and `is_homophone_functions` functions) or employs traditional retrieval methods (via the `action_type` and `is_abbreviation` functions). This verification-based approach enables interpretable reasoning and provides a mechanism for iterative improvement of solutions.\n\nThe authors evaluate their system on the Cryptonite benchmark dataset and achieve 32.5% Top-1 accuracy with their Gemini-Flash Formaliser, outperforming GPT-4o (27.6%) on the same test samples. Additionally, they demonstrate that open-source models can achieve competitive performance (29.5%) when used throughout the pipeline.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Statistical Significance of Improvement over GPT-4o\n\nWe acknowledge the reviewer's point regarding the margin of error.  Since the improvement over GPT-4o on our sampled test set was within the simple margin of error, we performed the Bayesian IRT analysis presented in the paper that suggests a high probability (92%) that Gemini-Flash Formaliser is indeed better than GPT-4o. Beyond numerical improvements, a key contribution is the verifiable reasoning process itself, offering interpretability not available in black-box models.\n\n\n### Contribution of Formalization/Verification Components\n\nWe agree that demonstrating the isolated numerical contribution of the formalization and verification steps is challenging.  However, the ablation studies (Table 1 - AB lines) clearly show that removing these components significantly degrades performance, indicating they are not merely incremental but essential for the system's success.  These steps provide crucial interpretability and enable systematic error correction through feedback, which fine-tuning alone cannot achieve.  The focus is not solely on maximizing Top-1 accuracy, but on reasoned solutions.\n\n\n### Lack of Timing/Cost Metrics\n\nWe appreciate the reviewer's point about timing and computational cost.  Quantitatively comparing proprietary API-based models (like GPT-4o) with local open-source models on FLOPs is problematic due to the unknown parameter count/architecture of the proprietary models.  Moreover, the proprietary models are likely run on very capable hardware - so wall-clock timing comparisons also do not make much sense.  We mention the total cost was under $100 USD to highlight the feasibility and accessibility of our approach, especially using open-source models.  That being said, our research prioritized reasoning and interpretability over raw speed : Generating multiple candidate answers, followed by multiple wordplay samples, can been framed as inference-time computation (using 9B models) rather than using a large proprietary model.  \n\n\n### Limited Task Domain\n\nWe recognize the domain-specificity of cryptic crossword solving.  However, we argue that it serves as a rigorous and complex testbed for reasoning, requiring multi-faceted language understanding and logic.  While the specific DSL is tailored to crosswords, the underlying principles of our approach – decomposition, formalization, and verification with feedback – are intended to be more broadly applicable to other reasoning tasks.  Cryptic crosswords, with their clearly defined rules and solutions, allow for precise evaluation and iterative refinement of these principles.\n\n\n### Missing Reference (ToRA)\n\nWe thank the reviewer for pointing out the relevance of tool-integrated reasoning frameworks and specifically for suggesting the ToRA paper [Gou et al., 2024].  We agree that our work shares conceptual similarities with this approach, and we will add a discussion of ToRA and cite this paper in the related work section.  A key novelty in our approach is the use of Python itself as the \"tool\" for verification within an NLP task, enabling a flexible and interpretable verification process without relying on pre-defined formal DSLs for the entire reasoning chain.\n\n\n### Single Dataset Testing (Cryptonite)\n\nWe acknowledge that testing is primarily on Cryptonite.  Appendix A.3 details the rationale for choosing Cryptonite over the Guardian dataset [Rozner et al., 2021], including dataset size; source ('gold standard' Times/Telegraph clues); and the consistent use of Cryptonite's train/val/test splits for focusing on reasoning vs the Wordplay Dataset.  \n\n\nWe believe our rebuttal addresses the insightful points raised in your review and provides important clarifications regarding the statistical significance of our results, the contribution of the formalization and verification components, the practical considerations of timing and cost, the domain-specificity of our approach, and the connection to tool-integrated reasoning frameworks. We would be grateful if you would consider re-evaluating your rating in light of these responses.  Thank you again for your time and constructive feedback, which has been invaluable in improving our work.", "claims_and_evidence": "- The paper's claim about providing verifiable reasoning procedure for cryptic clue solutions is well-supported through examples and is a clear strength of the approach.\n- The paper's primary claim of achieving state-of-the-art results on the Cryptonite dataset is supported by experimental evidence, but the claimed improvements over GPT-4o (32.5% vs 27.6%) may be within the margin of error (stated as ±3.3% at 200 samples). \n- While the experimental results support the conclusion that fine-tuned open-source models with less parameters can achieve competitive performance, the contribution of the Python formalization and verification components is less clearly established. The improvement from the Formaliser over the fine-tuned models is not particularly substantial and may fall within the margin of error, failing to conclusively validate the additional mechanisms' contribution beyond fine-tuning.", "methods_and_evaluation_criteria": "### Methods:\nThe paper's method (especially the formalisation and verification parts) is derived from observation and emulation of human solver processes for cryptic crossword clue solving. By explicitly modeling how humans parse, reason through, and verify cryptic clues, the authors have designed a specialized verification workflow tailored to this specific task. The systematically decomposition aligns well with the inherent structure of cryptic puzzle solving.\n\n### Evaluation Criteria: \nThe basic metrics are appropriate for the task at hand:\n- Using the Cryptonite dataset is justified as it represents a standard benchmark in the field\n- The Top-1 exact match accuracy is an appropriate primary metric for cryptic crossword solving\n- The \"Partial Correctness Metrics\" in the appendix provide additional insights into system performance with varying levels of letter hints", "theoretical_claims": "The paper makes no theoretical claims.", "experimental_designs_or_analyses": "The paper fails to provide comprehensive timing and computational cost metrics. The formalization process substantially increases the pipeline's complexity, requiring additional model calls, verification steps, and potential rewrites. Without detailed per-step and end-to-end timing data, it is impossible to fairly compare this approach against simpler, more direct baselines. While the authors mention total costs under $100 USD, they do not break down the inference time or computational requirements at each stage and each inference experiment of their pipeline, making it difficult to assess the practical efficiency trade-offs between accuracy gains and increased computational overhead.", "relation_to_broader_scientific_literature": "Pros: The approach draws on the Draft, Sketch, and Prove frameworks, taking a decomposition-then-formal verification approach. The analogy is apt, and the paper provides valuable insights into designing verification functions without a formal DSL for verification. To some extent, it is inspiring for addressing key challenges in current AI research in the code and math domain, especially in developing O1-style reinforcement learning training methods where verification is critical.\n\nCons: However, current implementations are limited to relatively restricted task domains. Whether this approach can be generalized to more complex scenarios—especially those where human behavior is not easily observable or where specialized tools and sufficiently powerful verification models are not available—remains an open question. The broader applicability of this verification-centric approach to less structured reasoning tasks warrants further exploration.", "essential_references_not_discussed": "While the paper effectively connects to literature on mathematical reasoning and code verification, there's a potential connection to tool-integrated reasoning frameworks that isn't discussed. The paper frames its approach primarily in relation to Draft, Sketch, and Prove frameworks, but the methodology of using Python as an external verification mechanism for natural language reasoning problems also shares conceptual similarities with tool-integrated reasoning approaches. For example, the recent works in mathematical problem solving fields like [1] present many framework where language models leverage external tools to enhance reasoning capabilities. \n\n[1] Gou, Z., et al. (2024). ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving. https://arxiv.org/abs/2309.17452", "other_strengths_and_weaknesses": "### Strengths:\n- The paper provides good coverage of cryptic crossword conventions and mechanisms\n- The step-by-step formalization of cryptic reasoning and the feedback mechanism for refining formalized proofs is well-designed\n\n### Weaknesses:\n- (Minor) Testing is limited to a single dataset (Cryptonite) rather than exploring transferability to other crossword sources", "other_comments_or_suggestions": "No other comments.", "questions_for_authors": "No other questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper argues for the use of UK-style cryptic crossword clues as a source of test material for reasoning in AI models.
The case is well made:
- the volume of examples is considerably higher than other sources e.g. math olympiad, and new examples are produced daily by major news sources;
- the type of reasoning required is somewhat orthogonal to mathematics, but is certainly considered to require human-level skills
- the agentic verifier introduced is interesting, generating verifiable Python code.

Criticisms:
- Restricted to one source of examples.  The paper and appendix argue in terms of dataset leakage and dataset size that only the Cryptonite benchmark (520K examples) should be used, and not the Guardian one (140K); but many of these arguments apply more to the fine-tuning scenario, and could be otherwise overcome.  The argument is lyrical ("reluctance to dataset-hop"), but again this is refuted by paper's observation that new examples are produced at the rate of perhaps thousands per year.  If number of examples is a metric, then the union of datasets is larger than any single one.
It is notable that the appendix observes that "the Guardian clues can often be rather haphazard".  It is clear that such haphazard clues will be less suitable for the python verifier in this paper, so that scores would be expected to be lower.  The other arguments are valid, but the relative difficulty should be addressed more directly than as one sentence in a page of reasoning.  The absence of across/down annotation is again simply a contributor to difficulty, not a fundamental stumbling block.

- Comparison to reasoning models.  Models such as OpenAI 01 are dismissed in lines 206-214, without any evidence of an attempt to empirically test them.  When brought up by a reviewer, this point was ignored.  This is an important comparison - even if those models are better than the proposed one, the proposed work is valuable - but ignoring them or trying to argue them away weakens this paper.

A non-criticism is the failure to provide cost metrics - as the authors present the dollar cost of less than $100, this is sufficient to argue that as a research activity, this is not cost-prohibitive.  As emphasized by the authors, the objective is to discover frontiers of reasoning ability more than to create a low-cost system to solve crossword clues.
Decision: Accept (poster)