Decrypting Cryptic Crosswords: Semantically Complex Wordplay Puzzles as a Target for NLP

1 Introduction . Modern computational models have made great progress at handling a variety of natural language tasks that require interpreting rich syntactic and semantic structures [ 6 ; 30 ; 25 ; 32 ] . However , in NLP [ 2 ; 26 ; 3 ] as in other areas of AI [ 22 ] , machines still lag humans on tasks that require flexible problem solving , rapid learning of unseen tasks , and generalization to new domains . Just as complex games mastered by human experts , such as chess , Go , and video games , have proved a fertile ground for developing more flexible AI [ 36 ; 35 ; 27 ] , we propose that creative language games are a rich area for developing more flexible NLP models . In particular , we argue that linguistic tasks involving metalinguistic reasoning pose an important and significant challenge for state-of-the-art computational language systems . One such domain is cryptic crossword puzzles . Cryptics are the most popular style of crossword in the United Kingdom and appear in major newspapers like The Times and The Guardian . Cryptic clues have two parts : a definition and a wordplay cipher that , when placed adjacent to each other , read like fluent natural language . For example , consider this NLP-centric cryptic crossword clue : “ But everything ’ s really trivial , initially , for a transformer model ( 4 ) ” with answer “ BERT ” . “ ( 4 ) ” is the enumeration and specifies the number of letters in the answer . Solvers must identify which 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . part of the clue is definition and which is wordplay . The definition should be a semantically valid description of the answer word : “ a transformer model ” can mean “ BERT ” . The wordplay part is the remainder of the clue : “ But everything ’ s really trivial , initially ” . The word initially in this context is used as an indicator , which tells the solver to take the initial letter of each of the preceding words ( but everything ’ s really trivial ) to produce “ BERT ” . Because both the wordplay and the definition give the same 4-letter word ( which is what the enumeration , “ ( 4 ) ” , calls for ) , we can be confident that “ BERT ” is the correct answer . The clue just discussed is an example of the initialism clue type , which is one of roughly 15 major clue types . Other types can require anagramming words , finding words hidden in longer phrases , performing synonym substitutions , substituting a word for a soundalike ( e.g. , “ hiring ” for “ high ring ” ) , or composing a number of these manipulations . See Table 1 for examples of several other clue types with answer “ BERT ” and Appendix A for examples of actual clues from the dataset . As with American-style clues , cryptics require world knowledge and linguistic flexibility to match the definition , but also considerable attention to meta-linguistic concepts to solve the wordplay . In this paper we study the language task of solving individual cryptic clues , rather than full puzzles since , unlike American-style crosswords , cryptics generally have unique answers and so are less reliant on grid constraints in order to achieve a complete solution . While cryptics pose a challenge to novice solvers unfamiliar with their structure , experts flexibly combine world , domain-specific , and linguistic knowledge to solve novel clues . Experts know the rules that govern cryptic clues , but they also reason about them flexibly and apply them to solve novel clues . In the psychology literature , it has been claimed that cryptics depend on domain-general intelligence and are an ideal domain for studying the “ aha moment ” in humans [ 11 ; 10 ] because experts can solve them with high accuracy ( but rarely at first glance ) , they can be easily generated , and they require drawing on a diverse range of cognitive abilities . Therefore , we believe that cryptic crosswords are an excellent domain for developing computational language systems that “ learn and think like humans ” [ 22 ] , posing an interesting and important challenge for modern machine learning . Our main contributions are , first , a cleaned dataset of cryptic crosswords clues from theguardian.com , consisting of 142,380 clues from 5,518 puzzles over 21 years ; 1 and second , a novel curriculum learning approach , in which the model is first fine-tuned on related , synthetic tasks ( e.g. , an augmented word descrambling task ) before tackling actual cryptic clues . This method meaningfully improves on a standard T5 seq-to-seq approach and on the best model of Efrat et al . [ 7 ] —concurrent work that presents a similar dataset and similar neural baseline using T5 . In this paper , we aim not only to present a dataset and propose a novel solution but also to characterize the problem and motivate its importance . To that end , we elucidate the task with three non-neural baselines and fine-tune T5 [ 31 ] , a Transformer-based [ 38 ] encoder-decoder , as a neural baseline . Since the character-level wordplay inherent to cryptics might be challenging to language models with subword tokenization ( T5 uses SentencePiece [ 21 ] ) , we study whether T5 has or can acquire metalinguistic knowledge . In Section 6.1 we examine whether T5 learns meta-features of the task related to answer length . In Section 6.3 we use a descrambling task to assess whether T5 understands the character composition of words and whether the model can make use of linguistic and meta-linguistic information simultaneously . Our results show , perhaps surprisingly , that the subword-tokenized T5 model is quite robust to character-level challenges . Moreover , the descrambling task may serve as a useful benchmark task in guiding the development of new approaches on the cryptics task . Given the compositional nature of cryptic clues , we investigate the extent to which the model generalizes under increasingly difficult data splits . In Section 6.2 we introduce a new form of disjoint data split to address T5 ’ s robustness to inflection : a word-initial disjoint split that segments clue– answer pairs based on the first two letters of the answer . In Section 6.4 we examine the systematicity of the model ’ s answer generation by perturbing the wordplay portion of anagram clues , showing that its behavior is partially consistent with human solving strategies . Although our novel curricular approach considerably improves performance on this task , fully solving cryptics remains a challenge for modern machine learning , with expert humans still outperforming machines . Therefore , we hope that our dataset will serve as a challenging benchmark for future work . 2 Related work . While there is an existing literature on puns and wordplay in NLP [ 17 ; 20 ; 24 ] as well as on solving American-style crosswords [ 13 ; 23 ; 34 ] , there has been relatively little work using NLP methods on cryptics , which require processing highly compositional language and disambiguating surface-level from hidden meaning . Hart and Davis [ 16 ] laid out a computational framework for solving the problem , and Hardcastle [ 14 , 15 ] proposed some rule-based solutions . The app Crossword Genius from Unlikely AI solves cryptic clues and gives human-understandable explanations . Because its method is proprietary and not available for open testing , we do not report it as a baseline but note that it is competitive . Deits [ 5 ] offers a rule-based solution , which can also output explanations , and which we test on our dataset . Most recently , in work concurrent to ours , Efrat et al . [ 7 ] release a similar dataset of cryptic crossword clues and present a neural baseline using T5-large . In Section 6.5 , we discuss how our new data split and curricular approach improve on their work . Our curricular approach fits into the space of recent work on pre-finetuning [ 37 ; 1 ] and curricular approaches for compositional tasks [ 12 ; 40 ] . Our approach is loosely related to the pre-finetuning of Aghajanyan et al . [ 1 ] but differs in that our curriculum is composed of fewer tasks that are all closely related to the primary task and synthetically generated . Our approach resembles Geva et al . [ 12 ] in that we attempt to endow language models with a specific kind of reasoning by training in a multi-task setup over synthetic data . In the vein of Wu et al . [ 40 ] , which trains on synthetic datasets to encode inductive bias , our approach can be understood as encoding wordplay functional biases . Whether large pretrained Transformer models generally pick up on meta-linguistic features like word length and the character composition of words is an open question . Brown et al . [ 4 ] explore a set of restricted word unscrambling tasks in the few-shot ( versus fine-tuned ) setting for GPT-3 . Probing results from Itzhak and Levy [ 18 ] suggest that information about the character composition of words is present in the embeddings of pretrained , subword-tokenized models . This seems to 1We release the dataset along with all code to reproduce the results in this paper at https : //github.com/ jsrozner/decrypt . confirm our result that subword-tokenized models do have more knowledge of character composition and meta-properties of words than we might have expected . 3 Dataset and task . We present a cleaned dataset of 142,380 cryptic crossword clues from 5,518 puzzles published in The Guardian from July 1999 to October 2020 . We also introduce a challenging “ word-initial ” disjoint split after observing that T5 is robust to inflection . Overall , the dataset has 55,783 unique answers , giving a mean frequency of 2.55 clues per unique answer . Answers in the dataset consist of one to six words , with most ( 97.4 % ) having one ( 83.3 % ) or two ( 14.1 % ) words . Full details of dataset preprocessing are given in Appendix B , and in addition to releasing the full dataset , we include code to fully replicate our data download , pre-processing pipeline , and split generation in the repository . 3.1 Task . We frame the problem as a standard seq-to-seq task from inputs ( clues with length enumeration ) to outputs ( answers ) . For example , one input could be But everything ’ s really trivial , initially , for a transformer model ( 4 ) , with output BERT . This is consistent with how clues are presented to human solvers . Full details of the input–output specification for each model are provided in Appendix C , along with other experimental details . 3.2 Splits . As motivated in the introduction ( and in Section 6.2 ) , we consider three splits . The naive ( random ) split is a standard 60/20/20 random split into train , dev , and test . The disjoint split ensures that all clues with the same answer appear in only one of train , dev , or test . For example , if any clue for which the answer is “ BERT ” is in the train set , then all clues for which the answer is “ BERT ” are in the train set . The disjoint split is used to test composition and generalization . It prevents an approach that relies only on lexical similarity across clues ( like KNN ) from succeeding on the task . Finally , the word-initial disjoint split is designed to address T5 ’ s robustness to inflection . For this split , we enforce that all clues whose answers start with the same two letters will appear in the same set . For example , all clues that have answers starting with ‘ ab ’ like “ abacus , ” “ abdicate , ” “ abdicates , ” will be grouped , ensuring that inflections or spelling variations of the same base word occur in a single split .

Summary: Authors 1) provide a dataset of cryptic crosswords; 2) explore baselines performances on this dataset; 3) provide interesting insights on those performances and come up with a pretraining strategy that increases performances.  Briefly, cryptic crosswords are puzzles that require both mastering definitions and wordplay (i.e. reordering of letters, extracting letters, composing parts of several words to form a novel word, etc.) to be solved. Authors provide a large dataset of such puzzles (in english), that they introduce as a stepping stone to solve more complex real-life settings.  They show that carefully crafted non-neural baselines are not very good (either rule-based following previous work on this, or KNN) and that even T5 (state of the art text-to-text transformer pretrained on many tasks, such as translation, summarization, etc.) cannot be finetuned for optimal performances on this dataset. Worst, when considering adversarial splits, such that for instance all puzzles with answers starting with the same subword are found in the same split, T5 performances drop dramatically.  They hint at a solution for this problem via curriculum learning, and devise the following strategy: first pretrain T5 on a series of tasks resembling the final wordplays (e.g. unscrambling a word using its definition: "etalp + flower part" --> "petal") and only then finetune T5 on the actual cryptic crosswords dataset. T5 performances indeed increase, but authors note that they are far from human performances, or even acceptable performances for a machine.