Decrypting Cryptic Crosswords: Semantically Complex Wordplay Puzzles as a Target for NLP

1 Introduction . Modern computational models have made great progress at handling a variety of natural language tasks that require interpreting rich syntactic and semantic structures [ 6 ; 30 ; 25 ; 32 ] . However , in NLP [ 2 ; 26 ; 3 ] as in other areas of AI [ 22 ] , machines still lag humans on tasks that require flexible problem solving , rapid learning of unseen tasks , and generalization to new domains . Just as complex games mastered by human experts , such as chess , Go , and video games , have proved a fertile ground for developing more flexible AI [ 36 ; 35 ; 27 ] , we propose that creative language games are a rich area for developing more flexible NLP models . In particular , we argue that linguistic tasks involving metalinguistic reasoning pose an important and significant challenge for state-of-the-art computational language systems . One such domain is cryptic crossword puzzles . Cryptics are the most popular style of crossword in the United Kingdom and appear in major newspapers like The Times and The Guardian . Cryptic clues have two parts : a definition and a wordplay cipher that , when placed adjacent to each other , read like fluent natural language . For example , consider this NLP-centric cryptic crossword clue : “ But everything ’ s really trivial , initially , for a transformer model ( 4 ) ” with answer “ BERT ” . “ ( 4 ) ” is the enumeration and specifies the number of letters in the answer . Solvers must identify which 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . part of the clue is definition and which is wordplay . The definition should be a semantically valid description of the answer word : “ a transformer model ” can mean “ BERT ” . The wordplay part is the remainder of the clue : “ But everything ’ s really trivial , initially ” . The word initially in this context is used as an indicator , which tells the solver to take the initial letter of each of the preceding words ( but everything ’ s really trivial ) to produce “ BERT ” . Because both the wordplay and the definition give the same 4-letter word ( which is what the enumeration , “ ( 4 ) ” , calls for ) , we can be confident that “ BERT ” is the correct answer . The clue just discussed is an example of the initialism clue type , which is one of roughly 15 major clue types . Other types can require anagramming words , finding words hidden in longer phrases , performing synonym substitutions , substituting a word for a soundalike ( e.g. , “ hiring ” for “ high ring ” ) , or composing a number of these manipulations . See Table 1 for examples of several other clue types with answer “ BERT ” and Appendix A for examples of actual clues from the dataset . As with American-style clues , cryptics require world knowledge and linguistic flexibility to match the definition , but also considerable attention to meta-linguistic concepts to solve the wordplay . In this paper we study the language task of solving individual cryptic clues , rather than full puzzles since , unlike American-style crosswords , cryptics generally have unique answers and so are less reliant on grid constraints in order to achieve a complete solution . While cryptics pose a challenge to novice solvers unfamiliar with their structure , experts flexibly combine world , domain-specific , and linguistic knowledge to solve novel clues . Experts know the rules that govern cryptic clues , but they also reason about them flexibly and apply them to solve novel clues . In the psychology literature , it has been claimed that cryptics depend on domain-general intelligence and are an ideal domain for studying the “ aha moment ” in humans [ 11 ; 10 ] because experts can solve them with high accuracy ( but rarely at first glance ) , they can be easily generated , and they require drawing on a diverse range of cognitive abilities . Therefore , we believe that cryptic crosswords are an excellent domain for developing computational language systems that “ learn and think like humans ” [ 22 ] , posing an interesting and important challenge for modern machine learning . Our main contributions are , first , a cleaned dataset of cryptic crosswords clues from theguardian.com , consisting of 142,380 clues from 5,518 puzzles over 21 years ; 1 and second , a novel curriculum learning approach , in which the model is first fine-tuned on related , synthetic tasks ( e.g. , an augmented word descrambling task ) before tackling actual cryptic clues . This method meaningfully improves on a standard T5 seq-to-seq approach and on the best model of Efrat et al . [ 7 ] —concurrent work that presents a similar dataset and similar neural baseline using T5 . In this paper , we aim not only to present a dataset and propose a novel solution but also to characterize the problem and motivate its importance . To that end , we elucidate the task with three non-neural baselines and fine-tune T5 [ 31 ] , a Transformer-based [ 38 ] encoder-decoder , as a neural baseline . Since the character-level wordplay inherent to cryptics might be challenging to language models with subword tokenization ( T5 uses SentencePiece [ 21 ] ) , we study whether T5 has or can acquire metalinguistic knowledge . In Section 6.1 we examine whether T5 learns meta-features of the task related to answer length . In Section 6.3 we use a descrambling task to assess whether T5 understands the character composition of words and whether the model can make use of linguistic and meta-linguistic information simultaneously . Our results show , perhaps surprisingly , that the subword-tokenized T5 model is quite robust to character-level challenges . Moreover , the descrambling task may serve as a useful benchmark task in guiding the development of new approaches on the cryptics task . Given the compositional nature of cryptic clues , we investigate the extent to which the model generalizes under increasingly difficult data splits . In Section 6.2 we introduce a new form of disjoint data split to address T5 ’ s robustness to inflection : a word-initial disjoint split that segments clue– answer pairs based on the first two letters of the answer . In Section 6.4 we examine the systematicity of the model ’ s answer generation by perturbing the wordplay portion of anagram clues , showing that its behavior is partially consistent with human solving strategies . Although our novel curricular approach considerably improves performance on this task , fully solving cryptics remains a challenge for modern machine learning , with expert humans still outperforming machines . Therefore , we hope that our dataset will serve as a challenging benchmark for future work . 2 Related work . While there is an existing literature on puns and wordplay in NLP [ 17 ; 20 ; 24 ] as well as on solving American-style crosswords [ 13 ; 23 ; 34 ] , there has been relatively little work using NLP methods on cryptics , which require processing highly compositional language and disambiguating surface-level from hidden meaning . Hart and Davis [ 16 ] laid out a computational framework for solving the problem , and Hardcastle [ 14 , 15 ] proposed some rule-based solutions . The app Crossword Genius from Unlikely AI solves cryptic clues and gives human-understandable explanations . Because its method is proprietary and not available for open testing , we do not report it as a baseline but note that it is competitive . Deits [ 5 ] offers a rule-based solution , which can also output explanations , and which we test on our dataset . Most recently , in work concurrent to ours , Efrat et al . [ 7 ] release a similar dataset of cryptic crossword clues and present a neural baseline using T5-large . In Section 6.5 , we discuss how our new data split and curricular approach improve on their work . Our curricular approach fits into the space of recent work on pre-finetuning [ 37 ; 1 ] and curricular approaches for compositional tasks [ 12 ; 40 ] . Our approach is loosely related to the pre-finetuning of Aghajanyan et al . [ 1 ] but differs in that our curriculum is composed of fewer tasks that are all closely related to the primary task and synthetically generated . Our approach resembles Geva et al . [ 12 ] in that we attempt to endow language models with a specific kind of reasoning by training in a multi-task setup over synthetic data . In the vein of Wu et al . [ 40 ] , which trains on synthetic datasets to encode inductive bias , our approach can be understood as encoding wordplay functional biases . Whether large pretrained Transformer models generally pick up on meta-linguistic features like word length and the character composition of words is an open question . Brown et al . [ 4 ] explore a set of restricted word unscrambling tasks in the few-shot ( versus fine-tuned ) setting for GPT-3 . Probing results from Itzhak and Levy [ 18 ] suggest that information about the character composition of words is present in the embeddings of pretrained , subword-tokenized models . This seems to 1We release the dataset along with all code to reproduce the results in this paper at https : //github.com/ jsrozner/decrypt . confirm our result that subword-tokenized models do have more knowledge of character composition and meta-properties of words than we might have expected . 3 Dataset and task . We present a cleaned dataset of 142,380 cryptic crossword clues from 5,518 puzzles published in The Guardian from July 1999 to October 2020 . We also introduce a challenging “ word-initial ” disjoint split after observing that T5 is robust to inflection . Overall , the dataset has 55,783 unique answers , giving a mean frequency of 2.55 clues per unique answer . Answers in the dataset consist of one to six words , with most ( 97.4 % ) having one ( 83.3 % ) or two ( 14.1 % ) words . Full details of dataset preprocessing are given in Appendix B , and in addition to releasing the full dataset , we include code to fully replicate our data download , pre-processing pipeline , and split generation in the repository . 3.1 Task . We frame the problem as a standard seq-to-seq task from inputs ( clues with length enumeration ) to outputs ( answers ) . For example , one input could be But everything ’ s really trivial , initially , for a transformer model ( 4 ) , with output BERT . This is consistent with how clues are presented to human solvers . Full details of the input–output specification for each model are provided in Appendix C , along with other experimental details . 3.2 Splits . As motivated in the introduction ( and in Section 6.2 ) , we consider three splits . The naive ( random ) split is a standard 60/20/20 random split into train , dev , and test . The disjoint split ensures that all clues with the same answer appear in only one of train , dev , or test . For example , if any clue for which the answer is “ BERT ” is in the train set , then all clues for which the answer is “ BERT ” are in the train set . The disjoint split is used to test composition and generalization . It prevents an approach that relies only on lexical similarity across clues ( like KNN ) from succeeding on the task . Finally , the word-initial disjoint split is designed to address T5 ’ s robustness to inflection . For this split , we enforce that all clues whose answers start with the same two letters will appear in the same set . For example , all clues that have answers starting with ‘ ab ’ like “ abacus , ” “ abdicate , ” “ abdicates , ” will be grouped , ensuring that inflections or spelling variations of the same base word occur in a single split .

Summary: This paper presents a new dataset of cryptic crossword clues. The work presents an extensive evaluation study on this dataset with 3 non-neural baselines and one neural baseline, namely the T5 transformer-based model. The authors investigate as well a kind of pre-finetuning stage (a phase of finetuning before the final finetuning stage on the main dataset) as a way to improve the performance of the model on this task.  Contributions: New dataset of cryptic crossword clues that has the potential to foster more research on complex reasoning tasks  Strengths: - The writing is compelling and the paper is a very enjoyable read - The author do a detailed analysis to understand better the limitations and abilities of transformer-based models on the task, including interesting ways of looking at data splits - The authors investigate the use of a pre-finetuning stage as a way to improve the performance of the T5 model on the task - The evaluation is convincing with multiple baselines, non-neural and neural.  Weaknesses: Sometimes, the paper seemed to be a bit condensed