Decrypting Cryptic Crosswords: Semantically Complex Wordplay Puzzles as a Target for NLP

Abstract: Cryptic crosswords, the dominant crossword variety in the UK, are a promising target for advancing NLP systems that seek to process semantically complex, highly compositional language. Cryptic clues read like fluent natural language but are adversarially composed of two parts: a definition and a wordplay cipher requiring character-level manipulations. Expert humans use creative intelligence to solve cryptics, flexibly combining linguistic, world, and domain knowledge. In this paper, we make two main contributions. First, we present a dataset of cryptic clues as a challenging new benchmark for NLP systems that seek to process compositional language in more creative, human-like ways. After showing that three non-neural approaches and T5, a state-of-the-art neural language model, do not achieve good performance, we make our second main contribution: a novel curriculum approach, in which the model is first fine-tuned on related tasks such as unscrambling words. We also introduce a challenging data split, examine the meta-linguistic capabilities of subword-tokenized models, and investigate model systematicity by perturbing the wordplay part of clues, showing that T5 exhibits behavior partially consistent with human solving strategies. Although our curricular approach considerably improves on the T5 baseline, our best-performing model still fails to generalize to the extent that humans can. Thus, cryptic crosswords remain an unsolved challenge for NLP systems and a potential source of future innovation.


Reviews and author responses:

 Thank you for your responses.  I did not initially observe quite how contemporaneous this work and the Efrat et al paper were.  In light of that my review's criticism of this manuscript was overly negative regarding the novelty of its contribution.  I've updated my score to reflect this and apologize for the oversight.  

This paper describes a new corpus of cryptic crossword clues as a challenging NLP task.  Cryptic crossword clues are composed of a literal (synonym) clue and wordplay.  Cryptic wordplay has a conventionalized inventory of ways to clue an answer.  This is a compelling NLP challenge ``1) because it is hard and 2) because it requires a system to process both surface, syntactic and semantic content of the examples.  This is related to being able to understand statements like "Al and Bob got married and bought a house together, but not in that order".  The paper includes a curriculum training approach where a T5 model is presented with data to improve cryptic clue solving.  The corpus and task is a compelling contribution.  The specifics of the curriculum training are interesting but it is less obvious what their broader contribution will be. A more substantial distinction from the Efrat approach and data is necessary here.  This limits the contribution of this work, since it is not the first dataset or the first application of T5 to this task.  This indicates that the most substantial contribution is the curricular approach training.

While this is an interesting and challenging task, it is not clear that there is substantial novelty in this work.  The corpus, while valuable, is not the first corpus of cryptic clues nor is it clearly preferred over the Efrat corpus.  The T5 approach has been explored previously.  The major contribution is a training strategy to expose the model to surface manipulations that may help answer certain cryptic clues.  A more focused venue might be more appropriate for this work. A more thorough analysis of the types of wordplay that are addressed via the training approach would strengthen the presentation.

Line 52: typo around the Efrat citation?

Section 5.1: the descramble and anagrams appear to be most suited to anagram type clues.  Are the clues in the data set labeled by wordplay type?  If so, does this training help on other clues as well, or only anagrams?

Section 6.5: is there any reason to not explore the superset of the two corpora together?  Is there anything substantially different between the two? There is not, to my knowledge, negative societal impact to this work.  Though, of course, bias in clueing can be reflected in the model.

This paper provides a new question-answering benchmark composed of cryptic crossword clues and describes a curriculum learning-based approach to this task.   **Review Criteria**

Originality: The proposed task is new and interesting to NLP practitioners, primarily discussed only in the contemporaneous work by Efrat, et al. (2021). Cryptic crossword clues contain a mixture of linguistic and meta-linguistic phenomena 

Quality: The paper is technically sound, with convincing baselines (e.g., KNN BoW) and targeted experiments (e.g., Sections 6.3-6.4). The paper includes a thorough description of the task domain and motivates future work on cryptic crosswords.

Clarity: The paper clearly describes cryptic crosswords with illustrative examples but would benefit from additional comparison to American-style crosswords, which many readers may be more familiar with. The technical approach is clearly described and replicable. 

Significance: In my opinion, the proposed task is likely more significant than the proposed curriculum learning approach. Cryptic crossword clues are quite different from standard question-answering benchmarks and may require completely new architectural approaches (e.g., due to segmentation and meta-linguistic reasoning, cf. Section 6.3) from those used in traditional NLP tasks. This is perhaps different from NLU benchmarks like GLUE and SuperGLUE, which track progress in NLP without necessarily requiring specialized model architectures. Additional discussion along these lines would be appreciated.

**Notes on Individual Sections**

Introduction:
 - Maybe worth clarifying that this paper focuses only on questions, rather than full puzzles
 - Table 1 is hilarious; I love it. I would normally recommend including real clues rather than synthetic ones, but I think these clues sufficiently convey the conventions of cryptic crosswords. It might also be worth discussing that even though cryptic crossword clues often look impossibly difficult to people who have never seen them before, there's a relatively steep learning curve, such that knowledge of just a few conventions and their associated indicators can make most clues relatively tractable. [This is in contrast to American-style crossword clues, which can mostly be understood without much domain knowledge.]

Related Work:
 - Probably worth discussing relationship to open-domain question answering
 - When comparing to American-style crosswords, it might be worth mentioning that solving cryptic crosswords requires additional focus on language and less on search (if you think this is true). Specifically, because cryptic crosswords typically have more black squares than e.g., a  standard NYT puzzle, more clues are going to have unique answers. They may also rely less on trivia/general knowledge, etc.

Dataset:
 - Currently, the paper does not include a dataset release but instead provides code for scraping clue-answer pairs from The Guardian website. Releasing the raw data would be better for future-proofing if there aren’t licensing restrictions. I don’t know about The Guardian specifically, but normally full puzzles (including the grid) are copyrighted, but individual clues aren’t. This is why e.g., the OTSYS dataset (see [11]) is permitted.
 - Could you report other metrics like MRR or graphs of accuracy@k? While this is perhaps less true for cryptic crosswords vs. American-style crosswords, it’s probably worth noting that top-1 accuracy (or even top-10) is not a great metric for domains with non-unique answers?

Baselines:
 - 147: “an inherent limitation of this approach is that it is difficult to enumerate all possibilities” -> I wonder if you could discuss this further? Undoubtedly there is a long tail of linguistic tricks, knowledge, wordplay, etc., but is it really impossible to enumerate all the cryptic crossword conventions? I am not an expert on cryptics by any means, but I have solved a few, and I was under the impression that there is a relatively small number of clue types (maybe 20-30?) with conventionally associated indicators such as “hear” (which indicates homophony).
 - I don't have a great intuitive sense of what the Deits solver is doing from this relatively short description. How do the hand-coded grammar and the WordNet components interact? Is one the generator and the other used as a reranker, or is something more nuanced going on?
 - 167: “leaves considerable room for improvement on the task” -> I would certainly expect that specialized approaches could outperform these numbers, but it’s worth discussing that, especially for top-1 accuracy, the performance ceiling is probably quite a bit lower than 100%.
 - Section 4.5: clarify that these numbers are top-1 and top-10 accuracy respectively?

Curriculum Learning
 - If space permits, maybe include some very brief dataset statistics for ACW? 
 - "American crossword clues are usually straightforward synonyms or definitions" -> can you justify this claim? [I don't think this is true, but there is a domain of "general knowledge" crosswords for which it might be, cf. Hill, et al. (2015) "Learning to Understand Phrases by Embedding the Dictionary"]

Model Analysis:
 - Section 6.1: how do you specify answer length to the model? Just to make sure, this improves performance even after filtering for length, right? Does this happen only on multiword clues because there is additional information being provided that filtering can't account for? Or e.g., would you expect a similar effect on NYT puzzles where only the full answer length is given? 
 - SentencePiece tokenization (Sec 6.3) seems like it would be a problem for many of the meta-linguistic conventions in cryptic crosswords; should possibly be mentioned early in the paper, as a sign that the task might not just be a benchmark for current NLP models but instead require specialized architectures

**Nits** 
 - 13: “a a” -> “a”

UPDATES AFTER AUTHOR RESPONSE:
 - Thanks for the very thorough response! The updated information about answer length and tokenization is all very interesting.
 - I have increased my score to an 8 following the updated information about the dataset release.
 - Sorry about the confusion re: evaluation metrics! I had assumed that cryptic crossword clues did not generally have unique answers, akin to American-style crosswords. I think it would be helpful to mention this distinction in the paper as well.
 - I agree with other reviewers that Efrat et al. is concurrent and should not affect the decision of this work. Given that there was only one reviewer who supported rejection and their review was primarily based on similarity to Efrat et al., I think the paper should clearly be accepted.  I have two primary concerns about this work:

1. Dataset Availability and Licensing -- This paper's primary contribution is the proposal of a new task (cryptic crosswords) scraped from The Guardian. However, the authors only release code for scraping the dataset, rather than the data itself. This approach is not future-proof and runs the risk of e.g., link rot and should only be taken if absolutely necessary. Given the discussion of licensing in the checklist, it is not clear to me that the authors are not allowed to release individual clues. Could the authors please provide some additional clarification about this decision to not release the data directly? [In my experience, usually full puzzle grids but not individual clues are copyrighted, which is why the XD and OTSYS datasets are allowed to exist publicly.]

2. Evaluation Metrics -- American-style crossword clues do not have unique answers and rely on full puzzle grids to determine uniqueness. Although I am less familiar with UK-style crosswords, I expect the same rule generally holds (although to a lesser extent). Because of this, top-1 and even top-10 accuracy may be inappropriate evaluation metrics for clue-answering, and I recommend including (1) a discussion of whether clues generally have only one acceptable fill and (2) additional evaluation metrics like MRR. 

 This is the second of two response comments. The first comment appears below; please read it first.

## Model Analysis discussion

### Answer length (Section 6.1)

- First, regarding how we specify length to the model: We briefly describe this on lines 124-126 (Section 4): “for models that can interpret the answer-length enumeration as a textual token (KNN, T5), we append it to the input string (e.g., (4)). In this case, we report results both with and without appending the enumeration.” We also provide some additional details, including input-output examples in Appendix B. The Wordnet and Rule-based (Deits) baselines receive the length as an integer input and thus produce outputs only of the correct length, with filtering that is done “internal” to the algorithm. Because of this, we do not study these two models’ outputs when length is not “provided”. (See also the parenthetical comment on line 245: “The WordNet and rule-based approaches cannot ‘learn’ the meaning of the length enuemration, so we omit them from this discussion.”)

- Second, regarding the impact of length enumeration on performance:  Yes, providing the length enumeration improves performance even after filtering: Table 2 reports top-1 and top-10 accuracy after filtering for length (see caption). We see in that Table that including lengths (compare adjacent rows for KNN and T5) always improves performance, on all datasets, splits, and metrics. And providing length enumeration also improves performance without filtering (no surprise), which we see by comparing adjacent rows in Table 3b (sample metrics without filter).

- > “Does this happen only on multiword clues because there is additional information being provided that filtering can't account for? Or e.g., would you expect a similar effect on NYT puzzles where only the full answer length is given?”

    This result holds for both single and multiword clues: if we further break down the final row of Table 3b into single-word and multi-word for the test set, we observe the following:
    |Dataset | % sample contains answer| % outputs with correct length| % outputs correct word count |
    |---|---|---|---|
    |All (as reported in final row, Table 3b)| 28.1| 48.5| 97.9|
    |Single-word| 28.4| 48.4| 99.2|
    |Multi-word| 26.5| 48.8| 91.3|

    So overall, we see that the increase from 16.2% outputs with correct length to ~48% holds irrespective of segmentation. We also see that correct word count improves for both, but that the improvement is less for multi-word. This could be because there is less training data and the long tail of more-than-two-word answers is challenging. Remember also that there are far more single-word (83.3%) than two-word answers (14.1), and many more two-word than three-plus-word answers (2.6%) (see line 91).

    Thus, we should expect an improvement on American-style crosswords like NYT puzzles, though the distribution of number of words will play a role. There may also be more substantial systematic patterns in cryptics that clue the model to produce multiword answers, as compared to American style crosswords.

## SentencePiece tokenization and T5
Regarding SentencePiece tokenization, we agree with the reviewer’s intuition that this is a possible limitation of the T5 model and acknowledge it in our submission checklist (1b). However, an important issue is that there are not good character-level models with the breadth and complexity of T5 to handle the semantic complexity of the cryptic domain. Moreover, given that we shared the reviewer’s concerns, we studied the limitations of SentencePiece tokenization in Sections 6.1 (meta-linguistic properties like length) and 6.3 (character level manipulations like descrambling) and uncovered interesting evidence that T5 *can* actually handle certain kinds of character-level manipulation. This suggests that T5 may actually be better suited for the task than it first appears. Given T5’s relative success on the experiments in T5 as well as an additional unreported experiment that we discuss below, a purely character-level model might not actually be the right choice here.

### T5 can descramble letters
First, Section 6.3 shows that T5 can do word descrambling under a random split (64/92% top1/top10 performance). If we take these as rough performance ceilings, our best curricular approach (20/42% respectively) still has substantial room to improve. A possible counterargument is that there is less headroom on the disjoint split: (12/24% performance on descramble vs 6/20% with our best curricular model). 
### T5 can interpret length constraints
Second, in Section 6.1, we show that T5 does a reasonably good job of learning the correspondence between length enumeration and actual word length -- i.e. the sum of lengths of constituent SentencePiece tokens -- (48% of top 10 outputs have the correct length). This suggests that T5 both can “understand” the length-enumeration part of the input and also that it learns the interaction between lengths of different tokens in a word with the length enumeration. 

### Character-based models
Lastly, we ran some unreported experiments in which we attempted to provide character-level information to T5, for example, by putting a space between each character or providing each word with and without spaces. We saw that the model achieved similar performance to the baseline with these modifications. So, given that the results mentioned above do provide evidence that T5 can understand lengths and constituent letters, we did not pursue these methods further.

### Additional remarks
We recognize that our mention of SentencePiece comes quite late in the text (Section 6.3) and for CR will mention in the Introduction this possible limitation and also that we study this potential limitation in our experiments in Section 6.1 and 6.3 (which *do* demonstrate that T5 is in fact quite robust to character-level challenges!).



# Clarifications and additional details
Finally, we will add clarification and additional details that will appear in CR:

- We will clarify that this paper paper focuses on individual clues rather than full puzzles (though further work combining these approaches with a CSP could tackle full puzzles)

- Section 4.5 - we will clarify the accuracy metrics

- Brief dataset statistics for ACW: we will include the same information as for the cryptic dataset: number of unique answers and mean frequency (number of clues with the same answer)


 We thank the reviewer for this thoughtful review. We address the two main weaknesses raised by the reviewer and then the minor comment.

# Human Evaluation
We agree that human evaluation, especially segmented by clue types in the way we did for our analysis of the curricular learning approach, would be a great way to augment this research. (In fact, we are planning data collection for such an experiment!) The reviewer is correct that novice solvers generally require some amount of instruction to be able to solve these clues; however, once humans are shown some of the rules, they flexibly apply and combine them to solve new clues (i.e. achieving few-shot learning). Friedlander & Fine (psych researchers, citations [9] and [10] in our manuscript) have some interesting work exploring cryptic solving for both experts and novices and elucidate a number of ways in which cryptics draw on interesting problem-solving and linguistic ability.


# Benchmarking systems that learn and think like humans
More broadly, we agree that cryptic crosswords are quite different from more common NLU benchmark tasks like GLUE and SuperGLUE in that they require a level of metalinguistic reasoning that is not generally necessary for most tasks. Still, as the reviewer suggests, cryptics offer a mechanism for studying aspects of language models having to do with interesting kinds of compositionality, expertise, etc. In particular, although good performance on cryptics is not necessarily something we should expect out of the box from LMs that do well on common benchmarks, on the other hand, a system that can “learn and think like humans” should be able to shift between levels of reasoning (e.g. reasoning about whether a given word takes on a metalinguistic function or is, say, part of a phrasal definition). Since people do this flexibly (see the Friedlander and Fine citations), we argue that we want our systems to do it as well. So although cryptics at first seem like a fairly niche task, we hope that this benchmark may push systems to succeed at other tasks that may require meta-cognitive abilities and involve new linguistic understanding. We will clarify this in our text. 

# Additional comments
- We will augment the appendix with example clues from the dataset while keeping the BERT examples in the main text.

 We thank the reviewer for the review and for catching the citation typo. We first address the reviewer’s primary concern (distinction from Efrat and novelty), and then move on to the reviewer’s other suggestions and comments.


# Distinction from Efrat et al.
A major focus of this review is about the relationship between our work and a paper by Efrat et al. that presents a similar data set. “The corpus, while valuable, is not the first corpus of cryptic clues nor is it clearly preferred over the Efrat corpus. The T5 approach has been explored previously.” We note that our work was done contemporaneously with the Efrat et al. work (not in response to it) and we added a discussion and comparison after we became aware of it (but, because there is value in having independent data sets, did not attempt to combine our data set with theirs). As far as we know, the Efrat et al. work has not been peer reviewed or appeared at a conference. We hope that our work can stand on its own scientific merit, and we think that the existence of concurrent groups interested in this topic speaks to its general interest and is not a reason for rejection! Below we detail a number of ways in which our work goes beyond what Efrat et al. do.

Though the reviewer is correct that Efrat concurrently presented a related dataset and also employed T5, our paper identifies a weakness in Efrat’s creation of a “disjoint split”, achieves much better performance with a curricular approach, and develops numerous experiments that characterize the task and the performance of a transformer-based approach. 

## Section 6.5: Explicit comparison
In Section 6.5 we discuss the two major ways in which our contribution improves on Efrat’s work:
One important weakness in Efrat’s work is overlooking the effect of common word endings: we study the robustness of T5 to different word suffixes in Section 6.2, introducing a novel word-initial disjoint split.
Our other major contribution is the curricular approach using related datasets. We show substantial improvement over Efrat’s performance with this approach.

## Additional contributions
Beyond the two major differences discussed in Section 6.5, our paper makes numerous additional contributions:

First, we provide two additional baselines (Wordnet and KNN BoW) to characterize the task and assess the extent to which lexical similarity in clues informs their solution.

Second, our section on model analysis (Section 6) both improves upon Efrat’s work and also provides numerous additional experiments:

* In Section 6.1, we more carefully study the effect of enumeration on both the length of outputs and the number of word generations, going beyond Efrat’s characterization that was purely in terms of overall accuracy. 
* As mentioned above, our Section 6.2 more fully characterizes the robustness of T5 to different datasplits.
* In Section 6.3, we provide a minimal task baseline that can be used to evaluate whether a given architecture can accomplish the sort of wordplay necessary to solve anagram clue types. 
* In Section 6.4, we develop an approach to characterize the extent to which T5 learns a systematic approach to clue solving by selecting a specific clue subset to control for various confounds. 

# Segmenting by Type of Wordplay

Second, you raise a good question about segmenting post-curricular training improvement by the type of wordplay. In our research we did create a number of other curricular training datasets (including for other, non-anagram, clue types) but did not see substantial gains over purely the anagram approach. To understand this further, we opted to focus on a study of the anagram clue type, showing first that curricular training can improve performance, and second, that bringing together different corpora does not always improve performance. In our view, the meager effect of including other clue types in curricular training follows this trend: a single dataset can improve performance on a subset, but we were not able to find a good way to combine multiple datasets in the curricular approach. This could be a good avenue for further work.

# Responses to Additional Questions

Finally, addressing the reviewer's last two questions:
## Section 5.1
Regarding Section 5.1: First, clues in the dataset are not labeled by wordplay type. We mention on line 226 that we algorithmically label the anagram subset of clues, and we provide code to replicate this labeling in our code supplement. We did also develop an algorithmic labeling strategy for some other clue types but did not report the performance on them in the paper. Other clue types that were similar to anagram (like hiddens) saw modest improvement with the anagram training. Training on a synthetic dataset of hidden clues improved performance on the hidden subset but did not significantly improve performance overall. We did not have space to include this in the main manuscript, but could include segmented subset performance in the appendix.

## Section 6.5
Regarding Section 6.5:
Our work was done contemporaneously with Efrat’s and, until the time the work was nearly complete, neither group knew about the other’s interests in the topic. So it was natural for us to publish our own data set. 

Our analysis of the vocabulary content (the solution space) of the two datasets suggests that there is not a substantial difference between the two corpora. In particular, a language model trained on one of the two would probably perform similarly on the other, though we did not test this hypothesis. It might be reasonable to assess whether the larger dataset has more lexical similarity amongst clues with similar answers. We think it is great that future researchers will be able to test their solvers on not one but two modern, independently constructed cryptic crossword data sets. 

 Thank you to this reviewer for the detailed and helpful comments, and we are glad they were amused by our example clues :-).
We will respond first to this reviewer’s two primary concerns and then address the other comments. Our response is too long for a single comment, so we break it into two parts.

#  Primary Comments

## 1 Dataset Availability and Licensing
In response to this comment, we have consulted with someone in the crossword community who has publicly released American crossword data sets. That person has previously consulted a copyright lawyer for similar datasets and is confident that we can release clue text as long as we remove metadata that would enable easily reconstructing full puzzles. We thus agree with the reviewer, and we will modify the paper, our code supplement, and our github repository to release the full data set publicly. We will ourselves assume responsibility for the data release.

## 2 Evaluation metrics
One difference between American-style and cryptic crosswords is that, for cryptics, the combination of the definition and the wordplay ensures that each clue has a unique answer that is not dependent on the grid. (Whereas in an American crossword, each letter is checked by being in both a down and an across, this is not the case for cryptics since it’s expected that the clue uniquely specifies an answer.) Thus, we think the top-1 and top-10 metric is appropriate but and also conservative in that the grid can give additional information in the form of additional letters. But those crossing letters are not usually *necessary* to solve the clue in the way they are for American crosswords. That said, we have run various experiments (not included in this version of the paper) looking at how performance can be improved by including partial answer information from the grid, and are interested in this for future work. The short answer is that T5 is very capable of incorporating additional character information, for example, with prompting of the form r_s_ for an answer ROSE However, since we did not implement a constraint solver, we did not evaluate this method on a full puzzle.

# Other comments
 
## Comparison to American-style crosswords

Please see also our discussion in Evaluation metrics above.

Addressing "American crossword clues are usually straightforward synonyms or definitions": To clarify this, we simply mean that American-style clues are straightforward compared to cryptics in that they do not (by and large) have a wordplay component. Moreover, most American-style clues satisfy the condition that, in a sentence, one could substitute a clue with the correct answer. Still, we certainly acknowledge that answers are not always just synonyms or definitions and can include things like fill-in-the-blanks, hypernyms/hyponyms, and trivia. We will clarify these distinctions in the text.

## Relation to common NLU benchmarks

Regarding more common NLU benchmarks like GLUE and SuperGLUE: We agree that this task is quite different in that it requires a level of metalinguistic reasoning that is not generally necessary for most tasks. Thus, the reviewer is correct that good performance on this task is not necessarily something we should expect out of the box from all LMs that do well on common benchmarks. On the other hand, a system that can “learn and think like humans” should be able to shift between levels of reasoning (e.g. reasoning about whether a given word takes on a metalinguistic function or is, say, part of a phrasal definition). Since people do this flexibly (see the Friedlander and Fine work), we argue that we want our systems to do it as well. So although it may seem at first that this task does not fit neatly into the GLUE or superGLUE benchmarks, we hope that it may push systems to succeed at tasks that may require meta-cognitive abilities and involve new linguistic understanding. We will clarify this in our text. 

## Relation to open-domain QA

The reviewer’s point about the relation to open-domain question answering is also well taken, and perhaps provides a nice window into how this task could be relevant to improving performance on other NLP tasks! As the reviewer points out, cryptics do have a small “trivia” portion that is principally linguistic (i.e., definitional), but solving the “question” primarily relies upon a form of compositional logical reasoning and segmentation that is itself rooted in linguistics. We agree that solving cryptics requires much more focus on language and less on search; we also note that cryptics generally have only one valid solution even before considering grid constraints. 

## Human solving of cryptics

We appreciate the point about the learning curve. This is an aspect we hope to explore in further work that would involve a human evaluation component. The Friedlander and Fine work has some discussion of this, too. We will add a sentence to the introduction acknowledging this point.

## Choice of metrics
Regarding other metrics like MRR or accuracy@k: 
First, unlike American crosswords that require checking answers through crossing letters, cryptics are intended to have only a single correct answer that can be uniquely verified by matching up the wordplay and definition. So in almost all cases, the answer will be unique. Still, the reviewer has good intuition that seeing how k affects correctness may inform our understanding of the models. We did examine other values of k, including ones larger than 10, but in the interest of brevity did not report them, and, as we discuss in Section 3.3, decided to focus on top-1 (exact correctness) and top-10 (useful for assessing how a full solver system might perform). We could add to the CR appendix a plot of accuracy@k for the models we test.

## Baselines discussion
### Enumeration and flexibility
“147: an inherent limitation...is that it is difficult to enumerate all possibilities”: You are correct that most cryptic clues will combine a number of manipulations; we agree that it’s probably around 20-30. One example we like is “Auntie is primed for college (3)”, with answer “UNI”. The solver must take the prime-numbered letters of Auntie (positions 2,3,5) to produce “uni” for which college is a valid definition. We argue that this example shows that even with a full enumeration of cryptic manipulations, and a system that could flexibly combine them, there are at least a few “functions”--in this case taking prime letters--that might not have been included in that enumerated set. In a more abstract framing, for this type of “function” there is some mapping from words (like “primed”, or “odd”, or “even”) that map onto a letter-selection function, but specifying every possible such function is hard. For example, has a clue with the word Fibonacci ever been written? 

We make one additional point about full enumeration and the need for flexibility in systems that solve these puzzles: even if every rule could be enumerated, one probably cannot find a comprehensive list of every possible indicator for all clue types. For example, a new word might come about in English that could reasonably function as an anagram indicator: think something like “squelched”, or even the use of something like “loco” or “fou” (crazy in French) for a puzzle of a certain theme. In this way, the linguistic flexibility of word embeddings are of particular importance to solving cryptics! Still, the reviewer is correct that a large number of clues should be solvable if one fully enumerates the most common patterns and has a sufficiently flexible system. 

### Rule-based (Deits) solver details
Regarding Deits, the grammar is a CFG that specifies possible clue forms. An example could be “\\$Anagram \\$AnagramIndicator \\$Definition”. Terminals for Indicators come from custom lists of indicators (see also the discussion above about why full enumeration is challenging). One of the components of the CFG is a definition: the definition terminal will be matched to a word or set of words. The non-definition part of the grammar (“\\$Anagram \\$AnagramIndicator” in the above example) is evaluated to produce the result of the wordplay. Finally, the result of the wordplay is compared to the definition component using WordNet’s word similarity function. Parses with higher similarity are ranked higher.

### Performance ceiling
“167 room for improvement”: The reviewer is likely correct that the ceiling is lower than 100%, but even if it is only 70%, that is still quite a bit of space! We hope future work on this benchmark can demonstrate that the ceiling is actually quite high!

 We thank the reviewer and are glad they appreciated the data set, the word-initial disjoint split, and our comparison to the interesting work by Efrat et al.

Interesting task that could be a good challenge for neural models. I generally favor accepting new resource papers especially ones that present challenging tasks.  The authors propose a task around cryptic crosswords (of good size, 142K clues, 5K puzzles from the guardian.com). A clue example is "But everything’s really trivial, initially, for a language model (4)" , where the answer is BERT. The "cryptic" aspect here is that the first letter of the first four words spells out BERT (and thus the phrase "initially"). The authors present a good argument (both intuitively and empirically) that this task is challenging for off-the-shelf neural LMs. 

The authors first describe some carefully constructed splits to test more challenging model generalization (since it appears the naive split allows lots of memorizing). The word-initial disjoint split seems quite interesting. 

The authors then show that existing baselines generally do not perform well (even on the naive split). They then propose a curriculum learning approach where they add some related datasets (e.g. anagrams etc.) to help implicitly teach the model certain skills that can help with the downstream task. The resulting model outperforms the baselines but it is clear there is a large amount of headroom. 

The authors not only acknowledge the concurrent work of Efrat et al. 2021 (which proposes a similar task) but also go above and beyond it with an explicit comparison. I think that this is nice.  I don't have any concerns. 

The paper focuses on data and methods for studying compositional language through UK-style cryptic crossword puzzles. These puzzles contain a definition and a wordplay cipher which together read like fluent natural language. The output is a word or phrase related to the wordplay cipher for which the definition is a valid description. Expert humans typically use creativity and linguistic and world knowledge to solve this task. The paper makes two main contributions: (1) a new dataset of cryptic clues as a new benchmark for NLP systems that process compositional language, and (2) baselines and an improved curriculum learning approach in which the model is first fine-tuned on related tasks (e.g., unscrambling words). Their experiments suggest that there is significant room for improvement for future work on this new benchmark.  Clarity: The paper is well-written and easy to follow. It explains the nuances of the dataset and the motivation for the experimental setup very clearly.

Data and baseline results: This looks like a really difficult task requiring knowledge and reasoning that may not be solved with just more compute. While problems revolving around puns, wordplay, and easier crosswords have been studied before, the nature of the data and the challenges that come with it are novel. It has generally been hard to create NLP benchmarks that are GPT/BERT-proof especially on account of train-test overlap often resulting from random splits and known biases. The benchmark includes disjoint splits designed to better test generalization, and as expected, the T5 baselines performs worse when compared to the random split.

Curriculum learning method: Examples in the dataset often contain subproblems which can be learned using additional training from other tasks like American-style crosswords and word descrambling. This seems to result in further gains suggesting that T5 can still benefit from such supervision. While the curriculum learning method isn't particularly novel, it is a solid extension of the baselines and adds value to the paper. Given the nature of the paper, I think this is perfectly fine. 

Overall, the experiments are thorough, and suggest that there is significant room for improvement for future work.

Weaknesses:
Human evaluation: It is still unclear exactly how difficult the task is for humans. Presumably, these puzzles appear periodically in the newspaper sources you mentioned and it is reasonable to assume that readers attempt to solve them. How often do they succeed? I understand that such data is likely not available, but it would be nice to see some sort of human evaluation on a smaller sample of the data both to quantify difficulty and for quality control.

The paper argues that cryptic crosswords are an excellent domain for developing computational language systems that "learn and think like humans". Yet, it is unclear if these puzzles are solvable by non-experts. The paper even seems to suggest that  cryptics pose a challenge to novices. Would someone be able to solve these puzzles without significant training? If not, does it make sense to use a rather niche task to develop systems that learn and think like humans? To clarify, I think there is value to studying niche tasks revolving around complex compositional language. It could still tell us interesting things about models and language. I'm unconvinced that they are effective benchmarks for _evaluating_ systems that learn and think like humans.

(Minor) The cryptic crossword examples used in the paper all revolve around BERT, and were presumably constructed by the authors themselves. While they don't hinder readability (and are  fun!), it would be much nicer to see real examples from the dataset.  Yes.

Meta-review: The submission introduces a new task for solving cryptic crossword problems. The authors introduce carefully constructed splits of the dataset to test generalization, and show that the task is very challenging for existing pretrained models. The reviewers agree that the paper should not be penalized because of a recent similar arxiv paper from Efrat et al. 2021, and it is great that they acknowledged this paper in the submission. The submission contributes is an interesting new problem that may inspire future work, and I recommend acceptance.

Decision: accept