Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a new dataset of cryptic crossword clues. The work presents an extensive evaluation study on this dataset with 3 non-neural baselines and one neural baseline, namely the T5 transformer-based model. The authors investigate as well a kind of pre-finetuning stage (a phase of finetuning before the final finetuning stage on the main dataset) as a way to improve the performance of the model on this task.

Contributions:
New dataset of cryptic crossword clues that has the potential to foster more research on complex reasoning tasks

Strengths:
- The writing is compelling and the paper is a very enjoyable read
- The author do a detailed analysis to understand better the limitations and abilities of transformer-based models on the task, including interesting ways of looking at data splits
- The authors investigate the use of a pre-finetuning stage as a way to improve the performance of the T5 model on the task
- The evaluation is convincing with multiple baselines, non-neural and neural.

Weaknesses:
Sometimes, the paper seemed to be a bit condensed 
main_review: Intro:
Lines 26-27: What does flexible mean in flexible AI or flexible NLP models?
Line 82: in what sense is it competitive? (Since you don’t have access to it to test it)

Dataset:
Is there a reason why the authors did not collect a “clue type” for each entry so as to potentially train models on that additional dimension vs without and see how that affects the learning?

Section 3.2:
Thoughts on formulating the problem similar to a MCQ problem where the answer is one of several choices? 

Section 3.3:
- When determining if the answer is correct, how are the generated and golden answers compared? For example, is the generated answer lemmatized etc?
- Clarify more the following: “For instance, the best-performing crossword solver for American-style (non-cryptic) crosswords relies heavily on satisfying grid constraints.”

Section 5.1:
- The 7 in (7) for ACW, ACW-descramble, ACW-descramble-word is supposed to be the number of letters in the right answer, right? So I’m supposing this is a typo?
- It would be good to explain the other anagram indicators in the section so that the reader has a better appreciation of the dataset at hand. At the very least this could be done in the appendix which is not the case.

Section 5.3:
Some aspects of your work remind me of the Muppet paper from FAIR (Aghajanyan et al 2021). Your curriculum learning phase could be seen as analogous to their pre-finetuning phase and the overall theme is not far (e.g. some of the downstream tasks there could be commonsense reasoning or NLI). What’s interesting is that they found in almost all cases (datasets/tasks) the performance was hurt when only a few tasks were used for pre-finetuning and typically it’s when they pre-finetuned on more than 15 tasks that they began to see a consistent improvement on the final task. Your findings seem to be in contrast to that (i.e. with only a couple tasks for pre-finetuning you got an improvement), so I was wondering if you had any thoughts on that. 

Section 6.2:
It would be interesting to assess quantitatively the similarity between different clues linked to the same word so that we can actually see whether the results are actually due to said similarity and not for some other factor. That is, it could be that different clues could have the same answers but not necessarily a high lexical overlap.

Typos/Minor comments:
Line 62: missing [ ] 
Line 63: data set -> dataset
Line 65: finetune -> finetuned 
Line 66: It would be good to add a sentence or 2 in the intro to clarify more (at this point) what you mean by disjoint data split so that the reader has a clearer idea about the different contributions of the paper from the intro
Line 329: we we
Lines 359-360: Incomplete bibliographic entry

Presentation comment:
It’s somewhat annoying that Tables 2 and 3 appear quite far from the actual relevant sections. You can consider referring to the relevant section in Section 3.3 and keeping the first instances of table citations to Section 5 so that they appear closer to the section.

Missing references:
- Line 155: Add the same reference [28] to the C4 Corpus (if some reader is not familiar with T5, they might not know directly that the corpus was introduced in the same paper)
- Generally, the Related Work section does not cover any work that involves this idea of (pre-)finetuning on another task before the actual finetuning phase. Some papers that do that: Tafjord et al. 2019, Shwartz et al. 2020, Aghajanyan et al. 2021

General feedback:
The main contribution of this paper seems to me to be the presentation of a new dataset which, I believe, is important. The main ML component of the paper is essentially the curriculum learning approach. As I mentioned, similar ideas of pre-finetuning have been introduced in at least 3 papers that I am aware of, and so, there's not much novelty here in my opinion (despite the specifics of the method adopted here). The question is then whether this paper is a better fit for an NLP conference. I am still of the opinion that the paper has a valuable contribution and so giving it an accept.

limitations_and_societal_impact: The authors do not address the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: **What is the task?**

Cryptic crossword puzzles task

**What are the contributions of the paper?**

* Presented a dataset of cryptic crossword clues as a challenging new benchmark for NLP systems that seek to process compositional language in more creative, human-like ways.

* A novel curriculum approach, in which the model is first fine-tuned with related, synthetic tasks (e.g., an augmented word descrambling task) before tackling actual cryptic clues.

* Introduced a challenging word-initial disjoint data split to test composition and generalization and investigated model behavior by systematically perturbing the wordplay part of clues, showing that the model exhibits behavior partially consistent with human solving strategies.

main_review: **What has been done before?**

Efrat et al. —concurrent work that presents a similar dataset and similar neural baseline using T5.
While Efrat et al. [6] conclude that train/test disjointness is important, they do not fully consider T5’s robustness to plural and other inflections. The word-initial disjoint split that presented in the paper addresses this. The proposed curricular approach outperforms Efrat et al. work on different kinds of splits, especially word-initial disjoint split.

**What are the main results?**

* Results show that a curriculum of synthetic datasets can address this, substantially improving performance on the primary cryptics task. 

* The proposed curricular approach outperforms previous best work on different kinds of splits, especially word-initial disjoint split.


** Are others (researchers or practitioners) likely to use the ideas or build on them? **

This curricular approach may be useful in other settings where focused linguistic capabilities are required. Cryptic crossword task can serve as a good benchmark for those interested in building NLP systems that can apply linguistic and meta-linguistic knowledge in more creative, flexible, and human-like ways.



**Strengths:**

* In this paper, author(s)  aim not only to present a data set and propose a novel solution but also to characterize the problem and motivate its importance. 

* The claims are well supported by experimental results and look reproducible.

* The proposed curricular approach outperforms previous best work on different kinds of splits, especially word-initial disjoint split.

* Other researchers or practitioners likely to use the ideas or build on them.

Weakness/ concern:

* No formal comparison with human performance on the task

limitations_and_societal_impact: Author(s)  discuss  opportunities to improve our approach and the limitations of using subword-level tokenization on a wordplay task.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: - introduces a dataset comprised of cryptic crossword clues, and shows performance of a few baselines including a T5 prefix-LM are not great, and suggests the problem is hard for current models
- discusses use of related datasets/tasks and data augmentation to improve T5 baseline

main_review: Language needs harder tasks/benchmarks and this paper attempts to introduce one. The most important question to answer in the paper is is the problem difficult enough and unsolved to be a new challenge benchmark. I do not think this is answered by the experiments. The fact that a T5 base model doesn't do well is a good sign, and intuitively seems like a very hard problem. However, this particular baseline may not be well-suited to the problem since it tokenizes text into SentencePiece tokens, which makes it difficult to interpret character-length constraints given as input. To do this task properly, the model must learn the number of characters in each token, which presumably requires a lot of data, and may partially explain the poor performance. This limitation is not discussed in the paper. I'd like to see a baseline (preferably a pre-trained LM similar to T5) that takes in characters as input. It'd also be useful to see whether larger T5 models do better.

The "Curricular" model is really just supervised pre-training on related datasets (american crosswords) with some data-augmentation. It is not curricular learning, nor is it particularly novel.

In summary, there is not much in terms of new methodology, but the dataset is potentially interesting as new language benchmark. However, the experiments do not convince me that it is indeed hard enough to be one.


Questions/clarifications:

T5 prefix-lm is not the standard T5 model, which is encoder-decoder.
Why did you use the T5 prefix-lm model instead of encoder-decoder?


# Update after author response 
Again I think this dataset is potentially interesting, but I'd like to see character-based model baselines as this task involves counting characters in the output which SentencePiece tokenization severely obstructs. It'd be great to see results in a future paper using something like ByT5. If a good character-based model cannot solve this task, I would be convinced that this is a great new benchmark for NLP; in the absence of that it is hard to tell. At the very least, some of the points in the author response should be added to the discussion in the paper. I've updated my score assuming this is added.

"This suggests that T5 both can “understand” the length-enumeration part of the input and also that it learns the interaction between lengths of different tokens in a word with the length enumeration."
- I'm sure it can but it must learn the length of tokens to properly do the task, which a model that decodes characters would have a much easier job doing.

Thanks for clarifying that the model is in fact an encoder-decoder.

limitations_and_societal_impact: Limitations of T5 sentencepiece model are not sufficiently discussed for this character-based task.

I don't see any potential negative societal impacts.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: Authors 1) provide a dataset of cryptic crosswords; 2) explore baselines performances on this dataset; 3) provide interesting insights on those performances and come up with a pretraining strategy that increases performances.

Briefly, cryptic crosswords are puzzles that require both mastering definitions and wordplay (i.e. reordering of letters, extracting letters, composing parts of several words to form a novel word, etc.) to be solved. Authors provide a large dataset of such puzzles (in english), that they introduce as a stepping stone to solve more complex real-life settings.

They show that carefully crafted non-neural baselines are not very good (either rule-based following previous work on this, or KNN) and that even T5 (state of the art text-to-text transformer pretrained on many tasks, such as translation, summarization, etc.) cannot be finetuned for optimal performances on this dataset. Worst, when considering adversarial splits, such that for instance all puzzles with answers starting with the same subword are found in the same split, T5 performances drop dramatically.

They hint at a solution for this problem via curriculum learning, and devise the following strategy: first pretrain T5 on a series of tasks resembling the final wordplays (e.g. unscrambling a word using its definition: "etalp + flower part" --> "petal") and only then finetune T5 on the actual cryptic crosswords dataset. T5 performances indeed increase, but authors note that they are far from human performances, or even acceptable performances for a machine.

main_review: Globally, this paper is easy to follow, and explains each step with enough details so that it may be understood without problem. It should also be noted that authors have made strong efforts to be honest about their work: they explicitly state that they do not claim to solve the difficult task they introduce, rather provide insights into promising future work and, very importantly, compare themselves candidly to the contemporary and concurrent work of Efrat et al. 2021 -- which also introduces a cryptic crosswords dataset.

First of all, the strengths of the paper:
- As I said, paper is well crafted and is easy to read and understand.
- The dataset introduced is relevant to the NLP community and seems like a good diagnostic dataset for methods that should be able to deal with subtle variations in language (simulated by wordplays).
- Contrary to the work of Efrat et al. 2021, which introduces a similar dataset andalso uses a finetuned T5 baseline, authors of this paper also propose credible strategies to improve performances, via curriculum learning.
- Experiments done are sound, and convincing. In particular, the new split introduced is clever, and show efforts in understanding where T5 might shine by chance, rather than by skill.

Despite those strengths, the paper suffers from several flaws, which are difficult to overlook:
- The related work section is somewhat light: 10 lines in total, and only refers to cryptic crosswords. I understand that space is tight in conference papers, but authors should also refer to work related to other areas of this work, for instance curriculum learning using similar, simpler tasks. For instance, Wu et al. 2021 [1] and Geva et al. 2020 [2] are two different works that explore the same strategy but for numerical reasoning.
- Similar to the related work focusing only on crosswords, the abstract and introduction seldom mention the usefulness of solving this specific task, for real world applications. Despite authors claiming to "characterize the problem and motivate its importance", motivation is rather lacking except for a few lines (e.g. in the abstract "NLP systems that seek to process compositional language in more creative, human-like ways") which need argumentation.
- While the rule-based and T5 baseline are relevant, there still seem to be lacking a crucial baseline: any character based model, which we would intuitively think has a better chance at rearranging letters than subword-based models such as T5.
- There are no experiments on other data (such as maybe tweet sentiment analysis?), that would showcase the presented improvements as useful in real-life settings.

EDIT: after a response from the authors, I'm updating my score from 6 to 7: Good paper, accept.

All in all, I think this paper is a marginally above acceptance threshold, because many of its flaws can be seen as future work and do not cripple the work as a whole. The dataset is great, and in my opinion relevant to NLP community because it introduces complex language manipulation. The experiments and claims are sound, well-designed and well explained. However, I feel that it could be argued that the methods presented here would be better suited for Empirical Methods in NLP rather NeurIPS. Furthermore, crucial experiments, either by using a character-based baseline, or a real-life dataset, are missing. This problem is attenuated because introducing a dataset, its scope and analyzing some baselines is in itself a great piece of work. (I also don't count as a negative the concurrent work of Efrat et al. 2021, which is too recent). I strongly feel that at least the related work should be reworked to include more than cryptic puzzles.

If this paper were to be rejected, I would advise authors to carefully reframe the paper, so that the importance of the presented tasks is better highlighted. For instance, the abstract should start by introducing complex language manipulation tasks as a real world challenge, and then present cryptic crosswords as a relevant diagnostic dataset, rather than focus directly on crosswords. Similarly, introduction and related work should also focus on related areas (e.g. curriculum learning, tasks with fuzzy nlp, etc.), which are arguably more important than simply solving crossword puzzles. I would also include stronger baselines (character based models) and real-life experiments.


[1]: LIME: Learning Inductive Bias for Primitives of Mathematical Reasoning -- Wu et al. 2021 

[2]: Injecting Numerical Reasoning Skills into Language Models -- Geva et al. 2020
limitations_and_societal_impact: Authors have addressed most limitations of their work. It does not seem to have negative societal impact.
review_title: null

Meta-review: This paper presents a new task for evaluating language models, cryptic crossword puzzles, along with a new dataset to evaluate on this task. The authors also propose a curriculum learning approach that improves over a standard seq2seq baseline on the proposed task. Reviewers agree that this is a well written paper and the dataset is interesting to a lot of people. Reviewer T9VZ has a concern about adding a character-level baseline. I agree this would be something good to add, and I encourage the authors to explore this for the next version of the paper. I think the paper is above the bar for acceptance to NeurIPS, so I recommend acceptance.
Decision: Accept (Poster)