SlowGuess/NAACL_2021_OCR huggingface dataset:
In all rounds annotators provided a label for whether content is hateful or not. In rounds 2, 3 and 4, they also gave labels for the target (i.e., which group has been attacked) and type of statement (e.g., derogatory remarks, dehumanization, or threatening language). These granular labels help to investigate model errors and improve performance, as well as directing the identification of new data for future entry. For approximately half of entries in rounds 2, 3 and 4, annotators created "perturbations" where the text is minimally adjusted so as to flip the label (Gardner et al., 2020; Kaushik et al., 2020). This helps to identify decision boundaries within the model, and minimizes the risk of overfitting given the small pool of annotators.
Over the four rounds, content becomes increasingly adversarial (shown by the fact that target models have lower performance on later rounds' data) and models improve (shown by the fact that the model error rate declines and the later rounds' models have the highest accuracy on each round). We externally validate performance using the HATE-CHECK suite of diagnostic tests from Röttger et al. (2020). We show substantial improvement over the four rounds, and our final round target model achieves  $94\%$  on HATE-CHECK, outperforming the models presented by the original authors.
4Derived from https://hatespeechdata.com, in anonymized form.
Task	Rounds	Examples	vMER
NLI	4	170,294	33.24%
QA	2	36,406	33.74%
Sentiment	3	19,975	35.00%
Hate speech	4	41,255	43.90%
Table 1: Statistics for the initial four official tasks.
3.3 Dynabenchmarking NLP
Table 1 shows an overview of the current situation for the four tasks. Some tasks are further along in their data collection efforts than others. As we can see, the validated model error rate (vMER; the number of human-validated model errors divided by the total number of examples—note that the error rates are not necessarily comparable across tasks, since the interfaces and in-the-loop models are not identical) is still very high across all tasks, clearly demonstrating that NLP is far from solved.
4 Caveats and Objections
There are several obvious and valid objections one can raise. We do not have all the answers, but we can try to address some common concerns.
Won't this lead to unnatural distributions and distributional shift? Yes, that is a real risk. First, we acknowledge that crowdsourced texts are likely to have unnatural qualities: the setting itself is artificial from the perspective of genuine communication, and crowdworkers are not representative of the general population. Dynabench could exacerbate this, but it also has features that can help alleviate it. For instance, as we discussed earlier, the sentiment analysis project is using naturalistic prompt sentences to try to help workers create more diverse and naturalistic data.
Second, if we rely solely on dynamic adversarial collection, then we increase the risks of creating unnatural datasets. For instance, Bartolo et al. (2020) show that training solely on adversarily-collected data for QA was detrimental to performance on non-adversarily collected data. However, they also show that models are capable of simultaneously learning both distributions when trained on the combined data, retaining if not slightly improving performance on the original distribution (of course, this may not hold if we have many more examples of one particular kind). Ideally, we would combine adversarily collected data with
non-adversarial—preferably naturally collected—data, so as to capture both the average and worst case scenarios in our evaluation.
Finally, we note that Dynabench could enable the community to explore the kinds of distributional shift that are characteristic of natural languages. Words and phrases change their meanings over time, between different domains, and even between different interlocutors. Dynabench could be a tool for studying such shifts and finding models that can succeed on such phenomena.
What if annotators "overfit" on models? A potential risk is cyclical "progress," where improved models forget things that were relevant in earlier rounds because annotators focus too much on a particular weakness. Continual learning is an exciting research direction here: we should try to understand distributional shift better, as well as how to characterize how data shifts over time might impact learning, and how any adverse effects might be overcome. Because of how most of us have been trained, it is natural to assume that the last round is automatically the best evaluation round, but that does not mean that it should be the only round: in fact, most likely, the best way to evaluate progress is to evaluate on all rounds as well as any high-quality static test set that exists, possibly with a recency-based discount factor. To make an analogy with software testing, similar to checklists (Ribeiro et al., 2020), it would be a bad idea to throw away old tests just because you've written some new ones. As long as we factor in previous rounds, Dynabench's dynamic nature offers a way out from forgetting and cyclical issues: any model biases will be fixed in the limit by annotators exploiting vulnerabilities.
Another risk is that the data distribution might be too heavily dependent on the target model in the loop. When this becomes an issue, it can be mitigated by using ensembles of many different architectures in the loop, for example the top current state-of-the-art ones, with multiple seeds.[5]
How do we account for future, not-yet-in-the-loop models? Obviously, we can't—so this is a very valid criticism. However, we can assume that an ensemble of model architectures is a reasonable approximation, if and only if the models are not too bad at their task. This latter point is crucial: we
5ANLI does not show dramatically different results across models, suggesting that this is not necessarily a big problem yet, but it shows in R2 and R3 that ensembles are possible.
take the stance that models by now, especially in aggregate, are probably good enough to be reasonably close enough to the decision boundaries—but it is definitely true that we have no guarantees that this is the case.
How do we compare results if the benchmark keeps changing? This is probably the main hurdle from a community adoption standpoint. But if we consider, e.g., the multiple iterations of SemEval or WMT datasets over the years, we've already been handling this quite well—we accept that a model's BLEU score on WMT16 is not comparable to WMT14. That is, it is perfectly natural for benchmark datasets to evolve as the community makes progress. The only thing Dynabench does differently is that it anticipates dataset saturation and embraces the loop so that we can make faster and more sustained progress.
What about generative tasks? For now Dynabench focuses on classification or span extraction tasks where it is relatively straightforward to establish whether a model was wrong. If instead the evaluation metric is something like ROUGE or BLEU and we are interested in generation, we need a way to discretize an answer to determine correctness, since we wouldn't have ground truth annotations; which makes determining whether a model was successfully fooled less straightforward. However, we could discretize generation by re-framing it as multiple choice with hard negatives, or simply by asking the annotator if the generation is good enough. In short, going beyond classification will require further research, but is definitely doable.
Do we need models in the loop for good data?
The potential usefulness of adversarial examples can be explained at least in part by the fact that having an annotation partner (so far, a model) simply provides better incentives for generating quality annotation. Having the model in the loop is obviously useful for evaluation, but it's less clear if the resultant data is necessarily also useful in general for training. So far, there is evidence that adversarily collected data provides performance gains irrespective of the model in the loop (Nie et al., 2020; Dinan et al., 2019; Bartolo et al., 2020). For example, ANLI shows that replacing equal amounts of "normally collected" SNLI and MNLI training data with ANLI data improves model performance, especially when training size is small (Nie et al., 2020), suggesting higher data efficiency. How-
ever, it has also been found that model-in-the-loop counterfactually-augmented training data does not necessarily lead to better generalization (Huang et al., 2020). Given the distributional shift induced by adversarial settings, it would probably be wisest to combine adversarially collected data with nonadversarial data during training (ANLI takes this approach), and to also test models in both scenarios. To get the most useful training and testing data, it seems the focus should be on collecting adversarial data with the best available model(s), preferably with a wide range of expertise, as that will likely be beneficial to future models also. That said, we expect this to be both task and model dependent. Much more research is required, and we encourage the community to explore these topics.
Is it expensive? Dynamic benchmarking is indeed expensive, but it is worth putting the numbers in context, as all data collection efforts are expensive when done at the scale of our current benchmark tasks. For instance, SNLI has 20K examples that were separately validated, and each one of these examples cost approximately  $0.50 to obtain and validate (personal communication with SNLI authors). Similarly, the 40K validated examples in MultiNLI cost$ 0.64 each (p.c., MultiNLI authors). By comparison, the average cost of creation and validation for ANLI examples is closer to $1.00 (p.c., ANLI authors). This is a substantial increase at scale. However, dynamic adversarial datasets may also last longer as benchmarks. If true, then the increased costs could turn out to be a bargain.
We should acknowledge, though, that dynamic benchmarks will tend to be more expensive than regular benchmarks for comparable tasks, because not every annotation attempt will be model-fooling and validation is required. Such expenses are likely to increase through successive rounds, as the models become more robust to workers' adversarial attacks. The research bet is that each example obtained this way is actually worth more to the community and thus worth the expense.
In addition, we hope that language enthusiasts and other non-crowdworker model breakers will appreciate the honor that comes with being high up on the user leaderboard for breaking models. We are working on making the tool useful for education, as well as gamifying the interface to make it (even) more fun to try to fool models, as a “game with a purpose” (Von Ahn and Dabbish, 2008), for example through the ability to earn badges.
We introduced Dynabench, a research platform for dynamic benchmarking. Dynabench opens up exciting new research directions, such as investigating the effects of ensembles in the loop, distributional shift characterisation, exploring annotator efficiency, investigating the effects of annotator expertise, and improving model robustness to targeted adversarial attacks in an interactive setting. It also facilitates further study in dynamic data collection, and more general cross-task analyses of human-and-machine interaction. The current iteration of the platform is only just the beginning of a longer journey. In the immediate future, we aim to achieve the following goals:
Anyone can run a task. Having created a tool that allows for human-in-the-loop model evaluation and data collection, we aim to make it possible for anyone to run their own task. To get started, only three things are needed: a target model, a (set of) context(s), and a pool of annotators.
Multilinguality and multimodality. As of now, Dynabench is text-only and focuses on English, but we hope to change that soon.
Live model evaluation. Model evaluation should not be about one single number on some test set. If models are uploaded through a standard interface, they can be scored automatically along many dimensions. We would be able to capture not only accuracy, for example, but also usage of computational resources, inference time, fairness, and many other relevant dimensions. This will in turn enable dynamic leaderboards, for example based on utility (Ethayarajh and Jurafsky, 2020). This would also allow for backward-compatible comparisons, not having to worry about the benchmark changing, and automatically putting new state of the art models in the loop, addressing some of the main objections.
One can easily imagine a future where, in order to fulfill reproducibility requirements, authors do not only link to their open source codebase but also to their model inference point so others can "talk with" their model. This will help drive progress, as it will allow others to examine models' capabilities and identify failures to address with newer even better models. If we cannot always democratize the training of state-of-the-art AI models, at the very least we can democratize their evaluation.
We would like to thank Jason Weston, Emily Dinan and Kyunghyun Cho for their input on this project, and Sonia Kris for her support. ZW has been supported in part by the Canada 150 Research Chair program and the UK-Canada AI Artificial Intelligence Initiative. YN and MB have been supported in part by DARPA MCS N66001-19-2-4031, DARPA YFA17-D17AP00022, and ONR N00014-18-1-2871. CP has been supported in part by grants from Facebook, Google, and by Stanford's Institute for Human-Centered AI.
Cecilia Ovesdotter Alm, Dan Roth, and Richard Sproat. 2005. Emotions from text: Machine learning for text-based emotion prediction. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 579-586, Vancouver, British Columbia, Canada. Association for Computational Linguistics.
Joshua Attenberg, Panos Ipeirotis, and Foster Provost. 2015. Beat the machine: Challenging humans to find a predictive model's "unknown unknowns". J. Data and Information Quality, 6(1).
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
Max Bartolo, Alastair Roberts, Johannes Welbl, Sebastian Riedel, and Pontus Stenetorp. 2020. Beat the ai: Investigating adversarial human annotation for reading comprehension. Transactions of the Association for Computational Linguistics, 8:662-678.
Yonatan Belinkov and Yonatan Bisk. 2018. Synthetic and natural noise both break neural machine translation. In International Conference on Learning Representations.
Yonatan Belinkov, Adam Poliak, Stuart Shieber, Benjamin Van Durme, and Alexander Rush. 2019. Don't take the premise for granted: Mitigating artifacts in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 877-891, Florence, Italy. Association for Computational Linguistics.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632-642, Lisbon, Portugal. Association for Computational Linguistics.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165.
Dallas Card, Peter Henderson, Urvashi Khandelwal, Robin Jia, Kyle Mahowald, and Dan Jurafsky. 2020. With little power comes great responsibility. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9263-9274, Online. Association for Computational Linguistics.
Michael Chen, Mike D'Arcy, Alisa Liu, Jared Fernandez, and Doug Downey. 2019. CODAH: An adversarially-authored question answering dataset for common sense. In Proceedings of the 3rd Workshop on Evaluating Vector Space Representations for NLP, pages 63-69, Minneapolis, USA. Association for Computational Linguistics.
Alexis Conneau and Douwe Kiela. 2018. SentEval: An evaluation toolkit for universal sentence representations. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The PASCAL recognising textual entailment challenge. In Machine learning challenges: evaluating predictive uncertainty, visual object classification, and recognising textual entailment, pages 177-190. Springer.
Harm de Vries, Dzmitry Bahdanau, and Christopher Manning. 2020. Towards ecologically valid research on language user interfaces. arXiv preprint arXiv:2007.14435.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Emily Dinan, Samuel Humeau, Bharath Chintagunta, and Jason Weston. 2019. Build it break it fix it for dialogue safety: Robustness from adversarial human attack. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4537-4546, Hong Kong, China. Association for Computational Linguistics.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2368-2378, Minneapolis, Minnesota. Association for Computational Linguistics.
Javid Ebrahimi, Daniel Lowd, and Dejing Dou. 2018a. On adversarial examples for character-level neural machine translation. In Proceedings of the 27th International Conference on Computational Linguistics, pages 653-663, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. 2018b. HotFlip: White-box adversarial examples for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 31-36, Melbourne, Australia. Association for Computational Linguistics.
Kawin Ethayarajh and Dan Jurafsky. 2020. Utility is in the eye of the user: A critique of NLP leaderboard design. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4846-4853, Online. Association for Computational Linguistics.
Allyson Ettinger. 2020. What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Transactions of the Association for Computational Linguistics, 8:34-48.
Allyson Ettinger, Sudha Rao, Hal Daumé III, and Emily M. Bender. 2017. Towards linguistically generalizable NLP systems: A workshop and shared task. In Proceedings of the First Workshop on Building Linguistically Generalizable NLP Systems, pages 1-10, Copenhagen, Denmark. Association for Computational Linguistics.
Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hannaneh Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, and Ben Zhou. 2020. Evaluating models' local decision boundaries via contrast sets. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 1307-1323, Online. Association for Computational Linguistics.
Jon Gauthier, Jennifer Hu, Ethan Wilcox, Peng Qian, and Roger Levy. 2020. SyntaxGym: An online platform for targeted evaluation of language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 70-76, Online. Association for Computational Linguistics.
Atticus Geiger, Ignacio Cases, Lauri Karttunen, and Christopher Potts. 2018. Stress-testing neural models of natural language inference with
multiply-quantified sentences. arXiv preprint arXiv:1810.13033.
Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018. Breaking NLI systems with sentences that require simple lexical inferences. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 650-655, Melbourne, Australia. Association for Computational Linguistics.
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 107-112, New Orleans, Louisiana. Association for Computational Linguistics.
Braden Hancock, Antoine Bordes, Pierre-Emmanuel Mazare, and Jason Weston. 2019. Learning from dialogue after deployment: Feed yourself, chatbot! In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3667-3684, Florence, Italy. Association for Computational Linguistics.
Md Mosharaf Hossain, Venelin Kovatchev, Pranoy Dutta, Tiffany Kao, Elizabeth Wei, and Eduardo Blanco. 2020. An analysis of natural language inference benchmarks through the lens of negation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9106-9118, Online. Association for Computational Linguistics.
Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328-339, Melbourne, Australia. Association for Computational Linguistics.
William Huang, Haokun Liu, and Samuel R Bowman. 2020. Counterfactually-augmented snli training data does not yield better generalization than unaugmented data. arXiv preprint arXiv:2010.04762.
Paloma Jeretic, Alex Warstadt, Suvrat Bhooshan, and Adina Williams. 2020. Are natural language inference models IMPPRESSive? Learning IMPlicature and PRESupposition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8690-8705, Online. Association for Computational Linguistics.
Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2021-2031, Copenhagen, Denmark. Association for Computational Linguistics.