SlowGuess/NAACL_2021_OCR huggingface dataset:
Samuel R. Bowman
bowman@nyu.edu
George E. Dahl
Google Research, Brain Team
gdahl@google.com
Evaluation for many natural language understanding (NLU) tasks is broken: Unreliable and biased systems score so highly on standard benchmarks that there is little room for researchers who develop better systems to demonstrate their improvements. The recent trend to abandon IID benchmarks in favor of adversarially-constructed, out-of-distribution test sets ensures that current models will perform poorly, but ultimately only obscures the abilities that we want our benchmarks to measure. In this position paper, we lay out four criteria that we argue NLU benchmarks should meet. We argue most current benchmarks fail at these criteria, and that adversarial data collection does not meaningfully address the causes of these failures. Instead, restoring a healthy evaluation ecosystem will require significant progress in the design of benchmark datasets, the reliability with which they are annotated, their size, and the ways they handle social bias.
A large and impactful thread of research on natural language understanding (NLU) has focused on improving results on benchmark datasets that feature roughly independent and identically distributed (IID) training, validation, and testing sections, drawn from data that were collected or annotated by crowdsourcing (Maas et al., 2011; Bowman et al., 2015; Rajpurkar et al., 2016; Wang et al., 2019b). Recent methodological progress combined with longstanding issues in crowdsourced data quality has made it so state-of-the-art systems are nearing the maximum achievable values on most of these benchmarks and thus are unlikely to be able to measure further improvements (Devlin et al., 2019; Raffel et al., 2020). At the same time, these apparently high-performing systems have serious known issues and have not achieved human-level competence at their tasks (Ribeiro et al., 2020).
1. Good performance on the benchmark should imply robust in-domain performance on the task.
$\hookrightarrow$  We need more work on dataset design and data collection methods.
2. Benchmark examples should be accurately and unambiguously annotated.
$\hookrightarrow$  Test examples should be validated thoroughly enough to remove erroneous examples and to properly handle ambiguous ones.
3. Benchmarks should offer adequate statistical power.
$\hookrightarrow$  Benchmark datasets need to be much harder and/or much larger.
4. Benchmarks should reveal plausibly harmful social biases in systems, and should not incentivize the creation of biased systems.
$\hookrightarrow$  We need to better encourage the development and use auxiliary bias evaluation metrics.
Figure 1: A summary of the criteria we propose.
Progress suffers in the absence of a trustworthy metric for benchmark-driven work: Newcomers and non-specialists are discouraged from trying to contribute, and specialists are given significant freedom to cherry-pick ad-hoc evaluation settings that mask a lack of progress (Church and Hestness, 2019).
The plight of benchmark-driven NLU research has prompted widespread concern about the assumptions underlying standard benchmarks and widespread interest in alternative models of evaluation. As an especially clear example, the documentation for the recent DynaBench benchmark suite argues that "benchmarks saturate", "benchmarks have artifacts", "researchers overfit on benchmarks", and "benchmarks can be deceiving" and use these claims to motivate abandoning the IID paradigm in favor of benchmark data that is collected adversarially by asking a broad population of annotators to try to fool some reference neural network model.<sup>1</sup>
'https://dynabench.org/about
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4843-4855 June 6-11, 2021. ©2021 Association for Computational Linguistics
The DynaBench approach falls into the broader category of adversarial filtering (Paperno et al., 2016; Zellers et al., 2018; Nie et al., 2020; Le Bras et al., 2020). Adversarial filtering starts with a pipeline that produces candidate examples for the task, often through crowdsourcing, and then constructs a dataset by selecting those examples from the pipeline where one or more machine learning models fails to predict the correct label. This approach is appealing in that it guarantees that, at least in the short term, existing approaches to dataset construction can be patched to keep producing data that will challenge current systems.
However, collecting examples on which current models fail is neither necessary nor sufficient to create a useful benchmark. Among other points of concern, this approach can create a counterproductive incentive for researchers to develop models that are different without being better, since a model can top the leaderboard either by producing fewer errors than the adversary or by simply producing different errors, because the examples on which these new errors would be tested will not appear in the evaluation set. One could attempt to do this by, for example, pretraining new models that deliberately avoid any data that was used to pretrain the original adversary model, in order to minimize the degree to which the idiosyncratic mistakes of the new model line up with those of the old one. This incentive can slow progress and contribute to spurious claims of discovery.
This position paper argues that concerns about standard benchmarks that motivate methods like adversarial filtering are justified, but that they can and should be addressed directly, and that it is possible and reasonable to do so in the context of static, IID evaluation. We propose four criteria that adequate benchmarks should satisfy: benchmarks should offer a valid test of the full set of relevant language phenomena, they should be built around consistently-labeled data, they should offer adequate statistical power, and they should disincentivize the use of systems with potentially harmful biases. We then briefly survey some ongoing or promising research directions that could enable us to meet these challenges, including hybrid data collection protocols involving both crowdworkers and domain experts, larger-scale data validation, and auxiliary bias metric datasets attached to benchmarks.
The Problem Performance on popular benchmarks is extremely high, but experts can easily find issues with high-scoring models. The GLUE benchmark (Wang et al., 2019b; Nangia and Bowman, 2019), a compilation of NLU evaluation tasks, has seen performance on its leaderboard approach or exceed human performance on all nine of its tasks. The follow-up SuperGLUE benchmark project (Wang et al., 2019a) solicited dataset submissions from the NLP research community in 2019, but wound up needing to exclude the large majority of the submitted tasks from the leaderboard because the BERT model (Devlin et al., 2019) was already showing performance at or above that of a majority vote of human crowdworkers. Of the eight tasks for which BERT did poorly enough to leave clear headroom for further progress, all are now effectively saturated (Raffel et al., 2020; He et al., 2020). State-of-the-art performance on the highly popular SQuAD 2 English reading-comprehension leaderboard (Rajpurkar et al., 2018) has long exceeded that of human annotators.
Ample evidence has emerged that the systems that have topped these leaderboards can fail dramatically on simple test cases that are meant to test the very skills that the leaderboards focus on (McCoy et al., 2019; Ribeiro et al., 2020). This result makes it clear that our systems have significant room to improve. However, we have no guarantee that our benchmarks will detect these needed improvements when they're made. Most were collected by crowdsourcing with relatively limited quality control, such that we have no reason to expect that perfect performance on their metrics is achievable or that the benchmark will meaningfully distinguish between systems with superhuman metric performance. While the true upper bound on performance for any task (Bayes error) is not measurable, the fact that our systems have exceeded serious estimates of human performance leaves us with no reason to expect there to be much more headroom.
In addition, many of our best models display socially-relevant biases that render them inappropriate for deployment in many applications.2 Our best current benchmarks do little or nothing to dis
2The state-of-the-art T5 model, for example, shows far more sensitivity to irrelevant gender information than humans do when making coreference judgments, according to results on the SuperGLUE leaderboard with the DNC Winogender dataset (Rudinger et al., 2018; Poliak et al., 2018).
courage harmful biases and, by building largely on crowdsourced or naturally-occurring text data, they likely incentivize the development of models that reproduce problematic biases, at least to some degree.
The Goal This paper lays out four criteria that we would like our benchmarks to satisfy in order to facilitate further progress toward a primarily scientific goal: building machines that can demonstrate a comprehensive and reliable understanding of everyday natural language text in the context of some specific well-posed task, language variety, and topic domain. Among language understanding tasks, we focus on those that use labeled data and that are designed to test relatively general language understanding skills, for which the design of benchmarks can be especially difficult.
We distinguish between a task and a benchmark: A task, in our terms, is a language-related skill or competency that we want a model to demonstrate in the context of a specific input-output format. A benchmark attempts to evaluate performance on a task by grounding it to a text domain and instantiating it with a concrete dataset and evaluation metric. As a rough example, multiple-choice reading-comprehension question answering is a task, which the Cosmos benchmark (Huang et al., 2019) attempts to test using an accuracy metric over a specific sample of passages and questions from the English personal narrative domain. There is no general way to prove that a concrete benchmark faithfully measures performance on an abstract task. Nevertheless, since we can only evaluate models on concrete benchmarks, we have no choice but to strengthen the correspondence between the two as best we can.
We set aside the evaluation of computational efficiency and data efficiency, despite its relevance to many specific applications of language technology. We will not fully set aside issues of social bias. Even though it is possible for the same system to demonstrate both adept language understanding and harmful social prejudices,<sup>3</sup> ethical concerns prompt us to argue that community-wide benchmarks should identify and disincentivize potentially harmful biases in models. The widespread sharing of trained models among NLU researchers and en
gineers and the fast pace of NLP R&D work mean that it is easy for systems designed with scientific goals in mind to be deployed in settings where their biases can cause real harm. While recent initiatives around data documentation should reduce the accidental deployment of models built on inappropriate data (Bender and Friedman, 2018; Gebru et al., 2018), we see room to do more.
We will also set aside few-shot learning, in which tasks are made artificially difficult by training models only on small subsets of the available training data (as was prominently used for GPT-3 by Brown et al., 2020). This paper focuses instead on the case where one is interested in reaching excellent performance on some language task and is willing to collect data or otherwise expend resources to make that possible. While few-shot learning represents a potentially impactful direction for engineering research, and success on some task in a few-shot setting is clear evidence of success more generally, artificial constraints on the use of training data do not fit the broad goals laid out above and do not fit many applied settings.
3 Four Challenges
This paper focuses on four criteria, outlined in Figure 1, that we argue effective future benchmarks for NLU tasks should satisfy. We believe that no current benchmark for any difficult broad-domain NLU task satisfies all four:
3.1 Validity
If one system significantly outperforms another on some benchmark, then that result should be strong evidence that the higher-scoring system is actually better at the task tested by the benchmark. In other words, benchmarks are only useful for language understanding research if they evaluate language understanding. General-purpose benchmarks that are designed to cover tasks like paragraph reading comprehension over Wikipedia are only effective if they test the full range of skills that are required to understand and reason about paragraphs from Wikipedia.
This criterion is difficult to fully formalize, and we know of no simple test that would allow one to determine if a benchmark presents a valid measure of model ability. Minimally, though, it requires the following:
- An evaluation dataset should reflect the full range of linguistic variation—including words
3The performance of models like RoBERTa (Liu et al., 2019) or T5 (Raffel et al., 2020) on benchmarks like SuperGLUE that include some coverage of social bias is a good example of this, and typical human behavior is an even better example.
and higher-level constructions—that is used in the relevant domain, context, and language variety.
- An evaluation dataset should have a plausible means by which it tests all of the language-related behaviors that we expect the model to show in the context of the task.
- An evaluation dataset should be sufficiently free of annotation artifacts (as in Si et al., 2019; Sugawara et al., 2020b; Niven and Kao, 2019) that a system cannot reach near-human levels of performance by any means other than demonstrating the required language-related behaviors.
If a benchmark fully meets this challenge, we should expect any clear improvement on the benchmark to translate to similar improvements on any other valid and reasonable evaluation data for the same task and language domain.4
The rest of this section surveys common paradigms for constructing a benchmark dataset, and points to reasons that none offers a straightforward way to satisfy this criterion:
Naturally-Occurring Examples It is intuitively appealing to, where possible, build benchmark datasets based on naturally-occurring data distributions. This minimizes our effort in creating benchmarks and minimizes the risk that the benchmark is somehow skewed in a way that omits important phenomena. However, this is often not viable.
For tasks like reading comprehension or natural language inference that require multiple related texts (such as a passage and a question) as input, there is often no natural distribution that efficiently isolates the relevant task behaviors. One can find naturally-occurring distributions over questions, like those used to construct Natural Questions (Kwiatkowski et al., 2019), but these will generally be tied to the use contexts of a specific NLP product and will thus be limited by users' perceptions of the current abilities of that product.
Even for single-input tasks like coreference resolution or Cloze, for which any text corpus can be the basis for a benchmark, naturalistic distributions do nothing to separate skills of interest from factual world knowledge and can be overwhelmingly dominated by the latter, making them poor
metrics for incremental progress on NLU. Credible existing NLU-oriented benchmarks for such tasks are generally heavily curated (Paperno et al., 2016; Levesque et al., 2012; Sakaguchi et al., 2019).
Expert-Authored
Examples Expert
constructed datasets for language understanding like FraCaS (Cooper et al., 1996) and the Winograd Schema Challenge (Levesque et al., 2012) have been crucial for defining several new tasks and introducing them as objects of study. However, expert example construction isn't desirable for the creation of benchmarks for the use cases we focus on here.
Setting aside the logistical challenges of creating sufficiently large and diverse datasets by expert labor alone, expert authorship generally gives members of the research community direct, fine-grained control over the data on which their systems will be evaluated. Intentionally or unintentionally, this can produce data that is oriented toward linguistic phenomena that are widely studied and widely known to be important to the task at hand. While this can be helpful when building diagnostic datasets that focus on specific types of model failure (Cooper et al., 1996; Naik et al., 2018; Wang et al., 2019b), it is counterproductive when our goal is to build a broad-coverage benchmark dataset to set priorities and guide progress toward the solution of some task.
Dunietz et al. (2020) and Sugawara et al. (2020a) work around this issue by leaning on taxonomies of required phenomena from outside NLP. This is a direction worth pursuing, but it is not clear that appropriate taxonomies will be available for most NLU tasks of interest, or that these taxonomies will be broad and thorough enough to be straightforwardly implemented as datasets.
Crowdsourcing Most recent benchmarks for language understanding have been collected, at least in part, through crowdsourcing example construction, where non-expert annotators are given some freedom to construct examples based on a simple set of guidelines. This has an obvious appeal: Using non-expert annotators significantly lowers costs and using simple guidelines significantly reduces the risk that the resulting data will be skewed artificially toward phenomena of interest to experts.
However, straightforward standard practice, as was used to collect datasets like SNLI (Bowman et al., 2015) and SQuAD, seem to be relatively poor
4Though, of course, any model with non-zero test error could be presented with a potentially-unreasonable benchmark entirely consisting of its own test errors.
at producing difficult datasets that test the intended phenomena. Existing datasets focus heavily on repetitive, easy cases and often fail to isolate key behaviors (Jia and Liang, 2017; Tsuchiya, 2018; McCoy et al., 2019).
Adversarial Filtering Given a source of examples and a model, adversarial-filtering-style approaches build a benchmark based on samples from that source for which the model fails. Adversarial filtering can remove examples that are easy due to trivial artifacts, but it does not ensure that the resulting dataset supports a valid test of model ability, and it can systematically eliminate coverage of linguistic phenomena or skills that are necessary for the task but already well-solved by the adversary model. This mode-seeking (as opposed to mass covering) behavior by adversarial filtering, if left unchecked, tends to reduce dataset diversity and thus make validity harder to achieve.
In contrast with this benchmark data collection setting, adversarial competitions, in which one compares the difficulty of collecting valid task examples that are adversarial to each of several systems, could be part of a healthy evaluation ecosystem. Such an ecosystem might involve frequent formative evaluations on a conventional non-adversarial benchmark in conjunction with periodic organized evaluations in an adversarial setting.
3.2 Reliable Annotation
For our benchmarks to incentivize the development of sound new methods, the labels for their test examples should be reliably correct. This means avoiding three failure cases: (i) examples that are carelessly mislabeled, (ii) examples that have no clear correct label due to unclear or underspecified task guidelines, and (iii) examples that have no clear correct label under the relevant metric due to legitimate disagreements in interpretation among annotators. The first two cases straightforwardly compromise the validity of the benchmark, but the third is somewhat subtler.
Legitimate disagreement emerges when an example can be labeled in multiple ways depending on an annotator's choice between reasonable interpretations of the text of an example. Such disagreements might stem from dialectal variants in the interpretation of words or constructions or different reasonable interpretations of the actual state of the world. As a toy example, consider the question: Does Ed ate a burrito entail Ed ate a sandwich?
While most US English speakers would likely answer no, many pedants and regulatory officials have argued for yes (Florestall, 2008).
When a benchmark contains many instances of this kind of legitimate disagreement, a machine learning model will be able to study a benchmark dataset's training set for clues about typical human behavior that might allow it to perform better than any single human annotator. This effect could contribute to misleading reports of super-human performance on such benchmarks, where human performance reflects the behavior of humans who are reporting their own judgments, rather than attempting to predict the most frequently assigned label, as the model does. We observe evidence of this kind of ambiguity in existing benchmarks: For example, Pavlick and Kwiatkowski (2019) find that  $20\%$  of examples across several textual entailment datasets are significantly ambiguous, and Kwiatkowski et al. (2019) show that  $36\%$  of short answer annotations in Natural Questions differ significantly from the majority answer.
3.3 Statistical Power
Benchmark evaluation datasets should be large and discriminative enough to detect any qualitatively relevant performance difference between two models. This criterion introduces a trade-off: If we can create benchmark datasets that are both reliable and highly difficult for the systems that we want to evaluate, then moderate dataset sizes will suffice. However, if our benchmark datasets contain many examples that are easy for current or near-future systems, then we will need dramatically larger evaluation sets to reach adequate power.
In the context of a reliable dataset that is difficult for current systems, a  $1\%$  absolute accuracy improvement, such as that from  $80\%$  to  $81\%$ , may be an acceptable minimum detectable effect. In this case, an evaluation set of a few thousand examples would suffice under typical conditions seen in NLU (Card et al., 2020). Many, though not all, popular benchmark datasets satisfy this size threshold.
Since our systems continue to improve rapidly, though, we should expect to be spending more time in the long tail of our data difficulty distributions: If we build reliable datasets, much of their future value may lie in their ability to measure improvements in accuracy among highly accurate systems. For example, an improvement from  $98\%$  accuracy to  $98.1\%$  represents the same  $5\%$  relative improve
ment as we saw from  $80\%$  to  $81\%$ . To reliably detect this smaller absolute improvement, though, requires two orders of magnitude more evaluation data (Card et al., 2020).
3.4 Disincentives for Biased Models
A benchmark should, in general, favor a model without socially-relevant biases over an otherwise equivalent model with such biases. Many current benchmarks fail this test. Because benchmarks are often built around naturally-occurring or crowdsourced text, it is often the case that a system can improve its performance by adopting heuristics that reproduce potentially-harmful biases (Rudinger et al., 2017). Developing adequate methods to minimize this effect will be challenging, both because of deep issues with both the precise specification of what constitutes harmful bias and because of the limited set of tools that we have available to us.