SlowGuess/NAACL_2021_OCR huggingface dataset:
There is no precise enumeration of social biases that will be broadly satisfactory across applications and cultural contexts. This can be most easily illustrated with the example of biased associations between word representations for US English (as in Bolukbasi et al., 2016). Associations between race or gender and occupation are generally considered to be undesirable and potentially harmful in most contexts, and are something that benchmarks for word representations should discourage, or at least carefully avoid rewarding. If a set of word representations encodes typically Black female names like Keisha as being less similar to professional occupation terms like lawyer or doctor than typically white male names like Scott are, then a model using those representations is likely to reinforce harmful race or gender biases in any downstream content moderation systems or predictive text systems it gets used in.
Adequately enumerating the social attributes for which we might want to evaluate bias in some context can be difficult. For example, Indian castes, like racial categories in the United States, are often signaled by names and are an axis on which managers sometimes discriminate in hiring. Caste is a salient category of social bias in India that is subject to legal and institutional recognition. However, this bias also arises in some cases within the United States, where it has no such recognition (Tiku, 2020), and where it could be easily overlooked by non-specialist bias researchers.
Furthermore, building such a list of attributes
is also deeply political. Within living memory, popular and legal attitudes have changed significantly in the United States about attributes like race, gender, gender expression, sexual orientation, and disability. Attitudes on these issues continue to change, and new categories can gain recognition and protection over time. In many cases, this means that choosing whether to include some attribute in a computational metric of bias means choosing which group of people to align oneself with on a political issue. While there are clear ethical rules of thumb to follow when doing so,[5] making any particular choice is nonetheless likely to put researchers in conflict with established institutions in ways that can change quickly. Any strategy for handling bias in the context of NLP benchmarks will have to grapple with this difficult reality.
4 Sketching a Solution
Building new benchmarks that improve upon our four axes is likely to be quite difficult. Below we attempt to sketch out some possible directions for improvement along each axis.
4.1 Improving Validity
Building valid benchmarks will require significant new research into data collection methods, at least some of which will be specific to the task under study. We suspect that much of this work will involve improvements in crowdsourcing and the use of non-experts, as most of the annotation behind the tasks we discuss requires no expertise other than fluent knowledge of the language variety under study.
One promising direction involves methods that start from relatively high-quality crowdsourced datasets, then use expert effort to augment them in ways that mitigate annotation artifacts. The Build-it-Break-it challenge (Ettinger et al., 2017), the Open Reading Benchmark (Dua et al., 2019), and the Gardner et al. (2020) contrast sets, among their other features, allow expert annotators to add examples to a test set to fill perceived gaps in coverage or correct perceived artifacts in a starting set of crowdsourced examples. To the extent that crowdsourcing with non-experts can produce data that has broad coverage and high difficulty but retains some measurable artifacts or flaws, this compro
5The ACM code of ethics states, "when the interests of multiple groups conflict, the needs of those less advantaged should be given increased attention and priority."
mise approach may help to create usable benchmark datasets out of the results.
Another approach brings computational linguists directly into the crowdsourcing process. This was recently demonstrated at a small scale by Hu et al. (2020) with OCNLI: They show that it is possible to significantly improve data quality issues by making small interventions during the crowdsourcing process—like offering additional bonus payments for examples that avoid overused words and constructions—without significantly limiting annotators' freedom to independently construct creative examples.
Of course, implementing interventions like these in a way that offers convincing evidence of validity will be difficult.
4.2 Improving Handling of Annotation Errors and Disagreements
The use of standard techniques from crowdsourcing—generally involving multiple redundant annotations for each example—can largely resolve the issue of mistaken annotations. Careful planning and pilot work before data collection can largely resolve the issue of ambiguous annotation guidelines. Handling legitimate annotator disagreements can take two fairly different approaches, depending on the goals of the benchmark.
The simplest approach treats ambiguously labeled examples in the same way as mislabeled examples, and systematically identifies and discards them during a validation phase. For some tasks, it may still be possible to test models' handling of fundamentally ambiguous linguistic phenomena or domains using unambiguous examples: In the case of multiple-choice question answering, for example, one can construct examples where one answer candidates is only debatably correct, but all other candidates are unequivocally wrong. Any sound model would then be expected to select the debatable choice.
Alternatively, one can decline to assign single, discrete labels to ambiguous examples. This can involve asking models to predict the empirical distribution of labels that trustworthy annotators assign (Pavlick and Kwiatkowski, 2019; Poesio et al., 2019), or allowing models to predict any of several answer choices that are supported by trustworthy annotators (as in the SQuAD benchmark). This comes at the cost, though, of requiring many more
annotator judgments per evaluation example.
4.3 Improving Statistical Power
In principle, achieving adequate statistical power is straightforward: we simply estimate the number of examples required to reach the desired statistical power for any plausible short-to-medium term system evaluation for the task, and collect that number of examples. In practice, however, costs can become prohibitive.
For a relatively simple task like NLI, labeling an existing example likely requires a bare minimum of 45 seconds (Vania et al., 2020), and creating a new example requires at least one minute (Bowman et al., 2020). Even if we use these very optimistic numbers to estimate annotation speed, a ten-way-annotated dataset of 500,000 examples will still cost over $1 million at a $15/hr pay rate. Recruiting more experienced annotators or encouraging annotators to work more carefully could increase this figure dramatically. While such an amount of money is not completely out of reach in a well-funded field like NLP, investments of this kind will inevitably be rare enough that they help reinforce the field's concentration of data and effort on a few high-resource languages and tasks.
For settings in which large datasets are necessary, we see no clear way to avoid high costs. Gamification, in the style of the ESP game or ZombiLingo (Von Ahn and Dabbish, 2004; Fort et al., 2014), promises to offer free human labor, but at the cost of the expert time needed to refine the task definition into a game that is widely enjoyable. This approach also introduces severe constraints on the kinds of data collection protocols that can be used and raises tricky new ethical issues (Morschheuser and Hamari, 2019). Ultimately, the community needs to compare the cost of making serious investments in better benchmarks to the cost of wasting researcher time and computational resources due to our inability to measure progress.
4.4 Disincentives for Biased Models
Because there is no one-size-fits-all definition of harmful social bias, there is little prospect of creat-
This figure ignores platform fees and makes the additional optimistic assumption that only  $10\%$  of fully-annotated examples will be discarded because of annotator disagreement.
To put this number in context, public estimates of the cost of OpenAI's GPT-3 (Brown et al., 2020) exceed  $10M (Wiggers, 2020), and in machine translation, Meng et al. (2019)'s use of 512 Nvidia V100 GPUs for three months would have cost over$ 1M USD on commodity cloud infrastructure.
ing a benchmark for language understanding that is guaranteed to never reward the development of harmfully biased models. This is not a compelling reason to accept the status quo, and we nonetheless have a clear opportunity to mitigate some of the potential harms caused by applied NLP systems before those systems are even developed. Opting not to test models for some plausible and potentially-harmful social bias is, intentionally or not, a political choice.
While it would be appealing to try to guarantee that our evaluation data does not itself demonstrate evidence of bias, we are aware of no robust strategy for reliably accomplishing this, and work on the closely-related problem of model bias mitigation has been fraught with false starts and overly optimistic claims (Gonen and Goldberg, 2019).
A viable alternate approach could involve the expanded use of auxiliary metrics: Rather than trying to fully mitigate bias within a single general dataset and metric for some task, benchmark creators can introduce a family of additional expert-constructed test datasets and metrics that each isolate and measure a specific type of bias. Any time a model is evaluated on the primary task test set in this setting, it would be evaluated in parallel on these additional bias test sets. This would not prevent the primary metric from unintentionally and subtly rewarding biased models, but it would combat this effect by more directly highlighting and penalizing bias in models. In addition, the fact that these metrics would target specific types of biases would make it easier for benchmark maintainers to adapt as changing norms or changing downstream applications demand coverage of additional potential harms.
For several tasks, metrics like this already exist, at least for gender in English, in the form of auxiliary test sets meant to be combined with a pre-existing training set (Rudinger et al., 2018; Webster et al., 2018; Kiritchenko and Mohammad, 2018; Li et al., 2020). Even so, refining these metrics and developing new ones will likely require us to face many of the same challenges that we highlight in this paper for benchmark design more generally.
The larger challenge in implementing this approach, however, is a matter of community structure and incentive design. Methods papers dealing with tasks for which metrics already exist rarely report numbers on these metrics. Even for the SuperGLUE benchmark, which requires users to com
pute test set metrics on the DNC Winogender test set in order to reveal test set results for any other target task, a large majority of papers that report test set numbers omit this metric and decline to report potentially unflattering bias numbers (Raffel et al., 2020; Pruksachatkun et al., 2020; Schick and Schütze, 2020; He et al., 2020).
The difficulty, then, is in developing community infrastructure to encourage the widespread reporting of metrics that address the full range of relevant likely harms. This could plausibly involve peer review norms, explicit publication venue policies, stricter versions of the SuperGLUE approach for which users can only retrieve aggregate performance numbers, without a precise separation of the primary and bias-oriented metrics, or even the introduction of professional licensing standards.
Of course, ensuring that bias is measured and reported is not enough to prevent bias-related harms from emerging in practice: It is also necessary to ensure that those who build and deploy NLP products will take these metrics seriously and respond to them appropriately. And, of course, even if a system encodes no social bias at all, it can still be deployed in ways that produce unfair or unjust outcomes. These difficult issues are beyond the scope of a paper on benchmark design.
The NLP and ML research communities are increasingly interested in issues surrounding data and evaluation. This section surveys relevant positions and issues that don't quite fit our schema.
Welty et al. (2019) advocate for the more precise reporting of the focus and abilities of test sets and metrics in ML broadly, with a focus on issues surrounding statistical power. Bender and Friedman (2018) and Gebru et al. (2018) advocate for explicit freestanding datasheets documenting dataset releases of all kinds, with a focus on making potential harmful mismatches between data and application visible, and Hutchinson et al. (2021) argue along similar lines for a broader program of transparency and stakeholder engagement in data creation. Dodge et al. (2019) lay out a set of best practices for results reporting, with a focus on the impact of hyperparameter tuning on model comparison. Ethayarajh and Jurafsky (2020) advocate for the inclusion of efficiency considerations in leaderboard design. Boyd-Graber and Borschinger (2020) describe ways that trivia competitions can
provide a model for carefully-considered dataset design.
Church and Hestness (2019) revisit the arguments that motivated the NLP community's shift toward quantitative benchmarking in the early 1990s and warn that the overwhelming success of this shift has indirectly laid the groundwork for the widespread use of poor-quality benchmarks. Blodgett et al. (2020) challenge researchers working on social bias in NLP to focus more precisely on specific types of harm to specific populations of users, a challenge that our broad position piece does not fully meet.
NLP has had longstanding debates over the types of tasks that best test substantial language understanding skills. Many task-specific papers contribute to this debate, as does a prominent recent thread advocating for an increased focus on grounding of various kinds by Bender and Koller (2020), Bisk et al. (2020), Zellers et al. (2020), and others.
Benchmarking for NLU is broken. We lay out four major criteria that benchmarks should fulfill to offer faithful, useful, and responsible measures of language ability. We argue that departing from IID evaluation (as is seen with benchmark datasets collected by adversarial filtering) does not help to address these criteria, but lay out in broad strokes how each criterion might be addressed directly.
Nonetheless, important open research questions remain. Most centrally, it is still unclear how best to integrate expert effort into crowdsourced data collection, and we do not yet see a clear institutional model by which to ensure that bias metrics are built and used when they are most needed.
This paper advocates for reforms to a set of benchmarking practices that have so far largely failed to address issues of social bias, and that have thereby helped create a false sense of security among those building applied systems. While this paper offers no complete and satisfactory solutions, it proposes measures that should contribute to harm reduction.
We thank Emily Bender, Iacer Calixto, Haokin Liu, Kyunghyun Cho, Will Huang, Jamie Kiros, and audiences at CMU, Google, and Apple for feedback on these ideas.
This project has benefited from financial support to SB by Eric and Wendy Schmidt (made by recommendation of the Schmidt Futures program), Samsung Research (under the project Improving Deep Learning using Latent Structure), and Intuit. This material is based upon work supported by the National Science Foundation under Grant No. 1922658. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
Emily M. Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587-604.
Emily M. Bender and Alexander Koller. 2020. Climbing towards NLU: On meaning, form, and understanding in the age of data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5185-5198, Online. Association for Computational Linguistics.
Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, Nicolas Pinto, and Joseph Turian. 2020. Experience grounds language. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8718-8735, Online. Association for Computational Linguistics.
Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of "bias" in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5454-5476, Online. Association for Computational Linguistics.
Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam Kalai. 2016. Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS'16, page 4356-4364, Red Hook, NY, USA. Curran Associates Inc.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632-642, Lisbon, Portugal. Association for Computational Linguistics.
Samuel R. Bowman, Jennimaria Palomaki, Livio Baldini Soares, and Emily Pitler. 2020. New protocols and negative results for textual entailment data collection. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8203-8214, Online. Association for Computational Linguistics.
Jordan Boyd-Graber and Benjamin Borschinger. 2020. What question answering can learn from trivia nerds. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7422-7435, Online. Association for Computational Linguistics.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Dallas Card, Peter Henderson, Urvashi Khandelwal, Robin Jia, Kyle Mahowald, and Dan Jurafsky. 2020. With little power comes great responsibility. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9263-9274, Online. Association for Computational Linguistics.
Kenneth Ward Church and Joel Hestness. 2019. A survey of 25 years of evaluation. Natural Language Engineering, 25(6):753-767.
Robin Cooper, Dick Crouch, Jan Van Eijck, Chris Fox, Johan Van Genabith, Jan Jaspars, Hans Kamp, David Milward, Manfred Pinkal, Massimo Poesio, et al. 1996. Using the framework. Technical report, LRE 62-051 D-16, The FraCaS Consortium.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A. Smith. 2019. Show your work: Improved reporting of experimental results. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the
9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2185-2194, Hong Kong, China. Association for Computational Linguistics.
Dheeru Dua, Ananth Gottumukkala, Alon Talmor, Sameer Singh, and Matt Gardner. 2019. ORB: An open reading benchmark for comprehensive evaluation of machine reading comprehension. arXiv preprint 1912.12598.
Jesse Dunietz, Greg Burnham, Akash Bharadwaj, Owen Rambow, Jennifer Chu-Carroll, and Dave Ferrucci. 2020. To test machine comprehension, start by defining comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7839-7859, Online. Association for Computational Linguistics.
Kawin Ethayarajh and Dan Jurafsky. 2020. Utility is in the eye of the user: A critique of NLP leaderboards. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4846-4853, Online. Association for Computational Linguistics.
Allyson Ettinger, Sudha Rao, Hal Daumé III, and Emily M. Bender. 2017. Towards linguistically generalizable NLP systems: A workshop and shared task. In Proceedings of the First Workshop on Building Linguistically Generalizable NLP Systems, pages 1-10, Copenhagen, Denmark. Association for Computational Linguistics.
Marjorie Florestall. 2008. Is a burrito a sandwich? Exploring race, class, and culture in contracts. Michigan Journal of Race and Language, 14:1.
Karen Fort, Bruno Guillaume, and Hadrien Chastant. 2014. Creating Zombilingo, a game with a purpose for dependency syntax annotation. In Proceedings of the First International Workshop on Gamification for Information Retrieval, pages 2-6.
Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hannaneh Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, and Ben Zhou. 2020. Evaluating models' local decision boundaries via contrast sets. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 1307-1323, Online. Association for Computational Linguistics.
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daume III, and Kate Crawford. 2018. Datasheets for datasets. arXiv preprint 1803.09010.
Hila Gonen and Yoav Goldberg. 2019. Lipstick on a pig: Debiasing methods cover up systematic gender