SlowGuess/NAACL_2022_OCR huggingface dataset:
When it comes to automatic metrics, many practitioners believe more is better. A participant noted "the current status is you kind of just throw it all at the wall. So if there's another automatic metric, great, let's add that to the suite of metrics we're running" [P13], and another that they "ended up having a lot of metrics because none of them is a superset of any others. They all have issues" [P17].  $54\%$  of survey participants also agree or strongly agree it is a good practice to evaluate with as many automatic metrics as possible [SQ22]. Yet a "kitchen sink" approach to evaluation risks obfuscating quality with quantity, yielding apparent model improvements that may not correlate with actual performance, or may result in expectations to see improvements on certain metrics, even when those may not be necessary or desirable. This approach may exist partly to pay a "tech debt," where metrics are included mainly to compare to legacy baselines assessed with them.
Human evaluations are often assumed to be the gold standard, but the term refers to many things. A participant said they "always trust the human evaluations, that's the gold standard. If the human evaluation is not great, then we would not consider that a good solution" [P9]. This was also echoed by survey participants, with  $69\%$  believing automatic metrics should always or often be judged by how they correlate with human evaluations (74% academic, 63% non-academic) [SQ23]. Most participants assume human evaluations can be reliably used for NLG evaluation, with  $56\%$  agreeing or strongly agreeing that is the case for crowdsourcing;  $71\%$  for manual inspection by practitioners themselves; and  $89\%$  for metrics based on usage patterns in deployment settings [SQ20]. Given these variations in practitioners' trust in different types of human evaluations, the blanket use of the term "human evaluation" to refer to them collectively can make it hard to calibrate claims about human evaluation results.
Expert evaluations are believed to help elicit insights that are hard to obtain via crowdsourcing. Without proper training, a layperson might struggle to adequately assess generated text (Clark et al., 2021; Karpinska et al., 2021). One participant noted people "often get very impressed by neural text when researchers might be aware that it looks impressive because it stay[s] safe [and] just say[s] generic things" [P17]. In our survey,  $87\%$  of participants believe experts always or often contribute to the evaluation in ways standard crowdsourcing cannot [SQ23]. Experts may however be even less representative of those using a NLG system and may inject their shared biases into evaluations (van der Lee et al., 2019).
Practitioners say NLG evaluations are most constrained by datasets, annotators, lack of metrics, and inability to deploy. When asked to rank resources by how much they constrain their evaluations,  $59\%$  of survey participants ranked relevant datasets among their top 3 most limiting resources (48% academic, 70% non-academic), 50% ranked expert annotators, 42% the ability to deploy, 37% crowdsourcing costs (45% academic, 30% non-academic), 36% relevant metrics; with compute and engineering resources being least frequently ranked among top limiting resources.
Existing practices, metrics and data shape the evaluation choices of many practitioners. Participants noted that "academic benchmarks are very important [as] they're [an] easy measure to figure out whether your experiments are working" [P15] or that "we used all the basic standard automatic [metrics in] that suit" [P2]. When asked to rank considerations that guide their evaluations,  $55\%$  of survey participants ranked among their top 3 existing standards and practices (67% academic, 43% non-academic),  $36\%$  ranked dataset and metrics availability (45% academic, 26% non-academic), and  $29\%$  maintaining performance in deployed settings (12% academic, 46% non-academic) [SQ26]. This emphasis on existing practices, metrics and data may be due to challenges to developing them, expectations of audiences like a research community or a team, or to make it easy to compare with prior work. This reliance also makes their proper development and validation even more critical.
Poor performance is most often attributed to poor data quality, with some practitioners also believing "one [way] of driving quality upwards is by feeding in more data to these models" [P12]. Oth
ers echoed this belief, explaining how their "model tends to talk about things similar to its training data; whatever you feed into the model, that's what the model [will] output" [P7] as "language models are kind of bottom up, you have a pile of historical data and you learn the patterns of language used in that data" [P11]. In our survey,  $62\%$  of practitioners agree or strongly agree that output quality is primarily impacted by the quality of the training data (55% academic, 70% non-academic) [SQ22]. Practitioners may thus rely mostly on expanding or curating datasets to improve performance, overlooking other approaches (e.g., optimization methods, UX design) or offloading issues onto dataset builders (Sambasivan et al., 2021).
Practitioners believe datasets should be formally evaluated, but that rarely happens.  $71\%$  of survey participants say there is always or often a need to evaluate the quality of training datasets (77% academic, 63% non-academic), and  $87\%$  say so for evaluation datasets (97% academic, 77% non-academic) [SQ23]. An interview participant, however, pointed to the "concern of who is in charge of validating all these datasets. I'm sure some slightly harmful things can fall through the cracks, that no one takes a look at just due to the sheer size of these datasets" [P3]. Datasets that are not carefully evaluated may not be well-matched to what they aim to measure (Blodgett et al., 2021) or give rise to harms (e.g., Prabhu and Birhane, 2021).
3.4 Ethical Considerations and Impacts
Finally, we solicited practitioners' reflections on ethical issues or adverse impacts stemming from their projects, and if they try to measure or mitigate such issues. We use the term fairness and inclusion (F&I) to refer to a broad range of adverse impacts, like allocational and representational harms (Barocas et al., 2017; Blodgett et al., 2020). Instead of imposing our definitions, we deliberately asked broadly about F&I issues and other adverse impacts and ethical considerations to understand how practitioners conceptualize them.
Practitioners worry about adverse impacts, but NLG evaluations rarely include assessments of F&I issues. A participant emphasized that "developing [an] ethically correct [system that] takes care of things like safety and even fairness and inclusion is really important, [but] tackling that is not trivial" [P10]. While this interest in possible adverse impacts is encouraging, about  $37\%$  of survey
participants thought their project had no adverse impacts [SQ28], and only  $20\%$  report systematically measuring F&I issues beyond toxicity (10% academic, 30% non-academic) [SQ31]. Furthermore, while some practitioners were "glad things like [stereotyping benchmarks] are coming out, [although those are] pretty insufficient," [P12] and "there are currently no good evaluation practices for [F&I issues]" [SP]. Thus, their "metrics don't capture [F&I] right now" [P1] or were not "specifically designed to capture [F&I] issues" [P8].
To incorporate F&I in their evaluations, practitioners want more ready-to-use and scalable metrics. Participants also described how there are "20 metrics trying to measure gender and occupation [bias], but that's so narrow and [it's only] one indicator we have [and] often we don't know how [it] might relate to other forms of bias" [P15], and how there is no "ImageNet[-like option] for responsible AI [to do things at that] scale for language" [P7] that "practitioners and product teams [can] leverage in an easier way to scale these things up" [P14]. A notable  $34\%$  of survey participants also agree or strongly agree it is possible to develop automatic metrics to adequately assess F&I [SQ33]. This preference for scale and ease-of-use, however, may nudge practitioners towards relying on new metrics before their validity and reliability are properly assessed (e.g., Blodgett et al., 2021).
For some practitioners, F&I is a concern only for deployed models or systems. A participant noted they did not think "much [about ethical issues, as] the models [they] built never got deployed anywhere, never will most likely [get deployed], [they were] mostly just a proof of concept" [P8]. Furthermore,  $34\%$  of survey respondents agree or strongly agree F&I issues are less important for academic prototypes (26% academic, 43% non-academic), while  $18\%$  agree or strongly agree F&I is only relevant if NLG systems are deployed, and  $33\%$  that F&I issues can only be effectively measured in deployed settings (26% academic, 40% non-academic) [SQ33]. Models and prototypes are however sometimes repurposed, and deployed systems built upon prototypes or academic work not meant to be deployed may give rise to F&I issues.
F&I work is often seen as separate from NLG work, or deferred to someone else.  $36\%$  of survey participants agree or strongly agree F&I is a separate research area from NLG evaluation (26% academic, 47% non-academic) [SQ33]. Interview
participants also echoed this, with even those doing F&I work "having [separate papers] dedicated to studying toxicity or bias in language" since "the only concerns reviewers [have] is how many benchmarks we ran our model [on], they're not concerned about whether our [model] is more or less biased than [other models]" [P17]. Participants also defer the responsibility of F&I work to others working, e.g., on user facing features and "who [will later] further refine [a model's output]," or to researchers as "product team[s don't] get enough time to research [F&I] metrics" [P1]. Deferring F&I work to later stages of NLG development or to others may produce workflows where teams must frequently react and correct arising issues.
Ethical considerations are often reduced to "toxic" speech. When probed about ethical issues, many interview participants were concerned about generating or reproducing hateful, offensive, or toxic language: "a big consideration is [that] a lot of the data [could] in some ways be toxic content of different varieties" [P15] and "so often what these models will do is generate something that is very very unsafe and in some cases toxic" [P10]. This narrow focus on toxicity may inadvertently crowd out other F&I issues. Surprisingly, despite this focus on toxic language, only  $30\%$  of survey participants reported systematically evaluating whether generated text contains such language (23% academic, 37% non-academic) [SQ31], meaning even these issues might be under-reported.
Practitioners hold a range of beliefs about what properties the language in their datasets or generated by NLG systems should embody.  $52\%$  of survey participants agree or strongly agree that generated text should adhere to standard grammar rules of the language it is generated in [SQ33], and  $57\%$  say neutrality is a property NLG systems should embody [SQ37]. Many also want their system or model to capture either a "neutral voice"  $(30\%)$ , "no voice"  $(8\%)$ , or not to capture any particular writing or speaking voice  $(33\%$  overall;  $45\%$  academic,  $20\%$  non-academic) [SQ35].
By contrast, only  $18\%$  believed the datasets they use do not capture any voice, and  $18\%$  say they capture a neutral voice [SQ34], while  $21\%$  also believe their systems or models do not capture any voice, and  $16\%$  say they capture a neutral voice (10% academic, 23% non-academic) [SQ36]. Notably, while  $13\%$  of non-academic participants think their system captures a specific, non-neutral voice, none of
the academic participants do; this is also reflected in the interviews, where multiple industry participants aim to capture a specific type of voice like matching a "company voice" [P6]. Such beliefs about language are consistent with pervasive language ideologies—"the cultural system of ideas about social and linguistic relationships" (Irvine, 1989)—including that some kinds of language (and their speakers) are inherently correct, neutral, or appropriate for public use (Rosa and Burdick, 2017), and thus that generated language can be neutral or voiceless. Since assumptions about language are rarely named or interrogated, it remains unclear how they shape practitioners' evaluations. But they can give rise to harm; for example, a belief that race-related language is inherently inappropriate if generated (Schlesinger et al., 2018) can erase language by or topics important to minoritized users.
The topics and linguistic style of generated text are rarely systematically measured. When asked if they measured the linguistic style of generated text, a participant, under the assumption that "these models [just] follow the training data," noted they "don't necessarily analyze the generated output" [P9]. Only  $25\%$  of survey participants report systematically measuring what topics their system tends to generate, while  $18\%$  report systematically measuring the linguistic style of generated text (10% academic, 27% non-academic) [SQ31] (notably, though, 43% report informally assessing style or tone, e.g., via eyeballing). Coupled with assumptions about neutral or voiceless language, this suggests that the over- or under-representation of topics, perspectives, or language varieties may not be generally conceptualized as F&I issues and measured. NLG systems may, however, unintentionally prioritize "voices" over-represented in datasets, especially when assumed neutral and not examined.
To mitigate F&I issues, practitioners seek control over what is generated, as many do not want "an uncontrolled model to spew out random information" [P2]. Others want their systems "to divert the conversation into [less sensitive] domains" [P6] by "trying to control [what is generated using] different [e.g.] types of sampling" [P7]. While such mechanisms may prevent the generation of some problematic texts, most are just "blunt means of controlling the generation" [P12], and can lead to e.g., erasure harms if topics important to minoritized groups are deemed inappropriate (Schlesinger et al., 2018; Dodge et al., 2021): "[R]efusing to talk
about these issues probably means you are distributionally refusing to talk about things marginalized populations care about more” [P5].
Blocklists are often the go-to mitigation technique for F&I issues. A majority of interview participants report using blocklists as an F&I mitigation approach—including to control system outputs—and  $21\%$  of survey participants report always or often using blocklists (7% academic, 37% non-academic) [SQ30], with notably more use in non-academic settings. However, as one participant noted, “we have blocklists and phrasal blocklists [to help] prevent some of the major categories of failure, and they’re understandable and explainable and adjustable, which is the upside. But is the coverage good? I don’t think it is” [P12].
We identify practices and assumptions that shape the evaluations of NLG systems, which we argue deserve further attention due to their potential adverse impacts, particularly when those practices and assumptions go partly stated, or are not stated at all. Our findings suggest a number of approaches and future directions, including some raised by survey participants when asked how to improve NLG evaluation [SQ38].
First, the range of implicit assumptions and choices we uncover indicates that future evaluations might be strengthened by making these assumptions and choices explicit, including how quality criteria are conceptualized, how manual analysis is conducted and shapes decisions, and whether some evaluation approaches gate others (as automatic evaluations often gate human ones).
Participants also suggested several opportunities for evaluation; one respondent raised the possibility of lab studies to explore people's perceptions and uncover failure modes without fully deploying a system, while another emphasized the need to study systems in interaction given the importance of social and task context. Industry participants described a range of practices, including user engagement metrics, red teaming, and blocklists, which may be less well-studied in academia.
The challenges surrounding the use of automatic metrics—particularly the pressure towards a "kitchen sink" approach despite widespread recognition of many existing metrics' limitations—may not have an easy remedy. A "kitchen sink" approach should be thoroughly justified, and not used
only because practitioners lack clarity about their goals and quality criteria. Beyond clarity about goals and quality criteria, a better understanding of evaluation datasets and metrics—whose perspectives and language are contained in datasets, what datasets and metrics can and cannot capture, and how metrics are connected to user or downstream outcomes—may also clarify their appropriateness for different settings, and disincentivize simply evaluating with as many of them as possible.
Furthermore, human evaluations, particularly those grounded in specific deployment settings or involving extensive participation, tend to be costly. Not only is such work expensive, but it may also involve qualitative or participatory approaches less familiar to NLP practitioners (than using automatic metrics), or may slow down product releases or publications. Industry teams often constrained by tight deadlines may be especially unlikely to adopt qualitative or participatory approaches in the absence of readily available, well-validated methods and guidance for their use. While such approaches will never be as inexpensive as many automatic metrics, user studies or design workshops with users to help develop and solidify methods and guidance might lower barriers to their broader adoption. Finally, shifting publication incentives towards work grounded in deployment scenarios, as well as work engaging with methods from other disciplines (e.g., HCI or sociolinguistics) may encourage the investment of time and resources needed to carry out more thorough evaluations.
Turning to ethical considerations, our participants often raised the need for investment in the development of resources for conceptualizing and measuring F&I issues. These include better frameworks of harms (to help anticipate issues beyond toxicity), appropriate datasets, additional frameworks to guide (and scale) qualitative evaluations, task-specific metrics to combat automatic metric reuse without proper validation, and approaches for measuring latent qualities such as voice, style, and topic (which remain longstanding open questions). Other opportunities include broader disciplinary shifts. Several participants desired clearer community standards addressing not only best practices in modeling, but also meta-level questions—e.g., who makes decisions on what ethical considerations to prioritize (for evaluation and beyond), or how to develop repeatable mechanisms for surfacing, reporting, and addressing failures as they arise.
Our interview and survey studies—which are intended to uncover community assumptions, constraints, and practices—necessarily surface those of some but not all practitioners. Although we aimed to recruit as widely as possible, our participants were recruited via snowball sampling, seeded with researchers and industry groups that were well-known to us, as well as on social media (and therefore biased by our followers). Additionally, our interviews and surveys were only in English. Those who participated largely focus on research, work on English language NLG systems, and live and work in the Global North. While this may reflect the NLG research community and who has been able to participate in it, our findings may not reflect the assumptions or practices of practitioners working in other settings.
The results of our studies are also rather descriptive, and (though we think they are comprehensive) they cannot provide an exhaustive set of assumptions, practices, or F&I concerns, and should be interpreted accordingly. While our goal is to encourage further reflections about practices, the factors that shape them, and possible implications, we may risk instead inadvertently discouraging certain evaluation practices.
We sincerely thank Adam Atkinson, Alessandro Sordoni, Angelina Wang, Chad Atalla, Dallas Card, Eric Yuan, Jackie Chi Kit Cheung, Lucy Li, Marion Zepf, Tong Wang, the MSR FATE Group, and our anonymous reviewers for their support, and helpful reviews and feedback. We also want to thank our interview and survey participants for taking the time to participate in our research and for sharing their experiences with us. This research has been supported in part by a Stanford Graduate Fellowship.
Gavin Abercrombie, Amanda Cercas Curry, Mugdha Pandya, and Verena Rieser. 2021. Alexa, Google, Siri: What are your pronouns? gender and anthropomorphism in the design and perception of conversational assistants. In Proceedings of the 3rd Workshop on Gender Bias in Natural Language Processing, pages 24-33, Online. Association for Computational Linguistics.
Solon Barocas, Kate Crawford, Aaron Shapiro, and Hanna Wallach. 2017. The problem with bias: Allocative versus representational harms in machine learning. In Proceedings of SIGCIS, Philadelphia, PA.
Anya Belz, Simon Mille, and David M. Howcroft. 2020. Disentangling the properties of human evaluation methods: A classification system to support comparability, meta-evaluation and reproducibility testing. In Proceedings of the 13th International Conference on Natural Language Generation, pages 183-194, Dublin, Ireland. Association for Computational Linguistics.
Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of "bias" in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5454-5476, Online. Association for Computational Linguistics.
Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, and Hanna Wallach. 2021. Stereotyping Norwegian salmon: An inventory of pitfalls in fairness benchmark datasets. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1004-1015, Online. Association for Computational Linguistics.
Margarita Boyarskaya, Alexandra Olteanu, and Kate Crawford. 2020. Overcoming failures of imagination in AI infused system development and deployment. Navigating the Broader Impacts of AI Research Workshop at NeurIPS 2020.
Amanda Cercas Curry and Verena Rieser. 2018. #MeToo Alexa: How conversational systems respond to sexual harassment. In Proceedings of the Second ACL Workshop on Ethics in Natural Language Processing, pages 7-14, New Orleans, Louisiana, USA. Association for Computational Linguistics.
Kathy Charmaz. 2014. Constructing grounded theory. SAGE.
Elizabeth Clark, Tal August, Sofia Serrano, Nikita Hahuong, Suchin Gururangan, and Noah A. Smith. 2021. All that's 'human' is not gold: Evaluating human evaluation of generated text. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7282-7296, Online. Association for Computational Linguistics.
Jesse Dodge, Maarten Sap, Ana Marasovic, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. 2021. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. In Proceedings of the