SlowGuess/NAACL_2022_OCR huggingface dataset:
Kaitlyn Zhou  
Stanford University  
katezhou@stanford.edu
Su Lin Blodgett  
Microsoft Research  
sublodge@microsoft.com
Adam Trischler  
Microsoft Research  
adtrisch@microsoft.com
Hal Daume III  
Microsoft Research & University of Maryland  
me@hal3.name
Kaheer Suleman  
Microsoft Research  
kasulema@microsoft.com
Alexandra Olteanu  
Microsoft Research  
aloltea@microsoft.com
There are many ways to express similar things in text, which makes evaluating natural language generation (NLG) systems difficult. Compounding this difficulty is the need to assess varying quality criteria depending on the deployment setting. While the landscape of NLG evaluation has been well-mapped, practitioners' goals, assumptions, and constraints—which inform decisions about what, when, and how to evaluate—are often partially or implicitly stated, or not stated at all. Combining a formative semi-structured interview study of NLG practitioners  $(N = 18)$  with a survey study of a broader sample of practitioners  $(N = 61)$ , we surface goals, community practices, assumptions, and constraints that shape NLG evaluations, examining their implications and how they embody ethical considerations.
Evaluating natural language generation (NLG) models and systems—that generate new or altered text—is notoriously difficult, as often there are many valid ways to express similar content in text. This difficulty is often compounded by the need for NLG systems to meet a variety of goals—often measured in competing and imperfect ways—depending on their deployment settings (Sai et al., 2021). Such challenges have been mapped out for a variety of NLG tasks, systems, and settings (Novikova et al., 2017; Howcroft et al., 2020; Liang and Li, 2021), including examinations of how their design, deployment, and evaluation can give rise to ethical concerns (Cercas Curry and Rieser, 2018; Schlesinger et al., 2018; Hovy et al., 2020; Sheng et al., 2021; Abercrombie et al., 2021).
These challenges also mean that practitioners seeking to evaluate their NLG systems must decide on their goals for the system (e.g., help users write fluently), what quality criteria to use to reflect those goals (e.g., output fluency), and how to operationalize the measurement of those criteria (e.g.,
perplexity or crowd judgments). Such decisions are often guided by deployment settings, by evaluation norms in a research community (Gkatzia and Mahamood, 2015), by assumptions about "proper" NLG model or system behavior, as well as by real-world constraints (Sai et al., 2022; Gehrmann et al., 2021). Although the landscape of quality criteria and operationalizations has been surveyed, much less is known about how practitioners' goals, assumptions, and constraints shape their decisions about which criteria to use, when, how, and why. Limited visibility into how these factors shape NLG evaluations can, however, make it difficult to anticipate issues these factors might give rise to, or even what we can learn about research progress.
In this paper, we surface goals, assumptions, community practices, and constraints that NLG practitioners<sup>1</sup> work with when evaluating NLG systems, including limitations and ethical implications. These tacit elements of NLG evaluation are challenging to ascertain from literature surveys, as they are often unstated or only partially stated.
To reveal otherwise latent aspects of NLG evaluation practice, we apply a mixed-methods approach from human-computer interaction (HCI). Specifically, we conduct a formative semi-structured interview study  $(N = 18)$  and a survey study  $(N = 61)$  of NLG practitioners ( $\S 2$ ) in order to identify common themes related to practitioners' goals ( $\S 3.1$ ), practices ( $\S 3.2$ ), and assumptions and constraints ( $\S 3.3$ ) when evaluating NLG systems, including ethical considerations ( $\S 3.4$ ). By recognizing more tacit elements of NLG evaluation, our work aims to provide scaffolding for discerning the issues they may give rise to and help re-think NLG evaluations.
<sup>1</sup>In this study, we dub "practitioners" all those with experience in researching, developing, deploying, or evaluating NLG models/systems. This includes both academic and industry researchers, as well as applied scientists and engineers.
Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 321 - 348 July 10-15, 2022 ©2022 Association for Computational Linguistics
2 Background and Methods
To examine NLG evaluation practices—along with the goals, assumptions, and constraints that inform them—and whether they incorporate assessments of possible adverse impacts, we first conducted a formative semi-structured interview study of NLG practitioners (§2.1). To probe the themes emerging from our interviews at greater scale, we subsequently conducted a survey study with a broader sample of practitioners (§2.2).
Participant recruitment and IRB. We recruited NLG practitioners using snowball sampling (Parker et al., 2019), via targeted emails and social media posts, with a request to share with relevant groups. Each interviewee received a  $25 gift card for a 45 minute video-conference interview, while survey participants could enter a raffle for one of ten$ 50 gift cards. Both studies were IRB approved, and we obtained informed consent from all participants.
2.1 Interview Study
To design our study, we drew on the measurement modeling framework (Jacobs and Wallach, 2021) to tease apart practitioners' conceptualization of quality criteria from how they measure those criteria. We also investigated how current NLG evaluation practice grapples with possible adverse impacts, such as fairness and inclusion issues (Sheng et al., 2021; Weidinger et al., 2021).
We began our interviews with a) background questions on participants' NLG projects and experience to establish context. Subsequent questions broadly asked about b) practitioners' overall project goals, c) the quality criteria they want to assess their system on, and d) how they measure those criteria—with c) and d) following the measurement modeling framework. We also asked participants about adverse impacts of their work, and whether they measure or assess them.
Before conducting the final interviews, we first ran pilot interviews  $(N = 5)$  with practitioners from both product and research organizations to identify possible clarity issues with our questions and study protocol. The full interview protocol is available in Appendix A. Overall, we interviewed 18 practitioners from 12 organizations (Table 1).
Identifying interview themes. To analyze interview transcripts, we used a bottom-up approach rooted in grounded theory (Strauss and Corbin, 1997; Charmaz, 2014), following Robertson et al. (2021). We iteratively coded and thematically
Sector	Interview	Survey
Academia	8	31
Industry	8	17
Academia & Industry	2	10
Gov./NGO	-	3
Role
Professor	2	12
Researcher or Postdoc	9	16
Graduate Student	5	17
Manager	1	3
Engineer or Data Scientist	1	6
Other	-	7
Total	18	61
Table 1: Overview of interview and survey participants.
sorted interview excerpts by looking for relations with or among already assigned codes. Specifically, we first distributed the interview transcripts across all authors for open coding, with the resulting codes being then discussed by all authors to identify and agree upon clusters of common themes.
To organize codes into themes, we drew again on measurement modeling to distinguish between codes for what practitioners want to measure (quality criteria) and codes for how practitioners operationalize those measurements (evaluation practices). To examine which factors affect practitioners' decisions about what criteria they prioritize and operationalize, we also consider the goals, assumptions and constraints that underpin their evaluation practices, clustering codes into related themes. Finally, we identify themes on how practitioners conceptualize and embody possible adverse impacts.
2.2 Survey Study
We designed our survey around themes from the interview study, starting again with questions about participants' background, and organizing the remaining questions under a few broad themes—a) quality criteria and goals, b) evaluation practices, c) evaluation rationales (including assumptions and constraints), and d) adverse impacts and ethical considerations—with different sets of questions matching different themes. Similarly with the interview study, before sharing and advertising our survey, we piloted it with a few industry researchers  $(N = 3)$  to identify possible clarity issues with our survey questions and protocol. See Appendix B for full survey script. To filter out spammers, we relied on 3 of the open-text questions—the description of participants' occupation, description of their NLG project, and their reflections on how to improve NLG evaluation. After removing spammers, we analyzed responses from 61 participants (Table 1).
3 Interview and Survey Findings
Across both studies, we observe common themes in participants' responses that we overview below. Interview participant quotes—anonymized and paraphrased for brevity and clarity—are followed by "P" and a participant ID, while quotes from survey responses are only marked by "SP." Statistics based on survey responses are followed by "SQ" and the question number. We report disaggregated statistics as (X% academic, Y% non-academic) when gaps exist between academic and non-academic participants. We note, however, that the statistics we report are only indicative as our respondent sample is skewed towards industry and academic researchers (Table 1), and not necessarily representative of the broader NLG community.
3.1 Evaluation Goals and Quality Criteria
What practitioners want their NLG systems or models to do (goals) and what they aim to assess to determine whether those goals are met (quality criteria) both shape which evaluation practices they are likely to follow. Participants' reflections on what success looks like for their projects and which quality criteria they deem critical could thus help discern the implications of the resulting practices.
The goals of NLG work are often disconnected from deployment settings or users. Our participants describe diverse goals, like imbuing bots with social skills or helping users write fluently. However, many practitioners do not have clarity about or consider possible deployment settings or users, e.g., “[W]e were not really thinking [about users], usually I try not to think about [users]” [P8]. This is especially true when participants had no deployment plans, including in non-academic settings—26% of all survey participants (39% academic, 13% non-academic) [SQ16]. A lack of clarity about possible deployment scenarios, however, can make it difficult to assess how appropriate certain metrics are (Blodgett et al., 2021) or even to imagine adverse impacts (Boyarskaya et al., 2020).
While practitioners conduct NLG evaluations to report to a variety of audiences, paper reviewers and the research community are most often mentioned. While we see differences between academic and non-academic participants—with the latter thinking more about specific audiences like their team or manager (19% academic, 53% non-academic) or other teams (16% academic, 50% non-academic)—77% of survey participants
say one goal of their evaluation was to report results in a paper (90% academic, 63% non-academic) and 59% to report to a research community (67% academic, 50% non-academic) [SQ13]. If reporting results in papers or to research communities represents a significant goal of many practitioners, however, then evaluations already legible to a research community may be over-incentivized, and quality criteria and measurements unfamiliar or uncommon to that community may be discouraged.
Quality criteria like correctness, usability, coherence, and fluency are often prioritized. While our participants mention a diversity of criteria,  $77\%$  of survey respondents deemed correctness among most important criteria (83% academic, 70% non-academic), followed by  $49\%$  usability (38% academic, 60% non-academic),  $47\%$  coherence, and  $44\%$  fluency; with criteria like readability (21%) and clarity (21%) being least frequently selected [SQ14]. Interview participants mention similar criteria, e.g., “[W]hout [even] figuring out how we measure [these criteria], we want the [text] to be informative [and] fluent” [P4]. This corroborates prior work that found task usefulness, fluency, grammaticality, relevance, or generic notions of output quality among most commonly used criteria for NLG evaluation (Howcroft et al., 2020; Liang and Li, 2021). This may stem from an assumption that criteria like correctness, coherence, and fluency, as intrinsic properties of text, are easier to measure and better-suited to automatic metrics (Gkatzia and Mahamood, 2015) than more extrinsic criteria related to how text is used (i.e., usability).
User engagement is often foregrounded in deployment settings. When systems are deployed, practitioners want "users to be happy [and] engaged" [P12] and may prioritize feedback and engagement-based metrics, like "whether [the generated suggestions] would be accepted by the end user" [SP] and "the number [of] clicks [or] of times people accept your suggestions" [P7]. However, users and their feedback may not be representative and may, e.g., "skew[] heavily towards enterprise customers" [P12]. In and of itself, focusing on specific, non-representative groups is not necessarily problematic, but designing to meet the needs of an existing user base might have exclusionary outcomes if that user base is not reflective of a larger population.
Practitioners may conflate different quality criteria if they lack a shared understanding of
their meanings, or if the criteria lack clear conceptualizations. As in prior work (Belz et al., 2020; Gkatzia and Mahamood, 2015; Howcroft et al., 2020; Liang and Li, 2021; Sai et al., 2022), we see a tendency to conflate different quality criteria. A participant noted that “[linguists tell] me [fluency and grammaticality] are clearly distinct from each other [but in NLP] most people don’t know that distinction” [P8], and another that “a lot of these descriptive [terms for quality criteria] have overlap” which is particularly noticeable when “trying to evaluate them by hand” [P3]. Different understandings of criteria or the conflation of criteria may have consequences beyond the concern that good measurements require clarity about what is being measured (Blodgett et al., 2021). If e.g., grammaticality is understood as adherence to Mainstream U.S. English grammar rules (often the case due to standard language ideologies (Rosa and Flores, 2017)), then it will exclude other language varieties, like Appalachian English (Luhman, 1990) or African American Language (Lanehart, 2015) and thus their speakers. Given the focus on fluency, this exclusion may be compounded if grammaticality and fluency are conflated.
Quality criteria are also conflated with measurements of those criteria. When asked about key performance criteria, some participants instead talk about what metrics they use, e.g., "we have to make [a text] generator that's good [and] we ensured the perplexity [was] on par with the state of the art perplexity. Perplexity was [the] performance metric we [used]" [P17]. To determine if perplexity is a good metric for the "goodness" of a text generator, however, we must clearly articulate what quality criteria a good generator should meet. Without this clarity, we may focus too strongly on a metric—like perplexity—and risk losing the ability to assess the metric's appropriateness and distinguish between the metric and the original goal.
3.2 Evaluation Practices
Existing evaluation practices can also provide insight into what is deemed important to evaluate, what is overlooked during evaluation, and how quality criteria are being operationalized.
Although routine, the use of automatic metrics may lack justification and validation, as "[e]veryone is in this in-between phase of just using everything they can find" [P3]. Automatic metrics commonly include training/validation loss, per
plexity, and string or word matching metrics (e.g., BLEU, ROUGE, METEOR, NIST), with  $85\%$  of survey participants reporting always or often using automatic metrics (94% academic, 77% non-academic) [SQ18]. Metrics developed for one task ("BLEU was [developed for] MT" [P3]) are also often adapted or used as-is for other tasks ("we [compute] BLEU for language modeling" [P8]), with  $57\%$  of participants saying they always or often use metrics developed for other NLG tasks (68% academic, 47% non-academic) [SQ18]. This practice can be risky if metrics validated in one setting do not capture key criteria in other settings.
Different types of human evaluation are often used to complement automatic metrics; one participant noted they conduct "human evaluation[s] just 'cause NLG metrics are still so messy" [P3]. When our interview participants discussed what human evaluations they conducted, they described several types, including crowdsourcing, user feedback or engagement metrics, expert evaluations, and qualitative examinations by practitioners or their colleagues. Overall,  $89\%$  of survey participants report always or often using some form of human evaluation (94% academic, 83% non-academic) [SQ18]. This also means "human evaluation" is often imprecisely used to refer to many types of evaluations and settings.
Automatic metrics often gate which models are evaluated by crowdsourcing. While  $34\%$  of survey participants report always or often performing crowdsourced evaluations [SQ18], many interview participants also say they select models based on their performance on automatic metrics before investing in crowdsourced evaluations, since "[there's a] huge search space [and we] pick those that get good automatic metric values" [P12]. Indeed,  $47\%$  of those who report performing crowdsourced evaluations also note crowdsourcing is always or often contingent on performing well on automatic metrics (57% academic, 36% non-academic) [SQ18]. However, this practice might overlook models for which human evaluations anticorrelate with automatic metrics, as "automatic metrics may show something and the human evaluation may show the complete opposite. Unless you do human evaluations, you will not have a thorough picture of [how well your system is doing]" [P2].
Human evaluations may draw on usage or feedback data in deployed settings, but many practitioners lack access to either deployment set-
tings or usage data.  $30\%$  of survey participants report conducting evaluations with users in deployed settings, with a sizeable gap among academics  $(12\%)$  and non-academics  $(46\%)$  [SQ18]. Interview participants also noted "this gap between academic [evaluations and the] things that matter when you actually deploy something out in the world" [P15]. Since feedback from users of a deployed system might more accurately reflect their in-situ needs than feedback from e.g., crowdworkers, practitioners without access to deployment settings may miss critical evaluation data and have little visibility into possible adverse impacts.
Practitioners manually assess generated texts, but rarely follow a formal method when doing so. For some participants, manual assessment is more of an art than a science, and many do not follow any established methods or a self-devised protocol when doing so: "it just comes down to me reading a lot of samples and then choosing the one which overall seems to be better" [P5]. Indeed, while  $77\%$  of survey participants report always or often manually examining generated texts, of those who perform such qualitative examinations, only  $38\%$  report always or often applying a formal method (30% academic, 47% non-academic) [SQ18], and even fewer describe such a method, with a few using a labeling scheme or a sampling protocol for selecting which examples to assess [SQ19]. While common, the practice of (informally and) manually examining generated text to assess quality is rarely detailed in the literature. It can be problematic if results of error analyses (Van Miltenburg et al., 2021) are shown without describing how assessments were made. Practitioners also may not be representative of those who use NLG systems or are reflected in generated text (Hovy and Spruit, 2016).
While expert evaluations are sought after, it can be unclear what type of expertise is needed. Practitioners also seek out evaluators with certain expertise, like English teachers or linguists, e.g., "we hire English teachers [who are] experts, do a much better job and give very detailed feedback" [P8]. In our survey,  $25\%$  of participants report seeking out expert judges for evaluation purposes (32% academic, 17% non-academic) [SQ18]. Yet even "experts" may need to "[see] a lot of [generated] text and [internalize] the sort of a language [a model tends to produce]" [P17].
Practitioners often know little about those rep
resented in or contributing to data creation and annotation, or to crowdsourced evaluation. Only  $18\%$  of survey participants report always or often having demographic information about the annotators or authors who created the datasets they use [SQ18]. Of those who report using crowdsourcing, only  $30\%$  report always or often having demographic information about crowdsourcing judges [SQ18]. Such information can provide crucial insight into the speakers, communities, language varieties, and perspectives represented in and shaping datasets; without it, some datasets—whose text may read to some as if “it’s written by [a] small group of people” [P17)—risk being used as representative while excluding wide swaths of people.
Many practitioners re-purpose datasets created for other tasks, with  $59\%$  of survey participants reporting always or often doing so [SQ18], while only  $48\%$  of survey participants report always or often using datasets created in-house for their project (42% academic, 53% non-academic) [SQ18]. Datasets' suitability should however be carefully interrogated before being re-purposed. If datasets custom-built or validated for other tasks do not properly capture the new task's peculiarities or the desired quality criteria, they can pose threats to the validity of the task and its evaluation.
3.3 Rationales for Evaluation Practices
To further probe participants about assumptions, constraints and other considerations that shape their evaluation practices, we also explicitly asked them about what guided their NLG evaluations.
While many practitioners recognize their limitations, automatic metrics are often assumed reliable enough. A participant noted they "wouldn't trust any sort of automatic measure of a text generation system [as they need] more than just a good BLEU or ROUGE score before [they'd] sign off on using a language model" [P11], while others questioned whether automatic metrics "capture anything meaningful" [P13] when assessing latent constructs like creativity.
Despite these and other documented shortcomings (Gkatzia and Mahamood, 2015; Novikova et al., 2017; Kuribayashi et al., 2021; Liang and Li, 2021), practitioners do rely broadly on automatic metrics:  $50\%$  of survey participants agree or strongly agree that automatic metrics represent reliable ways to assess NLG systems or models [SQ20], while  $43\%$  say that metrics developed
for one NLG task can be reliably used or adapted to evaluate other NLG tasks (32% academic, 53% non-academic) [SQ22]. One participant remarked that "automatic metric[s are] still more scalable and objective than human evaluation" [SP]. This trust in automatic metrics' reliability across tasks may entrench their use, possibly crowding out other measurement approaches, especially as publication processes may reward metrics that allow for easy comparison to existing models.