SlowGuess/NAACL_2022_OCR huggingface dataset:
2021 Conference on Empirical Methods in Natural Language Processing, pages 1286-1305, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Anuoluwapo Aremu, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna-Adriana Clinciu, Dipanjan Das, Kaustubh Dhole, Wanyu Du, Esin Durmus, Ondrej Dusek, Chris Chinenye Emezue, Varun Gangal, Cristina Garbacea, Tatsunori Hashimoto, Yufang Hou, Yacine Jernite, Harsh Jhamtani, Yangfeng Ji, Shailza Jolly, Mihir Kale, Dhruv Kumar, Faisal Ladhak, Aman Madaan, Mounica Maddela, Khyati Mahajan, Saad Mahamood, Bodhisattwa Prasad Majumder, Pedro Henrique Martins, Angelina McMillan-Major, Simon Mille, Emiel van Miltenburg, Moin Nadeem, Shashi Narayan, Vitaly Nikolaev, Andre Niyongabo Rubungo, Salomey Osei, Ankur Parikh, Laura Perez-Beltrachini, Niranjan Ramesh Rao, Vikas Raunak, Juan Diego Rodriguez, Sashank Santhanam, João Sedoc, Thibault Sellam, Samira Shaikh, Anastasia Shimorina, Marco Antonio Sobrevilla Cabezudo, Hendrik Strobelt, Nishant Subramani, Wei Xu, Diyi Yang, Akhila Yerukola, and Jiawei Zhou. 2021. The GEM benchmark: Natural language generation, its evaluation and metrics. In Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021), pages 96–120, Online. Association for Computational Linguistics.
Dimitra Gkatzia and Saad Mahamood. 2015. A snapshot of NLG evaluation practices 2005 - 2014. In Proceedings of the 15th European Workshop on Natural Language Generation (ENLG), pages 57-60, Brighton, UK. Association for Computational Linguistics.
Dirk Hovy, Federico Bianchi, and Tommaso Fornacciari. 2020. "you sound just like your father" commercial machine translation systems include stylistic biases. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1686-1690, Online. Association for Computational Linguistics.
Dirk Hovy and Shannon L. Spruit. 2016. The social impact of natural language processing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 591-598, Berlin, Germany. Association for Computational Linguistics.
David M. Howcroft, Anya Belz, Miruna-Adriana Clinciu, Dimitra Gkatzia, Sadid A. Hasan, Saad Mahamood, Simon Mille, Emiel van Miltenburg, Sashank Santhanam, and Verena Rieser. 2020. Twenty years of confusion in human evaluation: NLG needs evaluation sheets and standardised definitions. In Proceedings of the 13th International Conference on Natural Language Generation, pages
169-182, Dublin, Ireland. Association for Computational Linguistics.
Judith T. Irvine. 1989. When talk isn't cheap: Language and political economy. American Ethnologist, 16(2):248-267.
Abigail Z. Jacobs and Hanna Wallach. 2021. Measurement and fairness. Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency.
Marzena Karpinska, Nader Akoury, and Mohit Iyyer. 2021. The perils of using Mechanical Turk to evaluate open-ended text generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1265-1285, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Tatsuki Kuribayashi, Yohei Oseki, Takumi Ito, Ryo Yoshida, Masayuki Asahara, and Kentaro Inui. 2021. Lower perplexity is not always human-like. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5203-5217, Online. Association for Computational Linguistics.
Sonja Lanehart. 2015. The Oxford Handbook of African American Language. Oxford University Press.
Chris van der Lee, Albert Gatt, Emiel van Miltenburg, Sander Wubben, and Emiel Krahmer. 2019. Best practices for the human evaluation of automatically generated text. In Proceedings of the 12th International Conference on Natural Language Generation, pages 355-368, Tokyo, Japan. Association for Computational Linguistics.
Hongru Liang and Huaqing Li. 2021. Towards standard criteria for human evaluation of chatbots: A survey. arXiv preprint arXiv:2105.11197.
Reid Luhman. 1990. Appalachian english stereotypes: language attitudes in kentucky. Language in Society, 19(3):331-348.
Jekaterina Novikova, Ondrej Dusek, Amanda Cercas Curry, and Verena Rieser. 2017. Why we need new evaluation metrics for NLG. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2241-2252, Copenhagen, Denmark. Association for Computational Linguistics.
Charlie Parker, Sam Scott, and Alistair Geddes. 2019. Snowball sampling. SAGE research methods foundations.
Vinay Uday Prabhu and Abeba Birhane. 2021. Large image datasets: A pyrrhic win for computer vision? 2021 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1536-1546.
Ronald Robertson, Alexandra Olteanu, Fernando Diaz, Milad Shokouhi, and Peter Bailey. 2021. "I can't reply with that": Characterizing problematic email reply suggestions. In CHI Conference on Human Factors in Computing Systems (CHI '21). ACM.
Jonathan Rosa and Christa Burdick. 2017. Language Ideologies, pages 103-124. Oxford University Press.
Jonathan Rosa and Nelson Flores. 2017. Unsettling race and language: Toward a raciolinguistic perspective. Language in Society, 46(5):621-647.
Ananya B. Sai, Tanay Dixit, Dev Yashpal Sheth, Sreyas Mohan, and Mitesh M. Khapra. 2021. *Perturbation CheckLists* for evaluating NLG evaluation metrics. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7219-7234, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Ananya B. Sai, Akash Kumar Mohankumar, and Mitesh M. Khapra. 2022. A Survey of Evaluation Metrics Used for NLG Systems. ACM Comput. Surv., 55(2).
Nithya Sambasivan, Shivani Kapania, Hannah Highfill, Diana Akrong, Praveen Paritosh, and Lora M Aroyo. 2021. "everyone wants to do the model work, not the data work": Data cascades in high-stakes ai. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, CHI '21, New York, NY, USA. Association for Computing Machinery.
Ari Schlesinger, Kenton P. O'Hara, and Alex S. Taylor. 2018. Let's talk about race: Identity, chatbots, and ai. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, CHI '18, page 1-14, New York, NY, USA. Association for Computing Machinery.
Emily Sheng, Kai-Wei Chang, Prem Natarajan, and Nanyun Peng. 2021. Societal biases in language generation: Progress and challenges. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4275–4293, Online. Association for Computational Linguistics.
Anselm Strauss and Juliet M Corbin. 1997. Grounded theory in practice. Sage.
Emiel Van Miltenburg, Miruna Clinciu, Ondrej Dusek, Dimitra Gkatzia, Stephanie Inglis, Leo Leppänen, Saad Mahamood, Emma Manning, Stephanie Schoch, Craig Thomson, et al. 2021. Underreporting of errors in nlg output, and what to do about it. In Proceedings of the 14th International Conference on Natural Language Generation, pages 140-153.
Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. 2021. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359.
Appendix for: Deconstructing NLG Evaluation: Evaluation Practices, Assumptions, and Their Implications
A Interview Script
A.1 Introduction
We are conducting a semi-structured interview regarding your experience with natural language generation models and systems evaluation. We hope to learn about general practices, goals, constraints and assumptions researchers and practitioners make when evaluating NLG systems. Before we start, could I get verbal consent from you that it is okay to record this meeting and take notes?
We aim to have the interviews last 45 minutes, and we will cover about 15 high-level questions. Finally, we would love to hear your thoughts about this process and what can be improved. We will also send you a gift card via email. Do you have any questions before we begin?
A.2 Background Information
[IQ1] Could you describe your experience with developing, deploying, and researching NLG systems? [IQ2] Of the projects you've been involved with, can you describe one or two of them?
- What does it do? Why is it needed?
Who are your customers or stakeholders?
- Try to focus on one, but feel free to bring in examples and anecdotes from others as you see fit.
A.3 What are the stakeholders and goals?
[IQ3] What performance criteria are important for your system/project/model?
- What does success look like for your system/project/model?
- What are the goals of your system/project/model?
- Who are you optimizing for?
[IQ4] Can you describe the decision-making process for evaluating NLG models/systems on your team?
- For example, what does the process of adding a new evaluation metric look like?
- What constraints do you have on your project?
- Can you describe the makeup of your team? What is your role?
[IQ5] What are you trying to demonstrate with these results?
[IQ6] Are there any ethical issues or adverse impacts that you are concerned about in the context of your project?
- Are there also any F&I concerns you have related to your system/project/model?
A.4 What do you want to measure?
[IQ7] Which criteria does your evaluation aim to capture? Possible criteria can include: fluency, readability, coherence, naturalness, quality, correctness, usability, clarity, informativeness, accuracy. Feel free to add others.
[IQ8] [If nothing on fairness] Are your metrics trying to capture issues related to ethical AI or NLP? This can include qualities or metrics related to or for fairness and inclusion.
- Fairness issues: e.g., do you look for performance disparities, bias, lack of representation
- Inclusion issues: e.g., politeness, professionalism, accessibility, hate speech
What are barriers to evaluating for F&I?
- What F&I issues are and aren't being captured well?
[IQ9] How would you characterize your experience with practices and metrics that aim at examining issues related to ethical AI/NLP? Possible answers could be: no experience, a bit of experience, I have worked on it, the focus of my work
A.5 What do you measure? How?
[IQ10] Are you using automatic metrics like BLEU, ROUGE, METEOR, NIST in the development process? When?
- Which ones? Why?
- Are you using metrics based on some sort of "gold" or "reference" responses/datasets?
- Are your datasets made in-house specifically for your project or re-used from prior work?
- Do you have metadata about who created the references and how they were created?
- Is your system optimizing for a specific sort of prototypical voice or persona that is captured by these references?
[IQ11] Are you using human evaluations? When?
- If applicable, is this contingent on a certain performance on automatic metrics?
- If system is deployed, do you conduct evaluations with real users? How?
- What is your typical budget for evaluations both time and money?
- Which criteria do these human evaluations capture that your automatic metrics miss?
[IQ12] If your evaluation metrics give conflicting results, which metrics do you trust?
- Which evaluation metrics do you consider to be the most useful in your development process?
[IQ13] Are there any other approaches you use to evaluate?
[IQ14] Do you try to measure how representative the content produced by your NLG system/model is?
- What does your model avoid "talking" about and what it tends to "talk" about?
- Do you use red teaming to test for worst-case scenarios and/or blocklists to avoid known issues? If no, can you describe what it could be?
- Are there any noticeable tendencies in the content produced by your systemmodel?
[IQ15] Do you measure the linguistic style and tone of your model?
- Are you aware of whether the content produced by your system/model has a certain tone?
- Is there a specific linguistic style or tone that you are aiming for?
- Do you measure what the style or tone of the content generated by your model is?
- Does it tend to represent a certain type of voice or persona?
- If applicable, do you know if users respond differently to the different voices captured by your system/model? If no, can you describe what it could be?
A.6 Reflections on your practices
[IQ16] Finally, are you aware of other evaluation practices? Are your practices in line? Is there anything else you'd like us to know?
B Online Survey Questions
B.1 Consent Form
[SQ1] Do you understand and consent to these terms? R: Yes, no
[SQ2] Please type your initial here if we may contact you in the future to request consent for uses of your identifiable data that are not covered in this consent form. R: Write-in
[SQ3] Please type your initial here if we may contact you in the future with information about follow-up or other future studies. R: Write-in
B.2 Background
[SQ4] Do you have experience with developing, deploying, or researching natural language generation (NLG) systems or tasks?
- No, I have never worked with or on NLG
- I have limited experience with NLG
- Developing, deploying and researching NLG is part of my day-to-day job
I am an expert in NLG
[SQ5] Do you have experience with evaluating natural language generation systems or tasks?
- No, I have never worked with or on NLG evaluation
- I have limited experience with NLG evaluation
- NLG evaluation is part of my day-to-day job
I am an expert in NLG evaluation
[SQ6] What type of NLG system(s) or task(s) have you worked on or with? Check all that apply.
- Machine translation, Paraphrasing and summarization, Question answering, Style transfer, Image captioning, Storytelling and narrative generation, Writing assistant, Content and text planning, Affect/emotion generation, Dialogue system, NLG for embodied agents and robots, Prompt completion, Other
[SQ7] What is the title of your occupation? R: Write-in
[SQ8] What sector(s) are you employed in? (Select all that apply)
- Academia, Industry, Government, Non-profit, Other
[SQ9] How many years of experience do you have in your current field of occupation?
- Less than 1 year, 1 - 4 years, 5 - 9 years, 10-20 years, More than 20 years
[SQ10] Is research publication the goal of your work?
- Never, Rarely, Sometimes, Often, Always
[SQ11] How many people are working on your particular NLG system or task? (How big is your team? If your project is open-sourced, approximately how many regular contributors do you have?)
1-2,3-4,5-9,10-19,20+
[SQ12] In a couple of sentences, what is the overarching goal of your NLG system or task? Feel free to include additional pertinent context for the project such as domain, dataset, task, user base etc. If you have worked on multiple NLG systems and/or tasks, please pick the one you have the most experience with, in terms of evaluation. Please answer the remainder of the survey questions with this system in mind.
R: Write-in
B.3 Quality criteria and goals: what do you want and try to measure at evaluation time?
For the following questions, please consider the NLG system or task you briefly described earlier.
[SQ13] When you evaluate your system/model, what are the goals of this evaluation? Check all that apply.
- Evaluation for reporting in a paper.
- Evaluation for reporting to a research community.
- Evaluation for reporting to your team or manager.
- Evaluation for reporting to other teams or third-party platforms.
- Evaluation for model development, tuning and hyperparameter search.
- Evaluation for ensuring the on-going performance of a deployed system.
Write-in
[SQ14] Of the following, which are the most important performance criteria to evaluate for your NLG system or task? Select up to three or write your own. For more detailed definitions, please refer to the appendix here: https://aclanthology.org/2020.inlg-1.23.pdf
- Fluency (the output text 'flows well', and is not a sequence of unconnected parts)
- Readability (the output text is easy to read)
- Coherence (the output text is presented in a well-structured, logical and meaningful way)
- Naturalness (the output text is likely to be used or be written by a native speaker)
- Correctness (the output text is correct/true relative to the input)
- Usability (the output text is usable in the context of a given task/application)
- Clarity (the output text is easy to understand)
- Informativeness (the amount of information conveyed by the output text)
Write-in
[SQ15] Are there other criteria (than those listed above) that are important for your NLG system or task?
[SQ16] Is your work (or will your work be) part of an NLG system that is deployed to users in real settings (users who use the system for their own reasons, self-incentivized, and are not paid as test subjects/crowd workers)?
- Yes, no, maybe
[SQ17] Is your goal or one of your future goals to personalize the voice of your system for users? By voice we mean that it may capture a particular style of speaking or writing, or a particular persona.
B.4 Evaluation practices: what do you do during evaluation?
[SQ18] For the following statements, please answer how often you or others on your team do the following. Never, Rarely, Sometimes, Often, Always, (N/A)
- Use automatic evaluation metrics
- Use automated metrics originally developed for other NLG tasks (off the shelf).
- Use online crowdsourcing for model/system evaluation (e.g., Mechanical Turk)
- If used, online crowdsourcing evaluation is contingent on your system or model achieving a certain performance level on some automatic metrics.
- During development, you or others on your team manually examine or evaluate the outputs of your NLG system or model.
- Follow a formal method or procedure when you or your team members are manually inspecting/evaluating the system/model output (beyond just "eyeballing"; e.g., applying a coding scheme)
- Evaluate with users in actual deployed settings (users who use the system for their own reasons, self-incentivized, and are not paid as test subjects/crowd workers)
- Seek out judges or annotators with certain specific expertise to assist with your system or model evaluation (e.g., linguistics)
- Use datasets that were made in-house specifically for your project
- Re-use datasets (as-is or modified) that were created for other tasks or work (typically by others outside your team)
- Have demographic information about the judges, annotators, or writers who created the datasets or reference solutions you use
- Have demographic information about online crowdsourcing judges or annotators who evaluated the outputs of your system/task
[SQ19] If you follow a formal method or procedure for your manual evaluation or inspection of the system/model output, could you please describe your method? R: Write-in
B.5 Evaluation rationales: why do you follow certain evaluation practices?
[SQ20] For the system or task, you described earlier, do you agree or disagree that the following evaluation methods are reliable ways to assess performance? Strongly agree, agree, neither agree nor disagree, disagree, strongly disagree
- Using standard automatic metrics (e.g., BLEU, ROUGE, METEOR, NIST, perplexity)
- Researchers and practitioners themselves manually examining the system/ model output
- Using online crowdsourced evaluations (e.g., Mechanical Turk)
- Metrics based on how the system or model is being used in actual deployed settings (by users who use the system for their own reasons, self-incentivized, and are not are paid as test subjects/crowd workers)
[SQ21] For the system or task, you described earlier, other than those listed above, are there other evaluation methods you believe are (or would be) reliable ways to assess the performance of your system or on your task? R: Write-in
[SQ22] In general, please answer if you agree or disagree with the following statement about evaluating NLG systems or tasks: Strongly agree, agree, neither agree or disagree, disagree, strongly disagree
- Metrics developed for one NLG task or system can be reliably used or adapted to evaluate other NLG tasks or systems.
- It is a good practice to evaluate NLG systems or models on as many automatic metrics as possible.
- The quality of NLG systems or models' output is primarily impacted by the quality of the training data.
- Online crowdsourced evaluations are a good proxy for evaluations with users.
[SQ23] In general, please answer how often the following statements are true: Never, Rarely, Sometimes, Often, Always, (N/A)
- Automatic metrics should be judged by the degree to which they correlate with human evaluations (e.g., online crowdsourced, expert evaluation, team evaluation).
- Judges or annotators with specific qualifications or expertise can contribute to the evaluation in ways that standard crowdsourcing cannot.
- The quality of training datasets needs to be formally evaluated.
- The quality of evaluation datasets needs to be formally evaluated.
B.6 Constraints and other considerations
[SQ24] Rank which of the following resources currently limit the evaluation for your particular system or task the most. Top choice being the most limiting resource.
Insufficient access to relevant datasets for evaluation
Insufficient access to relevant metrics for evaluation
Insufficient access to online paid crowdsourcing (due to cost, availability, time, or any other reason)
- Insufficient access to being able to deploy systems or models to real world settings
- Insufficient access to users who use the system for their own reasons (self-incentivized, rather than because they are paid as test subjects/crowd workers)
Insufficient access to judges or annotators with certain qualifications or expertise
Insufficient access to compute resources
- Insufficient access to engineering resources (e.g., lack of engineers, lack of time)
- Other
[SQ25] If there are other limiting resources for evaluation, please elaborate. R: Write-in
[SQ26] Rank which of the following considerations currently guide your evaluation practices the most. Top choice being the most influential consideration.
- Potential concerns raised by research paper reviewers
- Community guidance, standards, or practices
- The need to convince yourself or your team
- The need to convince other stakeholders
Availability of relevant datasets and metrics
Financial considerations
- The number of anticipated users that the system is expected to reach
Public relations/public image
- The need for the system to perform well in real deployment settings (i.e., ecological validity)
- Other
[SQ27] If there are other considerations for evaluation, please elaborate. R: Write-in
B.7 Adverse impacts and ethical considerations
[SQ28] Are there any ethical issues or adverse impacts that you're concerned about in the context of your project, and if so, what are they? R: Write-in
[SQ29] Are there any of the ethical issues or adverse impacts you mentioned above that are particularly hard to evaluate or that you are unable to evaluate? If so, what are they? R: Write-in
B.8 Adverse impacts and ethical considerations (continued)
[SQ30] Do you intentionally try to block certain type of content from your generation? (e.g., by using blocklists or classifiers, cleaning the training data, etc.)
- Never, Rarely, Sometimes, Often, Always, (N/A)
[SQ31] For the following questions, for your system, how do you measure the following? Fairness and Inclusion states that NLG systems should treat all people equally, empowering and engaging everyone by providing equal benefit and access to e.g., opportunities and resources. Systematically, Informally (eyeballing), Indirectly (through another metric), No evaluation/not sure
- Which topics your system or model tends to generate and what it tends to avoid generating
- The linguistic style or tone of the content generated by your system or model
- Toxic, hateful, or offensive language produced by your system or model
- Other fairness and inclusion issues aside from toxic, hateful, and offensive language
[SQ32] If you assess other fairness and inclusion issues, please briefly explain what do you measure and how. R: Write-in