SlowGuess/NAACL_2021_OCR huggingface dataset:
Boaz Shmueli $^{1,2,3,*}$ , Jan Fell $^{2}$ , Soumya Ray $^{2}$ , and Lun-Wei Ku $^{3}$ $^{1}$ Social Networks and Human-Centered Computing, TIGP, Academia Sinica  
 $^{2}$ Institute of Service Science, National Tsing Hua University  
 $^{3}$ Institute of Information Science, Academia Sinica
The use of crowdworkers in NLP research is growing rapidly, in tandem with the exponential increase in research production in machine learning and AI. Ethical discussion regarding the use of crowdworkers within the NLP research community is typically confined in scope to issues related to labor conditions such as fair pay. We draw attention to the lack of ethical considerations related to the various tasks performed by workers, including labeling, evaluation, and production. We find that the Final Rule, the common ethical framework used by researchers, did not anticipate the use of online crowdsourcing platforms for data collection, resulting in gaps between the spirit and practice of human-subjects ethics in NLP research. We enumerate common scenarios where crowdworkers performing NLP tasks are at risk of harm. We thus recommend that researchers evaluate these risks by considering the three ethical principles set up by the Belmont Report. We also clarify some common misconceptions regarding the Institutional Review Board (IRB) application. We hope this paper will serve to reopen the discussion within our community regarding the ethical use of crowdworkers.
1 Revisiting the Ethics of Crowdsourcing
The information age brought with it the internet, big data, smartphones, AI, and along with these, a plethora of complex ethical challenges. As a result, there is growing concern and discussion on ethics within the research community at large, including the NLP community. This is manifested in new ethics-focused workshops, ethics conference panels and relevant updates to peer review forms.
While ethics in NLP has multiple aspects, most recent attention focuses on pressing issues related to the societal impact of NLP. These include discrimination, exclusion, over-generalization, bias,
and fairness (Hovy and Spruit, 2016; Leidner and Plachouras, 2017). Other works are concerned with the ethical implications of NLP shared tasks (Parra Escartín et al., 2017), and introducing ethics into the NLP curriculum (Bender et al., 2020).
A substantial amount of NLP research now takes advantage of crowdworkers — workers on crowdsourcing platforms such as Amazon Mechanical Turk (known also as AMT or MTurk), Figure Eight<sup>1</sup>, Appen, Upwork, Prolific, Hybrid, Tencent Questionnaire, and Baidu Zhongbao, as well as internal crowdsourcing platforms in companies such as Microsoft and Apple. Workers are recruited to label, evaluate, and produce data. In the pre-internet era, such tasks (e.g. part-of-speech (POS) tagging) were done by hiring expert annotators or linguistics students. However, these are now mostly replaced by crowdworkers due to lower costs, convenience, speed, and scalability.
Overall, the general consensus in the literature is that as long as the pay to the crowdworkers is "fair" (minimum hourly wage or above), there are no further ethical concerns, and there is no need for approval by an Institutional Review Board $^2$  (with some exceptions). For example, Hovy and Spruit (2016) mention that "[w]ork on existing corpora is unlikely to raise any flags that would require an IRB approval", with a footnote that there are "a few exceptions". Fort et al. (2011) mention that only "[a] small number of universities have insisted on institutional review board approval for MTurk experiments". As another example, NLP students are being taught that "paid labeling does not require IRB approval" since "[i]t's not an experiment
${}^{1}$  Previously CrowdFlower; acquired by Appen in 2019.
$^{2}$ Institutional Review Boards (IRBs) are university-level, multi-stakeholder committees that review the methods proposed for research involving human subjects to ensure that they conform to ethical principles. IRBs are also known by various other names, such as Research Ethics Boards (REBs) and Research Ethics Committees (RECs). Non-academic organizations may employ similar committees.
*Corresponding author: shmueli@iis.sinica.edu.tw
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3758-3769 June 6-11, 2021. ©2021 Association for Computational Linguistics
Year	Accepted Papers	Papers Using Crowdsourcing	Payment Mentioned	IRB Mentioned
ACL	EMNLP	NAACL	All
2015	318	312	186	816	59 (7%)	4 (7%)	0
2016	328	264	182	774	82 (11%)	15 (18%)	0
2017	302	323	—	625	57 (9%)	12 (21%)	3
2018	381	549	332	1262	136 (11%)	17 (13%)	1
2019	660	683	423	1766	189 (11%)	32 (17%)	5
2020	779	754	—	1533	180 (12%)	42 (23%)	5
Total	2768	2885	1123	6776	703 (10%)	122 (17%)	14
Table 1: Papers using crowdsourced tasks in top NLP conferences, 2015-2020. The columns show, from left to right: conference year; number of accepted papers at ACL, EMNLP, and NAACL; total number of accepted papers; number of accepted papers using crowdsourced tasks (percentage of papers using crowdsourced tasks); number of papers using crowdsourced tasks that mention payment (percentage of papers using crowdsourced tasks that mention payment); number of papers using crowdsourced tasks that mention IRB review or exemption.
with human subjects” (Carnegie-Mellon University, Language Technologies Institute, 2020). Indeed, NLP papers that involve crowdsourced work rarely mention a review by an ethics board.
In this work, we wish to revisit the ethical issues of crowdsourcing in the NLP context, highlight several issues of concern, and suggest ways forward. From our survey of top NLP conferences, we find that crowdsourcing tasks are growing rapidly within the community. We therefore establish a common understanding of research ethics and how it relates to crowdsourcing. We demonstrate that the existing ethical framework is often inadequate and does not seek to protect crowdsourced workers. We then dispel common misunderstandings regarding the IRB process that NLP researchers might harbor. And finally, we outline how to apply ethical values as guidelines to minimize potential harms, and conclude with recommendations.
2 The Rise and Rise of NLP Crowdsourcing
To get a sense of the extent and growth of crowdsourced tasks within the NLP research community, we analyzed the proceedings of three top NLP conferences: ACL, EMNLP, and NAACL (also known as NAACL-HLT). We scanned the annual proceedings of these conferences in the six years from 2015 to 2020, looking for papers that mention direct employment of crowdsourced workers. All together,
6776 papers were accepted for publication in these 16 conferences<sup>4</sup>. In total, we identified 703 papers that use crowdworkers as part of their research.<sup>5</sup> The results are summarized in Table 1.
Renumeration For each paper that uses crowdsourced labor, we checked whether the authors discuss payment or labor-related issues. Out of the 703 papers, 122 (17%) discuss payment, either by detailing the amount paid per task, the worker's hourly wages, or declaring that wages were paid ethically. While in some cases researchers emphasize fair payment (e.g., Nangia et al. (2020) ensured "a pay rate of at least $15/hour"), many other papers are more concerned about the cost of dataset acquisition, and thus mention the cost per task or the total dataset cost, but not the hourly compensation.
IRB Review Finally, we also checked whether authors mention a review by an IRB (or equivalent body) for their research. We found very few papers that mention an IRB approval or exemption — a total of 14 papers — which make up only  $2\%$  of the works that use crowdsourcing.
Growth We see that research papers using crowdsourced tasks have made up a relatively constant  $11 - 12\%$  of all research papers in the last three years. As research production grows exponentially, we expect a corresponding increase in the number of crowdsourced tasks.[6]
${}^{3}$  ACL is the Annual Meeting of the Association for Computational Linguistics,EMNLP is the Conference on Empirical Methods in Natural Language Processing,and NAACL is the Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
<sup>4</sup>NAACL was not held in 2017 and 2020; it is skipped every three years.
${}^{5}$  MTurk is used in 80% of tasks.
$^{6}$ ACL, EMNLP, and NAACL are only three of many other publication venues in the NLP community.
2.1 Categorizing Crowdsourced Tasks
To understand the many nuanced ways in which researchers presently use crowdworkers in NLP tasks, we examined the tasks performed in each of the papers that use crowdsourcing. We found that NLP crowdsourced tasks generally fall into one of three categories, which we designate as labeling, evaluation, and production. We found that the categories account for  $34\%$ ,  $43\%$ , and  $23\%$  of crowdsourcing tasks respectively. The results are summarized in Table 2, which also lists common action verbs that researchers use to describe the work performed by crowdworkers.
Task Category	Pct.	Action Verbs
Labeling	34%	annotate, label, identify, indicate, check
Evaluation	43%	rate, decide, score, judge, rank, choose
Production	23%	write, modify, produce, explain, paraphrase
Table 2: Tasks in the papers that use crowdsourcing, broken down by category. We also list typical action verbs used by the authors to describe the task.
Labeling entails the processing of existing data by the crowdworker and then the selection or composition of a label or labels for that data<sup>7</sup>. Labeling tasks augment the data with human-supplied labels. The augmented data are often used for training machine learning models. We further divide labeling tasks into two: objective and subjective labeling. In objective labeling, the desired label is factual, and does not depend on the worker. Classical examples are the tagging of sentences with named entities and part-of-speech (POS) labeling, and text transcription. In contrast, subjective labeling comprises of tasks where labels may depend on the worker's personality, cultural background, opinion, or affective state. Examples include emotion labeling, detecting sarcasm in a tweet, or deciding whether a text constitutes hate speech or not.
In evaluation tasks, the worker is presented with data, for example a sentence, tweet, paragraph, or dialogue, and then requested to evaluate and score the data — mostly text — according to predefined criteria, such as fluency, coherence, originality, or structure. These tasks are often used by researchers to evaluate natural language generation (NLG) models. Similar to subjective labeling,
scores given by the workers may reflect the interpretation, values, or beliefs of the worker.
Finally, in production tasks, workers are asked to produce their own data, rather than label or evaluate existing data. In the NLP context, this often amounts to text elicitation or text generation. Examples include captioning a photo or video clip, writing a story given a sequence of images, or composing questions and answers. In this category we also include text translation. The produced data is often used for model training or evaluation.
While the majority of the studies use only one type of task—labeling, evaluation, or production—we found that in  $10\%$  of the papers that use crowdsourcing, researchers used two or more types of tasks in the same study. The combination of production and evaluation is particularly common; researchers often ask workers to generate data, which in turn is used to train a model; they then use workers to evaluate the model's performance.
2.2 Surveys and Gamification
Although not common, some papers also collect personal information from workers. For example, Yang et al. (2015) and Ding and Pan (2016) conduct personality surveys among its crowdworkers. Pérez-Rosas and Mihalcea (2015) collect demographic data from the workers which included "their gender, age, country of origin, and education level". Finally, we also found a few papers that add elements of gaming to their crowdsourced tasks, e.g. Niculae and Danescu-Niculescu-Mizil (2016) and Urbanek et al. (2019).
3 The Rules and Institutions of Research Ethics
Given the increasing use of crowdsourced NLP tasks, how can researchers ensure ethical concerns are reasonably addressed? Should a researcher make a judgement call and decide which tasks pose a risk of harm to the worker, and which are benign? To answer such questions, we will first explore the existing ethical framework used by researchers in the biomedical, social, and behavioral sciences.
3.1 The Genesis of Modern Research Ethics
The roots of contemporary research ethics originate in the 19th century, when researchers made unparalleled discoveries, but also engaged in hazardous, and frequently deadly, experimentation without great concern for the human subjects involved as
<sup>7</sup>We refrain from using the terms "annotation" and "annotators" as these terms are overloaded and often used for non-annotation work.
long as these trials advanced the boundaries of scientific knowledge (Ivy, 1948). The dominant ethics paradigm at the time was largely devoid of now-common principles surrounding therapeutic benefits, scientific validity, full knowledge, or subject consent (Lederer, 1995). Examples include researchers infecting intellectually disabled orphans with gonorrhea, or puncturing a healthy and unaware woman with the nodules of a leper patient to observe the clinical course of these diseases (Shamoo and Resnik, 2009).
Such incidents were common before the 1940s, and academia and public discourse were generally ignorant of them and of research ethics in general (Rothman, 1987). The revelation of the Nazi concentration camp experiments at the end of World War II was a watershed moment (Gillespie, 1989) and led to an early precursor of contemporary research ethics, namely the Nuremberg Code of 1947 (Jonsen, 1998). Not long after, the fallout of the Tuskegee Syphilis Study in the US prompted the formalization of research ethics at American universities (Caplan, 1992). In the study, which took place between 1932 and 1972, a total of 399 socio-economically disadvantaged African-American males with latent syphilis infections were recruited through the false promise of "free healthcare". Yet, these subjects were actually left without therapy even as effective treatment became available, with the objective to observe the clinical course of the disease (Brandt, 1978).
3.2 The Belmont Principles and IRBs
From thereon, beginning with biomedical research, the notion of research ethics has been institutionalized in the US at the university level through institutional review boards (IRBs), as well as national legislation (Israel, 2015). Gradually, research ethics have become a concern also in the social and behavioral sciences, as demonstrated by the 1978 Belmont Report created by the National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research (Jonsen, 2005). This report became the basis of the 1991 Federal Policy for the Protection of Human Subjects in the United States, more commonly known as the Common Rule (Owen, 2006) and superseded by the Final Rule (2018). Chiefly, the Final Rule aims to ensure that the following three basic principles listed in the Belmont Report (National Commission for the Protection of Human Subjects of Biomedical and
Behavioral Research, 1978) are met:
1. Respect for persons, which includes "the requirement to acknowledge autonomy and the requirement to protect those with diminished autonomy";
2. Beneficence, which mandates considering whether the benefits resulting from the research can outweigh the risks; and
3. Justice, requiring that the burden — and benefits — of the research are equally shared among potential subjects.
The Final Rule is codified in Title 45, Code of Federal Regulations, Part 46 and applies to all government-funded research in the United States (Israel, 2015). Virtually all universities in the United States apply this regulation to human subjects research projects irrespective of funding source (Klitzman, 2015). Specifically, the Final Rule requires that most research involving human subjects receives approval from an IRB. The IRB is a special university-level committee that reviews research proposals to verify that they comply with ethical standards. While it is difficult to assess the effectiveness of IRBs, and the process of ethical review is sometimes criticized as overly bureaucratic which may hamper low-risk social science (Resnik, 2018; Schrag, 2010), glaring ethical lapses have been rare after 1974 (Klitzman, 2015).
3.3 Are IRBs Universal?
The three ethical principles outlined by the Belmont Report — Respect for persons, Beneficence, and Justice — are also the stated principles guiding the actions of more recently formed ethics boards around the world, as well as the underlying principles of relevant policies of intergovernmental organizations, including the Universal Declaration on Bioethics and Human Rights (UNESCO, 2006) and the International Ethical Guidelines for Biomedical Research Involving Human Subjects (Council for International Organizations of Medical Sciences, 2002). Consequently, many countries worldwide have modeled national regulations after the Final Rule or its predecessor, the Common Rule (Capron, 2008). Both regulations have also influenced editorial policies of academic journals (e.g., the Committee on Publication Ethics, 2020). The Final Rule can thus be considered a cross-disciplinary defacto standard for research ethics in Western/US-
influenced academic settings (Gontcharov, 2018), including India, Japan, Korea, and Taiwan.
Though we find that a global agreement on human-subjects ethics is emerging, countries still vary in the extent to which relevant policies are accepted, framed, implemented, or enforced.
4 Do NLP Tasks Constitute Research Involving Human Subjects?
These formal rules and institutions have been established to protect the rights and interests of humans subjects involved as participants in scientific research. Thus, we must determine whether the rules and institutions of research ethics are even applicable to crowdsourced studies. At the core of this determination are two fundamental questions:
1. Are crowdsourcing tasks research?
2. Are crowdworkers human subjects?
In the following, we address these two questions.
4.1 Are Crowdsourcing Tasks Research?
The Final Rule defines research as follows:
Research means a systematic investigation, including research development, testing, and evaluation, designed to develop or contribute to generalizable knowledge. Activities that meet this definition constitute research for purposes of this policy, whether or not they are conducted or supported under a program that is considered research for other purposes. (45 CFR 46.102(1), Final Rule, 2018)
From this definition it is evident that rather than a concrete research behavior on part of the NLP researcher, it is the purpose of the research behavior that classifies said behavior as research under the Final Rule. In other words, all categories of crowdsourced tasks summarized in Section 2, i.e., labeling, evaluation, and production, may be considered part of research so long as the intended outcome is to create generalizable knowledge. Typically, this encompasses academic settings where research behavior takes place (course assignments by students being a prominent exception), but does not include research conducted in industry settings (Meyer, 2020; Jackman and Kanerva, 2015).
4.2 Are Crowdworkers Human Subjects?
The Final Rule defines human subjects as follows:
Human subject means a living individual about whom an investigator (whether professional or student) conducting research:
(i) Obtains information or biospecimens through intervention or interaction with the individual, and, uses, studies, or analyzes the information or biospecimens; or
(ii) Obtains, uses, studies, analyzes, or generates identifiable private information or identifiable biospecimens.
(45CFR 46.102(e)(1),Final Rule,2018)
Clearly, if the researcher obtains identifiable private information (IPI) as part of the crowdsourced task — e.g. name, date of birth, email address, national identity number, or any other information that identifies the worker — then (ii) holds and the worker is considered a human subject.
Even if the researcher does not obtain any IPI, the worker may still be considered a human subject under (i) in certain cases. This is because NLP researchers interact with crowdworkers through MTurk or analogous platforms when they publish the task, and obtain information through this interaction. It is also evident that academic NLP researchers make use of this information as they conduct a given study, and hence it is "used, studied, or analyzed". If the information is about the crowdworker then they are considered human subjects and (i) is met. However, Final Rule does not expand on what constitutes information about the individual. According to University of Washington, Office of Research (2020), for example, about whom means that the "data or information relates to the person. Asking what [crowdworkers] think about something, how they do something, or similar questions usually pertain to the individuals. This is in contrast to questions about factual information not related to the person."
Whether the information obtained in an NLP task is about the worker can initially seem like an easy-to-answer question. For example, Benton et al. (2017) write:
[R]earch that requires the annotation of corpora for training models involves hu
man annotators. But since the research does not study the actions of those annotators, the research does not involve human subjects. By contrast, if the goal of the research was to study how humans annotate data, such as to learn about how humans interpret language, then the research may constitute human subjects research.
However, we believe that this is not so clear-cut. First, one might argue that although in a POS labeling task we do not obtain information about the worker, other labeling tasks might be harder to classify. For example, when researchers ask a worker to compose a story given a sequence of photos, do they obtain information about the worker? And if so, what kind of information? Similar questions can be asked about tasks related to emotion classification (which might reveal a worker's personality or mood), composing questions and answers (which point to areas of interest and cultural background), or identifying hate speech (which can indicate political orientation).
Second, platforms might automatically provide information that can be considered by some to be about the individual. Even in the most benign and "objective" tasks such as POS tagging, MTurk supplies researchers with information on the amount of time taken to complete each task. This information is sometimes collected and used by NLP researchers (e.g., Sen et al., 2020).
In summary, we have shown that in an academic context, NLP crowdsourcing tasks are research, but that the categorization of crowdworkers as human subjects can, in some cases, be a gray area that is open to interpretation. The Final Rule was designed to address ethical issues in medical research, and later in behavioral sciences; lawmakers and experts involved did not anticipate its use in new domains such as crowdsourcing. Therefore, its application to online data collection, and crowdsourcing in particular, can be ambiguous and unsatisfactory. Thus, while in some cases the protections and procedures mandated under the Final Rule apply, in others they might not. As a consequence, some NLP crowdsourcing tasks may not require an IRB application, and this may happen even if crowdworkers are at risk.
5 Dispelling IRB Misconceptions
As we now see, if workers constitute human subjects, an IRB application is required. To clarify any other misconceptions that might be present within the community, we list key points related to the IRB process and dispel misconceptions around them. While not exhaustive, the list can serve both researchers and reviewers.