Incorporating Worker Perspectives into MTurk Annotation Practices for NLP
Authors: Olivia Huang
Eve Fleisig
Dan Klein
Venue: EMNLP 2023 Main (2023)
Keywords: data collection
annotators
mechanical turk
AI ethics
data quality
TL;DR: We survey MTurk workers regarding issues in current MTurk data collection practices, examining how we can improve data quality and protect worker rights when collecting NLP data.

Abstract: Current practices regarding data collection for natural language processing on Amazon Mechanical Turk (MTurk) often rely on a combination of studies on data quality and heuristics shared among NLP researchers. However, without considering the perspectives of MTurk workers, these approaches are susceptible to issues regarding workers' rights and poor response quality. We conducted a critical literature review and a survey of MTurk workers aimed at addressing open questions regarding best practices for fair payment, worker privacy, data quality, and considering worker incentives. We found that worker preferences are often at odds with received wisdom among NLP researchers. Surveyed workers preferred reliable, reasonable payments over uncertain, very high payments; reported frequently lying on demographic questions; and expressed frustration at having work rejected with no explanation. We also found that workers view some quality control methods, such as requiring minimum response times or Master's qualifications, as biased and largely ineffective. Based on the survey results, we provide recommendations on how future NLP studies may better account for MTurk workers' experiences in order to respect workers' rights and improve data quality.

Incorporating Worker Perspectives into MTurk Annotation Practices for NLP
Olivia Huang Eve Fleisig
Dan Klein UC Berkeley
UC Berkeley
UC Berkeley
(efleisig, oliviahuang, klein)@berkeley.c edu edu
Abstract
opinions on public forums such as Reddit, then de- veloped survey that asked MTurk workers about open questions in the following areas:
Current practices regarding data collection for natural language processing on Amazon Me- chanical Turk (MTurk) often rely on com- bination of studies data quality and heuris- tics shared among NLP researchers. However, without considering the perspectives of MTurk workers. these approaches are susceptible to issues regarding workers' rights and poor re- sponse quality. We conducted critical liter- ature review and survey of MTurk workers aimed at addressing open questions regarding best practices for fair payment, worker privacy, data quality, and considering worker incentives. We found that worker preferences are often at odds with received wisdom among NLPP re- searchers. Surveyed workers preferred reliable, reasonable payments payments; reported frequently lying demo- graphic questions; and expressed frustration having work rejected with no explanation. We also found that workers view some quality control methods, such as requiring minimum response times or Master's qualifications, as biased and largely ineffective. Based on the survey results, we provide recommendations on how future NLP studies may better account for MTurk workers' experiences order re- spect workers' rights and improve data quality.
Task clarity. How can requesters ensure that workers fully understand task and complete it accurately? Surveyed workers expressed desire for more examples and more detailed instructions, and nearly half desired more information about the downstream context in which an annotation will be used. In addition, they indicated that task clarity is a major factor in their judgment of how responsible and reliable requester is (Section 3).
Payment. What is an appropriate level of pay on MTurk? Are there tradeoffs to different amounts or methods of pay? Respondents indicated that there often a threshold pay below which they will refuse to complete task, suggesting that pay rates below minimum wage are not only unethi- cal, but also counterproductive to worker recruit- ment. However, very high pay attracts spammers, which moreover forces attentive workers com- plete tasks more quickly before spammers take them all, meaning that response quality likely de- creases across all workers. Workers expressed mixed opinions over bonus payments, suggesting that they are most effective if they are added to a reasonable base pay and workers trust the requester to deliver the bonus. (Section 4).
uncertain, very high
1 Introduction
Amazon Mechanical Turk (MTurk) is an online survey platform that has become increasingly pop- ular for NLP annotation tasks (Callison-Buur and Dredze, 2010). However, data collection on MTurk also runs the risk of gathering noisy, low-quality responses (Snow et al., 2008) or violating workers' rights to adequate pay, privacy, and overall treat- ment (Xia et al., 2017; Hara al., 2018: Kim et al.. 2021; Lease et al., 2013; Shmueli et al., 2021). We conducted a critical literature review that identi- fies key concerns in five areas related to NLP data collection on MTurk and observed MTurk worker
What methods best ensure worker pri- vacy on MTurk? How do workers react to per- ceived privacy violations? Workers report that they often respond to questions for personal informa- tion untruthfully if they are concerned about pri- vacy violations. Nearly half of workers reported lying on questions about demographic information. which means that tasks relying on demographic data collection should be very careful about collect- ing such information on MTurk (if at all). Some of these issues can be partially mitigated if requesters
Privacy.

---

have good reputations among workers and clear about the purpose of personal data collection, since spam requesters are common on MTurk. (Section 4).
2 Survey Design
We conducted a literature review and examined posts on r/mturk, a Reddit forum that MTurk work- ers use to discuss the platform, to identify the areas of uncertainty discussed in sections 3-7. Then, to collect data on MTTurk workers' experiences, we conducted Qualtrics survey posted task on MTurk. Our survey was broadly subdivided into sections on payment, sensitive questions, response quality, and miscellaneous issues (mainly task clar- ity and context). As a preliminary filter for re- sponse quality, we required that all respondents have 97% HIT approval rate, at least 100 HITs completed, minimum of 18 years of age, and En- glish fluency. This is a lower HIT approval rate than is typically used, which can increase the rate of spam; however, aimed collect opinions of workers who might be excluded by high approval rate filters, and therefore manually reviewed the data afterwards to remove spam responses by ex- amining responses to the required free text fields. We collected 207 responses from our survey over one week, of which 59 responses were dropped for spam incompleteness (we discuss further im- plications of high spam rates in Section 8). Each annotator was paid $2.50 to complete the survey, based on the estimated completion time and a min- imum wage of $15/hour. Appendices A and contain details on informed consent, the full text of the survey, and data cleaning details.'
Response quality. What methods best ensure high-quality responses and minimize spam? Re- questers must consider how traditional response filters have the side effect of influencing workers' perception of the task, and in turn, their response quality. The rising importance of incorporating worker perspectives underscored by the establish- ment of organizations such as Turkopticon, which consolidates workers' reviews of requesters and advocates for MTurk workers' rights, including petitioning to "end the harm that mass rejections cause" (Turkopticon, 2022). We find that workers consider some common quality control methods in- effective and unfair. For example, workers reported that minimum response times often make them simply wait and do other things until they hit the minimum time, while allowing only workers with Masters qualifications to complete a task excludes all workers who joined after 2019, since the qualifi- cation is longer offered. In addition, workers very concerned about their approval rates because requesters often filter for workers with very high approval rates. Respondents expressed frustration over having work rejected, especially automatically or without justification, and receiving no pay for their labor. Workers thus avoid requesters with low approval rates or poor communication (Section 6).
The following sections address each area of con- cern raised in Section 1, discussing previous re- search on MTurk annotation practices, the open questions raised by the literature, the survey ques- tions we included on those topics, and the results and implications of the survey responses.
Sensitive content. Tasks in which workers anno- tate sensitive content can pose a psychological risk, particularly since MTurk workers often face mental health issues (Arditte et al., 2016). Though it is common practice to put general content warnings before such tasks (e.g. "offensive content"), nearly third of surveyed workers expressed desire for specific content warnings (e.g., "homophobia") de- spite this being uncommon in current research prac- tices (Section 7).
Task Clarity
On MTurk, requesters publish "human intelligence tasks" (or HITs) for MTurk workers to complete in exchange for monetary incentive. Best practices for question phrasing and ordering consistent with overall survey design guidelines, which en- courage including clear instructions for how use the platform, clearly outlining requirements for ac- ceptance, including label definitions and examples, and avoiding ambiguous language (Gideon, 2012). To look for strategies for minimizing confusion in survey questions, our survey asked what additional
Common themes emerge across these topics re- garding the importance of maintaining good rep- utation requester, understanding worker incen- tives, and communicating clearly with workers. In Section 8, we discuss the implications of worker preferences for survey design and provide recom- mendations for future data collection on MTurk.
'This study underwent IRB review and annotators pro- vided informed consent prior to participation.

---

provided information, such as context and purpose of the task, would be most helpful for reducing ambiguity or confusion.
Information Responders Desired When Confused
Purpose of question
Information Type
To ensure that results are accurate and useful, it is important to carefully conceptualize the task (de- fine what quality is being measured by the study) and operationalize (decide how the study will measure the construct defined by the conceptu- alization) along with clear compensation details (discussed further in Section and Jacobs and Wallach, 2021). In a Reddit thread on r/mturk, worker u/dgrochester55 (2022) stated that "we are not likely to take your survey seriously if you requester aren't 't taking your survey seriously" due to issues such as "poor pay, irrelevant subject" or being "badly put together." Thus, task clarity not only ensures that the worker can provide accu- rate answers, but also serves to increase their effort levels and willingness to complete the task itself.
Context
Platform where labels will be used
Other
33
70
Responses
Figure 1: Types of information that respondents felt would have helped them answer questions they found confusing.
for such qualities in annotation tasks. To better understand how context can influence annotation tasks, asked MTurk workers what additional information most useful to them when given a question with an unclear answer.
Our survey sought further clarify the impor- tance of task clarity by asking whether influences effort level and willingness to start a task. In ad- dition, we asked a free response question about common areas of confusion to observe common pitfalls in designing MTurk tasks.
3.2 Survey Results on Task Clarity
We found that task clarity and fair payment are key factors that determine both MTurk workers' willingness to start a task and the amount of effort they put into it. In our survey, 26.3% of respon- dents stated that difficulty of understanding the task influences whether they start the task, and 31.6% indicated that it influences the amount of effort they put into the task. One respondent suggested providing additional, more comprehensive exam- ples, since requesters "often skimp on these or only provide super basic, obvious examples." When re- spondents were asked to rank the importance of different qualities in determining how "reasonable" and "reliable" the requester is (from one to six), clarity of instructions with an average rank of 2.71 (payment was ranked highest; see Section 4). These results reinforce the importance of task clarity in improving workers' perceptions of the task and its requester.
3.1 Context
One way to improve task clarity is to provide addi- tional context for the task. Context is commonly provided in two ways. The first way to provide additional words, sentences, or paragraphs in the annotated text itself. Providing context can increase disagreement among responses, and can be useful if the goal is to mimic the distribution of differ- ing human judgments (Pavlick and Kwiatkowski, 2019). The second way is to introduce background information about the speaker, such as age, gender, race, socioeconomic status, etc. In tasks such as an- notating for the presence of harmful language, such information can affect an annotator's perception of text as influenced by individual biases.
ranked second highest
Sometimes, context is inherent in the language itself, instead of explicitly provided. For example, some annotators may incorrectly interpret African American English (AAE) as offensive (Sap et al., 2019). Other forms context include grammat- icality, location, and relative positions of power between the speaker and the audience.
Workers often reported being confused by an- notation tasks that involve sentiment analysis and subjective ratings. For such questions, the most commonly preferred additional information was the purpose of the question (69.1%), context (50.0%), and the platform the annotations would be used for (34.9%). This suggests that providing an ex- planation for the downstream use case of the task significantly aids workers in understanding tasks and providing quality annotations.
Thus, it is important to note that even without the presence of additional context, text inherently contextual. It is important to identify and control

---

4 Payment
payments affect effort, how payment influences likelihood of starting or completing the task, and how payment affects the degree to which requesters are perceived as "reasonable" and "reliable." We also asked MTurk workers about their MTurk in- come relative to the opportunity cost of their work, as well as their own thresholds for the minimum pay rate at which they would do task.
MTurk's platform, which enables researchers to collect large amount of data at low cost, does not currently regulate compensation beyond a S0.01 minimum per task that disregards task completion time. The mean and median hourly wages for MTurk workers are $3.13 and 31.77 per hour, re- spectively. 95% of MTurk workers earn below min- imum wage requirements in their geographic loca- tion (Hara et al.,, 2018), and improvements in house- hold income for U.S. MTTurk workers lag behind the general U.S. population (Jacques and Kristensson, 2019). Low payment is not only unfair to workers, but can also serve as an additional sign to work- ers of an unreliable requester to avoid due to high potential of having HIT rejected. w/Bermin299 (2022) claims that "Requestors that pays cheaply are some of the most trigger happy people when it comes to handing
4.1 Survey Results on Payment
Survey results indicate reasonable payment be around minimum wage; significantly higher or lower payment appears to be detrimental to the quality of responses. Respondents' minimum pay rate at which they would be willing to complete task was $13.72 per hour on average (median of $12 per hour). 61.5% of respondents stated that the rationale behind their threshold is that they want to earn a wage on MTurk comparable what they would make elsewhere. These numbers are significantly above the current mean and median, though still below minimum wage. Despite this, the survey surprisingly indicates that 70.4% of re- spondents make more on MTurk than they would make elsewhere, and 22.4% make the same amount. However, payment should not be set too high, one worker stated that "[e "le]xtremely high pay rates cause rushing in order to complete as many pos- sible prior to the batch getting wiped out by other workers."
rejections."
Most researchers agrec on using minimum wage as baseline hourly rate for MTurrk work, though exact recommendations vary due to differences in the minimum wage worldwide (Hara et al., 2018).
A common sentiment among MTurk workers is to pay at rate of $15 per hour (u/Sharpsillerzz, 2022).
Whiting et al. (2019) found that workers are likely to overestimate their work time by 38% of the ob- served task time. Thus, the amount of time used to determine minimum wage can roughly account for this overestimation if using workers' self-reported work times as a metric. Previous research is di- vided on whether extra pay beyond minimum wage improves response quality. Callison-Buu Burch and Dredze (2010) gave anecdotal evidence that unusu- ally high payments, such as $1 for very short task, may encourage cheating.
Workers responded that both higher normal pay- ments and bonus payments increase the amount of effort they put into a task. 73.0% of respon- dents indicated that bonus payments will increase their effort levels while 49.3% selected high(er) payments. However, some respondents noted that bonus payments are only a significant incentive if they are from a reputable requester. Uncer- tainty about whether the bonus will be paid is the main concern, with one worker noting that "regular payments are guaranteed within 30 days, whereas bonuses take however long the req decides take pay you (assuming they even do)." Thus, the effectiveness of bonus payments varies with workers' perception of the requester's reliability and reputation.
However, Snow et al. (2008) found that non- guaranteed payments, which are paid only after submitted work is approved for work quality, are sometimes effective at improving response qual- ity. MTurk's bonus payments can also be non- guaranteed payments, and they are given to work- ers after they have completed the task in addition to the advertised payment rate.
Given the wide range of payment options and formats, along with varying guidelines the ef- fectiveness of different payment options, we in- clude multiple questions in our survey regarding MTurk workers' perception and response to pay- ments. These included workers' perception of nor- mal and bonus payments, how bonus versus normal
Privacy
Demographic questions are commonly used in sur- veys to collect respondents" information. However, some MTurk workers may view these questions as

---

threats to their privacy and decide not to complete the survey. To clarify which questions are more commonly viewed as "sensitive," our survey asked MTurk workers to indicate types of questions, in- cluding demographic questions and questions about offensive topics or mental health, that cause them to not complete task or answer untruthfully. We also asked for reasons would cause them to answer question untruthfully, including whether they be- lieve that the question is privacy violation or that it will be used against them.
Questions that Responders Lie On
Age
Gender
Political Stance
Education
Question
Employment
Sexual Orientation
Race
Offensive Content
Disability
Mental Health
Other
Responses (%)
Figure Types of personal questions that workers re- port will cause them to answer untruthfully. Over half of annotators report that questions about age will make them answer untruthfully, and over 40% report the same about gender.
It is especially important to build trust and the perception of fairness between the requester and the annotator for surveys that ask more personal questions. Xia et al. (2017) found that crowdwork- ers' major areas of concern regarding data privacy involved collection of sensitive data, information processing, information dissemination, invasion into private lives, and deceptive practices. To mini- mize these concerns, they proposed providing in- formation about the requester. being specific about why any private information is collected and how it will be used, not sharing any personal informa- tion with third parties, and avoiding using provided emails for spam content outside the context of the survey. Turkopticon's Guidelines for Academic Requesters recommend that requesters provide in- formation that clearly identifies who the requester are and how to communicate with them, which indi- cates that the requester is "accountable and respon- sible", as well as avoiding collection of workers' personally identifying information (Turkopticon, 2014). Building this, our survey asked what additional information, such as content warnings and mental health resources, would be most helpful in surveys with sensitive questions.
with all of their activities on Amazon's 's platform, including responses to MTurk tasks and activity on Amazon.com (Lease et al., 2013). As result, any requester with access to a Worker ID (which is automatically included with survey responses) can identify the associated MTurk worker's personal in- formation through a simple online search. This can return private information such as photographs, full names, and product reviews (Lease et al., 2013). As this is a vulnerability in MTurk's s system that cannot be removed by the requester, it is common for requesters to simply drop the Worker IDs from the dataset before continuing with any data analy- sis. In our survey, we ask MTurr workers whether they are aware of the risk of their Worker IDs being attached to MTurk responses and if that knowledge is concern that influences their behavior.
5.1 Survey Results on Privacy
Our survey results indicate that concerns about privacy often lead to untruthful answers. Over- all, 79.6% of respondents stated they will answer question untruthfully if they feel that it is a pri- vacy violation, and 17.1% stated they will do so if they have concerns that the questions will be used against them. In regards to Worker IDs, 51.3% of workers stated that they are aware that they exist, but has not caused them to change their behavior.
Best practices on demographic questions include ensuring inclusivity for questions on gender iden- tity by allowing for open ended responses and con- sidering that terminology changes over time (e.g. "How do you currently describe your gender iden- tity?"), ) distinguishing between ethnicity and race, and permitting annotators to indicate multiple re- sponses for race, ethnicity, and gender in case they identify with multiple categories (Hughes et al., 2016). In our survey, we asked MTurk workers if they encountered questions where the responses provided did not adequately capture their identity.
In addition, workers often feel that demographic questions do not capture their identity. This is most often an issue for questions about gen- der (38.2%). age (38.2%), and sexual orienta- tion (37.5%). Workers also frequently answer demographic questions untruthfully (Figure 2). especially those regarding age (52.0%), gender
Another MTurk worker vulnerability is the use of Worker IDs. Each MTurk worker has an identify- ing Worker ID attached to their account, associated

---

(41.5%), and education (32.2%). 9.87% of respon- dents indicated that other questions caused them to answer untruthfully, and most frequently men- tioned questions requesting a phone number, name, or zipcode. Other respondents were concerned that their education level could be held against them.
in response quality (Kummerfeld, 2021). Mean- while, researchers such as Ashida and Komachi (2022) used 98% HIT approval rate to allow more workers to attempt HITs. This aligns with views expressed by MTTurk workers that a 99% approval rate is unreasonably difficult to obtain, as it re- quires near perfect record when HITs are often rejected without a logical reason explanation (u/L Lu/Laaagnww,ww, 2019). Furthermore, a 99% HIT approval rate largely serves to unfairly dis- criminate against newer workers (rather than low quality workers), due to the relative margin er- ror based on the total number of HITs completed (u/ptethesen, 2022). Turkopticon encourages pro- viding clear conditions for rejection due to the per- manent mark that rejection leaves on worker's record (Turkopticon, 2014).
6 Response Quality
Response quality can be maximized by ensuring that the task is clear and well-designed through a preliminary pilot task (annotation tasks released before the actual task to fine-tune the survey) and filterng respondents to include those who are more likely honestly complete the task (Gideon, 2012). Requesters can also work towards maximizing re- sponse quality by using various functionalities pro- vided by MTurk's filtering software.
Many MTurr requesters also use time as a means of filtering responses both as workers are complet- ing tasks (e.g. through timers) and after receiv- ing the complete dataset, by removing the top and bottom percentile of response times (Justo et al., 2017). In response one such task warning that "speeders" would be rejected, however, an r/mturk user stated that they "did that survey, made some lunch, checked my text messages then submitted" (u/gturker, 2022). Such forum posts suggest that these filters encourage workers to work on other tasks to pad time rather than spend more time think- ing about their answers.
6.1 MTurk Qualifications
MTuurk provides several mechanisms for improving response quality. These include Masters Qualifica- tions (which filter for "Masters Workers", selected on the basis of successful completion of a wide range of MTurk tasks over time), System Quali- fications (e.g. HIT approval rate, location), and Premium Qualifications (e.g., income, job func- tion). MTurk also has "Qualifications Types you have created," which allow requesters to assign spe- cific workers with a custom score between 0- 100 to determine their eligibility for tasks. One chal- lenge filtering MTurk workers is balaneing the of filters to minimize spam responses while still allowing enough real workers respond in order to collect a large enough dataset. Filtering on location is common because it helps to filter out MTurk workers who may be unqualified to complete tasks that require proficiency in certain languages or experience with certain culture or environment (Karpinska et al., 2021).
While several papers outline strategies to en- sure high response quality, there is little work on how such measures are received by MTurk workers themselves. Thus, our survey asked MTurk work- ers about how strategies such as limiting response time ranges and adding time limits affect their re- sponse quality. In addition, we asked if any of these measures are perceived as unreasonable. Lastly, we asked MTTurk workers to provide their own input on reasonable checks for response quality and whether any existing qualifications are perceived as unfair or unreasonable.
To prevent overfiltering respondents at the be- ginning, some papers suggest keeping numerical thresholds relatively low. Peer et al. (2014) stated that high-reputation and high-produuttivity workers can be filtered using 500 approved HITs and 95% HIT approval rate, while a looser lower bound to avoid low-reputation and low -wwoduduttivity workers is 90% approval rate and 100 approved HIT's How- ever, more recent research has found that around 2500 approved HITs and 99% HIT approval rate a more effective filter for MTurk surveys, since ap- proval rates below that result in a significant drop
6.2 Recent LLM Developments and Concerns
The rise of large language models (LLMs) poses significant risk to response integrity. Veselovsky et al. (2023) found that 33 to 46% of workers used LLMs complete a text summarization task on MTurk. Furthermore, these responses may be dif- ficult to detect, as LLMs such as ChatGPT outper- form humans on some annotation tasks (Gilardi

---

al., 2023), In our survey, we found that required, open-ended text responses made it casiest spot spam respondents who might pass unnoticed in rat- ing or classification tasks, but such manual methods may become less effective in the face of better text generation tools. However, multi-stage filtering procedures, such as Zhang et al. (2022)'s pipeline that uses qualification task and a longer endurance task to identify a high-quality worker list before the main task begins, can help identify a trustwor- thy pool of workers to minimize spam responses. Al-generated text detectors may also help to flag potential spam responses in these filtering pipelines (Veselovsky et al., 2023).
start task. Thus, good practice for rejecting HITs is is provide a clear rationale for why a HIT is being rejected as feedback for future work.
Sensitive Content
MTurk surveys involving more sensitive NLP tasks can pose psychological risks. MTTurk workers are more likely to have mental health issues than the general population, making them vulnerable population (Arditte et al., 2016). NLP tasks that pose potential risk to workers include labeling hate speech applicable the worker's own back- ground, identifying adult or graphic content, or questions about workers' mental health. There is usually higher risk with tasks that involve language that provides strong emotional stimuli (e.g. offen- sive tweets), which can potentially cause trauma (Shmueli et al., 2021). In addition, work may be more personally damaging when the offensive con- tent directly applies to the worker (Sap al., 2020). may be beneficial to modify studies being com- pleted by targeted demographic groups with addi- tional safeguards and warnings in the survey re- garding potential harm or offense.
6.3 Survey Results on Response Quality
One common metric to filter for response quality is response time. Overall, respondents indicated that task timers and time estimates are well-calibrated. However, we also found that time-based filters are largely unhelpful and counterproductive. When given minimum required response times, 11.5% of workers reported that they spend the additional time working on other things to extend their completion time. 22.4% of survey respondents reported having had HIT unfairly rejected due to responding too fast. Meanwhile, time limits can reduce response quantity and quality. One respondent explained that "it is frustrating when you have put in great deal of conseientious effort only to find the time has expired and you cannot submit HIT." These results align closely with the comments made in the Reddit thread described in the previous section 2022a).
In addition to exposing workers to poten- tial psychological harm, the presence and fram- ing of sensitive questions can cause workers leave tasks incomplete or answer untruthfully. AllThh Way83 (2022b) described pro- viding "accurate and thoughtful answers 99% of the time, the exception being in regards to my own mental health.. Requesters have been known to overstep their lane and contact the police for a wel- fare check and I' m not opening myself up that possibility for an extra $2, ya know?" Thus, it is vital for researchers to identify potentially sensi- tive content and take steps to protect workers from possible harm.
AlTTh
The filter most commonly seen as unreason- able the Masters qualification (55.9%), because MTurk has reportedly stopped giving out, One respondent explained that "Masters is a dead qual. Hasnt been issued to anyone since 2019. It is the single biggest gatekeeper to success for newer turk- ers like myself." Thus, MTTurk requesters who re- quire Masters qualification unknowingly filter out all workers that joined after this time.
Generally, researchers are aware of the presence of offensive content, reflected in content warn- ings included in the beginning of published pa- pers (c.g., Sap al., 2020). However, there is a clear need for standardized best practices due to the highly vulnerable nature of such MTTurk work.
On average, respondents indicated that a reason- able HIT approval rate is 93.6%, while the median response was 98%. Overall, MTurk workers re- ported a strong incentive to maintain high HIT ap- proval rate, as 40.1% of survey respondents stated that the approval rate of the requester influences the amount of effort they put into their responses and 36.2% state that it influences whether they even
A relevant example is Facebook's partnership with Sama, company that pays workers to identify illegal or banned content on social media (Perrigo, 2022). An interview with the workers revealed high levels of psychological distress due insuf- ficient breaks and subpar in in-person resources. an online setting such as MTurk, such personal-

---

ized care is even less accessible, making more difficult provide support for vulnerable workers. These workers' experiences exemplify the difficul- ties faced by the large, invisible workforce behind MTurk and powerful NLP applications, also de- scribed by Gray and Suri (2019) in their discussion of the typically unregulated "ghost work" that pow- ers machine learning.
Information Desired for Sensitive or Harmful Tasks
Ceneral content warnings
Information Type
Specific content warnings
Mental health resources
Other
Common safeguards include listing clear descrip- tions of the benefits and risks of the task, providing mental health resources for workers, using data safety monitoring boards, adhering to institutional review board policies, and following confidential data code and informed consent procedures (Kim et al., 2021). We look to further add to this list of best practices by asking survey respondents whether there questions that will immediately deter them from completing a task answer untruthfully and preferred actions or resources pro- vided by requesters to minimize possible harm.
Responses 01)
Figure 3: Information desired for tasks about sensitive or harmful content. Nearly a third of annotators desire specific content warnings (e.g., "homophobia") though general content warnings (e.g.. "offensive content") are more common in current tasks.
levels. We suggest paying around the minimum wage of $15 per hour (slightly above the median minimum threshold of $12, and in accordance with the views expressed by r/MTurk users), but not too much higher, which decreases response quality. Additional budget may be used as bonus payments, which are effective incentives when provided by reputable accounts.
7.1 Survey Results on Sensitive Content
Sensitive survey questions, such as questions that ask workers to annotate harmful or offensive con- tent (e.g. hate speech), not only expose workers potential harm, but also often lead to inaccu- rate incomplete responses. When we asked for specific types of questions that cause workers to de- cide to not complete a task, 23.7% of respondents indicated offensive content, and 19.1% 1% indicated questions concerning mental health. As seen in Figure 2, 20.4% of workers reported answering questions on offensive questions untruthfully and 18.4% did so for questions regarding mental health. In survey that contains potentially sensitive questions, 13.1% of respondents most preferred general content warnings (whether the survey con- tains sensitive content), 30.9% preferred specific content warnings (the type of sensitive content the survey contains), and 23.7% preferred men- tal health resources (Figure 3). These numbers indicate that far higher number of workers prefer specific content warnings than there currently in MTurk studies.
Minimize the collection of private information, and be transparent. Workers are highly sensi- tive to questions that they see violations of pri- vacy, such demographic questions. It is thus prudent minimize their use, provide a brief expla- nation for their importance, clarify that the answers are confidential and will not be held against the re- spondent, and include an option for respondents not to answer. To protect worker privacy, always drop Worker IDs before working with the dataset. Also follow worker-centered policies such Turkopti- con's 's Guidelines for Academic Requesters (Turkop- ticon, 2014) maintain transparent requester poli- cies and communication.
Avoid time-based filters and the Masters qualifi- cation. Our results indicate that both lower and upper bounds on completion time are flawed meth- ods of quality filtering. Thus, we discourage reject- ing HITs based on time, barring humanly impossi- ble extremes. In addition, to avoid unnecessarily filtering out workers who joined MTurk later, do not use the outdated Masters' Qualification. For high- responses, 98% HIT approval rate filter adheres to both worker preferences and re-
8 Recommendations
We consolidate our findings on worker perspectives to arrive at several recommendations.
Pay at least minimum wage, but not exception- ally higher. Payment can influence MTurk work- ers' perception of the task as well as their effort
We note that living wages vary by region and inflation, so the ideal pay is likely to change over time and may depend on the region from which MTurr workers are recruited.

---

search findings, it exactly equals the median value proposed in our survey results. Clearly out- line the requirements for task approval to increase trust between the worker and the requester.
Ethical Considerations
Understanding the factors that make MTurk work- ers more likely to trust requester and thus more likely to provide personal information accept tasks with unusual payment schemes (e.g., low nor- mal pay and high bonus pay) could be misused by spam requesters or phishers to maliciously acquire more information from MTurr users or extract their labor without fair pay. We understand that these are potential consequences of our research, but hope that they are outweighted by the potential bene- fits of good-faith requesters understanding how to improve the effects of their tasks workers.
Consider effects on worker approval rate when controlling for quality. Since MTurk spam rates are high (as we saw firsthand), quality filtering is key to obtaining good results. However, workers are sensitive the fact that drops in approval rate bar them from future tasks, and so are both less willing to accept tasks from requesters with lower approval rates, and frustrated when work is rejected without explanation. As a result, is best to pro- vide explanations when rejecting workers. Paid qualification tasks that filter and/or train workers before they begin larger main task, such as the pipeline proposed by Zhang et al. (2022), can help minimize spam responses without needing to re- ject many responses on the main task. As text gen- eration tools become more powerful, Al-generated text detectors may also help to flag potential spam responses (Veselovsky et al., 2023).
A Acknowledgments
Many thanks to Nicholas Tomlin and Eric Wallace for their feedback on earlier drafts of this paper.
References
Kimberly A Arditte, Demet Çek, Ashley M Shaw, and Kiara R Timpano. 2016. The importance of assessing clinical phenomena in Mechanical Turk research.
Provide context for subjective tasks. When pos- sible, preface questions with an explanation of the purpose of the annotation task, such as the down- stream use case. Additional context is also helpful, such as whether the text is from a longer passage or was sourced from a specific platform.
Mana Ashida and Mamoru Komachi. 2022. Towards automatic generation of messages countering online Proceedings
hate speech and microaggressions. of the Sixth Workshop on Online Abuse and Harms (WOAH), pages 11-23, Seattle, Washington (Hybrid). Association for Computational Linguistics.
Provide general and specific content warnings. If questions in sensitive areas such as offensive content or mental health are necessary, provide general and specific warnings for sensitive content, explanation of the questions' purpose, and an overview of how the data will be used.
Chris Callison- Burooh and Mark Dredze. 2010. Creating speech and language data with Amazon's 's Mechanical Turk. Proceedingy of the NAACL HLT 2010 Work- shop on Creating Speech and Language Data with Amazon's 's Mechanical Turk, CSLDAMT '10, page 1-12, USA. Association for Computational Linguis- tics.
Lior Gideon. 2012. The art of question phrasing. In Handbook of Survey Methodology for the Social Sci- ences, pages 91-107. Springer.
Limitations
We surveyed MTurk workers from the United States; the views of these workers may differ from those of workers other parts of the world. In addition, because the survey was completed by the first 207 workers to see the posted task, there may be sampling bias in the workers who answered (i.e., ones who go on MTurk more often or are quicker to answer tasks are more likely to have filled out the survey). Future studies with a larger sample pool of workers from different parts of the world, and on different crowdworking platforms, could help to examine the extent to which these results generalize.
Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. ChatGPT outperforms workers for text- annotation tasks. ArXiv, abs/2303.1500
Mary L. Gray and Siddharth Suri. 2019. Ghost work how to stop Silicon Vulley from building new globul underclass. Houghton Mifflin Harcourt.
Kotaro Hara, Abigail Adams, Kristy Milland, Saiph Savage. Chris Callison-Buree and Jeffrey P. Bigham. 2018. data-driven analysis of workers' carnings on Amazon Mechanical Turk. In Proceedings of the 2018 CHI Conference on Human Factors in Com- puting Systems, CHI '1, page 1-14, New York, NY, USA. Association for Computing Machinery.

---

Jennifer Hughes, Abigail Camden, and Tenzin Yangchen. 2016. Rethinking and updating demo- graphic questions: Guidance improve descriptions of research samples. Psi Chi Journal of Psychologi-
Maarten Sap. Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019. The risk of racial bias in hate speech detection. In ACI.
cal Research, 21:138-151.
Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Juraf- sky, Noah A. Smith, and Yejin Choi. 2020. Social bias frames: Reasoning about social and power im- plications of language. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pages 5477-5490, Online. Association for Computational Linguistics.
Abigail 7. Jacobs and Hanna Wallach. 2021. Measure- ment and fairness. In Proceedings of the 2021 ACM Conference Fairness, Accountability. and Truns- parency, FAceT '21, page 375-385, New York, NY, USA. Association for Computing Machinery.
Jason T. Jacques and Per Ola Kristensson. 2019. Crowd- worker economics in the gig economy. In Proceed- ings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI '19, page 1-10, New York, NY, USA. Association for Computing Machin- ery.
Boaz Shmueli. Jan Fell, Soumya Ray, and Lun-Wei Ku. 2021. Beyond fair pay: Ethical implications of NLP crowdsoureing. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3758-3769, Online. Association for Computational Linguisties.
Raquel Justo, M. Torres, and José Alcaide. 2017. Mea- suring the quality of annotations for a subjective crowdsourcing task. pages 58 68.
Rion Snow, Brendan O'Connor, Daniel Jurafsky, and Andrew Ng. 2008. Cheap and fast - but is it good? evaluating non- con-expert annotations for natural lan- guage tasks. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Process- ing, pages 254-263, Honolulu, Hawaii. Association for Computational Linguistics.
Marzena Karpinska, Nader Akoury, and Mohit lyyer. 2021. The perils of using Mechanical Turk evalu- ate open-ended text generation.
Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1265-1285, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Turkopticon. 2014. Guildeines for academic requesters.
Turkopticon. 2022. End the harm of mass rejections.
Jane Paik Kim, Katie Ryan, Tenzin Tsungmey. Max Kasun, Willa A. Roberts, Laura B. Dunn, and Laura Weiss Roberts. 2021. Perceived protectiveness of research safeguards and influences on willingness to participate in research: A novel mturk pilot study. Journal of Psychiatric Research, 138:200-206.
w/Bermin299. 2022. Getting to the point you can't trust any requestor paying less than a $1 (surveys). Reddit,
w/dgrochester55. 2022. How seriously do you answer questions? Reddit.
w/tturkrr 2022. Gotta love the transparency of a seem- ingly reasonable requester! Reddit.
Jonathan K. Kummerfeld. 2021. Quantifying and avoid- ing unfair qualification labour in crowdsourcing. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume Short Papers). pages 343-349, Online. Association for Computational Linguistics.
LalughinggA/Theeaaaae 2022a. Gotta love the trans- parency of a seemingly reasonable requester! Reddit.
ALaaghhnggAllTheeeeee. Way83, 2022b. How seriously do you answer questions? Reddit.
Matthew Lease, Jessica Hullman, Jeffrey Bigham, Michael Bernstein, Juho Kim, Walter Lasecki, Saci- deh Bakhshi, Tanushree Mitra, and Robert Miller. 2013. Mechanical Turk is not anonymous. SSRN Electronic Journal.
w/Lushangdewww. 2019, question for workers with approval rate "greater than 99%". Reddit.
u/ p/tttheeeen 2022. Rejection from simran g. Reddit.
w/Sharpsilverz. 2022. Requesting speakers read out loud for about 7 minutes- what is ethical pay? Reddit.
Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent disagreements in human textual inferences. Transac- tions of the Association for Computational Linguis- tics, 7:677-694
Veniamin Veselovsky, Manoel Horta Ribeiro, and Robert West. 2023. Artificial artificial artificial in- telligence: Crowd workers widely use large lan- guage models for text production tasks. ArXiv, abs/2306.07899.
Eyal Peer, Joachim Vosgerau, and Alessandro Acquisti. 2014. Reputation sufficient condition for data quality on Amazon Mechanical Turk. Behavior re- search methods, 46: 1023-1031.
Mark E. Whiting, Grant Hugh, and Michael S. Bemstein. 2019. Fair Work: Crowd Work Minimum Wage with One Line of Code, volume 7.
Billy Perrigo. 2022. Inside Facebook's African sweat- shop. TIME.

---

Huichuan Xia, Yang Wang, Yun Huang, and Anuj Shah. 2017. "Our privacy needs to be protected at all costs": Crowd workers' privacy experiences on Amazon Me- chanical Turk. Proc. ACM Hum.. Comp-tt Interact., (CCSCW).
knowledge of the risks involved in MTurk surveys. For example, participants may learn about the fact that Worker IDs are shared to survey requesters. In addition, the purpose of the study itself is to improve the MTurk survey experience for MTurk workers by making it safer, less invasive, and more reasonable. Thus, the MTurk workers completing the task will benefit from the survey findings. Risks/Discomforts
Lining Zhang, João Sedoc, Simon Mille, Yufang Hou, Sebastian Gehrmann, Daniel Deutsch, Elizabeth Clark, Yixin Liu, Miruna Clinciu, Saad Mahamood, and Khyathi Chandu. 2022. Needle in haystack: An analysis of finding qualified workers on mturk for summarization.
Although the survey does not explicitly ask any demographic questions, it asks about participants' perceptions and opinions on sensitive questions such as personal demographic information and of- fensive content. These questions may be a possible area of discomfort for the participant. As with all research, there is chance that confidentiality could be compromised; however, are taking precau- tions to minimize this risk.
A Informed Consent
A.1
Key Information
You are being invited to participate the online ver- sion of our research study on Amazon Mechanical Turk (MTurk). Participation in research is com- pletely voluntary. The purpose of the study is to examine the best practices and common pain points involved in conducting annotation tasks on MTurk. The study will take approximately 10 minutes, and you will be asked to respond to questions regarding your experience and opinion on conducting annota- tion tasks. Risks and/or discomforts may include potentially sensitive questions that may cause dis- comfort in the survey respondent. There is no direct benefit to you. The results from the study may help improve general MTurk workers' experiences in future MTurk annotation tasks.
Confidentiality
Your study data will handled as confidentially as possible. If results of this study are published or presented, individual names and other personally identifiable information will not be used. Amazon Mechanical Turk automatically stores a list contain- ing the Worker IDs of all participants who work on survey requester's task, but these Worker IDs will not be linked to the anonymous Qualtrics responses. will not downloaded, and will only be used ensure payment. We will delete the Worker IDs on the Mechanical Turk platform immediately once participants have been paid. We will not obtain any identifiers linked to individual Qualtrics survey responses and there is no other potentially iden- tifiable information collected in the survey. The Qualtrics survey responses will be downloaded im- mediately after the required number of responses is reached. After they are downloaded, they will be deleted from the Qualtries platform within one week. The survey data will be secured on two password password-protected computers, and it will be re- tained for no more than 6 months. After this time period, only the aggregate stats will be retained. We will not be transferring any identifiable data after results are downloaded from Amazon Mechanical Turk. When the research is completed, we will save the data for possible use in future research done by us or others. We will retain these records for up to 6 months after the study is over. The same measures described above will be taken to protect confiden- tiality of this study data. Your personal information may be released if required by law. Authorized
Introduction and Purpose
(Researcher names) We would like invite you take part in our research study, which concerns observing MTurk workers' experiences on the plat- form in order to improve understanding on best practices and common pain points involved in con- ducting annotation tasks. The key research areas involve 1) fair payment methods and its effective- ness in incentivizing workers, 2) administration of sensitive questions, 3) methods for maximizing re- sponse quality, and 4) miscellancous questions on survey design and MTurk workers' experiences. Procedures
If you agree to participate in this research, we will ask you to complete the attached Qualtrics survey included in the MTurk task. The survey will in- volve questions about payment, administration of sensitive questions, maximizing response quality, survey design, and MTurk workers' experiences, and should take about 10 minutes complete. Benefits
There is no direct benefit to you from taking part in this study. It is hoped that the research will increase

---

representatives from the following organizations may review your research data for purposes such as monitoring or managing the conduct of this study: University of California. Identifiers might re- moved and the de-identified information used for future research without your additional informed consent.
Compensation
Participants will be compensated with $2.50 upon completing the survey on Amazon Mechanical Turk.
Rights
Participation research is completely voluntary. You are free to decline to take part in the project. You can decline to answer any questions and are free to stop taking part in the project at any time. Whether you choose to participate, to answer any particular question, continue participating in the project, there will be no penalty you or loss of benefits to which you are otherwise entitled. Questions [Contact Information]
you agree to take part in the research, please print copy of this page to keep for future refer- ence, then check the "Accept" box below.
B Survey and Data Cleaning Details
Figures 4 through contain the full text of the survey.
After data was collected, the mandatory free text responses were manually reviewed by two authors for spam, including free text responses that were identical for multiple survey responses, direct copies of text from the question with nothing added; numbers in response questions that do not ask for numbers; or responses that were incoherent. We also removed incomplete responses (where the survey was only partially answered).

---

MTurk Annotation Practices for NLP Survey Questions
How the amount money you make on Mechanical Turk (MTurk) compare amount you would able make elsewhere?
make more on MTurk
make the same amount money
make elsewhere the real world
(Read before completing questions 2-4) There couple payment features offered on Amazon Mechanical Turk. The relevant ones are defined below:
Normal payments only: Standard listed payments paid exchange for completing task. Payments are contingent the task being approved for work quality.
Bonus payments in addition to normal payments: Additional payments given to workers after they have completed the task addition to the advertised payment rate.
Q2 Opinion payment type:
(1=Dislike, 5=Like)
Bonus Payments
Normal Payments
Q3 (Optional) Please elaborate on your above response:
How payment type influence of effort you put your responses?
(1=Reduced response effort, 5=Increased response effort)
3
Bonus Payments
Normal Payments
Q5 (Optional) Please elaborate on your above response:
Page 1 of 7
Figure 4: Full survey (page 1).

---

How the payment type influence the likellhood that you will complete the task?
(1=Much lower likelihood completion, 5=Much higher likelihood of completion)
Bonus Payments
Normal Payments
Q7 (Optional) Please elaborate on your above response:
Do you have minimum threshold for payment per unit determines whether you complete task? (If so, please indicate dollar amount per unit time, e.g. $15 per minutes)
No Minimum
Yes, have a minimum:
Q9 As follow for your previous response, what the rationale behind minimum threshold? (Select all that apply)
want my payment comparable the wage can eam elsewhere
The payment level indicates reliable Requestor
It's not worthwhile for complete tasks that pay less than that rate
Other
Page 2 of 7
Figure 5: Full survey (page 2).

---

types questions that will cause you to decide not complete (Select that apply)
Demographic: Race
Demographic: Gender
Demographic: Sexual Orientation
Demographic: Age
Demographic: Education
Demographic: Employment
Demographic: Disability
Demographic: Political stance
Questions containing offensive content (e.g. tasks containing passages asking to identify the presence of harmful language)
Questions concerning your mental health
Other
Q11 Under what circumstances would you respond to a question untruthfully?
When presented with questions that feel like privacy violation. (Please describe specific questions you've come across that apply)
Concerns that the question responses may be used against (Please describe specific questions you've come across that apply)
Other
Q12 Are there certain types questions that will cause you to answer untruthfully? (Select that apply)
Demographic: Race
Demographic: Gender
Demographic: Sexual Orientation
Demographic: Age
Demographic: Education
Demographic: Employment
Demographic: Disability
Demographic: Political stance
Questions containing offensive content (o.g. tasks containing passages asking to identify the presence language)
Questions concernin your mental health
Other
Page 3 of 7
Figure 6: Full survey (page 3).

---

you any demographic questions and/or their response options fully capture your identity?
Demographic: Race
Demographic: Gender
Demographic: Sexual Orientation
Demographic: Age
Demographic: Education
Demographic: Employment
Demographic: Disability
Demographic: Political stance
Other
Q14 (Optional) Please elaborate your above response:
Q15 you were considering completing a survey containing potentially sensitive or harmful questions, what information would you like to be provided to you?
General content warnings (e.g. "offensive content")
Specific content warnings (e.g. "homophobia")
Mental health resources
Other (Please elaborate)
Q16 In MTurk tasks, although name is not included with the survey results, WorkerIDs that uniquely identify each responder are stored with your responses to MTurk tasks. Were aware of this, and is this concern for you?
aware,
concern
aware, and concern, but has caused change my behavior when
completing tasks
was aware, and concern that may cause me to avoid answering certain questions
am aware, and concern that may cause me to avoid working on certain tasks
not aware,
a concem
not and this knowledge will affect my responses and selection surveys complete in the future
Page 4 of 7
Figure 7: Full survey (page 4).

---

specifies you must complete within certain absolute No, haven't seen such a
you encountered
range response (e.g. between minutes hour) be approved? If how that impacted the way you complete the task?
have such task haven't worked
Spent time doing other things to extend work time
Spent more time thinking about responses for each task (for minimum required)
Other (Please elaborate)
Q18 you completed that specifies certain percentile response times (e.g. responses in the bottom and top 5%) will rejected? If so, how that impacted way you complete
No, haven't seen such a task
have seen such task but haven't worked on
Spent time doing things to extend
Spent time thinking about responses for each task minimum required)
Other (Please elaborate)
Q19 Tasks on MTurk typically list the expected amount time needed to complete the task. What is your perception this?
1=Estimates are shorter than actual completion time 5=Estimates longer than actual completion time
Q20 Sometimes, tasks on MTurk include a timer that limits the amount of time one can spend on task. Please answer the following questions regarding this feature:
1
3
4
What is your general perception of this? (1=Negative, 5=Positive)
reasonable is the timer? (1=Time limit short, 5=Time limit is far above the time needed)
How does this timer influence your response quality? (1=) put less effort, put more effort)
Page 5 of 7
Figure 8: Full survey (page 5).

---

requirements workers must qualify complete the Have any them felt unreasonable?
Many
Custom qualifications (if so, list an example)
Premium qualifications (e.g. language fluency; household income; employment)
Masters qualificttion
Hit approval rate
Number of HITs approved
Q22 (Optional) Please elaborate your above response:
Q23 What do you think reasonable hit approval rate to require for task?
Provide percentage:
Don't know
Q24 Which of the factors below would most effectively encourage you to put more effort into your responses? (Select all that apply)
High(er) payment
Bonus payments
Cleariprotective data sharing policies
Difficulty understanding task
Readability and of understanding
Approval rate requester
Accurate time estimation
Other Turkers' experience with the requester (as shared on platforms such as Reddit)
Which of the below would most effectively encourage you decide to start task? (Select that apply)
High(er) payment
Bonus payments
Ciear/protective data sharing policies
Difficulty understanding task
Readability and of understanding task
Approval rate of requester
Accurate time estimation
Other Turkers' experience with the requester (as shared on platforms such Reddit)
Page 6 of 7
Figure 9: Full survey (page 6).

---

you ever had response unfairly rejected? so, indicate what reasons:
Failing attention checks
Inaccuracy response
Completing too short time
Completing in too time
Wasn't told why
Other (Please elaborate)
influence your perception how "reasonable" "reliable" requestor on MTurk (where most influential, influential)
Please following aspects
High payment
Number of previously published tasks
Clarity instructions
Length of task
Design accessibility of
Other Turkers' experience with the requester (as shared on platforms such Reddit)
Reddit) and ask way. Examples of this could include: 1) highlighting the answer to question passage, 2) Given pieces text, selecting the preferred/more fluent/better response, indicating if there is positive negative sentiment the provided text. Have you completed annotation task MTurk?
Q28 Annotation tasks tasks MTurk that provide textual passage (e.g.
the worker to evaluate
Yes
Q29 What part of annotation tasks (as described above) do most commonly find confusing?
Have you ever come across questions where you were not sure about the correct or answer to put? what would have most helpful you in answering the question? elaborate your choice your specific experience(s). (Select apply)
Context
Purpose question
Platform that annotations would used (for annotation tasks)
Other
Q31 (Optional) Please elaborate above response:
Page 7 of 7
Figure 10: Full survey (page 7).


Reviews:
[{"review_id": "dTSXnZR59w", "reviewer": "Reviewer_PCxw", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for taking the time to provide helpful feedback on our paper. We will integrate your comments into the camera-ready version:\n- We will mention the importance of replicating the study on alternative platforms as future work.\n- We will add a reference to Jaques and Kristensson, 2019 in the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The authors consolidate many useful practices that are know by word of mouth, but have, to the best of my knowledge, not been consolidated yet.\n- The authors report very important insights that, after conducting crowdsourcing studies for various years, a new to me (e.g., regarding paying too much).\n- The authors conduct a large-scale survey and report all survey details.\n- The paper is well written and follows a clear and comprehensive structure by grouping literature survey results and questionnaire results together per aspect.\n- Including Reddit comments as a complementing source to the literature review and the questionnaire is a good idea and makes the paper more tangible. ", "reasons_to_reject": "I do not see a real reason to reject this paper!\n- Minor: The missing direct direct link to NLP can be criticised, but given the important role of crowdsourcing in NLP, I argue that this paper is suitable for the conference anyways.\n- Minor: The paper only focuses on MTurk, replicating the study on alternative platforms would be interesting, however, this cannot be held against the authors as they already conducted a large scale study which, I think, is preferable to a series of smaller studies.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors take a three-fold approach to investigate crowdworker perspectives on MTurk: They (i) review related work, (ii) evaluate public comments (from Reddit), and (iii) conduct a large scale survey on MTurk.\nThe results are grouped along five guiding themes: Task clarity, payment, privacy, response quality, and sensitive content.\nImportantly, the paper consolidates information that is partly already known by experienced practitioners (e.g., that using the Master qualification is unfair to new workers) but not available as a united resource yet.\nIn addition, it reports interesting findings (e.g., regarding not paying too much) that I consider to be equally surprising and helpful.\nHaving had this paper a couple years ago would have saved me a lot of trouble and money, and I would recommend the paper to all my students working on crowdsourcing on MTurk.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "- Crowdworker Economics in the Gig Economy, Jaques and Kristensson 2019. (https://dl.acm.org/doi/10.1145/3290605.3300621)"}, "extra_text": {}}, {"review_id": "2PI2RdZfom", "reviewer": "Reviewer_NbBA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 1, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for taking the time to provide feedback on our paper. If you have any specific questions or concerns, we would be happy to address them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper presents strong, logical and actionable steps for better data collection. The paper considers various aspects of the data collection, instead of being limited to a single aspect.\n\nThe paper is very well written (as someone who is not familiar with the relevant literature, I had no issues following the points in the paper)", "reasons_to_reject": "Not being familiar with the literature, I don’t see any apparent reasons to reject the paper.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper argues for including perspective of MTurk workers for improving collected data quality and improving worker rights. They prescribe best practices for data collection using MTurk, considering various aspects including fair pay, privacy concerns and sensitive content.\n\nThe paper is very well written (as someone who is not familiar with the relevant literature, I had no issues following the points in the paper)", "excitement": 3, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "wOJO9FmwSQ", "reviewer": "Reviewer_sUkx", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for taking the time to provide helpful feedback on our paper. We will integrate your comments into the camera-ready version. \n\n> “Although users claim this paper is for NLP tasks, most content of this paper is general for different domains of AI and I didn't find some specific conclusion used for NLP tasks.”\n\nAlthough much of the content would be useful to any AI researcher doing crowdsourcing work, crowdsourcing is particularly critical to NLP (as reviewer PCxw noted). The percent of NLP papers involving crowdsourcing grew steadily from 2017 to 2020, with 12% of accepted papers in major conferences (ACL, EMNLP, and NAACL) in 2020 explicitly describing crowdsourcing as part of their research methods (Shmueli et al., 2021). In addition, we include several sections that focus on issues specific to text data (e.g. 3.1 and 3.2 on clarity in annotation tasks, and 6.2 on annotators’ use of LLMs for writing text assumed to be human-authored). We came at this from the perspective of NLP researchers and did not focus on investigating issues that might be relevant to other domains, such as computer vision, but are glad that the findings might be more widely applicable. We can expand on the sections that are especially specific to NLP for the camera-ready version.\n\n> “Only 207 workers completed the questionnaire in this paper, hence, the results may be less representative.”\n\nWe would argue that 207 workers is a reasonable size for a survey (compare 80 respondents in Kim et al., 2021 and 435 respondents in Xia et al., 2017). In addition, because our results include manual analysis of qualitative, open-ended feedback from survey responses, we prioritized detail of analysis over volume of responses. However, we will note the potential non-representativeness of the sample as a limitation.\n\n> “Not sure if this paper is suitable to be a research track paper in EMNLP. Authors conclude some phenomena from MTurk annotators and provide their own suggestions.”\n\n\nThe ACL’23 Peer Review Policies note that “NLP is an interdisciplinary field, relying on many kinds of contributions: models, resource, survey, data/linguistic/social analysis, position, and theory” (Section 4.2). Much of NLP research is heavily dependent on crowdsourced data collection. Given that crowdsourcing is both fundamental to NLP and one of the aspects most vulnerable to errors, our paper contributes to the research community by (1) providing an overview of best practices that are often spread by word-of-mouth, with no centralized resource to aid NLP researchers collecting crowdsourced data; (2) evaluating public MTurk worker comments and surveying workers, which provides novel findings about flaws of current accepted practices; and (3) providing recommendations to help NLP researchers. We argue that our results, which are grounded in the research literature and our survey findings, can directly help future NLP researchers to collect crowdsourced data with improved worker recruitment, labor practices, and data quality.\n\nReferences\n\nJane Paik Kim, Katie Ryan, Tenzin Tsungmey, Max Kasun, Willa A. Roberts, Laura B. Dunn, and Laura Weiss Roberts. 2021. Perceived protectiveness of research safeguards and influences on willingness to participate in research: A novel mturk pilot study. Journal of Psychiatric Research, 138:200–206.\n\nBoaz Shmueli, Jan Fell, Soumya Ray, and Lun-Wei Ku. 2021. Beyond Fair Pay: Ethical Implications of NLP Crowdsourcing. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3758–3769, Online. Association for Computational Linguistics.\n\nHuichuan Xia, Yang Wang, Yun Huang, and Anuj Shah. 2017. \"Our privacy needs to be protected at all costs\": Crowd workers’ privacy experiences on Amazon Mechanical Turk. Proc. ACM Hum.-Comput. Interact., 1(CSCW).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Manual data annotations are essential in various AI tasks. MTurk is a popular platform for annotations. This paper reveals multiple aspects of the data annotations. The authors conduct a survey and conclude the potential issues that can exist in the current MTurk workers.\n\n2. The suggestions from this paper can provide some useful advice for the future MTurk usage.", "reasons_to_reject": "1. Although users claim this paper is for NLP tasks, most content of this paper is general for different domains of AI and I didn't find some specific conclusion used for NLP tasks.\n\n2. Only 207 workers completed the questionnaire in this paper, hence, the results may be less representative.\n\n3. Not sure if this paper is suitable to be a research track paper in EMNLP. Authors conclude some phenomena from MTurk annotators and provide their own suggestions.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper conducts a critical literature review and a survey of MTurk workers to address open questions regarding best practices for fair payment, worker privacy, data quality and worker incentives.\nAuthor conclude their findings in this paper and provide suggestions on how to improve the quality of data annotation from MTurk.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This manuscript seeks to deal with the confusion that arises from the terms (and research on) values, morals, and ethics. The manuscript surveys prior literature from outside of NLP on these concepts, then surveys NLP literature, and suggests future avenues for NLP.

In general, the reviewers are positive towards this manuscript and particularly appreciate:

1. That there is confusion of the terms and this manuscript may provide a starting point to address such confusion
2. The manuscript provides a comprehensive report on how the terms have been used
3. The manuscript provides quantitive support for their recommendations and findings
4. The manuscript highlights flaws in terminology in prior work

However, the reviewers had the following contention:

1. The manuscript would do well to consider additional related terms such as norms, beliefs, customs, behaviors, and ideologies.
2. It can be hard to gain a sense of the bigger picture from how results are presented
3. The selection of papers is narrow and does not take into consideration topics where the terms in question are implicit.
4. It is unclear how to implement recommendations for common terminology in future work

Wrt. 1 both authors and reviewers agree that this manuscript can serve as a starting point for such considerations. To this effect, authors will reframe writing to make the bigger picture (2) clearer. While the authors do not address 3, the scope of addressing this is incredibly large and is in my opinion more appropriate for future work. Finally, the authors correctly identify that NLP researchers creating vocabularies may lead to further inaccuracies and limit opportunities for interdisciplinary collaboration.

5193

This manuscript presents a study on practices and experiences of annotation on amazon mechanical turk. Towards the goal of understanding crowdwork, they review prior work, public discussions on reddit, and conduct a survey on MTurk. As a result they provide several recommendations based on their findings.

Reviewers are generally positive towards this manuscript, particularly as it provides evidence-based recommendations for best-practices for annotation.

Of the concerns highlighted, only the concern that the manuscript applies to more fields than NLP is reasonable. While the reviewer is correct that best practices for annotation work is not only applicable to NLP, work in NLP very often requires human labelling and the paper is therefore also very applicable to the ACL community.

A slight concern that I have: The book Ghost Work by Suri and Grey is missing from the references, which would further support and contextualize the claims and findings. Moreover, Turkopticon, the organization that has been working on crowd worker’s rights seems to have been neglected, and the findings from their work and their advocacy should be included in the final manuscript, should it be accepted.
Decision: Accept-Main