A Needle in a Haystack: An Analysis of High-Agreement Workers on MTurk for Summarization
Year: 2023

Abstract:
To prevent the costly and inefficient use of resources on low-quality annotations, we want a method for creating a pool of dependable annotators who can effectively complete difficult tasks, such as evaluating automatic summarization. Thus, we investigate the recruitment of high-quality Amazon Mechanical Turk workers via a two-step pipeline. We show that we can successfully filter out subpar workers before they carry out the evaluations and obtain highagreement annotations with similar constraints on resources. Although our workers demonstrate a strong consensus among themselves and CloudResearch workers, their alignment with expert judgments on a subset of the data is not as expected and needs further training in correctness. This paper still serves as a best practice for the recruitment of qualified annotators in other challenging annotation tasks.

heading: Future Work
text: GOLD and SILVER workers who complete all HITs are added to a maintained worker list as high-agreement annotators for future tasks. To ensure their general performance for the true annotation task, a reference-based task to evaluate information coverage between summaries is conducted with these workers later. While serving as a best practice beyond its scope, our study has the following contributions:
• establish a cost-effective recruitment pipeline on MTurk to consistently build a pool of annotators for high-agreement annotations. • successfully recruit 12 out of 200 (6%) superior annotators for text summarization evaluation, while reducing costs and guaranteeing high agreement. • rigorously demonstrate that the annotators identified through our pipeline can match or surpass the IAA of expert annotators and standard statistical techniques, though further calibration may be required for correctness. 2 Related Work
Challenges of Human Evaluation Compared to automatic evaluation metrics for NLG tasks like BLEU (Papineni et al. , 2002) and ROUGE (Lin, 2004), human annotations from non-expert annotators on MTurk can reach an agreement with gold standards or expert judgments (Callison-Burch, 2009). Although recent works leverage language models like BERT (Devlin et al. , 2019) to get better automatic evaluations (Zhang et al. , 2020), human judgments are still indispensable in identifying nuances in specific language tasks (Manning et al. , 2020). Finding qualified workers to carry out the evaluations is crucial. This is especially true for tasks like text summarization, which lacks consensus on evaluation protocols (Fabbri et al. , 2021) and is often inconsistent with previous human evaluations (Hardy et al. , 2019). However, human evaluation from non-expert crowdsourcing platforms have low quality (Gillick and Liu, 2010) and a sim-
ple qualification filter is not sufficient to identify qualified workers (Berinsky et al. , 2012; Robinson et al. , 2019). Some studies applied quality control mechanisms to filter out poor quality annotations, resulting in a relatively low pass rate for a variety of tasks (Graham et al. , 2017, 2018; Mille et al. , 2019). The fact that up to 70% of the HITs are eventually discarded indicates a huge resource waste. Even with qualified workers, human annotations might still be adversely affected by factors like incomplete instructions or unfair wages paid to annotators (Huynh et al. , 2021), and workers need clear references, schemes, or standards to follow (Howcroft et al. , 2020; Karpinska et al. , 2021). Thus, our study serves as a detailed reference for finding qualified MTurk workers for a summarization evaluation task and further identifying those who can assist in a large number of annotations. Inter-Annotator Agreement For annotations without true labels or those evaluated with a qualitative scale such as Likert scale (Likert, 1932), the inter-annotator agreement (IAA) among MTurk workers measures the reliability of the annotations. For example, Cohen’s Kappa (Cohen, 1960) measures IAA between a pair of results of the same length from two annotators, while Krippendorff’s Alpha (Hayes and Krippendorff, 2007) measures the agreement of a set of results from any number of annotators, even with unequal sample sizes. Both range from −1 to 1, with 1 indicating complete agreement. Further studies also continue to mitigate annotator bias through complementary methods to IAA (Amidei et al. , 2020), aimed at high-quality annotations. In our study, we utilize both Cohen’s Kappa and Krippendorff’s Alpha as the measurement of annotation reliability. 3 Methods
In this section, we detail how the workers were recruited and which tasks were carried out. 2
3. 1 MTurk Qualification Settings
To narrow down the pool of our target workers, we set a few pre-defined qualifications for workers on MTurk before publishing the qualification task: (i) the Location is set to “UNITED STATES (US)”; (ii) the Number of HITs Approved is set to be “greater than 1000” to target workers who are already experienced on MTurk; (iii) the HIT Approval Rate (%) is set to be “greater than or
2Appendix A. 9 shows instructions given during the tasks. equal to 99” to target workers who are able to finish tasks with high quality and have stable performance. We also set the task visibility as “Private”, which means our tasks are visible to any worker, but only workers who meet all qualification requirements can preview and accept. Paolacci et al. (2010) show that the annotations collected with the “Location” setting on MTurk are representative of the population of our target country in terms of demographic data. This helps mitigate biases introduced by samples from traditional recruitment methods like college undergraduate samples (Buhrmester et al. , 2011). We set qualification settings (ii) and (iii) based on previous work (Whiting et al. , 2019; Oppenlaender et al. , 2020; Kummerfeld, 2021) and our own experience on MTurk. Workers who meet all qualification requirements are eligible to participate in the qualification task. 3. 2 Qualification Task
Summarization task In summarization, the input is the text of a document and the output is a short summary. We evaluate a summary S according to 6 dimensions based on the criteria taxonomy presented in Howcroft et al. (2020), and workers are asked for a binary answer as to whether a dimension is satisfied in a summary or not:
• Understandability: can the worker understand S and is S worth being annotated. • Compactness: S does not contain duplicated information. • Grammaticality: S is free from grammatical & spelling errors. • Coherence: S is presented in a clear, wellstructured, logical, and meaningful way. • Faithfulness: all of the information in S can be found in the article; S accurately reflects the contents of the article. • Saliency: S captures the most important information of the article and does not include parts of the article that are less important. Training and qualification There are two main parts of the qualification task. The training part guides the workers through the above evaluation dimensions and instructs them on how to annotate. The definition of each dimension is illustrated with positive and negative examples, and full annotation examples are shown (summary and binary rating for each dimension). Then, workers are required to write an instruction summary in their own words
to make sure they have understood the task and are ready to annotate. The qualification part tests the worker’s understanding of the task. Three documents are provided, each with one summary. The worker reads the document and annotates the corresponding summary according to each dimension. The ratings are then compared to expert ratings provided by the authors of this paper. The last document comes with an attention check to test whether a worker is just randomly assigning scores without reading: a highlighted instruction asks the worker to ignore the task and select specific answers. Finally, an optional field is provided to collect feedback. Worker categorization Upon finishing their task, workers are categorized into four types:
• GOLD. The GOLD workers pass the attention check and annotate every dimension of every document in the qualification part correctly. • SILVER. The SILVER workers pass the attention check and make only one mistake when annotating each dimension of the documents in the qualification part. • BRONZE. The BRONZE workers pass the attention check and make more than one mistake when annotating each dimension of the documents in the qualification part. • BLOCK. The BLOCK workers fail to pass the attention check. The GOLD and SILVER workers are assigned a qualification score and proceed with the endurance task. Besides, we conducted multiple rounds of the qualification task to avoid influence from the time or day when the task was conducted and randomly sampled workers (Arechar et al. , 2017; Berinsky et al. , 2012). 3. 3 Endurance Task
The endurance task is designed to test whether a worker can reliably perform a large number of annotations. The workers who finish all HITs of this task are assigned the highest qualification score and are added to a maintained worker list. The endurance task comprises 10 HITs. For each HIT, a document and 4 corresponding summaries generated by different models are provided; each HIT takes around 5 minutes to finish (approximately an hour for all HITs). To keep the task simple we only evaluate each summary on one dimension, but to ensure that the task is challenging enough we (i) use the most subjective of the 6 di-
mensions, Saliency, and (ii) use a more fine-grained 10-point Likert scale (from 1 to 10). Rationale for choosing 10 HITs Our motivation is two-fold: to find workers who were able to complete many tasks and whose annotations are better than random. As the number of HITs increases, the number of remaining workers drops from 26 to 12. The survival rate defined by the Kaplan–Meier estimator (Kaplan and Meier, 1958) is 38. 59% when the number of HITs is set to 10 which is an estimate of a worker’s capacity to be able to complete many tasks. We empirically found that we need a minimum of 8 HITs completed by a worker in order to validate that their annotations are statistically significantly different from random noise (see Table 2). 3. 4 Reference-based Task Finally, to test whether the selected MTurk workers actually perform better at annotating summaries in general, we conduct a reference-based task that comprises 30 HITs. In each HIT, a reference summary and 4 candidate summaries are provided. The worker is asked to assign each candidate summary two scores (“can2ref” score and “ref2can” score)
on a scale from 1 to 5. The “can2ref” score indicates whether all of the information in the candidate summary can also be found in the reference summary, while the “ref2can” score checks the converse coverage direction. A score of 1 means that almost no information in one summary can be found in the other, while a score of 5 indicates complete information coverage. The worker is provided with instructions and examples of the rating at the beginning of the task. 4 Results
4. 1 Annotation Data and Cost
The collected experimental data not only contained annotation results but also metadata reflecting annotator behaviors. 3 The cost of annotation on MTurk included both the wages paid to MTurk Workers and the fees paid to MTurk (which may vary according to the task). A worker who participated in the qualification and the endurance tasks earned $8. 5 ($1 for the qualification task plus $7. 5 for the endurance task) on average, while a worker who participated only in the qualification task (i. e. who did not qualify) earned $1 on average. Given the total cost of $514 for the entire pipeline which yielded 12 workers, the cost of identifying a qualified worker is $42. 8. For details, the breakdown of the cost is shown in Table 3. 4. 2 Qualification Task Results
We conducted four rounds of the qualification task, each round included 50 MTurk workers (see Table 1). This choice of multiple rounds aimed to guarantee the stability of the annotation results (Berinsky et al. , 2012; Arechar et al. , 2017). The overall pass rate of the attention check was 0. 69; thus, 62 workers in total did not pass the attention check and
3The data and code used for the analysis of all tasks are available at https://github. com/GEM-benchmark/ MTurkRequirementPipeline. were categorized as BLOCK. Out of 200 MTurk workers, there were only 8 GOLD workers and 18 SILVER after the qualification task. Thus, only 26 MTurk workers (13% of all participants) qualified for the endurance task. For each round, we calculated Krippendorff’s Alpha4 to measure the agreement among annotators. The highest Krippendorff’s Alpha was 0. 33 reached by the first round, and the average Krippendorff’s Alpha of all four rounds was 0. 25. In addition, the exclusion of BLOCK workers led to an increase in Krippendorff’s Alpha, compared to the value calculated on all workers. The highest Krippendorff’s Alpha without BLOCK workers was 0. 44 (second round), and the average Krippendorff’s Alpha of all four rounds increased to 0. 41. These results showed that, as expected, BLOCK workers seemed to lack good-faith effort in the task and likely yielded low quality annotations. 4. 3 Endurance Task Results
We published the same endurance task for GOLD and SILVER workers separately, and reported IAA using Cohen’s Kappa and Krippendorff’s Alpha among each type of worker; we also reported similar IAA results from combined GOLD and SILVER workers. We additionally collected endurance task results from volunteer researchers unrelated to this paper for a comparison between MTurk workers and NLG “experts”. SILVER Workers There were 18 SILVER workers after the qualification task, 13 of whom accepted the endurance task. However, only 8 SILVER workers finished all 10 HITs–a yield rate of around 44% given the number of SILVER workers entering this task. To calculate the IAA, we considered the annotation scores of all summaries (40 ratings) for each of the 8 workers and calculated Cohen’s Kappa for each worker pair; the highest Cohen’s
4https://pypi. org/project/krippendorff/
Kappa was 0. 451 between workers S22 and S43. To avoid influence from a possible unstable performance at the beginning of the task, we also tried to omit the first two HITs, that is, we only used 32 ratings when calculating Cohen’s Kappa; the resulting improvement for Cohen’s Kappa was very low. In addition, we calculated Krippendorff’s Alpha on the entire annotation results for all summaries and workers, and it reached 0. 358. GOLD Workers There were 8 GOLD workers after the qualification task and 6 of them accepted the endurance task. However, only 4 GOLD workers finished all 10 HITs, for a yield rate of around 67% given the number of GOLD workers entering this task. This rate was higher than that of SILVER workers. We calculated pairwise Cohen’s Kappa using all the scores, and the highest IAA score increased to 0. 48, compared to 0. 45 for SILVER workers. There was no significant improvement after omitting the first two HITs. Krippendorff’s Alpha for the GOLD workers reached 0. 443, which is higher than with SILVER workers (0. 358). GOLD and SILVER Workers To investigate IAA of worker pairs across GOLD and SILVER workers, we combined the results of these two categories of workers and calculated pairwise Cohen’s Kappa. The highest pairwise Cohen’s Kappa on the 40 ratings per worker was 0. 55; see the matrix in Figure 2. Again, omitting the first two HITs also did not change the scores much. For Krippendorff’s Alpha, the value was 0. 396, which fell in the range between the SILVER worker’s (0. 358) and GOLD worker’s (0. 443) values. 5
In Appendix A. 2, we show a breakdown of the results per text position in each HIT (correlations for all first texts, for all second texts, etc. ) for each of the three subgroups (SILVER, GOLD, GOLD AND SILVER); the possibly sightly darker heat maps
5Note that the relatively low Krippendorff’s Alpha scores may in part be due to the large size of the scale (10 points). could indicate higher correlations for the second text of each HIT. Comparison to Expert Ratings To get an idea of the quality of qualified MTurk workers according to our approach, we compared their IAA with the IAA obtained by conducting the same endurance task with three researchers as NLG “experts”. The pairwise Cohen’s Kappa for all 40 ratings only reached 0. 268 (see Table 10 in Appendix A. 3). The IAA among the experts was comparatively lower than the GOLD and SILVER workers, indicating that qualified workers identified by our tasks reached a better agreement at least for the endurance task. Thus, it seems possible to recruit high-quality workers using our pipeline. Detection of Abnormal Workers From Cohen’s Kappa scores shown in Figure 2, the worker S426 had much lower agreement scores (heatmap in the yellow colors on the row and column corresponding to the worker). Recent studies have uncovered the presence of bots on MTurk (Webb and Tangney, 2022). To understand the reason for this worker’s lower agreement with other workers, we analyzed their online behavior using the metadata extracted from their annotation results. Figure 3 shows the timeline of each of the 10 HITs as a horizontal gray line. The timelines are plotted from top to bottom, corresponding to the first to the last HIT in the endurance task. The X-axis represents the duration between the time of acceptance and submission, which is normalized by the duration for each HIT (ranging from 0 to 1). Different marks present each annotator behavior, as shown in the legend. Among these behaviors, blue points represent the time when the MTurk worker
6S42 stands for the second SILVER worker from Round 4
assigned a score for one of the four summaries, and the corresponding number on top represents the summary index (valued from 0 to 3). Orange crosses denote the suggested reading time of the article in each HIT, given the average human reading speed of 130 words per minute. 7 If the suggested reading time after normalization was longer than the duration, we marked the orange cross as 1 at the time of submission which is at the end of the gray line. Most of the orange crosses were marked at the end of the timelines in Figure 3 (right), indicating this worker assigned scores and submitted the HIT in less time than it usually takes for a human to even finish reading the article. This result demonstrates that this worker may not have put in good faith in the endurance task, which possibly explains the low IAA with other workers. By removing this worker and calculating Krippendorff’s Alpha again within GOLD and SILVER workers, the IAA increased to 0. 454 (compared to 0. 396 when including the worker). 4. 4 Reference-based Task Results
To test the reliability of our qualified workers and compare them to workers who do not undergo our selection process, we launched the reference-based task (see Section 3. 4), which is open to our qualified workers as well as to any other workers satisfying basic qualification settings. Qualified Workers after Pipeline We published the reference-based task to the 12 MTurk workers from four rounds who have passed both the qualification and the endurance task. All 12 workers accepted this task but only 8 workers finished 30 HITs within a week. There are two scores to evaluate the information coverage between each candidate summary and the reference summary. We use the “can2ref” score to represent whether all information in the candidate summary can be found in the reference summary, and the “ref2can” score to represent the converse coverage. For both types of scores, we calculated Cohen’s Kappa for every worker pair (given 4 candidate summaries per HIT, 30 HITS per worker). Cohen’s Kappa for “can2ref” score ranges from 0. 15 to 0. 71, with a relatively high IAA between the first GOLD workers from the first two rounds (G11 and G21). Similarly, Cohen’s Kappa for “ref2can” score ranges from 0. 14 to 0. 66. 7https://wordstotime. com/
Finally, Cohen’s Kappa for the combined scores ranges from 0. 15 to 0. 68 (see Figure 4), demonstrating that the agreement numbers are stable across multiple measures. Krippendorff’s Alpha for the above scenarios (“can2ref” score, “ref2can” score, and combined) are 0. 558, 0. 508, and 0. 534. Baseline MTurk Workers For comparison, we published the same reference-based task to MTurk workers who did not participate in our previous experiments. 276 MTurk workers participated and each worker finished on average 2 HITs (In total 30 HITs × 20 Assignments/HIT). Krippendorff’s Alpha for “can2ref”, “ref2can”, and the two combined were extremely low, at 0. 087, 0. 077, and 0. 080 respectively, demonstrating the necessity of a high-quality recruitment pipeline. We experimented with the following approaches to investigate whether we could increase the agreement between random MTurk workers to a level comparable to qualified workers from our pipeline. IAA with Median Among the 20 assignments of each HIT, we randomly divided the workers into 4 groups of 5 workers and took the median of each group representing a “new
worker” (Lau et al. , 2014). Then, we concatenated the results of 20 HITs for the 4 “new workers” to calculate IAA. Krippendorff’s Alpha scores increased to 0. 191, 0. 185, and 0. 188 respectively. Filter on Timing and Number of Finished HITs To exclude unqualified workers whose annotations may decrease IAA, only workers who (i) spent more than the suggested reading time8 and (ii) finished 3 or more HITs were selected for calculation of IAA. This resulted in 25 workers remaining, but Krippendorff’s Alpha remained almost the same as calculated without the filter. Statistical Filter (MACE) We applied the MultiAnnotator Competence Estimation (MACE) (Hovy et al. , 2013; Paun et al. , 2018) to identify reliable workers based on competence scores calculated on annotations. The workers with competence scores above a threshold were kept. We additionally calculated Spearman’s coefficient (Spearman, 1904) within the groups of our pipeline and MACE (see Table 4). We report the results of additional failed attempts to improve Spearman’s coefficient across these two groups, in Table 12 in the Appendix. In summary, the most effective methods to improve agreement numbers among random workers were median grouping and MACE. IAA on median scores can raise Krippendorff’s Alpha to almost 0. 2. MACE increases Krippendorff’s Alpha as the threshold increases, but at the cost of an incomplete HIT coverage (27/30 and 18/30 respectively for the threshold of 0. 6 and 0. 7 in Table 4) and fewer workers per HIT (1. 9 and 1. 2, respectively, for the threshold of 0. 6 and 0. 7 in Table 4). Similarly, Spearman’s coefficient of MACE workers
8We performed the same timing analysis as in Section 4. 3. can be increased above our pipeline workers’ only at the same expense as above. CloudResearch MTurk Workers To further test our pipeline, we conducted the same reference-based task on the CloudResearch platform (cloudresearch. com), which helps researchers recruit high-quality annotators. We recruited the same number (eight) of CloudResearch workers as our pipeline. The Krippendorff’s Alpha and Cohen’s Kappa9 for CloudResearch workers is slightly lower than our pipeline workers (see Table 5 and Figure 9). Additionally, we found that our pipeline workers have a higher task acceptance rate. This results in a shorter experimental period compared to the task conducted on CloudResearch. Analysis of Correctness Across Annotation Sources We randomly sampled 50 annotation questions from the reference-based task to test correctness, which is defined as the alignment with expert judgments. 10 In addition, we also compared the expert judgment with scores generated by GPT models: GPT-3. 5 (“text-davinci-003”) and ChatGPT which are built on InstructGPT (Ouyang et al. , 2022), and GPT-4 (OpenAI, 2023). Scores are aggregated by taking the median within groups of pipeline, MACE, and CloudResearch workers, as
9The range of Cohen’s Kappa is slightly smaller for CloudResearch workers. 10Fifty random samples were chosen in order to differentiate between MACE and pipeline assuming 20% superiority in terms of correctness. well as experts. 11 For ChatGPT we ran inference 5 times with default parameters (temperature=1, top_p=1) and took the median. To obtain GPT-3. 5 and GPT-4 scores temperature was set to 0 with a single run. We did not find that pipeline workers were superior to MACE workers in terms of correctness. Pipeline and CloudResearch workers had a significant Spearman’s correlation with each other (see Figure 5), which indicates a reproduction of the recruitment procedure on CloudResearch at a lower cost