“Get Their Hands Dirty, Not Mine”: On Researcher-Annotator Collaboration and the Agency of Annotators (2024)

Abstract: Annotation quality is often framed as post-hoc cleanup of issues caused by annotators. This position paper discusses how and why this narrative limits the scope of improving data quality. We call to consider annotation as a procedural collaboration, outlining three key points in this direction: (1) An issue can be either annotatoror researcher-oriented, where one party is accountable and the other party may lack ability to fix it; (2) yet, they can co-occur and/or have similar consequences, and any specific problem we observe may be a complicated combination; (3) therefore, we need a new language to capture the nuance and holistically describe the full procedure to resolve these issues. To that end, we propose to study how agency is manifested in annotation and picture how this perspective benefits the community more broadly.

heading: 1 Introduction
text: The pursuit of data quality in NLP and AI usually takes a post-hoc, outcome-oriented approach, targeting the immediate goal of refining the data obtained. This includes the vast amount of ML-based post-processing (Raykar et al., 2009; Khetan and Oh, 2016), and recent notions for replacing crowd work (Gilardi et al., 2023; Alizadeh et al., 2023). While more works recently seek to dig deeper, attention is still largely placed on retrospectively analyzing artifacts in existing datasets (Malaviya et al., 2022; Gururangan et al., 2018; Sap et al., 2022).
Emphasizing the output of a reliable set of annotations over the full data collection procedure often leads data issues to be attributed as undesired noises or artifacts from annotators’ behaviors, biases, or failure (Rodrigues and Pereira, 2018; Han et al., 2020, etc.) With this narrative, annotators are implicitly stationed as the source of problems, and researchers as the party who then find, correct, and/or discard the flawed data. This could lead to an oversight of various systematic failures that
attributed more to researchers, e.g., a flaw in the early stage of task design (Gururangan et al., 2018; Pyatkin et al., 2023; Gadiraju et al., 2017, etc.)
Complicating this further, these independent issues – with potentially distinct sources and solutions – often co-occur, interact, and converge in forming the eventual, real-world problems we encounter. This creates a subtle chasm between theory and practice: while various independent issues are well-documented, annotation is still often understood only via retrospective observations. On the one hand, many works discuss the extensive intricacies of annotation tasks as well as what researchers should look for; On the other hand, improving data quality in practice remains a whaca-mole game: How data were collected doesn’t seem relevant until something pops up; we then guesstimate what happened way back and attempt a fix, and again use it as normal until the next issue stands out. As users downstream are agnostic of the data collection details, this mode is limited.
It is especially hard in retrospective analysis, if users are motivated at all, to further search through the long list of known issues and narrow down to some exact match in the literature with their guesstimates. This calls for a universal language that systematically captures how issues arise and interact in annotation pipelines, one that composes and compares the practices of the researchers and annotators in a coherent, generalizable framework.
In this position paper, we urge the community to rethink the acquisition of annotations as a bilateral, collaborative process: the researchers as task designers and the annotators as task accomplishers, where responsibility is shared among both parties. To better describe flaws in such procedures, we propose to study the agency of annotators, i.e., how much capacity to act is allocated by task designers to an annotator toward the best outcome. We set forth an essential agency (mis)alignment problem: while researchers intend to provide a certain
8773
degree of annotator agency, the task setup they implement may entail a different level of agency, leading to undesired outcomes. With this perspective, we can better understand the annotation pipeline with a procedural lens and match downstream observations with their root causes. We hope this will aid in the broader mission to empower data work with additional transparency and accountability.
2 Tracking down the “Problems” We See
Our statement is based upon several pioneering pieces that consider the annotation process beyond annotators’ behaviors and their outcomes. Parmar et al. (2023) were among the first to point out that existing work targets annotator-related bias; instead, they measure instruction bias: artifacts introduced as early as in the instructions, which directly leads to downstream biases. Huang et al. (2023) surveys workers’ perspectives and how they view their assigned roles. They show that the clarity of tasks impacts workers’ willingness and effort level needed to complete them. Plank (2022) review the human label variation “problem”: disagreements in annotation may not always be noise, but instead suggest important properties of the data and task (e.g. hard cases). Finally, Rottger et al. (2022) highlight that the definition of annotation is essentially different within two contrasting paradigms, “descriptive” and “prescriptive” annotation, based on whether subjectivity should be encouraged.
These individual pieces share a common insight:
implementing a task with crowd work is only a fraction of the data-annotation pipeline. While this phase has many of its own flaws, it also reflects subtleties from earlier stages such as how researchers plan out and describe their tasks.
Consider two hypothetical failure cases for a research group seeking annotations that “represent a good output from a chatbot”. In one case, the researchers used this generic description in quotes as the sole instruction, and the annotations they got were full of artifacts and inconsistencies. In the other case, several annotators improperly used ChatGPT when they were clearly advised not to, and thus the results are unusable. For annotators in the first case, the best work would not possibly exceed a guesstimate of the researchers’ intention: the “chatbot” in question and what would be a “good output” is agnostic. However, things could be largely different if the researchers take a moment to encode the fundamentals of their goal in the task design. Conversely, in the second case, as long as some annotators insist on the improper move, a researcher won’t be able to easily get rid of the trouble of extra work and wasted funds. While both can be reported as a quality issue, the actual cause and solution are distinct: one centering on the role of the researchers and the other on the annotators.
Compounded observations and atomic issues To add to the above, any observed problem in the real world can often be a compound of various contributors: a lack of motivation found among an-
notators may well be individual negligence (annotator), overly low payment (researcher), or – more often – a subtle mixture in the middle. The annotation procedure essentially follows the law of the minimum: any part going wrong would be sufficient to undermine the outcome. An oversight early in the instruction or protocol similarly determines the lower bound of annotation quality, but tackling the annotators’ side in these cases has little effect. The borders are further blurred as issues centering different roles can have similar appearances. For instance, in the first failure case above, an annotator may misunderstand the task and resort to ChatGPT with no bad intention but just because they believe that’s how they should find “good chatbot outputs”.
To sort out the complexity of real-world observations, we outline atomic issues (which derive from a single root cause) in Figure 1, categorized by accountability, i.e., which party would be the center cause for the issue, and further detail them in Appendix A to link the related literature for interested readers. These issues on the spectrum can interact and even converge, forming the compounded problems we encounter when retrospectively examining the efficacy of annotations.
Note that there are no fewer researcher-centered issues than the commonly known annotatorcentered ones. Moreover, the inherently asymmetric relationship between researchers and annotators can create a risky situation: deficiencies on the former party may have a high impact, but are masked by the fact that annotators are the ones who perform the work. For instance, breakdowns in a strict manual might be more likely attributed to annotator laziness than a failure to properly deliver instructions. While issues are well discussed individually in the literature, understanding real-world fallacies demands a holistic model that involves all these subtleties throughout the pipeline. We aim to draw attention to the complexity of the interplay between these issues in reality, and further propose a method to describe it in the next paragraph.
The Agency Problem Researcher-centered issues may appear similar to annotator-centered ones, but their solutions are distinct. How should such intricacies be collected and modeled in any specific task? When task designers conceive a task, it might seem at first blush that they take control of all that needs to be controlled. However, tasks when implemented provide varying (and sometimes unexpected) degrees of agency to annotators in the
course of their work. For example, meticulous manuals might be deployed to confine annotators’ operations in a labeling task, but there can be unforeseen or underspecified interaction modes in how the rules are deployed by annotators. In other words, there remains extra agency for annotators themselves to determine how to complete their tasks. This therefore leads to annotator-centered issues not being monitored by researchers.
We call attention to such agency misalignment problem in annotation workflows. On one hand, we have the notion of annotator’s agency: the extent to which an annotator is able to take (various) actions towards better annotation quality. Constructing any dataset demands a certain level of annotator’s agency; yet, a capable annotator’s agency is also the direct result of how a task is deployed by its designers. This leads to the other component, a task designer’s intended agency. Task designers conceive a task with an enhanced or constrained space of possible actions, along with the expectations that these actions will be taken (e.g., the strict manual in the previous example). They also might instead promote agency on the part of annotators to gather more heterogeneous data, such as in the case of chatbots. Nonetheless, these conceptions might not be fully achieved, thus creating misalignments between the conception and implementation.
We pose that misalignments between intended and realized agency are not only a major contributing factor in quality issues, but studying it also helps to organize and root out these issues. For instance, in the previous codebook example, misalignments between the designers’ intended lowagency task and the actual freedom annotators had to interpret and apply the codebook led to a series of breakdowns that reduced data quality. Yet, without this integrated view of the process, it is easy to attribute this breakdown solely to unruly annotators. This might lead a designer to attempt to recruit “better” workers, when an effective solution is instead to revise the task design.

heading: 3 Case Study: A tale of 4 datasets
text: To further our discussion of the agency misalignment problem, we will outline four classic dataset case studies, visualized as a detailed diagram in Figure 2. Each dataset represents a branch of NLP tasks it belongs to, but also features a specific implementation in question.
Consider 1 the Part-of-Speech (POS) tagging
setup of the Penn TreeBank (PTB, Marcus et al. 1993) as a base. Representing the “canonical” linguistic annotation setup, the task is rigidly grounded with linguistic definitions mapping each word (“the”) to a label (“DET”) on a largely one-toone basis. Annotators are systematically informed with a coherent definition and variance is discouraged. Both the researchers and annotators work under a shared set of rules to reduce irregularities. Thus, the intended and actual agency assigned are aligned at a minimal level1, as denoted in the lower left, and the resulting data generally matches the expectations established at the start.
In contrast, creativity is essential for downstream NLP tasks like 2 SQUAD (Rajpurkar et al., 2016). Making sure that annotators have agency is key to providing the necessary variability. To achieve this, a paragraph is shown and annotators are prompted to generate five questions and highlight text snippets as the corresponding answers. To boost the
1Note that this is not to say an aligned, low-agency task always leads to unanimity; rather, it means the task is implemented in such a way that a relatively small space of variation is allowed for any individual annotator. In fact, “genuine disagreements” with multiple acceptable answers have been widely found in POS tagging (Plank et al., 2014). This maps to the “grey zone” on the spectrum (included as “Intrinsic ambiguities of language”): fundamental factors beyond the control of both parties also contribute to observed “problems”.
variance, the task takes a self-guided form with free-form input, and users are hinted to “use your own words” and “avoid the same words”.
Nonetheless, the implementation did not deliver a full range of creative responses. While little constraints are imposed by the instructions, annotators are limited by the source of the text: all paragraphs come from the Introduction section of a Wikipedia item, which takes a highly similar form and revolves around a topic entity. Annotations, unsurprisingly, follow these patterns. Moreover, no actual elaborations or motivations are given to narrow down how (not) to use the same strategies across instances. Annotators, faced with similar sources that have similar patterns, may feel compelled to follow them. While they theoretically may have agency to answer freely (and are encouraged to do so), shortcut-taking and mimicry have been reported, essentially making the data “simpler” than conceived (Sugawara et al., 2018; Bartolo et al., 2020). In this case, a misalignment between expectations and the way the task format realized them led to problematic outcomes.
Underspecification of a task may also create issues. Consider the family of datasets regarding Toxic Language Detection like 3 Jigsaw’s Perspective API. Intrinsically, there is no single definition for what language is “toxic”. Annotators’
perception is related to (if not determined by) their personalities, demographics, and ideologies (Sap et al., 2022). Even if a designer were to construct an extremely detailed guideline, there is no guarantee individual differences may not be a factor (more so due to subjectivity Waseem et al. 2021). Moreover, commercial platforms will eventually need to make an exact decision whether or not to intervene with or remove a potentially toxic speech, further constraining the decision space as a yes-or-no choice downstream. In this case, misalignment between an expected, rigid outcome and the messy realities of human judgment can challenge the ability to gather useful annotations.
This contrast of varied individuals vs. one simplified decision is even more polarized in Human Feedback datasets represented by 4 Stanford Human Feedback (Ethayarajh et al., 2022), which compares Reddit upvotes to entail an A-or-B preference decision. Here, preference was conceived as a task with very low agency: the only operation possible is the up/downvote made, and it does not vary by topic or time. However, real-life preferences and upvotes are highly arbitrary, and it is not clear whether the latter is a decent proxy of the former – for instance, upvotes can be given to arbitrarily many replies (high-agency) but are restricted to strictly one of the two in pairs (low-agency). Misaligning the diverse nature of the implementation with a low-agency framework risks misrepresenting the ground truth embedded in the annotations.

heading: 4 Discussion
text: Improving annotation quality is all about the interplay of annotators and designers. Our framing of the agency misalignment problem essentially proxies a larger misalignment between what we think we’re building and what we end up with. Rather, anything could be part of a dataset; but the designers themselves should – and only are they able to – ensure it is the dataset, the exact one that we believe to serve a certain purpose. Given our datahungry models, the stakes are high. A piece of data from a coarse design may instantly have a consequence on a sophisticated application: to determine which model is better, to place rewards on some options while penalizing others, and – eventually – to decide what values are coded in our models and whose input is represented. Shifting our focus and transferring the responsibilities in data collection also means letting go of control over our models,
eventually adding to the already notorious myth of training data and evaluation (Paullada et al., 2021; Rogers, 2021; Howcroft et al., 2020).
We hope our proposed framing can be an entry point into understanding what our data are and how they are generated, as early as when we conceive it as a hypothetical task. We see two ways this framing might be used. First, as a diagnostic, examining issues of misalignment can help to explain the human factors underpinning what may seem like a simple issue of low-quality annotations. This can lead to fixes with greater impact across multiple task designs. Second, using this framing prospectively when designing a task can help to identify unforeseen complications as agency is allocated. For instance, thinking carefully about the degrees of freedom a codebook affords can help to identify its vulnerabilities to unusual annotator behavior.
In considering this issue, we also draw attention to discussions in the field of crowdsourcing related to working conditions (Kittur et al., 2013), techniques for shepherding crowd work (Dow et al., 2012), alternate collaborative annotation workflows (Chang et al., 2017), perspectivist judgments (Cabitza et al., 2023), as well as a wide range of quality control techniques under debate within the domain (Daniel et al., 2018; Rzeszotarski and Kittur, 2012; Zaidan and Callison-Burch, 2011).

heading: 5 Conclusion
text: We hope this position paper offers readers an additional perspective on how to approach annotation quality issues and their compounds via a cohesive and procedural lens. Moving forward, we urge the community to adopt more holistic assessments of annotation pipelines, and draw insights from the ways that human factors, crowdsourcing, and NLP researchers are innovating in how tasks are designed and assessed.

heading: Limitations
text: This position paper emphasizes the procedural collaboration between researchers and annotators and initializes discussions on how that collaboration can be modeled. The work, as a succinct proposal, is necessarily limited for a rather broad shift from the deeply rooted discourse and ecosystem. The spectrum and case studies may not cover all related issues on the table, and subsequent work would be necessary to quantify the misalignment of agency and validate the expressiveness of such a model.
Nonetheless, we hope that our analysis will serve as an entry point for further discussion towards a better future of data collection, as well as a bridge connecting our field to the immense work in related fields navigating toward a shared mission.

heading: Ethical Considerations
text: Data as the ingredients of models determines what decisions they make and what values are encoded. Our work is in line with this direction and aims to help understand and improve data collection for that reason. While our work does not contain new datasets or experiments with humans, we propose a new way to analyze the annotation procedure based on agency as defined. This may influence how data collection practices and the resulting datasets are described and assessed in the future. Yet, we believe these new perspectives in this position paper would enrich current views in a positive way.

heading: Acknowledgements
text: We thank our reviewers and the members of the Artifact Lab @ Cornell InfoSci for their detailed feedback and suggestions on our drafts.

heading: A Details of the atomic issues along the Spectrum of Accountability
text: We provide more details and recent examples for the atomic issues that we identify and cluster along the Spectrum of Accountability (Fig. 1). We hope this could provide readers with the general landscape of the issues of interest – especially readers who are not specifically familiar with the field of annotation and crowdsourcing – and initiate future research and discussions. Note that the discussion
here for each issue is by no means an exhaustive review of that direction, but we seek to provide an appropriate starting point for interested readers to search into the broader related literature.

heading: A.1 Annotator-Centered Issues
text: The usability of annotated data is directly determined by the work of annotators. For a specific task, annotators might have not performed their work effectively and properly (None-optimal Work Mode). More broadly, one’s role of (outsourcing) workers might be compromised or varied due to individual conditions (Individual Variations).
Non-optimal work mode is a major source of issues where the annotation work is compromised as the result of annotators’ (purposeful) behaviors for a specific task, even when seemingly qualified after adequate consideration of the researchers (e.g. via a pre-test or a filter). This would be especially relevant for crowdsourcing scenarios where task designers usually only have limited and distant supervision of the implementations (“outsourcing to an undefined, generally large group”, as defined by Jeff Howe (2009)). These cases can be further categorized by whether and to what extent the undesired mode is intentional and destructive. On one extreme, it is practically impossible to locate a participant with a determined and strategic malicious intent (Gadiraju et al., 2015), since pre-task screening could be deluded in most cases. In a milder scenario, users may not intend to sabotage the requests, but (knowingly) react to the task without adequate care as instructed (Curran, 2016; Brühlmann et al., 2020).2 Annotators can also strategically develop shortcuts that approximate the actual requirements and instructions with a simplified but inaccurate implementation (Malaviya et al., 2022).3 These have since inspired a broad literature in quality control and estimation (Daniel et al., 2018; Rzeszo-
2Any human worker would have a natural error rate and this is not the major referent in this context. Instead, we refer to the cases where we see a systematic pattern of carelessness in one’s work.
3Malaviya et al. 2022 refer to these broadly as cognitive heuristics (Tversky and Kahneman, 1974; Gigerenzer and Todd, 1999), yet majorly concerns the deliberate application of these strategies. We avoid the subtlety introduced by the vast literature in the Psychology field surrounding this terminology, as we seek to make a clear, three-way distinction that corresponds to different accountabilities: the strategic, intentional shortcutting of annotators (Shortcutting in the spectrum), the intuitive, less-aware cognitive patterns (Underlying & collective patterns), and an act of expedience faced with unclear instructions or flawed task design.
tarski and Kittur, 2011; Zaidan and Callison-Burch, 2011). With the recent progress in the field of Artificial Intelligence, another issue is the improper use of AI tools. The extreme ease of access to proxies like large language models has proliferated this type of improper work mode (when they are not suitable for the task) (Veselovsky et al., 2023).
Individual Variations are another group of issues that are less specific to the implementation of any single task. Rather, they concerns the characteristics and roles of individual annotators on a higher level. For one thing, factors that are out of the reach of researchers, e.g., the mood (Zhuang and Gadiraju, 2019) when a worker starts on their task, may always have a direct impact on the output. There are also more fundamental aspects in the profile of annotators that subtly impact the way they work. One important aspect is the motivation or “reason” that an annotation worker is in their role (Martin et al., 2014). Crowd work may, for instance, merely be an option of entertainment over a cup of tea (that even pays!) for one retired scholar, but is the lifeline of another unemployed single mother. While both can be well qualified, the outcome can have different patterns due to personal status. In the former case, the annotator might be happy to provide an especially detailed and critical response with feedback, but would only take on an extremely small portion of the data, and has long intervals between their work periods. In the latter, the annotator may be willing to efficiently complete tasks in large batches, but the speed and incentive might be prioritized over higher quality, and they might be subject to the effects of fatigue. Similarly, the background and mindset of annotators can contribute to notable variations encoded in the outcome, which would be both a rich source of high-agency tasks and a major challenge to monitor and mitigate inappropriate biases (Sap et al., 2022).

heading: A.2 Researcher-Centered Issues
text: Like the annotators’ side, researchers can run into atomic issues both from the specific design choices in a task (Task Design) and from the broader considerations for annotations as a (societal) system.
System While data collection usually has the most to do with a research purpose, it should not be omitted that the researchers are the payers and employers of crowd work. Various issues can emerge within the operation of this system with unequal roles. The most classic one is perhaps the topic of
fair pay and, as a result, the perceived fairness and incentives of workers (not to forget the researchers’ own budgets are in the play as well [Ye et al., 2017; Huang et al., 2023, among many others]). Recent studies also pose questions on the broader ethics in annotation and crowdsourcing performed by workers (Shmueli et al., 2021). In the role of employers, researchers are also in charge of the selection and filtering of participating workers. This complements the improper work mode of annotators: researchers should, as the upstream designer and employer, guarantee the effectiveness of their data by considering who would be suitable workers, before attributing to individual failures.
Task Design The design of a task is the very core of its eventual usability. An overlooked design factor might directly lead to the failure of communication with workers and undetected deviations from the supposed scenario (as our discussions of the agency are concerned). Various hierarchies work together to form a valid task and the staging could well be a whole separate paper on its own. Things to consider involve the abstract, early-stage conception of protocols and procedures to use (Nangia et al., 2021), whether a task-specific protocol will work out in practice (Huang et al., 2022), as well as how effective this task itself is effective and complete at all (e.g., the classic findings of Gururangan et al. (2018) on the strong yet overlooked cues in Natural Language Inference). Meanwhile, it also spans to the very details when handing over to the annotation workers, such as whether the text layout has been frustratingly long or difficult (Sugawara et al., 2022) and how the annotation User Interface should be displayed (Sullivan Jr. et al., 2022).

heading: A.3 The “Grey Zone” – Underlying Limitations
text: Finally, we would also like to highlight the more fundamental and universal sources of issues which we call the grey zone. Intricate as the cognitive system and language ability of humans are, many times a failure in a task does not simply go to the party that designs or implements it. For instance, ambiguity is the very nature of natural language (Piantadosi et al., 2012), and we have noted that even a low-agency, aligned task can have intrinsic disagreements as perceived by different people(Plank et al., 2014). This could be a gold mine for understanding human language, yet also add trouble to many machine learning setups with one “ground
truth” label. Besides, many interesting cognitive patterns and biases per se – ones we might use collectively and unconsciously – are yet to be identified and/or understood. It would be insufficient to conclude that annotators are to blame as they apply such patterns or that researchers did not foresee these highly probable heuristics in their data. Included in the spectrum is an interesting case that confirms the Pollyanna hypothesis (our word choices include significantly more positive terms than negative ones) across languages (Dodds et al., 2015).

heading: Future Work