## 6 AI Agents for Annotation Workflow and Quality
### Annotation Quality and Consistency  
Beyond automating the workflow, AI agents are enhancing the quality and consistency of annotations [80; 81; 82]. LLM agents distill domain knowledge into the labeling process and augment data to cover edge cases. Here we present how AI agents serve as diligent reviewers, where they check for bias, resolve conflicts between annotators, and detect errors that humans miss. These advances improve not only the speed of annotation but its reliability and fairness. Figure 4 illustrates the working of the intelligent quality checker agent in the annotation process.  
* Knowledge Distillation and Augmentation: AI agents dramatically amplify annotation efforts by distilling knowledge from powerful models and augmenting datasets with synthetic examples [83; 84]. The core idea of knowledge distillation in this context is to use LLM to generate labeled data or guidance that trains a small language model (SLM). [PERSON] et al. [85] propose a knowledge distillation scheme where the system first analyzes the SLM weaknesses, then an LLM synthesizes new training examples specifically targeting those weak spots. Alongside distillation, LLM-based agents contribute to data augmentation for annotation. Instead of relying solely on human-curated examples, these LLM agents generate large quantities of plausible data points with labels [85; 86].
* Bias and Fairness Checking: Maintaining high annotation quality and consistency is paramount, and AI agents play diverse roles in quality control. One contribution is through detecting and mitigating bias in labels, where LLM agents act as a second pair of eyes to review annotations for potential mistakes or biases. A recent study [64] used GPT-4 to re-evaluate crowdsourced annotations in an event extraction task, and the LLM flagged roughly 24% of the human-provided labels as debatable or likely errors. Another study [87] on political Twitter data found that zero-shot GPT-4 labeling not only achieved higher accuracy but also exhibited equal or lower bias compared to human annotators.
* Inter-annotator Agreement Conflict Resolution: When multiple annotators provide labels for the same item, disagreements often arise, resulting in ambiguity or differing interpretations. AI agents assist in resolving these conflicts to achieve higher inter-annotator agreement (IAA). For example, an LLM agent organizes the rationale and highlights where guidelines might be unclear. After iterative debate and guideline refinement, the annotations converge, and the guidelines are updated to codify the resolved distinctions [76]. [PERSON] et al. [88] explore chain-of-thought prompting and majority voting among LLM agents to imitate a panel of annotators, producing more consistent labels at scale. This results in higher consistency than individual annotators by aggregating multiple perspectives.
* Automated Quality Control: AI agents excel at the tedious yet critical task of quality control in annotation projects. They perform automated checks on the annotated data to catch errors, inconsistencies, or low-quality labels much faster than manual review. One common technique is to employ an LLM as a quality reviewer: after initial annotation (by humans or another model), the LLM is prompted to verify the label given the input and the guidelines [89]. Another strategy is the \"LLM-as-a-judge\" approach, where an ensemble of LLMs reviews existing labels and flags those likely to be incorrect. [PERSON] et al. [90] implement this by prompting multiple diverse LLMs to label the same data and measuring their agreement against the original label.