## 6 AI Agents for Annotation Workflow and Quality
### Active Data Selection and Adaptive Annotation  
Effective data annotation increasingly leverages adaptive methods that prioritize efficiency, quality, and responsiveness to evolving data challenges. Here, we provide several AI-driven strategies, emphasizing how LLM-based AI agents optimize active sampling, active learning, guideline adaptation, and continuous learning. * Active Sampling Strategies: Active sampling strategies aim to maximize the information gained per annotation while minimizing labeling effort [67; 69]. AI agents enhance this process by intelligently prioritizing data using LLM-driven insights. For example, ActiveLLM [66] addresses the \"cold start\" problem in few-shot learning by leveraging GPT-4 and similar models to select informative instances even with minimal initial data. The LLM-guided sampling significantly boosts downstream classifier performance, outperforming both standard active learning and other few-shot methods. In parallel, efficiency-oriented approaches like ActivePrune [65] use LLMs to prune large unlabeled pools before selection by employing a two-stage filtering, which not only outperforms prior pruning methods but also cuts total annotation time by up to 74%. * LLM-driven Active Learning: LLM-driven active learning allows the AI agents to actively select and annotate data in each iteration [68]. One notable framework, LLMaAA [70], treats an LLM as an \"active annotator\" within the loop to decide what to label next in order to provide high-quality pseudo-labels. Modern approaches [70; 71] generate new unlabeled instances that are predicted to be informative, rather than drawing only from a fixed pool. In these approaches, AI agents pose hypothetical or synthesized examples on the fly, expanding the training set with \"high-value\" queries that a human or the LLM itself can then label [71]. * On-the-fly Guideline Adaptation: In terms of annotation guidelines, AI agents assist in adapting on the fly to improve consistency [72; 73; 74]. Complex information extraction \begin{table}
\begin{tabular}{p{56.9 pt} p{56.9 pt} p{56.9 pt} p{56.9 pt} p{56.9 pt} p{56.9 pt}} \hline \hline
**Ref.** & **Workflow** & **Mechanisms** & **Impact** & **Domain** & **Remarks** \\ \hline [PERSON] et al. [64] & Post-label aggregation via federated clients & FedAvg on raw, disgreeing labels; preserves annotator privacy & Higher Macro-F\({}_{1}\) than majority vote across 8 NLP tasks & Subjective text (sentiment, hate, toxicity) & Disagreement becomes a _signal_—federated agents upgrade label robustness without extra human passes \\ \hline [PERSON] et al. [65] & Pre-label pool pruning before active learning & KenLM perplexity filter + quantized-LLM scoring & Cuts AL wall-clock time \(\downarrow\)74\% while matching quality & Translation, summarization, sentiment corpora & LLM “gatekeeper” trims pools so expensive AL heuristics stay fast yet effective \\ \hline [PERSON] et al. [66] & Cold-start and few-shot selection & Zero-shot GPT-4 picks first-batch items, fully auditable & +17-24 pp accuracy, seconds-level runtime & GLUE-style text classification & General-purpose LLM agent eliminates AL cold-start pain at minimal cost \\ \hline [PERSON] et al. [67] & In-model label refinement & Uncertainty-weighted fusion; KLE regularizer benchmarks & More robust group-emotion accuracy on 3 benchmarks & Vision (crowd images) & Probabilistic “self-grader” agent down-weights noisy features mid-pipeline \\ \hline [PERSON] et al. [68] & End-to-end orchestration across data products & Human checkpoints, KG-edge uncertainty, federated governance & Scales to 1.5 M projects and 27 M centers while cansuring auditability & Multimodal procurement data & Multi-agent mesh shows how systemic quality gates span every annotation stage \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Agent-centric workflows and their quality controls. (IE) tasks come with detailed guidelines that vanilla LLMs struggle to follow. To address this issue, [PERSON] et al.