## 8 Evaluating AI Agents in Data Annotation
### Performance Metrics for AI Agents  
The assessment of AI agents in data annotation requires comprehensive performance metrics that quantify technical effectiveness. Annotation throughput represents a fundamental efficiency indicator, measuring processing speed relative to human benchmarks [114]. Streaming graph-summarization tasks such as OSNet process millions of edges  
\begin{table}
\begin{tabular}{p{56.9 pt} p{56.9 pt} p{56.9 pt} p{56.9 pt} p{56.9 pt}} \hline
**Ref.** & **Type** & **Agent Role** & **Metrics** & **Properties** & **Remarks** \\ \hline [PERSON] et al. [109] & Benchmark, Performance & Benchmark harness testing LLMs as autonomous agents (8 environments) & Success Rate, F1, reward, avg. turns & CoT prompting; temp=0 & GPT-4 leads 27 models; exposes gaps in long-horizon reasoning for OSS \\ \hline [PERSON] et al. [110] & Performance, Economic & Few-shot web agent adaptor (meta-learned planner) & Element Acc., Op. F1, Step SR, Overall SR (+4–7 pp) & Prompt-token cost; seconds-level latency & 1-2 multimodal demos boost success 21–45\% while keeping compute low. \\ \hline [PERSON] et al. [111] & Performance, Economic & Multi-robot active-learning coordinator & mIoU 2.5 pp, Top-1 Acc. & Data-upload volume 90 \% & Synchronized selection matches pool-based AL with a fraction of bandwidth. \\ \hline [PERSON] et al. [112] & Performance, Economic & LLM generator, assessor, and utilizer pipeline for taxonomy + labeling & Coverage \(>\) 99.5 \%; Acc./Rel. & Cost $0.0006 vs 50.082 per call; human-vs-LLM agreement & Lightweight classifiers match GPT-4 quality at a fraction of runtime cost. \\ \hline [PERSON] et al. [113] & Performance, User-centric & Explainable-AI assistant for anxiety annotation & Spearman \(\rho\) = 0.261 & 8-item Likert trust scale; behavior-trust correlation & Trust rises over time but dips after errors; validates user-centric metrics. \\ \hline [PERSON] et al. [114] & Performance, User-centric & Web agent for real-time stimulus display & Reaction-time stats; ANOVA \(F\) = 2.997 (\(p\approx\) 0.05) & Scalable in-browser logging; bilingual participant pool & Low-cost agent captures valence shifts from code-switching at scale. \\ \hline \end{tabular}
\end{table}
Table 6: Evaluation metrics and roles of AI agents in data annotation.  
per second, giving a realistic upper bound for agent throughput on dynamic data [115]. Synthetic generators like ROLL create billion-node scale-free graphs in minutes, letting us benchmark label-throughput without privacy constraints [116]. Label quality evaluation typically involves comparison with gold-standard annotations or the analysis of consistency patterns across datasets [117]. The scalability dimension addresses an agent's capacity to handle increasingly complex and voluminous datasets while maintaining performance levels [111]. Recent research emphasizes adaptability across diverse annotation domains as a critical capability metric [110]. Illustrative studies include [53], who evaluated LLMs against human annotators in relevance detection using precision metrics and Cohen's kappa to quantify agreement levels. The innovative \"Turking Test\" developed by [PERSON] and [PERSON] [118] examined whether LLM-generated annotations successfully adhered to established guidelines for natural language inference tasks. [PERSON] et al. [119] implemented a pragmatic evaluation approach by training models on AI-labeled data and comparing their performance against models trained with human-labeled datasets, demonstrating that LLMs achieved 90-95% of human-level accuracy in classification scenarios. These methodologies collectively illuminate both capabilities and limitations of AI agents in generating reliable annotations for machine learning applications.