## 3 Revisiting Data Annotation
### Traditional Approaches  
Traditional data annotation methods rely heavily on human effort. These include manual annotation by individual experts and crowdsourcing through distributed workforces. Manual annotation involves trained annotators using specialized tools to label data with high precision. In computer vision, tools like LabelImg, VGG Image Annotator (VIA), and LabelMe ([[https://github.com/wkentaro/labelme](https://github.com/wkentaro/labelme)]([https://github.com/wkentaro/labelme](https://github.com/wkentaro/labelme)), accessed on 20 June 2025) allow annotators to draw bounding boxes or polygons around objects in images and assign class labels. In natural language processing (NLP), platforms like Prodigy ([[https://prodi.gy/](https://prodi.gy/)]([https://prodi.gy/](https://prodi.gy/)), accessed on 20 June 2025) support tasks such as named entity recognition and sentiment analysis. These methods excel in accuracy and context understanding, especially for nuanced tasks that require domain expertise. Crowdsourcing platforms (e.g., MTurk from Amazon, Appen, and Prolific) distribute micro-tasks to a global pool of workers to enable the rapid annotation of large datasets at lower costs. For instance, MTurk ([[https://www.mturk.com/](https://www.mturk.com/)]([https://www.mturk.com/](https://www.mturk.com/)), accessed on 20 June 2025) has been used to annotate product reviews for sentiment analysis by leveraging the collective effort of many annotators. However, crowdsourcing requires careful task design and quality control mechanisms such as majority voting to ensure consistency. Workers often lack specialized knowledge or sustained attention.  
Despite their strengths, traditional approaches face significant limitations that hinder scalability and efficiency. Manual annotation is time-consuming and costly. Annotating the ImageNet dataset with over 14 million images required thousands of hours and substantial financial investment [39]. Human annotators are also prone to errors and inconsistencies, particularly in subjective tasks like emotion detection, where inter-annotator agreement falls as low as 60 to 70% [36]. Crowdsourcing, while more scalable, often yields inconsistent results due to diverse annotator backgrounds. One study found that up to 30% of MTurk annotations required correction because of quality issues [38]. Furthermore, both methods struggle with large-scale datasets and tasks that demand deep domain knowledge. Crowdsourcing also raises privacy concerns when handling sensitive data. These challenges have driven the development of ML-based approaches that automate parts of the annotation process, reduce costs, and improve consistency as shown in Figure 1.  
Figure 1: End-to-end AI-assisted data-annotation pipeline. Raw data of diverse modalities (text, images, audio, etc.) are ingested from external sources and pass through a pre-processing stage that standardizes formats and screens out corrupt samples. An auto-labeling module, coordinated by an AI agent manager, generates initial annotations at scale. All provisional labels are routed to a dedicated quality-assurance block; records that fail automated checks are escalated to a human reviewer for expert correction.