## 4 Machine Learning-Based Label Quality Assurance  
_Demo is accessible at [[https://label-qa.piculjantechnologies.ai/](https://label-qa.piculjantechnologies.ai/)]([https://label-qa.piculjantechnologies.ai/](https://label-qa.piculjantechnologies.ai/)). The accompanying API can be found at [[https://label-qa.piculjantechnologies.ai/?view=api](https://label-qa.piculjantechnologies.ai/?view=api)]([https://label-qa.piculjantechnologies.ai/?view=api](https://label-qa.piculjantechnologies.ai/?view=api)) (accessed on 16 May 2023)._  
In this section, we describe the proposed machine learning-based label quality assurance method. To contextualize this within the proposed process for data requirements within requirements engineering, let us assume that during the \"Requirements Elicitation\" activity, the objective is to build an object detection computer vision software product capable of detecting objects in images belonging to 80 classes (e.g., \"apple\", \"banana\", \"person\", \"umbrella\" and so on). It is assumed that the \"Legal and Regulatory Data Requirements\" activity is completed, as well as the \"Production-like Image Data Acquisition\" activity, which a dataset that accurately represents the problem is collected. For the \"Data Quality Assurance\" activity, we assume that all images are of acceptable sizes and qualities. The method starts with labeling a small random subset (approximately 5% of the full collected dataset) carefully to ensure that the labels are of very high quality. Let us call this the _small set_, and the remaining approximately 95% of the full collected dataset is the _large set_. For the purpose of this work and for building and evaluating the machine learning-based label quality assurance method, we rely on a publicly available labeled object detection dataset. Using the ground truth label information about a carefully annotated _small set_, large amounts of good and bad labels for every sample can be created to train a machine learning-based label quality assurance model. With this model, there is no need to assign a reviewer to examine all remaining samples from the _large set_ and find the ones with bad labels or to assign multiple annotators to all samples from the _large set_ to perform inter-annotator agreement. Instead, the model can determine whether a label is good or bad by being trained on a large number of good and bad labels. Once it has learned the patterns and features that distinguish good labels from bad ones, it can then evaluate the quality of new labels. Consequently, a single annotator can annotate one sample from the _large set_, and the trained model can assess the label quality, streamlining the annotation process and ensuring higher-quality labeled data. There is also no need to set up a full machine learning pipeline with a larger amount of labeled data of good quality to perform a deep active learning approach. Each image from _large set_ receives labels from a single annotator, which may be a machine learning model that generates labels, a program or a human. The label quality assurance neural network can immediately determine if a label is good or not and notify the annotator to pay more attention to the sample if it is classified as a bad sample. If the label quality assurance neural network produces a false negative prediction, the annotator can confirm that they have inspected the sample carefully and their confirmation serves as a strong signal that the sample is outside the _small set_ distribution used for training the model. The creation of a machine learning-based label quality assurance neural network within the proposed data requirements process is illustrated in Figure 5.  
Subsequently in the development process, the _small set_ is utilized by machine learning engineers and researchers as a validation set, while the _large set_ serves as a training set for the primary task being addressed: in this case, the creation of an object detection model.